Podcasts about eks

Cribl's Field CISO Ed Bailey discusses how customers can manage the quality and quantity of data by providing intelligent controls between data sources and destinations.Topics Include:Cribl company name originCompany helps organizations screen data to find valuable insightsEd Bailey was Cribl's first customer back in 2018Data growth of 25% yearly created seven-figure cost increasesCEOs and CIOs complained about explosive data storage costsUsers demanded more data while budgets remained constrainedBailey discovered Cribl through a random Facebook advertisementCribl Stream sits between data sources and destinationsNo new agents required, uses existing infrastructure connectionsReduced data growth from 28% to 8% within yearDevelopment cycles shortened from six weeks to two weeksBailey managed global security and telemetry data systemsOperated large Splunk instance across forty different countriesTeam spent time collecting data instead of extracting valueCribl provided consistent data control plane for operationsSmart engineers could focus on machine learning solutionsMigrated from terrible SIEM to better security platformData strategy should focus on business requirements firstNot all data has the same business valueTier one: Critical data goes to expensive platformsTier two: Important data stored in cheaper lakesTier three: Compliance data in low-cost object storageSIEM costs around one dollar per gigabyte storedData lakes cost twelve to eighteen cents per gigabyteObject storage costs fractions of pennies per gigabyteAWS partnership provides scalable infrastructure for rapid growthEC2, EKS, and S3 are heavily utilized servicesCribl Search finds data directly in object storageAvoids costly data movement for search and analysisParticipants:Edward Bailey – Field CISO, CriblSee how Amazon Web Services gives you the freedom to migrate, innovate, and scale your software company at https://aws.amazon.com/isv/

Explore FSx for Lustre's new intelligent storage tiering that delivers cost savings and unlimited scalability for file storage in the cloud. Plus, discover how the new Model Context Protocol (MCP) servers are revolutionizing AI-assisted development across ECS, EKS, and serverless platforms with real-time contextual responses and automated resource management. 00:00 - Intro, 00:52 - Introduction new storage class, 03:43 - MCP Servers, 07:18 - Analytics, 09:34 - Application Integration, 15:52 - Business Applications, 16:21 - Cloud Financial Management, 17:44 - Compute, 20:44 - Containers, 21:31 - Databases, 24:25 - Developer Tools, 25:42 - End User Computing, 25:58 - Gaming, 26:34 - Management and Governance, 28:35 - Marketplace, 28:51 - Media Services, 29:29 - Migration and Transfer, 30:01 - Networking and Content Delivery, 34:01 - Security Identity and Compliance, 34:43 - Serverless, 35:06 - Storage, 36:55 - Wrap up Show Notes: https://dqkop6u6q45rj.cloudfront.net/shownotes-20250613-185437.html

Cik labi ir jāpārzina automašīnas navigācijas sistēma, lai varētu nokārtot B kategorijas vadīšanas eksāmenu? Savukārt fotoradari pastiprināti kontrolēs OCTA un tehniskās apskates derīgumu. Raidījumā Kā labāk dzīvot plašāk skaidro Latvijas Apdrošinātāju asociācijas vadītājs Jānis Abāšins un Ceļu satiksmes drošības direkcijas valdes priekšsēdētājs Aivars Aksenoks. No 2025. gada jūnija, lai nokārtotu B kategorijas vadīšanas eksāmenu, jāapliecina arī zināšanas navigācijas sistēmas izmantošanā. Aivars Aksenoks atzīst, ka šis ir pēdējais CSDD jaunums, kam ilgi gatavojās un jau laicīgi brīdināja gan autoskolas, gan sabiedrību, ka šāds elements būs iekļauts braukšanas eksāmenā. "Būtiski, lai jaunie vadītāji, kārtojot eksāmenu, šīs zināšanas būtu ieguvuši un apliecinātu," atzīst Aivars Aksenoks. "Tā ir būtiska iemaņa, kas jāpārbauda, kā vadītājs spēj sadalīt uzmanību starp navigācijas ierīci un vienlaikus ceļu, ievērojot visus ceļu satiksmes noteikumus." Eksāmenā vadītāja uzdevums būs vadīt transporta līdzekli atbilstoši navigācijai, instruktors šajā laikā nekādas komandas nedos. Tas būs salīdzinoši īss posms, kas aptver 5 minūtes, lai saprastu, kā prot izmantot navigāciju. Tāpat autovadītājiem jāatceras, ka vairs nav tehniskās apskates uzlīmes mašīnām. Vadītājiem ir bažas, ka varētu palaist garām termiņu. Aivars Aksenoks skaidro, ka šādas uzlīmes nav nevienā tuvākajā kaimiņvalstī jau labu laiku un mierina, ka tehniskās apskates termiņu viegli var pārbaudīt elektroniski. CSDD sūta atgādinājumu septiņas dienas pirms tehniskās apskates termiņa un arī konkrētajā dienā, kad tehniskā apskate beidzas, atgādina.  "Cilvēks, kurš atbrauc uz tehnisko apskati, saņem arī protokolu. Ja ir kaut kur pazaudējis, to var ērti redzēt un izdrukāt arī CSDD e-pakalpojumu. To var izdarīt arī cilvēki, kuri nav reģistrējušies. Tas ir viens no pakalpojumiem, ko var veikt nereģistrējoties," norāda Aivars Aksenoks.

President Macron nektet for at kona dyttet ham og sa at den virale videoen var falsk. Eks-president Bidens folk skjulte hvor svekket han var blitt. Hva skjer når politikere prøver å dekke over virkeligheten? Og kan det påvirke det norske valget? Med utenriksjournalist Kristoffer Rønneberg og politisk redaktør Kjetil B. Alstadheim. Foto: Chalinee Thirasupa, Reuters/NTB

Hoje a conversa foi com o Guilherme Oki, um verdadeiro veterano do SRE e Cloud, que já navegou por ambientes de infraestrutura em fintechs, jogos e agora está numa startup stealth (sim, aquele mistério que te deixa curioso até o final). Falamos de Kubernetes em large scale, desafios de rede, geodistribuição e aquele eterno dilema do multi-cloud: usar ou fugir?Exploramos desde o que realmente significa trabalhar em "grande escala" (não, seu EKS com 10 nodes não conta), até questões mais cabeludas como Federation, eBPF, Cilium, e como lidar com as dores reais da escalabilidade em ambientes críticos.Tudo isso com uma pegada técnica, sem perder o bom humor. Cola com a gente nesse episódio que está simplesmente imperdível para quem vive ou quer viver no mundo de Kubernetes e infraestrutura moderna.Capítulos principais do episódio:00:00 - Abertura03:00 - O que é grande escala07:30 - Geodistribuição11:00 - Multi-cloud vale a pena?14:40 - Desafios de rede19:30 - Federation de clusters24:10 - Cilium e eBPF30:00 - Infra para jogos34:20 - Padronização em escala38:10 - Limites do Kubernetes42:00 - Controle com Cilium46:30 - Bugs e UDP50:40 - Gerenciado vs autonomiaLinks Importantes:- Guilherme Oki - https://www.linkedin.com/in/guilherme-oki-1a649b115/- João Brito - https://www.linkedin.com/in/juniorjbnParticipe de nosso programa de acesso antecipado e tenha um ambiente mais seguro em instantes!https://getup.io/zerocveO Kubicast é uma produção da Getup, empresa especialista em Kubernetes e projetos open source para Kubernetes. Os episódios do podcast estão nas principais plataformas de áudio digital e no YouTube.com/@getupcloud.

Vajadzēja sākties 30 dienu pamieram, ko pieprasa Ukraina kopā ar sabiedrotajiem, tomēr Krievija uzbrukumus turpina. Koalīcijas partneri viens otram atgādina par uzdevumu nekavēties tikt skaidrībā ar valsts tēriņu pārdali aizsardzībai. Skolu jaunatnes dziesmu un deju svētki šovasar pulcēs ap 37 550 dalībniekus. ASV un Ķīna paziņo, ka uz 90 dienām samazina savstarpēji noteiktos muitas tarifus. Eksāmenu sezonu vidusskolēniem atklāj sociālo zinātņu eksāmens augstākajā mācību līmenī. Vidējais patēriņa cenu līmenis Latvijā 2025. gada aprīlī, salīdzinot ar 2024. gada aprīli,  pieauga par  3,9%, liecina jaunākie Centrālās statistikas pārvaldes dati. Latvijas hokeja izlases rindas pasaules čempionātā pastiprinās uzbrucējs Ābols.

7. mail toimus Facebooki otsesaade, kus said läbi võetud kõik viimaste nädalate põletavamad teemad alates RFK Jr. kannapöördest, Trumpist, Iisraelist, Ukrainast, Venemaast, laborilekkest, uhhuu-jahist ja hitlerjugendist kuni ravimite raskete kõrvaltoimete ja pandeemialeppeni.   Muidugi tuli kõne alla ka see, et Katy Perry käis omast arust kosmoses ja fakt, et sind järgitakse juba sünnist saati igal päeva ja öösel? Kes seda teeb? Eks ikka see inimene, kes sinu elu ja heaolu eest vastutab… Vihje – sa saad seda inimest alati valida, aga mitte valimisjaoskonnas. Lisaks loosime kõikide live´i FB-video jagajate vahel välja kasti Lumiorava magneesiumivett.   Vaata videot: https://www.telegram.ee/ajaviide/fb-live-7-05-25-globaalmangud-uhhuu-jaht-kiirgused-pandeemialepe-ja-sind-jalgitakse-24-7  

Ruslan Kusov of SoftServe presents how their Application Modernization Framework accelerates ISV modernization, assesses legacy code, and delivers modernized applications through platform engineering principles.Topics Include:Introduction of Ruslan Kusov, Cloud CoE Director at SoftServeSoftServe builds code for top ISVsSuccess case: accelerated security ISV modernization by six monthsHealthcare tech company assessment: 1.6 million code lines in weeksBusiness need: product development acceleration for competitive advantageBusiness need: intelligent operations automationBusiness need: ecosystem integration and "sizeification" to cloudBusiness need: secure and compliant solutionsBusiness need: customer-centric platforms with personalized experiencesBusiness need: AWS marketplace integrationDistinguishing intentional from unintentional complexityPlatform engineering concept introductionSelf-service internal platforms for standardizationApplying platform engineering across teams (GenAI, CSO, etc.)No one-size-fits-all approach to modernizationSAMP/SEMP framework introductionCore components: EKS, ECS, or LambdaModular structure with interchangeable componentsCase study: ISV switching from hardware to software productsFour-week MVP instead of planned ten weeksSix-month full modernization versus planned twelve monthsAssessment phase importance for business case developmentCalculating cost of doing nothing during modernization decisionsHealthcare customer case: 1.6 million code lines assessedBenefits: platform deployment in under 20 minutesBenefits: 5x reduced assessment timeBenefits: 30% lower infrastructure costsBenefits: 20% increased development productivity with GenAIIntegration with Amazon Q for developer productivityClosing Q&A on security modernization and ongoing managementParticipants:Ruslan Kusov – Cloud CoE Director, SoftserveSee how Amazon Web Services gives you the freedom to migrate, innovate, and scale your software company at https://aws.amazon/isv/

Eks-politimannen Leif starter sin egen private etterforskning når han skjønner at han har blitt lurt. Hvordan fikk falske kjendis-annonser med blant andre Erna Solberg og Elon Musk nordmenn til å bli svindlet for millioner av kroner? Og hvem er det egentlig som står bak? Med journalist Nora Thorp Bjørnstad. Ansvarlig redaktør Gard Steiro.

Today's episode is with Paul Klein, founder of Browserbase. We talked about building browser infrastructure for AI agents, the future of agent authentication, and their open source framework Stagehand.* [00:00:00] Introductions* [00:04:46] AI-specific challenges in browser infrastructure* [00:07:05] Multimodality in AI-Powered Browsing* [00:12:26] Running headless browsers at scale* [00:18:46] Geolocation when proxying* [00:21:25] CAPTCHAs and Agent Auth* [00:28:21] Building “User take over” functionality* [00:33:43] Stagehand: AI web browsing framework* [00:38:58] OpenAI's Operator and computer use agents* [00:44:44] Surprising use cases of Browserbase* [00:47:18] Future of browser automation and market competition* [00:53:11] Being a solo founderTranscriptAlessio [00:00:04]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol.ai.swyx [00:00:12]: Hey, and today we are very blessed to have our friends, Paul Klein, for the fourth, the fourth, CEO of Browserbase. Welcome.Paul [00:00:21]: Thanks guys. Yeah, I'm happy to be here. I've been lucky to know both of you for like a couple of years now, I think. So it's just like we're hanging out, you know, with three ginormous microphones in front of our face. It's totally normal hangout.swyx [00:00:34]: Yeah. We've actually mentioned you on the podcast, I think, more often than any other Solaris tenant. Just because like you're one of the, you know, best performing, I think, LLM tool companies that have started up in the last couple of years.Paul [00:00:50]: Yeah, I mean, it's been a whirlwind of a year, like Browserbase is actually pretty close to our first birthday. So we are one years old. And going from, you know, starting a company as a solo founder to... To, you know, having a team of 20 people, you know, a series A, but also being able to support hundreds of AI companies that are building AI applications that go out and automate the web. It's just been like, really cool. It's been happening a little too fast. I think like collectively as an AI industry, let's just take a week off together. I took my first vacation actually two weeks ago, and Operator came out on the first day, and then a week later, DeepSeat came out. And I'm like on vacation trying to chill. I'm like, we got to build with this stuff, right? So it's been a breakneck year. But I'm super happy to be here and like talk more about all the stuff we're seeing. And I'd love to hear kind of what you guys are excited about too, and share with it, you know?swyx [00:01:39]: Where to start? So people, you've done a bunch of podcasts. I think I strongly recommend Jack Bridger's Scaling DevTools, as well as Turner Novak's The Peel. And, you know, I'm sure there's others. So you covered your Twilio story in the past, talked about StreamClub, you got acquired to Mux, and then you left to start Browserbase. So maybe we just start with what is Browserbase? Yeah.Paul [00:02:02]: Browserbase is the web browser for your AI. We're building headless browser infrastructure, which are browsers that run in a server environment that's accessible to developers via APIs and SDKs. It's really hard to run a web browser in the cloud. You guys are probably running Chrome on your computers, and that's using a lot of resources, right? So if you want to run a web browser or thousands of web browsers, you can't just spin up a bunch of lambdas. You actually need to use a secure containerized environment. You have to scale it up and down. It's a stateful system. And that infrastructure is, like, super painful. And I know that firsthand, because at my last company, StreamClub, I was CTO, and I was building our own internal headless browser infrastructure. That's actually why we sold the company, is because Mux really wanted to buy our headless browser infrastructure that we'd built. And it's just a super hard problem. And I actually told my co-founders, I would never start another company unless it was a browser infrastructure company. And it turns out that's really necessary in the age of AI, when AI can actually go out and interact with websites, click on buttons, fill in forms. You need AI to do all of that work in an actual browser running somewhere on a server. And BrowserBase powers that.swyx [00:03:08]: While you're talking about it, it occurred to me, not that you're going to be acquired or anything, but it occurred to me that it would be really funny if you became the Nikita Beer of headless browser companies. You just have one trick, and you make browser companies that get acquired.Paul [00:03:23]: I truly do only have one trick. I'm screwed if it's not for headless browsers. I'm not a Go programmer. You know, I'm in AI grant. You know, browsers is an AI grant. But we were the only company in that AI grant batch that used zero dollars on AI spend. You know, we're purely an infrastructure company. So as much as people want to ask me about reinforcement learning, I might not be the best guy to talk about that. But if you want to ask about headless browser infrastructure at scale, I can talk your ear off. So that's really my area of expertise. And it's a pretty niche thing. Like, nobody has done what we're doing at scale before. So we're happy to be the experts.swyx [00:03:59]: You do have an AI thing, stagehand. We can talk about the sort of core of browser-based first, and then maybe stagehand. Yeah, stagehand is kind of the web browsing framework. Yeah.What is Browserbase? Headless Browser Infrastructure ExplainedAlessio [00:04:10]: Yeah. Yeah. And maybe how you got to browser-based and what problems you saw. So one of the first things I worked on as a software engineer was integration testing. Sauce Labs was kind of like the main thing at the time. And then we had Selenium, we had Playbrite, we had all these different browser things. But it's always been super hard to do. So obviously you've worked on this before. When you started browser-based, what were the challenges? What were the AI-specific challenges that you saw versus, there's kind of like all the usual running browser at scale in the cloud, which has been a problem for years. What are like the AI unique things that you saw that like traditional purchase just didn't cover? Yeah.AI-specific challenges in browser infrastructurePaul [00:04:46]: First and foremost, I think back to like the first thing I did as a developer, like as a kid when I was writing code, I wanted to write code that did stuff for me. You know, I wanted to write code to automate my life. And I do that probably by using curl or beautiful soup to fetch data from a web browser. And I think I still do that now that I'm in the cloud. And the other thing that I think is a huge challenge for me is that you can't just create a web site and parse that data. And we all know that now like, you know, taking HTML and plugging that into an LLM, you can extract insights, you can summarize. So it was very clear that now like dynamic web scraping became very possible with the rise of large language models or a lot easier. And that was like a clear reason why there's been more usage of headless browsers, which are necessary because a lot of modern websites don't expose all of their page content via a simple HTTP request. You know, they actually do require you to run this type of code for a specific time. JavaScript on the page to hydrate this. Airbnb is a great example. You go to airbnb.com. A lot of that content on the page isn't there until after they run the initial hydration. So you can't just scrape it with a curl. You need to have some JavaScript run. And a browser is that JavaScript engine that's going to actually run all those requests on the page. So web data retrieval was definitely one driver of starting BrowserBase and the rise of being able to summarize that within LLM. Also, I was familiar with if I wanted to automate a website, I could write one script and that would work for one website. It was very static and deterministic. But the web is non-deterministic. The web is always changing. And until we had LLMs, there was no way to write scripts that you could write once that would run on any website. That would change with the structure of the website. Click the login button. It could mean something different on many different websites. And LLMs allow us to generate code on the fly to actually control that. So I think that rise of writing the generic automation scripts that can work on many different websites, to me, made it clear that browsers are going to be a lot more useful because now you can automate a lot more things without writing. If you wanted to write a script to book a demo call on 100 websites, previously, you had to write 100 scripts. Now you write one script that uses LLMs to generate that script. That's why we built our web browsing framework, StageHand, which does a lot of that work for you. But those two things, web data collection and then enhanced automation of many different websites, it just felt like big drivers for more browser infrastructure that would be required to power these kinds of features.Alessio [00:07:05]: And was multimodality also a big thing?Paul [00:07:08]: Now you can use the LLMs to look, even though the text in the dome might not be as friendly. Maybe my hot take is I was always kind of like, I didn't think vision would be as big of a driver. For UI automation, I felt like, you know, HTML is structured text and large language models are good with structured text. But it's clear that these computer use models are often vision driven, and they've been really pushing things forward. So definitely being multimodal, like rendering the page is required to take a screenshot to give that to a computer use model to take actions on a website. And it's just another win for browser. But I'll be honest, that wasn't what I was thinking early on. I didn't even think that we'd get here so fast with multimodality. I think we're going to have to get back to multimodal and vision models.swyx [00:07:50]: This is one of those things where I forgot to mention in my intro that I'm an investor in Browserbase. And I remember that when you pitched to me, like a lot of the stuff that we have today, we like wasn't on the original conversation. But I did have my original thesis was something that we've talked about on the podcast before, which is take the GPT store, the custom GPT store, all the every single checkbox and plugin is effectively a startup. And this was the browser one. I think the main hesitation, I think I actually took a while to get back to you. The main hesitation was that there were others. Like you're not the first hit list browser startup. It's not even your first hit list browser startup. There's always a question of like, will you be the category winner in a place where there's a bunch of incumbents, to be honest, that are bigger than you? They're just not targeted at the AI space. They don't have the backing of Nat Friedman. And there's a bunch of like, you're here in Silicon Valley. They're not. I don't know.Paul [00:08:47]: I don't know if that's, that was it, but like, there was a, yeah, I mean, like, I think I tried all the other ones and I was like, really disappointed. Like my background is from working at great developer tools, companies, and nothing had like the Vercel like experience. Um, like our biggest competitor actually is partly owned by private equity and they just jacked up their prices quite a bit. And the dashboard hasn't changed in five years. And I actually used them at my last company and tried them and I was like, oh man, like there really just needs to be something that's like the experience of these great infrastructure companies, like Stripe, like clerk, like Vercel that I use in love, but oriented towards this kind of like more specific category, which is browser infrastructure, which is really technically complex. Like a lot of stuff can go wrong on the internet when you're running a browser. The internet is very vast. There's a lot of different configurations. Like there's still websites that only work with internet explorer out there. How do you handle that when you're running your own browser infrastructure? These are the problems that we have to think about and solve at BrowserBase. And it's, it's certainly a labor of love, but I built this for me, first and foremost, I know it's super cheesy and everyone says that for like their startups, but it really, truly was for me. If you look at like the talks I've done even before BrowserBase, and I'm just like really excited to try and build a category defining infrastructure company. And it's, it's rare to have a new category of infrastructure exists. We're here in the Chroma offices and like, you know, vector databases is a new category of infrastructure. Is it, is it, I mean, we can, we're in their office, so, you know, we can, we can debate that one later. That is one.Multimodality in AI-Powered Browsingswyx [00:10:16]: That's one of the industry debates.Paul [00:10:17]: I guess we go back to the LLMOS talk that Karpathy gave way long ago. And like the browser box was very clearly there and it seemed like the people who were building in this space also agreed that browsers are a core primitive of infrastructure for the LLMOS that's going to exist in the future. And nobody was building something there that I wanted to use. So I had to go build it myself.swyx [00:10:38]: Yeah. I mean, exactly that talk that, that honestly, that diagram, every box is a startup and there's the code box and then there's the. The browser box. I think at some point they will start clashing there. There's always the question of the, are you a point solution or are you the sort of all in one? And I think the point solutions tend to win quickly, but then the only ones have a very tight cohesive experience. Yeah. Let's talk about just the hard problems of browser base you have on your website, which is beautiful. Thank you. Was there an agency that you used for that? Yeah. Herb.paris.Paul [00:11:11]: They're amazing. Herb.paris. Yeah. It's H-E-R-V-E. I highly recommend for developers. Developer tools, founders to work with consumer agencies because they end up building beautiful things and the Parisians know how to build beautiful interfaces. So I got to give prep.swyx [00:11:24]: And chat apps, apparently are, they are very fast. Oh yeah. The Mistral chat. Yeah. Mistral. Yeah.Paul [00:11:31]: Late chat.swyx [00:11:31]: Late chat. And then your videos as well, it was professionally shot, right? The series A video. Yeah.Alessio [00:11:36]: Nico did the videos. He's amazing. Not the initial video that you shot at the new one. First one was Austin.Paul [00:11:41]: Another, another video pretty surprised. But yeah, I mean, like, I think when you think about how you talk about your company. You have to think about the way you present yourself. It's, you know, as a developer, you think you evaluate a company based on like the API reliability and the P 95, but a lot of developers say, is the website good? Is the message clear? Do I like trust this founder? I'm building my whole feature on. So I've tried to nail that as well as like the reliability of the infrastructure. You're right. It's very hard. And there's a lot of kind of foot guns that you run into when running headless browsers at scale. Right.Competing with Existing Headless Browser Solutionsswyx [00:12:10]: So let's pick one. You have eight features here. Seamless integration. Scalability. Fast or speed. Secure. Observable. Stealth. That's interesting. Extensible and developer first. What comes to your mind as like the top two, three hardest ones? Yeah.Running headless browsers at scalePaul [00:12:26]: I think just running headless browsers at scale is like the hardest one. And maybe can I nerd out for a second? Is that okay? I heard this is a technical audience, so I'll talk to the other nerds. Whoa. They were listening. Yeah. They're upset. They're ready. The AGI is angry. Okay. So. So how do you run a browser in the cloud? Let's start with that, right? So let's say you're using a popular browser automation framework like Puppeteer, Playwright, and Selenium. Maybe you've written a code, some code locally on your computer that opens up Google. It finds the search bar and then types in, you know, search for Latent Space and hits the search button. That script works great locally. You can see the little browser open up. You want to take that to production. You want to run the script in a cloud environment. So when your laptop is closed, your browser is doing something. The browser is doing something. Well, I, we use Amazon. You can see the little browser open up. You know, the first thing I'd reach for is probably like some sort of serverless infrastructure. I would probably try and deploy on a Lambda. But Chrome itself is too big to run on a Lambda. It's over 250 megabytes. So you can't easily start it on a Lambda. So you maybe have to use something like Lambda layers to squeeze it in there. Maybe use a different Chromium build that's lighter. And you get it on the Lambda. Great. It works. But it runs super slowly. It's because Lambdas are very like resource limited. They only run like with one vCPU. You can run one process at a time. Remember, Chromium is super beefy. It's barely running on my MacBook Air. I'm still downloading it from a pre-run. Yeah, from the test earlier, right? I'm joking. But it's big, you know? So like Lambda, it just won't work really well. Maybe it'll work, but you need something faster. Your users want something faster. Okay. Well, let's put it on a beefier instance. Let's get an EC2 server running. Let's throw Chromium on there. Great. Okay. I can, that works well with one user. But what if I want to run like 10 Chromium instances, one for each of my users? Okay. Well, I might need two EC2 instances. Maybe 10. All of a sudden, you have multiple EC2 instances. This sounds like a problem for Kubernetes and Docker, right? Now, all of a sudden, you're using ECS or EKS, the Kubernetes or container solutions by Amazon. You're spending up and down containers, and you're spending a whole engineer's time on kind of maintaining this stateful distributed system. Those are some of the worst systems to run because when it's a stateful distributed system, it means that you are bound by the connections to that thing. You have to keep the browser open while someone is working with it, right? That's just a painful architecture to run. And there's all this other little gotchas with Chromium, like Chromium, which is the open source version of Chrome, by the way. You have to install all these fonts. You want emojis working in your browsers because your vision model is looking for the emoji. You need to make sure you have the emoji fonts. You need to make sure you have all the right extensions configured, like, oh, do you want ad blocking? How do you configure that? How do you actually record all these browser sessions? Like it's a headless browser. You can't look at it. So you need to have some sort of observability. Maybe you're recording videos and storing those somewhere. It all kind of adds up to be this just giant monster piece of your project when all you wanted to do was run a lot of browsers in production for this little script to go to google.com and search. And when I see a complex distributed system, I see an opportunity to build a great infrastructure company. And we really abstract that away with Browserbase where our customers can use these existing frameworks, Playwright, Publisher, Selenium, or our own stagehand and connect to our browsers in a serverless-like way. And control them, and then just disconnect when they're done. And they don't have to think about the complex distributed system behind all of that. They just get a browser running anywhere, anytime. Really easy to connect to.swyx [00:15:55]: I'm sure you have questions. My standard question with anything, so essentially you're a serverless browser company, and there's been other serverless things that I'm familiar with in the past, serverless GPUs, serverless website hosting. That's where I come from with Netlify. One question is just like, you promised to spin up thousands of servers. You promised to spin up thousands of browsers in milliseconds. I feel like there's no real solution that does that yet. And I'm just kind of curious how. The only solution I know, which is to kind of keep a kind of warm pool of servers around, which is expensive, but maybe not so expensive because it's just CPUs. So I'm just like, you know. Yeah.Browsers as a Core Primitive in AI InfrastructurePaul [00:16:36]: You nailed it, right? I mean, how do you offer a serverless-like experience with something that is clearly not serverless, right? And the answer is, you need to be able to run... We run many browsers on single nodes. We use Kubernetes at browser base. So we have many pods that are being scheduled. We have to predictably schedule them up or down. Yes, thousands of browsers in milliseconds is the best case scenario. If you hit us with 10,000 requests, you may hit a slower cold start, right? So we've done a lot of work on predictive scaling and being able to kind of route stuff to different regions where we have multiple regions of browser base where we have different pools available. You can also pick the region you want to go to based on like lower latency, round trip, time latency. It's very important with these types of things. There's a lot of requests going over the wire. So for us, like having a VM like Firecracker powering everything under the hood allows us to be super nimble and spin things up or down really quickly with strong multi-tenancy. But in the end, this is like the complex infrastructural challenges that we have to kind of deal with at browser base. And we have a lot more stuff on our roadmap to allow customers to have more levers to pull to exchange, do you want really fast browser startup times or do you want really low costs? And if you're willing to be more flexible on that, we may be able to kind of like work better for your use cases.swyx [00:17:44]: Since you used Firecracker, shouldn't Fargate do that for you or did you have to go lower level than that? We had to go lower level than that.Paul [00:17:51]: I find this a lot with Fargate customers, which is alarming for Fargate. We used to be a giant Fargate customer. Actually, the first version of browser base was ECS and Fargate. And unfortunately, it's a great product. I think we were actually the largest Fargate customer in our region for a little while. No, what? Yeah, seriously. And unfortunately, it's a great product, but I think if you're an infrastructure company, you actually have to have a deeper level of control over these primitives. I think it's the same thing is true with databases. We've used other database providers and I think-swyx [00:18:21]: Yeah, serverless Postgres.Paul [00:18:23]: Shocker. When you're an infrastructure company, you're on the hook if any provider has an outage. And I can't tell my customers like, hey, we went down because so-and-so went down. That's not acceptable. So for us, we've really moved to bringing things internally. It's kind of opposite of what we preach. We tell our customers, don't build this in-house, but then we're like, we build a lot of stuff in-house. But I think it just really depends on what is in the critical path. We try and have deep ownership of that.Alessio [00:18:46]: On the distributed location side, how does that work for the web where you might get sort of different content in different locations, but the customer is expecting, you know, if you're in the US, I'm expecting the US version. But if you're spinning up my browser in France, I might get the French version. Yeah.Paul [00:19:02]: Yeah. That's a good question. Well, generally, like on the localization, there is a thing called locale in the browser. You can set like what your locale is. If you're like in the ENUS browser or not, but some things do IP, IP based routing. And in that case, you may want to have a proxy. Like let's say you're running something in the, in Europe, but you want to make sure you're showing up from the US. You may want to use one of our proxy features so you can turn on proxies to say like, make sure these connections always come from the United States, which is necessary too, because when you're browsing the web, you're coming from like a, you know, data center IP, and that can make things a lot harder to browse web. So we do have kind of like this proxy super network. Yeah. We have a proxy for you based on where you're going, so you can reliably automate the web. But if you get scheduled in Europe, that doesn't happen as much. We try and schedule you as close to, you know, your origin that you're trying to go to. But generally you have control over the regions you can put your browsers in. So you can specify West one or East one or Europe. We only have one region of Europe right now, actually. Yeah.Alessio [00:19:55]: What's harder, the browser or the proxy? I feel like to me, it feels like actually proxying reliably at scale. It's much harder than spending up browsers at scale. I'm curious. It's all hard.Paul [00:20:06]: It's layers of hard, right? Yeah. I think it's different levels of hard. I think the thing with the proxy infrastructure is that we work with many different web proxy providers and some are better than others. Some have good days, some have bad days. And our customers who've built browser infrastructure on their own, they have to go and deal with sketchy actors. Like first they figure out their own browser infrastructure and then they got to go buy a proxy. And it's like you can pay in Bitcoin and it just kind of feels a little sus, right? It's like you're buying drugs when you're trying to get a proxy online. We have like deep relationships with these counterparties. We're able to audit them and say, is this proxy being sourced ethically? Like it's not running on someone's TV somewhere. Is it free range? Yeah. Free range organic proxies, right? Right. We do a level of diligence. We're SOC 2. So we have to understand what is going on here. But then we're able to make sure that like we route around proxy providers not working. There's proxy providers who will just, the proxy will stop working all of a sudden. And then if you don't have redundant proxying on your own browsers, that's hard down for you or you may get some serious impacts there. With us, like we intelligently know, hey, this proxy is not working. Let's go to this one. And you can kind of build a network of multiple providers to really guarantee the best uptime for our customers. Yeah. So you don't own any proxies? We don't own any proxies. You're right. The team has been saying who wants to like take home a little proxy server, but not yet. We're not there yet. You know?swyx [00:21:25]: It's a very mature market. I don't think you should build that yourself. Like you should just be a super customer of them. Yeah. Scraping, I think, is the main use case for that. I guess. Well, that leads us into CAPTCHAs and also off, but let's talk about CAPTCHAs. You had a little spiel that you wanted to talk about CAPTCHA stuff.Challenges of Scaling Browser InfrastructurePaul [00:21:43]: Oh, yeah. I was just, I think a lot of people ask, if you're thinking about proxies, you're thinking about CAPTCHAs too. I think it's the same thing. You can go buy CAPTCHA solvers online, but it's the same buying experience. It's some sketchy website, you have to integrate it. It's not fun to buy these things and you can't really trust that the docs are bad. What Browserbase does is we integrate a bunch of different CAPTCHAs. We do some stuff in-house, but generally we just integrate with a bunch of known vendors and continually monitor and maintain these things and say, is this working or not? Can we route around it or not? These are CAPTCHA solvers. CAPTCHA solvers, yeah. Not CAPTCHA providers, CAPTCHA solvers. Yeah, sorry. CAPTCHA solvers. We really try and make sure all of that works for you. I think as a dev, if I'm buying infrastructure, I want it all to work all the time and it's important for us to provide that experience by making sure everything does work and monitoring it on our own. Yeah. Right now, the world of CAPTCHAs is tricky. I think AI agents in particular are very much ahead of the internet infrastructure. CAPTCHAs are designed to block all types of bots, but there are now good bots and bad bots. I think in the future, CAPTCHAs will be able to identify who a good bot is, hopefully via some sort of KYC. For us, we've been very lucky. We have very little to no known abuse of Browserbase because we really look into who we work with. And for certain types of CAPTCHA solving, we only allow them on certain types of plans because we want to make sure that we can know what people are doing, what their use cases are. And that's really allowed us to try and be an arbiter of good bots, which is our long term goal. I want to build great relationships with people like Cloudflare so we can agree, hey, here are these acceptable bots. We'll identify them for you and make sure we flag when they come to your website. This is a good bot, you know?Alessio [00:23:23]: I see. And Cloudflare said they want to do more of this. So they're going to set by default, if they think you're an AI bot, they're going to reject. I'm curious if you think this is something that is going to be at the browser level or I mean, the DNS level with Cloudflare seems more where it should belong. But I'm curious how you think about it.Paul [00:23:40]: I think the web's going to change. You know, I think that the Internet as we have it right now is going to change. And we all need to just accept that the cat is out of the bag. And instead of kind of like wishing the Internet was like it was in the 2000s, we can have free content line that wouldn't be scraped. It's just it's not going to happen. And instead, we should think about like, one, how can we change? How can we change the models of, you know, information being published online so people can adequately commercialize it? But two, how do we rebuild applications that expect that AI agents are going to log in on their behalf? Those are the things that are going to allow us to kind of like identify good and bad bots. And I think the team at Clerk has been doing a really good job with this on the authentication side. I actually think that auth is the biggest thing that will prevent agents from accessing stuff, not captchas. And I think there will be agent auth in the future. I don't know if it's going to happen from an individual company, but actually authentication providers that have a, you know, hidden login as agent feature, which will then you put in your email, you'll get a push notification, say like, hey, your browser-based agent wants to log into your Airbnb. You can approve that and then the agent can proceed. That really circumvents the need for captchas or logging in as you and sharing your password. I think agent auth is going to be one way we identify good bots going forward. And I think a lot of this captcha solving stuff is really short-term problems as the internet kind of reorients itself around how it's going to work with agents browsing the web, just like people do. Yeah.Managing Distributed Browser Locations and Proxiesswyx [00:24:59]: Stitch recently was on Hacker News for talking about agent experience, AX, which is a thing that Netlify is also trying to clone and coin and talk about. And we've talked about this on our previous episodes before in a sense that I actually think that's like maybe the only part of the tech stack that needs to be kind of reinvented for agents. Everything else can stay the same, CLIs, APIs, whatever. But auth, yeah, we need agent auth. And it's mostly like short-lived, like it should not, it should be a distinct, identity from the human, but paired. I almost think like in the same way that every social network should have your main profile and then your alt accounts or your Finsta, it's almost like, you know, every, every human token should be paired with the agent token and the agent token can go and do stuff on behalf of the human token, but not be presumed to be the human. Yeah.Paul [00:25:48]: It's like, it's, it's actually very similar to OAuth is what I'm thinking. And, you know, Thread from Stitch is an investor, Colin from Clerk, Octaventures, all investors in browser-based because like, I hope they solve this because they'll make browser-based submission more possible. So we don't have to overcome all these hurdles, but I think it will be an OAuth-like flow where an agent will ask to log in as you, you'll approve the scopes. Like it can book an apartment on Airbnb, but it can't like message anybody. And then, you know, the agent will have some sort of like role-based access control within an application. Yeah. I'm excited for that.swyx [00:26:16]: The tricky part is just, there's one, one layer of delegation here, which is like, you're authoring my user's user or something like that. I don't know if that's tricky or not. Does that make sense? Yeah.Paul [00:26:25]: You know, actually at Twilio, I worked on the login identity and access. Management teams, right? So like I built Twilio's login page.swyx [00:26:31]: You were an intern on that team and then you became the lead in two years? Yeah.Paul [00:26:34]: Yeah. I started as an intern in 2016 and then I was the tech lead of that team. How? That's not normal. I didn't have a life. He's not normal. Look at this guy. I didn't have a girlfriend. I just loved my job. I don't know. I applied to 500 internships for my first job and I got rejected from every single one of them except for Twilio and then eventually Amazon. And they took a shot on me and like, I was getting paid money to write code, which was my dream. Yeah. Yeah. I'm very lucky that like this coding thing worked out because I was going to be doing it regardless. And yeah, I was able to kind of spend a lot of time on a team that was growing at a company that was growing. So it informed a lot of this stuff here. I think these are problems that have been solved with like the SAML protocol with SSO. I think it's a really interesting stuff with like WebAuthn, like these different types of authentication, like schemes that you can use to authenticate people. The tooling is all there. It just needs to be tweaked a little bit to work for agents. And I think the fact that there are companies that are already. Providing authentication as a service really sets it up. Well, the thing that's hard is like reinventing the internet for agents. We don't want to rebuild the internet. That's an impossible task. And I think people often say like, well, we'll have this second layer of APIs built for agents. I'm like, we will for the top use cases, but instead of we can just tweak the internet as is, which is on the authentication side, I think we're going to be the dumb ones going forward. Unfortunately, I think AI is going to be able to do a lot of the tasks that we do online, which means that it will be able to go to websites, click buttons on our behalf and log in on our behalf too. So with this kind of like web agent future happening, I think with some small structural changes, like you said, it feels like it could all slot in really nicely with the existing internet.Handling CAPTCHAs and Agent Authenticationswyx [00:28:08]: There's one more thing, which is the, your live view iframe, which lets you take, take control. Yeah. Obviously very key for operator now, but like, was, is there anything interesting technically there or that the people like, well, people always want this.Paul [00:28:21]: It was really hard to build, you know, like, so, okay. Headless browsers, you don't see them, right. They're running. They're running in a cloud somewhere. You can't like look at them. And I just want to really make, it's a weird name. I wish we came up with a better name for this thing, but you can't see them. Right. But customers don't trust AI agents, right. At least the first pass. So what we do with our live view is that, you know, when you use browser base, you can actually embed a live view of the browser running in the cloud for your customer to see it working. And that's what the first reason is the build trust, like, okay, so I have this script. That's going to go automate a website. I can embed it into my web application via an iframe and my customer can watch. I think. And then we added two way communication. So now not only can you watch the browser kind of being operated by AI, if you want to pause and actually click around type within this iframe that's controlling a browser, that's also possible. And this is all thanks to some of the lower level protocol, which is called the Chrome DevTools protocol. It has a API called start screencast, and you can also send mouse clicks and button clicks to a remote browser. And this is all embeddable within iframes. You have a browser within a browser, yo. And then you simulate the screen, the click on the other side. Exactly. And this is really nice often for, like, let's say, a capture that can't be solved. You saw this with Operator, you know, Operator actually uses a different approach. They use VNC. So, you know, you're able to see, like, you're seeing the whole window here. What we're doing is something a little lower level with the Chrome DevTools protocol. It's just PNGs being streamed over the wire. But the same thing is true, right? Like, hey, I'm running a window. Pause. Can you do something in this window? Human. Okay, great. Resume. Like sometimes 2FA tokens. Like if you get that text message, you might need a person to type that in. Web agents need human-in-the-loop type workflows still. You still need a person to interact with the browser. And building a UI to proxy that is kind of hard. You may as well just show them the whole browser and say, hey, can you finish this up for me? And then let the AI proceed on afterwards. Is there a future where I stream my current desktop to browser base? I don't think so. I think we're very much cloud infrastructure. Yeah. You know, but I think a lot of the stuff we're doing, we do want to, like, build tools. Like, you know, we'll talk about the stage and, you know, web agent framework in a second. But, like, there's a case where a lot of people are going desktop first for, you know, consumer use. And I think cloud is doing a lot of this, where I expect to see, you know, MCPs really oriented around the cloud desktop app for a reason, right? Like, I think a lot of these tools are going to run on your computer because it makes... I think it's breaking out. People are putting it on a server. Oh, really? Okay. Well, sweet. We'll see. We'll see that. I was surprised, though, wasn't I? I think that the browser company, too, with Dia Browser, it runs on your machine. You know, it's going to be...swyx [00:30:50]: What is it?Paul [00:30:51]: So, Dia Browser, as far as I understand... I used to use Arc. Yeah. I haven't used Arc. But I'm a big fan of the browser company. I think they're doing a lot of cool stuff in consumer. As far as I understand, it's a browser where you have a sidebar where you can, like, chat with it and it can control the local browser on your machine. So, if you imagine, like, what a consumer web agent is, which it lives alongside your browser, I think Google Chrome has Project Marina, I think. I almost call it Project Marinara for some reason. I don't know why. It's...swyx [00:31:17]: No, I think it's someone really likes the Waterworld. Oh, I see. The classic Kevin Costner. Yeah.Paul [00:31:22]: Okay. Project Marinara is a similar thing to the Dia Browser, in my mind, as far as I understand it. You have a browser that has an AI interface that will take over your mouse and keyboard and control the browser for you. Great for consumer use cases. But if you're building applications that rely on a browser and it's more part of a greater, like, AI app experience, you probably need something that's more like infrastructure, not a consumer app.swyx [00:31:44]: Just because I have explored a little bit in this area, do people want branching? So, I have the state. Of whatever my browser's in. And then I want, like, 100 clones of this state. Do people do that? Or...Paul [00:31:56]: People don't do it currently. Yeah. But it's definitely something we're thinking about. I think the idea of forking a browser is really cool. Technically, kind of hard. We're starting to see this in code execution, where people are, like, forking some, like, code execution, like, processes or forking some tool calls or branching tool calls. Haven't seen it at the browser level yet. But it makes sense. Like, if an AI agent is, like, using a website and it's not sure what path it wants to take to crawl this website. To find the information it's looking for. It would make sense for it to explore both paths in parallel. And that'd be a very, like... A road not taken. Yeah. And hopefully find the right answer. And then say, okay, this was actually the right one. And memorize that. And go there in the future. On the roadmap. For sure. Don't make my roadmap, please. You know?Alessio [00:32:37]: How do you actually do that? Yeah. How do you fork? I feel like the browser is so stateful for so many things.swyx [00:32:42]: Serialize the state. Restore the state. I don't know.Paul [00:32:44]: So, it's one of the reasons why we haven't done it yet. It's hard. You know? Like, to truly fork, it's actually quite difficult. The naive way is to open the same page in a new tab and then, like, hope that it's at the same thing. But if you have a form halfway filled, you may have to, like, take the whole, you know, container. Pause it. All the memory. Duplicate it. Restart it from there. It could be very slow. So, we haven't found a thing. Like, the easy thing to fork is just, like, copy the page object. You know? But I think there needs to be something a little bit more robust there. Yeah.swyx [00:33:12]: So, MorphLabs has this infinite branch thing. Like, wrote a custom fork of Linux or something that let them save the system state and clone it. MorphLabs, hit me up. I'll be a customer. Yeah. That's the only. I think that's the only way to do it. Yeah. Like, unless Chrome has some special API for you. Yeah.Paul [00:33:29]: There's probably something we'll reverse engineer one day. I don't know. Yeah.Alessio [00:33:32]: Let's talk about StageHand, the AI web browsing framework. You have three core components, Observe, Extract, and Act. Pretty clean landing page. What was the idea behind making a framework? Yeah.Stagehand: AI web browsing frameworkPaul [00:33:43]: So, there's three frameworks that are very popular or already exist, right? Puppeteer, Playwright, Selenium. Those are for building hard-coded scripts to control websites. And as soon as I started to play with LLMs plus browsing, I caught myself, you know, code-genning Playwright code to control a website. I would, like, take the DOM. I'd pass it to an LLM. I'd say, can you generate the Playwright code to click the appropriate button here? And it would do that. And I was like, this really should be part of the frameworks themselves. And I became really obsessed with SDKs that take natural language as part of, like, the API input. And that's what StageHand is. StageHand exposes three APIs, and it's a super set of Playwright. So, if you go to a page, you may want to take an action, click on the button, fill in the form, etc. That's what the act command is for. You may want to extract some data. This one takes a natural language, like, extract the winner of the Super Bowl from this page. You can give it a Zod schema, so it returns a structured output. And then maybe you're building an API. You can do an agent loop, and you want to kind of see what actions are possible on this page before taking one. You can do observe. So, you can observe the actions on the page, and it will generate a list of actions. You can guide it, like, give me actions on this page related to buying an item. And you can, like, buy it now, add to cart, view shipping options, and pass that to an LLM, an agent loop, to say, what's the appropriate action given this high-level goal? So, StageHand isn't a web agent. It's a framework for building web agents. And we think that agent loops are actually pretty close to the application layer because every application probably has different goals or different ways it wants to take steps. I don't think I've seen a generic. Maybe you guys are the experts here. I haven't seen, like, a really good AI agent framework here. Everyone kind of has their own special sauce, right? I see a lot of developers building their own agent loops, and they're using tools. And I view StageHand as the browser tool. So, we expose act, extract, observe. Your agent can call these tools. And from that, you don't have to worry about it. You don't have to worry about generating playwright code performantly. You don't have to worry about running it. You can kind of just integrate these three tool calls into your agent loop and reliably automate the web.swyx [00:35:48]: A special shout-out to Anirudh, who I met at your dinner, who I think listens to the pod. Yeah. Hey, Anirudh.Paul [00:35:54]: Anirudh's a man. He's a StageHand guy.swyx [00:35:56]: I mean, the interesting thing about each of these APIs is they're kind of each startup. Like, specifically extract, you know, Firecrawler is extract. There's, like, Expand AI. There's a whole bunch of, like, extract companies. They just focus on extract. I'm curious. Like, I feel like you guys are going to collide at some point. Like, right now, it's friendly. Everyone's in a blue ocean. At some point, it's going to be valuable enough that there's some turf battle here. I don't think you have a dog in a fight. I think you can mock extract to use an external service if they're better at it than you. But it's just an observation that, like, in the same way that I see each option, each checkbox in the side of custom GBTs becoming a startup or each box in the Karpathy chart being a startup. Like, this is also becoming a thing. Yeah.Paul [00:36:41]: I mean, like, so the way StageHand works is that it's MIT-licensed, completely open source. You bring your own API key to your LLM of choice. You could choose your LLM. We don't make any money off of the extract or really. We only really make money if you choose to run it with our browser. You don't have to. You can actually use your own browser, a local browser. You know, StageHand is completely open source for that reason. And, yeah, like, I think if you're building really complex web scraping workflows, I don't know if StageHand is the tool for you. I think it's really more if you're building an AI agent that needs a few general tools or if it's doing a lot of, like, web automation-intensive work. But if you're building a scraping company, StageHand is not your thing. You probably want something that's going to, like, get HTML content, you know, convert that to Markdown, query it. That's not what StageHand does. StageHand is more about reliability. I think we focus a lot on reliability and less so on cost optimization and speed at this point.swyx [00:37:33]: I actually feel like StageHand, so the way that StageHand works, it's like, you know, page.act, click on the quick start. Yeah. It's kind of the integration test for the code that you would have to write anyway, like the Puppeteer code that you have to write anyway. And when the page structure changes, because it always does, then this is still the test. This is still the test that I would have to write. Yeah. So it's kind of like a testing framework that doesn't need implementation detail.Paul [00:37:56]: Well, yeah. I mean, Puppeteer, Playwright, and Slenderman were all designed as testing frameworks, right? Yeah. And now people are, like, hacking them together to automate the web. I would say, and, like, maybe this is, like, me being too specific. But, like, when I write tests, if the page structure changes. Without me knowing, I want that test to fail. So I don't know if, like, AI, like, regenerating that. Like, people are using StageHand for testing. But it's more for, like, usability testing, not, like, testing of, like, does the front end, like, has it changed or not. Okay. But generally where we've seen people, like, really, like, take off is, like, if they're using, you know, something. If they want to build a feature in their application that's kind of like Operator or Deep Research, they're using StageHand to kind of power that tool calling in their own agent loop. Okay. Cool.swyx [00:38:37]: So let's go into Operator, the first big agent launch of the year from OpenAI. Seems like they have a whole bunch scheduled. You were on break and your phone blew up. What's your just general view of computer use agents is what they're calling it. The overall category before we go into Open Operator, just the overall promise of Operator. I will observe that I tried it once. It was okay. And I never tried it again.OpenAI's Operator and computer use agentsPaul [00:38:58]: That tracks with my experience, too. Like, I'm a huge fan of the OpenAI team. Like, I think that I do not view Operator as the company. I'm not a company killer for browser base at all. I think it actually shows people what's possible. I think, like, computer use models make a lot of sense. And I'm actually most excited about computer use models is, like, their ability to, like, really take screenshots and reasoning and output steps. I think that using mouse click or mouse coordinates, I've seen that proved to be less reliable than I would like. And I just wonder if that's the right form factor. What we've done with our framework is anchor it to the DOM itself, anchor it to the actual item. So, like, if it's clicking on something, it's clicking on that thing, you know? Like, it's more accurate. No matter where it is. Yeah, exactly. Because it really ties in nicely. And it can handle, like, the whole viewport in one go, whereas, like, Operator can only handle what it sees. Can you hover? Is hovering a thing that you can do? I don't know if we expose it as a tool directly, but I'm sure there's, like, an API for hovering. Like, move mouse to this position. Yeah, yeah, yeah. I think you can trigger hover, like, via, like, the JavaScript on the DOM itself. But, no, I think, like, when we saw computer use, everyone's eyes lit up because they realized, like, wow, like, AI is going to actually automate work for people. And I think seeing that kind of happen from both of the labs, and I'm sure we're going to see more labs launch computer use models, I'm excited to see all the stuff that people build with it. I think that I'd love to see computer use power, like, controlling a browser on browser base. And I think, like, Open Operator, which was, like, our open source version of OpenAI's Operator, was our first take on, like, how can we integrate these models into browser base? And we handle the infrastructure and let the labs do the models. I don't have a sense that Operator will be released as an API. I don't know. Maybe it will. I'm curious to see how well that works because I think it's going to be really hard for a company like OpenAI to do things like support CAPTCHA solving or, like, have proxies. Like, I think it's hard for them structurally. Imagine this New York Times headline, OpenAI CAPTCHA solving. Like, that would be a pretty bad headline, this New York Times headline. Browser base solves CAPTCHAs. No one cares. No one cares. And, like, our investors are bored. Like, we're all okay with this, you know? We're building this company knowing that the CAPTCHA solving is short-lived until we figure out how to authenticate good bots. I think it's really hard for a company like OpenAI, who has this brand that's so, so good, to balance with, like, the icky parts of web automation, which it can be kind of complex to solve. I'm sure OpenAI knows who to call whenever they need you. Yeah, right. I'm sure they'll have a great partnership.Alessio [00:41:23]: And is Open Operator just, like, a marketing thing for you? Like, how do you think about resource allocation? So, you can spin this up very quickly. And now there's all this, like, open deep research, just open all these things that people are building. We started it, you know. You're the original Open. We're the original Open operator, you know? Is it just, hey, look, this is a demo, but, like, we'll help you build out an actual product for yourself? Like, are you interested in going more of a product route? That's kind of the OpenAI way, right? They started as a model provider and then…Paul [00:41:53]: Yeah, we're not interested in going the product route yet. I view Open Operator as a model provider. It's a reference project, you know? Let's show people how to build these things using the infrastructure and models that are out there. And that's what it is. It's, like, Open Operator is very simple. It's an agent loop. It says, like, take a high-level goal, break it down into steps, use tool calling to accomplish those steps. It takes screenshots and feeds those screenshots into an LLM with the step to generate the right action. It uses stagehand under the hood to actually execute this action. It doesn't use a computer use model. And it, like, has a nice interface using the live view that we talked about, the iframe, to embed that into an application. So I felt like people on launch day wanted to figure out how to build their own version of this. And we turned that around really quickly to show them. And I hope we do that with other things like deep research. We don't have a deep research launch yet. I think David from AOMNI actually has an amazing open deep research that he launched. It has, like, 10K GitHub stars now. So he's crushing that. But I think if people want to build these features natively into their application, they need good reference projects. And I think Open Operator is a good example of that.swyx [00:42:52]: I don't know. Actually, I'm actually pretty bullish on API-driven operator. Because that's the only way that you can sort of, like, once it's reliable enough, obviously. And now we're nowhere near. But, like, give it five years. It'll happen, you know. And then you can sort of spin this up and browsers are working in the background and you don't necessarily have to know. And it just is booking restaurants for you, whatever. I can definitely see that future happening. I had this on the landing page here. This might be a slightly out of order. But, you know, you have, like, sort of three use cases for browser base. Open Operator. Or this is the operator sort of use case. It's kind of like the workflow automation use case. And it completes with UiPath in the sort of RPA category. Would you agree with that? Yeah, I would agree with that. And then there's Agents we talked about already. And web scraping, which I imagine would be the bulk of your workload right now, right?Paul [00:43:40]: No, not at all. I'd say actually, like, the majority is browser automation. We're kind of expensive for web scraping. Like, I think that if you're building a web scraping product, if you need to do occasional web scraping or you have to do web scraping that works every single time, you want to use browser automation. Yeah. You want to use browser-based. But if you're building web scraping workflows, what you should do is have a waterfall. You should have the first request is a curl to the website. See if you can get it without even using a browser. And then the second request may be, like, a scraping-specific API. There's, like, a thousand scraping APIs out there that you can use to try and get data. Scraping B. Scraping B is a great example, right? Yeah. And then, like, if those two don't work, bring out the heavy hitter. Like, browser-based will 100% work, right? It will load the page in a real browser, hydrate it. I see.swyx [00:44:21]: Because a lot of people don't render to JS.swyx [00:44:25]: Yeah, exactly.Paul [00:44:26]: So, I mean, the three big use cases, right? Like, you know, automation, web data collection, and then, you know, if you're building anything agentic that needs, like, a browser tool, you want to use browser-based.Alessio [00:44:35]: Is there any use case that, like, you were super surprised by that people might not even think about? Oh, yeah. Or is it, yeah, anything that you can share? The long tail is crazy. Yeah.Surprising use cases of BrowserbasePaul [00:44:44]: One of the case studies on our website that I think is the most interesting is this company called Benny. So, the way that it works is if you're on food stamps in the United States, you can actually get rebates if you buy certain things. Yeah. You buy some vegetables. You submit your receipt to the government. They'll give you a little rebate back. Say, hey, thanks for buying vegetables. It's good for you. That process of submitting that receipt is very painful. And the way Benny works is you use their app to take a photo of your receipt, and then Benny will go submit that receipt for you and then deposit the money into your account. That's actually using no AI at all. It's all, like, hard-coded scripts. They maintain the scripts. They've been doing a great job. And they build this amazing consumer app. But it's an example of, like, all these, like, tedious workflows that people have to do to kind of go about their business. And they're doing it for the sake of their day-to-day lives. And I had never known about, like, food stamp rebates or the complex forms you have to do to fill them. But the world is powered by millions and millions of tedious forms, visas. You know, Emirate Lighthouse is a customer, right? You know, they do the O1 visa. Millions and millions of forms are taking away humans' time. And I hope that Browserbase can help power software that automates away the web forms that we don't need anymore. Yeah.swyx [00:45:49]: I mean, I'm very supportive of that. I mean, forms. I do think, like, government itself is a big part of it. I think the government itself should embrace AI more to do more sort of human-friendly form filling. Mm-hmm. But I'm not optimistic. I'm not holding my breath. Yeah. We'll see. Okay. I think I'm about to zoom out. I have a little brief thing on computer use, and then we can talk about founder stuff, which is, I tend to think of developer tooling markets in impossible triangles, where everyone starts in a niche, and then they start to branch out. So I already hinted at a little bit of this, right? We mentioned more. We mentioned E2B. We mentioned Firecrawl. And then there's Browserbase. So there's, like, all this stuff of, like, have serverless virtual computer that you give to an agent and let them do stuff with it. And there's various ways of connecting it to the internet. You can just connect to a search API, like SERP API, whatever other, like, EXA is another one. That's what you're searching. You can also have a JSON markdown extractor, which is Firecrawl. Or you can have a virtual browser like Browserbase, or you can have a virtual machine like Morph. And then there's also maybe, like, a virtual sort of code environment, like Code Interpreter. So, like, there's just, like, a bunch of different ways to tackle the problem of give a computer to an agent. And I'm just kind of wondering if you see, like, everyone's just, like, happily coexisting in their respective niches. And as a developer, I just go and pick, like, a shopping basket of one of each. Or do you think that you eventually, people will collide?Future of browser automation and market competitionPaul [00:47:18]: I think that currently it's not a zero-sum market. Like, I think we're talking about... I think we're talking about all of knowledge work that people do that can be automated online. All of these, like, trillions of hours that happen online where people are working. And I think that there's so much software to be built that, like, I tend not to think about how these companies will collide. I just try to solve the problem as best as I can and make this specific piece of infrastructure, which I think is an important primitive, the best I possibly can. And yeah. I think there's players that are actually going to like it. I think there's players that are going to launch, like, over-the-top, you know, platforms, like agent platforms that have all these tools built in, right? Like, who's building the rippling for agent tools that has the search tool, the browser tool, the operating system tool, right? There are some. There are some. There are some, right? And I think in the end, what I have seen as my time as a developer, and I look at all the favorite tools that I have, is that, like, for tools and primitives with sufficient levels of complexity, you need to have a solution that's really bespoke to that primitive, you know? And I am sufficiently convinced that the browser is complex enough to deserve a primitive. Obviously, I have to. I'm the founder of BrowserBase, right? I'm talking my book. But, like, I think maybe I can give you one spicy take against, like, maybe just whole OS running. I think that when I look at computer use when it first came out, I saw that the majority of use cases for computer use were controlling a browser. And do we really need to run an entire operating system just to control a browser? I don't think so. I don't think that's necessary. You know, BrowserBase can run browsers for way cheaper than you can if you're running a full-fledged OS with a GUI, you know, operating system. And I think that's just an advantage of the browser. It is, like, browsers are little OSs, and you can run them very efficiently if you orchestrate it well. And I think that allows us to offer 90% of the, you know, functionality in the platform needed at 10% of the cost of running a full OS. Yeah.Open Operator: Browserbase's Open-Source Alternativeswyx [00:49:16]: I definitely see the logic in that. There's a Mark Andreessen quote. I don't know if you know this one. Where he basically observed that the browser is turning the operating system into a poorly debugged set of device drivers, because most of the apps are moved from the OS to the browser. So you can just run browsers.Paul [00:49:31]: There's a place for OSs, too. Like, I think that there are some applications that only run on Windows operating systems. And Eric from pig.dev in this upcoming YC batch, or last YC batch, like, he's building all run tons of Windows operating systems for you to control with your agent. And like, there's some legacy EHR systems that only run on Internet-controlled systems. Yeah.Paul [00:49:54]: I think that's it. I think, like, there are use cases for specific operating systems for specific legacy software. And like, I'm excited to see what he does with that. I just wanted to give a shout out to the pig.dev website.swyx [00:50:06]: The pigs jump when you click on them. Yeah. That's great.Paul [00:50:08]: Eric, he's the former co-founder of banana.dev, too.swyx [00:50:11]: Oh, that Eric. Yeah. That Eric. Okay. Well, he abandoned bananas for pigs. I hope he doesn't start going around with pigs now.Alessio [00:50:18]: Like he was going around with bananas. A little toy pig. Yeah. Yeah. I love that. What else are we missing? I think we covered a lot of, like, the browser-based product history, but. What do you wish people asked you? Yeah.Paul [00:50:29]: I wish people asked me more about, like, what will the future of software look like? Because I think that's really where I've spent a lot of time about why do browser-based. Like, for me, starting a company is like a means of last resort. Like, you shouldn't start a company unless you absolutely have to. And I remain convinced that the future of software is software that you're going to click a button and it's going to do stuff on your behalf. Right now, software. You click a button and it maybe, like, calls it back an API and, like, computes some numbers. It, like, modifies some text, whatever. But the future of software is software using software. So, I may log into my accounting website for my business, click a button, and it's going to go load up my Gmail, search my emails, find the thing, upload the receipt, and then comment it for me. Right? And it may use it using APIs, maybe a browser. I don't know. I think it's a little bit of both. But that's completely different from how we've built software so far. And that's. I think that future of software has different infrastructure requirements. It's going to require different UIs. It's going to require different pieces of infrastructure. I think the browser infrastructure is one piece that fits into that, along with all the other categories you mentioned. So, I think that it's going to require developers to think differently about how they've built software for, you know

EU Cloud Sovereignty & Open Source AlternativesMarket OverviewCurrent EU Cloud Market ShareAWS: ~33% market share (Frankfurt, Ireland, Paris regions)Microsoft Azure: ~25% market shareGoogle Cloud Platform: ~10% market shareOVHcloud: ~5% market share (largest EU-headquartered provider)EU Sovereign Cloud ProvidersFull-Stack European SolutionsOVHcloud (France)33 datacenters across 4 continents, 400K+ serversVertical integration: custom server manufacturing in RoubaixProprietary Linux-based virtualization layerSelf-built European fiber backboneIn-house distributed storage system (non-S3 compatible)Scaleway (France)Growing integration with French AI companies (e.g., Mistral)Custom hypervisor and management planeARM-based server architecturesDatacenters in France, Poland, NetherlandsGrowing rapidly in SME/startup segmentHetzner (Germany)Bare metal-focused infrastructureProprietary virtualization layer100% European datacenters (Germany, Finland)Custom DDoS protection systems designed in GermanyComplete physical/logical isolation from US networksOther European ProvidersDeutsche Telekom/T-Systems (Germany)Orange Business Services (France)SAP (Germany)Leading Open Source Cloud PlatformsTier 1OpenStackMost mature, enterprise-ready open source cloud platformComprehensive IaaS functionality with modular architectureKey components: Nova (compute), Swift (object storage), Neutron (networking)Strong adoption in telecommunications, research, government sectorsKubernetes"Cloud in a box" container orchestration platformNot a complete cloud solution but foundational componentCross-cloud compatibility (GKE, EKS, AKS)Key features: exceptional scalability, self-healing, declarative configurationFacilitates workload portability between cloud providersTier 2Apache CloudStackEnterprise-grade IaaS platformSingle management server architectureStraightforward installation, less architectural flexibilityMature and stable for productionOpenNebulaLightweight virtualization managementLower resource requirements than OpenStackStrong integration with VMware and KVM environmentsEmerging PlatformsRancher/K3sLightweight Kubernetes distributionOptimized for edge computingSimplified binary deployment modelGrowing edge computing ecosystemOKD (OpenShift Kubernetes Distribution)Upstream project for Red Hat OpenShiftDeveloper-focused capabilities on KubernetesGeopolitical & Strategic ContextGrowing US-EU tension creating market opportunity for European cloud sovereigntyEuropean emphasis on data privacy, rights-based innovation, and technological independencePotential bifurcation between US and European technology ecosystemsRising concern about Big Tech's influence on governance and sovereigntyEuropean cloud providers positioned as alternatives emphasizing human rights, privacyTechnical Independence ChallengesProcessor architecture dependencies (Intel/AMD dominance)European Processor Initiative and SiPearl developing EU alternativesFull software stack independence remains aspirationalNetwork equipment supply chain complexities

The Rise of Expertise Inequality in AIKey PointsSimilar to income inequality growth since 1980, we may now be witnessing the emergence of expertise inequality with AIProblem: Automation Claims Lack NuanceClaims about "automating coders" or eliminating software developers oversimplify complex realitiesExample: AWS deployment decisions require expertiseMultiple compute options (EC2, Lambda, ECS Fargate, EKS, Elastic Beanstalk)Each option has significant tradeoffs and use casesSurface-level AI answers lack depth for informed decision-makingExpertise Inequality DynamicsExperts Will ThriveDeep experts can leverage AI effectively They understand fundamental tradeoffs (e.g., compiled vs scripting languages)Can make optimized choices (e.g., Rust for Lambda functions)Know exactly what questions to ask AI systemsBeginners Will StruggleLack domain knowledge to evaluate AI suggestionsDon't understand fundamental distinctions (website vs web service)Cannot properly prompt AI systems due to knowledge gapsOrganizational ImpactDysfunctional organizations at riskHIPAA-driven (High-Paid Person's Opinion)University systemsCorporate bureaucraciesExpert individuals may outperform entire teamsExperts with AI might deliver in one day what organizations take a full year to completeAI Reality CheckCurrent generative AI is fundamentally:Enhanced Stack OverflowFancy search enginePattern recognition systemNot truly "intelligent" - builds on existing information servicesWill reach perfect competition as technologies standardizeOpen source solutions rapidly approaching commercial offeringsFuture PredictionsExperts become increasingly valuableBeginners face decreased demandDysfunctional organizations accelerate toward failure Expertise inequality may become as concerning as income inequalityConclusionThe AI revolution isn't replacing expertise - it's making it more valuable than ever.

2025 er kun lige startet, men vi har allerede set en masse scenekunst. Vi vender lige de ting, der er sket siden sidst: Line Kirsten har set en one-woman musical, og Clara har planlgat sin næste teaterrejse. Vi taler generelt om mange af de forestillinger, som rigtig mange taler om lige nu: EN POLTERABEND på Betty Nansen Teatret og EKS og JEPPE PÅ BJERGET på Det Kongelige Teater). Og så taler vi om den totalt udsolgte FUGLENE på Revolver på Teater Republique. Alle taler om teater lige nu.  I DETTE AFSNIT ANMELDER VI:  - En Polterabend (Betty Nansen Teatret) - Jeppe på bjerget (Det Kongelige Teater) - EKS (Det Kongelige Teater) - Fuglene (Teater Republique, Revolver)

Das Ausmass an sexualisierter Gewalt in der evangelischen Kirche ist grösser als bisher angenommen. Das zeigt eine breit angelegte Untersuchung in Deutschland. Wie ist die Situation in den reformierten Kirchen der Schweiz und was geschieht hinsichtlich Aufarbeitung und Prävention? Frau W erleidet als Kind sexuelle Übergriffe von ihrem Grossvater. Dieser ist ehrenamtlich als Kirchenpfleger in einer reformierten Kirchgemeinde aktiv und eine Respektsperson. Alle schauen weg. Im Alter von 55 Jahren beginnt Frau W, ihre Vergangenheit aufzuarbeiten und spricht erstmals über die Übergriffe und deren Auswirkungen in ihrem Leben. Frau N wird von einem reformierten Pfarrer während Jahren manipuliert und sexuell ausgebeutet. Der Täter benutzte das Wissen über ihr Privatleben und ihre religiösen Gefühle für seine Zwecke. Sexuelle Übergriffe gibt es nicht nur in der römisch-katholischen Kirche, sondern auch im Umfeld der evangelisch-reformierten Kirche. Welche Strukturen ermöglichten dort Übergriffe und sexualisierte Gewalt? Welche Strukturen erschweren es, diese aufzudecken? Dazu äussert sich in «Perspektiven» die Theologin Sabine Scheuter von der Anlaufstelle «Grenzverletzungen» der reformierten Kirche im Kanton Zürich. Welches Ausmass haben sexuelle Übergriffe in der evangelisch-reformierten Kirche der Schweiz? Die Synode der Evangelisch-Reformierten Kirche Schweiz EKS hat im Jahr 2024 eine breit angelegte Studie zu sexuellem Missbrauch in der Schweizer Gesellschaft abgelehnt. Der Rat der EKS hat inzwischen eine Arbeitsgruppe «Schutz der persönlichen Integrität» eingesetzt, die Kantonalkirchen haben Meldestellen eingerichtet und forcieren die Präventionsarbeit.

Eks-bibliotekar Carolyn Fjeld velger sine seks favorittbøker fra og om Stavanger i det vi går inn i jubileumsåret 2025. (00:00) Hjerne og hjerte (02:08) Alexander L. Kielland, En god samvittighet (06:01) Karsten Roedder, Knus ikke en elendig i porten (12:44) Tore Renberg, Kompani Orheim (16:36) Ole Kristian Ellingsen, Stormsinn (21:16) Sven Egil Omdal, Byen som formet Norge (23:24) Asgeir Lode, Varmestuen gjennom 50 år --- Innspilt på Sølvberget bibliotek og kulturhus i januar 2025. Medvirkende: Carolyn Fjeld og Åsmund Ådnøy. Produksjon: Åsmund Ådnøy.

Send us a textWhat if the future of AI lies in the palm of Amazon's hand? Discover Amazon's cutting-edge advancements in AI chip technology and their strategic moves in the generative AI landscape. We'll unpack the debut of the Tranium 2 chip and the anticipated Tranium 3, alongside the introduction of Amazon's foundational models, Nova Micro and Nova Canvas, designed to revolutionize text and image tasks. This episode promises to give you insights into how these innovations could reshape industries and solidify Amazon's standing in the AI market.Join Chris and me as we recount our vibrant experiences at AWS re:Invent, rubbing shoulders with over 57,000 tech enthusiasts and professionals. From meeting past podcast guests like Sharla and Dejuan to leading a breakout session, the event was a whirlwind of connections and inspiration. We also dive into Amazon's latest offerings, such as S3 Tables and the extension of EKS for hybrid cloud infrastructure, shedding light on how these developments could influence cloud strategies and operations.In a lighter vein, we ponder the possibilities and peculiarities of AWS's mysterious new data transfer terminals. Imagine a world with hidden data hubs, industrial IoT transfers, and enigmatic locations—complete with a sprinkle of humor. We invite you to laugh with us, question the implications, and share your own tales of AWS exploration. Let's navigate these cables to clouds together, with a mix of curiosity, insight, and a dash of humor.List of AWS announcements:https://aws.amazon.com/blogs/aws/top-announcements-of-aws-reinvent-2024/Tim's Breakout Session:https://www.youtube.com/watch?v=KNxUf15QNvECheck out the Fortnightly Cloud Networking Newshttps://docs.google.com/document/d/1fkBWCGwXDUX9OfZ9_MvSVup8tJJzJeqrauaE6VPT2b0/Visit our website and subscribe: https://www.cables2clouds.com/Follow us on Twitter: https://twitter.com/cables2cloudsFollow us on YouTube: https://www.youtube.com/@cables2clouds/Follow us on TikTok: https://www.tiktok.com/@cables2cloudsMerch Store: https://store.cables2clouds.com/Join the Discord Study group: https://artofneteng.com/iaatj

Nous terminons cette mini-série de re:Caps quotidiens avec deux habitués du podcast: Chrys et Jules qui nous parlent de ce qu'ils ont particulièrement apprécié pendant la conférence et des nouveautés qui pourraient être utiles pour eux. Sans divulgaché, nous parlons d'injection de pub automatiquement au bon endroit dans des vidéos, de nouveautés EKS et de SageMaker Studio.

Perang Lawan TPPO, Pemerintah Dorong Penguatan Pengawasan | Transformasi Armada: TNI AL Upayakan Kekuatan Maritim Modern | Peringan Milad ke-48 GAM, Eks-kombatan Kibarkan Bendera Bintang Bulan *Kami ingin mendengar saran dan komentar kamu terkait podcast yang baru saja kamu simak, melalui surel ke podcast@kbrprime.id

C'est une tradition le dimanche soir à re:Invent, il y a une première salve d'annonces et de de nouveautés. Cette année c'est 32 billets de blogs que nous venons de publier. J'ai retenu des nouveautés qui concernent EKS, Amazon S3, AWS Verfied Access, Data Migration Service Schema Converter, les policies IAM, et une nouvelle solution physique pour transférer de gros volumes de données.

Vi har ikke det endelige resultat endnu, men det ser rigtigt godt ud for Trump. Eks-præsidenten kan meget vel vinde alle svingstaterne. Mads Fuglede og David Trads giver den foreløbige dom over valgresultaterne.  Værter: David Trads, journalist og forfatter, og Mads Fuglede, historiker, USA-analytiker og medlem af Folketinget for Danmarksdemokraterne.See omnystudio.com/listener for privacy information.

Formand for Moderaternes folketingsgruppe Henrik Frandsen og medlem af Folketinget for Moderaterne Karin Liltorp er blevet forelagt den kritik, Jeppe Søe fremlægger i dette afsnit. Karin Liltorp har ingen kommentarer og Henrik Frandsen er ikke vendt tilbage.  _________________ Eks-moderat Jeppe Søe er dagens gæst i Borgerlig Tabloid. Han er primært i studiet for at tale politik. For han vil gerne fortsætte på Borgen  - men i hvilket parti? Og hvor er han egentlig velkommen?   Vi kommer dog ikke uden om, hvad han ellers har på hjerte her en lille måned efter, han - som følge af skandalen i Moderaterne - meldte sig ud for åben mikrofon i B.T.s mediepodcast Q&CO. See omnystudio.com/listener for privacy information.

In this episode of the Kubernetes Bytes podcast, Bhavin sits down with Alex Lines and Vara Bonthu from AWS to talk about the Data on EKS project. The discussion dives into why AWS decided to build the Data on EKS project and provide patterns for EKS customers to use to deploy data platforms, machine learning and GenAI tools on EKS clusters. They talk about what's included and what's not included with each of these patterns and whats coming down the line. Check out our website at https://kubernetesbytes.com/ Cloud Native News: https://kubernetes.io/blog/2024/08/16/kubernetes-1-31-prevent-persistentvolume-leaks-when-deleting-out-of-order/ https://kubernetes.io/blog/2024/08/16/matchlabelkeys-podaffinity/ https://kubernetes.io/blog/2024/08/15/kubernetes-1-31-volume-attributes-class/ https://roadmap.vcluster.com/changelog/vcluster-v020-ga https://www.cloudbees.com/blog/cloudbees-acquires-launchable-to-enable-development-teams-to-iterate-faster?Product=Launchable&Tag=Blog%2CAI%2CLaunchable%20Update Show links: https://awslabs.github.io/data-on-eks/ https://www.youtube.com/watch?v=G9aNXEu_a8k https://github.com/awslabs/data-on-eks https://www.linkedin.com/in/alex-lines-aws/ https://www.linkedin.com/in/varaprofile/ Timestamps: 00:01:45 Cloud Native News 00:12:15 Interview with Alex and Vara 00:58:21 Key takeaways

(01:00): Rapport om bundtrawl møder kritik. Medvirkende: Kristian Bøgsted, fiskeriordfører, Danmarksdemokraterne. (16:00): Dansker i Rusland siger Ukraines modangreb har overrasket russerne og er en ydmygelse. Medvirkende: Steen Sørensen, selvstændig, bosiddende i Moskva. (30:00): Eks-prostitueret lægger noget af ansvaret for utro mænd på kvinderne. Medvirkende: Nynne Frederiksen, tidligere prostitueret og escortpige. (39.00): Konservativt bande-forslag får kritik: "Det vil have tæt på ingen effekt". Medvirkende: Frederik Bloch Münster, folketingsmedlem for Konservative. Værter: Kasper Harboe og Nicolai Dandanell.See omnystudio.com/listener for privacy information.

Aglonā noslēdzas Vissvētākās Jaunavas Marijas debesīs uzņemšanas svētki. Pasaules Veselības organizācija izsludinājusi starptautiska mēroga ārkārtas situāciju sabiedrības veselības jomā saistībā ar jauna pērtiķu baku varianta izplatību Āfrikā. Katarā atsākas sarunas par pamieru Gazā. Liepājas cietuma spāru svētki sagaidīti trīs mēnešu ātrāk nekā gaidīts. Rīgā pie dzelzceļa stacijām veidos mobilitātes punktus. Sola lielākas ērtības iedzīvotājiem. Eksāmenu apelāciju skaits šogad ir divas reizes lielāks nekā pagājušajā gadā. Kāpēc tā?

Valgkampen i USA er nu helt åben. Reelt står der helt lige mellem Trump og Harris. Der er endnu ikke panik hos Republikanerne, men Harris har vitterligt momentum. JD Vance er en belastning for Trump. Eks-præsidenten skal til at finde nye måder at angribe Harris på. Det er, som om han fortsat kæmper mod Biden. Mads Fuglede og David Trads zoomer igen helt tæt på. Værter: David Trads, journalist og forfatter, og Mads Fuglede, historiker, USA-analytiker og medlem af Folketinget for Danmarksdemokraterne.See omnystudio.com/listener for privacy information.

Principle Season 3 Ep 11Sometimes one little change can make all the difference. Learn the one thing that changed business, parenting, everything. Sign up for Eks's tips to be a better parent at https://eksayn.com/subscribe

For some strange reason, “maintenance” has been in the news quite a bit lately. Is there ever a time when maintenance is enjoyable, or appreciated? SHOW: 814SHOW TRANSCRIPT: The Cloudcast #814SHOW VIDEO: https://youtube.com/@TheCloudcastNET CLOUD NEWS OF THE WEEK - http://bit.ly/cloudcast-cnotwCHECK OUT OUR NEW PODCAST - "CLOUDCAST BASICS"SHOW NOTES:AWS increased the price of longer-running EKS clusters by 6xBroadcom changes VMware licensing from perpetual to subscriptionBroadcom offers security patches to perpetual license customersIncreasing the Kubernetes support window to 1 yearDiscovering the XZ backdoor (Oxide and Friends, podcast)IS MAINTENANCE EVER APPRECIATED OR ENJOYABLE?Spent the day surrounded by maintenance activities (oil, AC, power-wash)The costs of maintenance are real and opportunityMaintenance often goes unappreciated and unseenNaming: Release Notes, Technical Debt, Chaos EngineeringTECHNICAL DEBT VS. MAINTENANCEShould we encourage a lack of maintenance vs. innovation as a priority?Should we encourage active maintenance with lower hard costs?Is there a way to put respect on maintenance? (e.g. OSS maintainers)Do we undervalue maintenance (e.g. Backup/Recovery, DisasterRecovery, etc.)?What maintenance best practices do you use? What are the good and bad of them?FEEDBACK?Email: show at the cloudcast dot netTwitter: @cloudcastpodInstagram: @cloudcastpodTikTok: @cloudcastpod

Eks neid aastaringe lisandub meil kõigil. Mõnele kohe meeldib seda suurelt tähistada, naudib sellega kaasnevat tähelepanu ja rõõmu. Mõnele aga tundub, et tegemist on neetud päevaga, ning sünnipäeva ümber keerlevad pidevad draamad, valupunktid ja pettumus. Nii need traumad alguse saavad. Aga mis seal ikka - järgmisel aastal on uued lootused, et sel korral tuleb raju värk. Kuniks jälle pisarad silmis... :D Kirjutage meile ka oma traumeeriv sünnipäeva kogemus: Meie anonüümse Google Forms lingi oma jutu poetamiseks leiad SIIT: https://tinyurl.com/2p8cddkc

Justin is a developer who's helped create Oscar winning movies at Disney Animation, built infrastructure for Disney+, and worked on EKS at AWS. He is now the Director of Developer Relations at Sidero Labs and host of the Ship it! podcast You can find Justin on the following sites: Twitter Website PLEASE SUBSCRIBE TO THE PODCAST Spotify Apple Podcasts Google Podcasts Amazon Music RSS Feed You can check out more episodes of Coffee and Open Source on https://www.coffeeandopensource.com Coffee and Open Source is hosted by Isaac Levin --- Support this podcast: https://podcasters.spotify.com/pod/show/coffeandopensource/support

Tänase saate lähtekohaks on šveitsi maalikunstniku Arnold Böcklini surematu teos "Surnute saar". Eks vaatame, kuhu see maal meie jutu kannab. Peatse kohtumiseni! H.

Eks-statsråder og stortingstopper har fått refs av Kontroll- og konstitusjonskomiteen, men hva betyr det? Vi går opp løpet. FpU-leder Simen Velle gikk ut, la seg flat, men har likevel fått i gang en debatt om menns problemer, både seksuelt og generelt. Noen vil kunne anklage Lars for å trekke stigen opp under seg. Sarah beklager til Frp-representant Erlend Wiborg etter en stor tabbe, og Kjetil og Trine forklarer hvordan de har løst en uenighet. Kjøp billetter til Aftenpodden live 5. mars, på www.ticketmaster.no og få nyhetsbrevet på www.aftenpodden.no Alle episodene av Aftenpodden kan du høre som abonnent hos Podme eller Aftenposten. Med Lars Glomnes, Trine Eilertsen, Sarah Sørheim og Kjetil Alstadheim.

Tahaks lennata või tahaks unnata? Eks siis tuleb koos Rein Langi ja Väino Koorbergiga lasta end lennutada Kuku-raadio lainesagedusel: vanamehed viinavabriku kõrvalt võtavad sedapuhku jutuks kodumaise lennunduse ja teised ärid, kus inimesi ja muud kraami suurtel kiirustel ja hulgakaupa liigutatakse. Räägime, kuidas meid üha rohkem sõjaärevuse lainele häälestatakse ning kes ja kuidas meile udu silmade ette ning ärevuse liiva laugudele puistavad. Vaatame, millisest blufist peab kriitiliselt mõtlev kodanik end uudiseid tarbides läbi närima, ning küsime, kas tal jätkub selleks ikka teravaid hambaid ja kainet mõistust. Sekka lisame ka riikliku tähtsusega informatsiooni laskesuusatamise ja „Eesti Laulu“ teemal.

Nääksumine ehk näägutamine ehk vingumine ehk mögisemine ehk... noh, saate aru küll, mida me öelda tahame. Eks igal naisel seda rohkem või vähem ette tulnud. Paistab, et tegu on normaalse nähtusega ja et meil kõigil parem hakkaks, toome mõned näited oma elust. Ja muidugi leiame ka selgituse kõigele sellele, et ikkagi õigustada oma nääksuvat käitumist :P

Endothelial keratoplasty (EK) procedures have become the preferred method of corneal transplantation in the United States. As the frequency of EKs increase, so do associated costs. Dr. Matt Feng interviews Drs. David S. Portney and Shahzad I. Mian about their recent analysis to assess for differences in surgical costs and surgery length based on type of EK, use of preloaded grafts, and performance of simultaneous cataract surgery with EK from their Ophthalmology article “Cost Drivers of Endothelial Keratoplasty: A Time-Driven Activity-Based Costing Analysis.” Cost Drivers of Endothelial Keratoplasty. Goldstein, Jenna K. et al. Ophthalmology, Volume 130, Issue 10, 1073 – 1079. Sign up for the next Ophthalmology Journal Virtual Club on March 6, 2024, at https://store.aao.org/ophthalmology-virtual-journal-club.html

Phil Estes, a principal engineer at AWS and a key contributor to the containerd project, discusses the evolution and roadmap of containerd, its relationship to Docker, and its journey to being a fully open source project under the Cloud Native Computing Foundation (CNCF).  00:00 Introduction and Guest Background 00:41 The Evolution of Container Runtimes 02:48 The Impact of Docker and Kubernetes 06:07 The Birth and Growth of containerd 09:13 The Role of the CNCF in Open Source Projects 11:09 Challenges and Opportunities in Open Source Contribution 18:49 The Importance of Mentorship in Open Source 23:47 Conclusion and Final Thoughts   Guest: Phil Estes is a Principal Engineer for Amazon Web Services (AWS), focused on core container technologies that power AWS container offerings like Fargate, EKS, and ECS. Phil is an active contributor and maintainer for the CNCF containerd runtime project, and participates in the Open Container Initiative (OCI) as the member of the Technical Oversight Board (TOB). He is also a current member of the 2023 CNCF Ambassador class and enjoys speaking on container technology topics and events worldwide.

Evelyn Osman, Principal Platform Engineer at AutoScout24, joins Corey on Screaming in the Cloud to discuss the dire need for developers to agree on a standardized tool set in order to scale their projects and innovate quickly. Corey and Evelyn pick apart the new products being launched in cloud computing and discover a large disconnect between what the industry needs and what is actually being created. Evelyn shares her thoughts on why viewing platforms as products themselves forces developers to get into the minds of their users and produces a better end result.About EvelynEvelyn is a recovering improviser currently role playing as a Lead Platform Engineer at Autoscout24 in Munich, Germany. While she says she specializes in AWS architecture and integration after spending 11 years with it, in truth she spends her days convincing engineers that a product mindset will make them hate their product managers less.Links Referenced:LinkedIn: https://www.linkedin.com/in/evelyn-osman/TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is Evelyn Osman, engineering manager at AutoScout24. Evelyn, thank you for joining me.Evelyn: Thank you very much, Corey. It's actually really fun to be on here.Corey: I have to say one of the big reasons that I was enthused to talk to you is that you have been using AWS—to be direct—longer than I have, and that puts you in a somewhat rarefied position where AWS's customer base has absolutely exploded over the past 15 years that it's been around, but at the beginning, it was a very different type of thing. Nowadays, it seems like we've lost some of that magic from the beginning. Where do you land on that whole topic?Evelyn: That's actually a really good point because I always like to say, you know, when I come into a room, you know, I really started doing introductions like, “Oh, you know, hey,” I'm like, you know, “I'm this director, I've done this XYZ,” and I always say, like, “I'm Evelyn, engineering manager, or architect, or however,” and then I say, you know, “I've been working with AWS, you know, 11, 12 years,” or now I can't quite remember.Corey: Time becomes a flat circle. The pandemic didn't help.Evelyn: [laugh] Yeah, I just, like, a look at that the year, and I'm like, “Jesus. It's been that long.” Yeah. And usually, like you know, you get some odd looks like, “Oh, my God, you must be a sage.” And for me, I'm… you see how different services kind of, like, have just been reinventions of another one, or they just take a managed service and make another managed service around it. So, I feel that there's a lot of where it's just, you know, wrapping up a pretty bow, and calling it something different, it feels like.Corey: That's what I've been low-key asking people for a while now over the past year, namely, “What is the most foundational, interesting thing that AWS has done lately, that winds up solving for this problem of whatever it is you do as a company? What is it that has foundationally made things better that AWS has put out in the last service? What was it?” And the answers I get are all depressingly far in the past, I have to say. What's yours?Evelyn: Honestly, I think the biggest game-changer I remember experiencing was at an analyst summit in Stockholm when they announced Lambda.Corey: That was announced before I even got into this space, as an example of how far back things were. And you're right. That was transformative. That was awesome.Evelyn: Yeah, precisely. Because before, you know, we were always, like, trying to figure, okay, how do we, like, launch an instance, run some short code, and then clean it up. AWS is going to charge for an hour, so we need to figure out, you know, how to pack everything into one instance, run for one hour. And then they announced Lambda, and suddenly, like, holy shit, this is actually a game changer. We can actually write small functions that do specific things.And, you know, you go from, like, microservices, like, to like, tiny, serverless functions. So, that was huge. And then DynamoDB along with that, really kind of like, transformed the entire space for us in many ways. So, back when I was at TIBCO, there was a few innovations around that, even, like, one startup inside TIBCO that quite literally, their entire product was just Lambda functions. And one of their problems was, they wanted to sell in the Marketplace, and they couldn't figure out how to sell Lambda on the marketplace.Corey: It's kind of wild when we see just how far it's come, but also how much they've announced that doesn't change that much, to be direct. For me, one of the big changes that I remember that really made things better for customers—thought it took a couple of years—was EFS. And even that's a little bit embarrassing because all that is, “All right, we finally found a way to stuff a NetApp into us-east-1,” so now NFS, just like you used to use it in the 90s and the naughts, can be done responsibly in the cloud. And that, on some level, wasn't a feature launch so much as it was a concession to the ways that companies had built things and weren't likely to change.Evelyn: Honestly, I found the EFS launch to be a bit embarrassing because, like, you know, when you look closer at it, you realize, like, the performance isn't actually that great.Corey: Oh, it was horrible when it launched. It would just slam to a halt because you got the IOPS scaled with how much data you stored on it. The documentation explicitly said to use dd to start loading a bunch of data onto it to increase the performance. It's like, “Look, just sandbag the thing so it does what you'd want.” And all that stuff got fixed, but at the time it looked like it was clown shoes.Evelyn: Yeah, and that reminds me of, like, EBS's, like, gp2 when we're, like you know, we're talking, like, okay, provision IOPS with gp2. We just kept saying, like, just give yourself really big volume for performance. And it feel like they just kind of kept that with EFS. And it took years for them to really iterate off of that. Yeah, so, like, EFS was a huge thing, and I see us, we're still using it now today, and like, we're trying to integrate, especially for, like, data center migrations, but yeah, you always see that a lot of these were first more for, like, you know, data centers to the cloud, you know. So, first I had, like, EC2 classic. That's where I started. And I always like to tell a story that in my team, we're talking about using AWS, I was the only person fiercely against it because we did basically large data processing—sorry, I forget the right words—data analytics. There we go [laugh].Corey: I remember that, too. When it first came out, it was, “This sounds dangerous and scary, and it's going to be a flash in the pan because who would ever trust their core compute infrastructure to some random third-party company, especially a bookstore?” And yeah, I think I got that one very wrong.Evelyn: Yeah, exactly. I was just like, no way. You know, I see all these articles talking about, like, terrible disk performance, and here I am, where it's like, it's my bread and butter. I'm specialized in it, you know? I write code in my sleep and such.[Yeah, the interesting thing is, I was like, first, it was like, I can 00:06:03] launch services, you know, to kind of replicate when you get in a data center to make it feature comparable, and then it was taking all this complex services and wrapping it up in a pretty bow for—as a managed service. Like, EKS, I think, was the biggest one, if we're looking at managed services. Technically Elasticsearch, but I feel like that was the redheaded stepchild for quite some time.Corey: Yeah, there was—Elasticsearch was a weird one, and still is. It's not a pleasant service to run in any meaningful sense. Like, what people actually want as the next enhancement that would excite everyone is, I want a serverless version of this thing where I can just point it at a bunch of data, I hit an API that I don't have to manage, and get Elasticsearch results back from. They finally launched a serverless offering that's anything but. You have to still provision compute units for it, so apparently, the word serverless just means managed service over at AWS-land now. And it just, it ties into the increasing sense of disappointment I've had with almost all of their recent launches versus what I felt they could have been.Evelyn: Yeah, the interesting thing about Elasticsearch is, a couple of years ago, they came out with OpenSearch, a competing Elasticsearch after [unintelligible 00:07:08] kind of gave us the finger and change the licensing. I mean, OpenSearch actually become a really great offering if you run it yourself, but if you use their managed service, it can kind—you lose all the benefits, in a way.Corey: I'm curious, as well, to get your take on what I've been seeing that I think could only be described as an internal shift, where it's almost as if there's been a decree passed down that every service has to run its own P&L or whatnot, and as a result, everything that gets put out seems to be monetized in weird ways, even when I'd argue it shouldn't be. The classic example I like to use for this is AWS Config, where it charges you per evaluation, and that happens whenever a cloud resource changes. What that means is that by using the cloud dynamically—the way that they supposedly want us to do—we wind up paying a fee for that as a result. And it's not like anyone is using that service in isolation; it is definitionally being used as people are using other cloud resources, so why does it cost money? And the answer is because literally everything they put out costs money.Evelyn: Yep, pretty simple. Oftentimes, there's, like, R&D that goes into it, but the charges seem a bit… odd. Like from an S3 lens, was, I mean, that's, like, you know, if you're talking about services, that was actually a really nice one, very nice holistic overview, you know, like, I could drill into a data lake and, like, look into things. But if you actually want to get anything useful, you have to pay for it.Corey: Yeah. Everything seems to, for one reason or another, be stuck in this place where, “Well, if you want to use it, it's going to cost.” And what that means is that it gets harder and harder to do anything that even remotely resembles being able to wind up figuring out where's the spend going, or what's it going to cost me as time goes on? Because it's not just what are the resources I'm spinning up going to cost, what are the second, third, and fourth-order effects of that? And the honest answer is, well, nobody knows. You're going to have to basically run an experiment and find out.Evelyn: Yeah. No, true. So, what I… at AutoScout, we actually ended up doing is—because we're trying to figure out how to tackle these costs—is they—we built an in-house cost allocation solution so we could track all of that. Now, AWS has actually improved Cost Explorer quite a bit, and even, I think, Billing Conductor was one that came out [unintelligible 00:09:21], kind of like, do a custom tiered and account pricing model where you can kind of do the same thing. But even that also, there is a cost with it.I think that was trying to compete with other, you know, vendors doing similar solutions. But it still isn't something where we see that either there's, like, arbitrarily low pricing there, or the costs itself doesn't really quite make sense. Like, AWS [unintelligible 00:09:45], as you mentioned, it's a terrific service. You know, we try to use it for compliance enforcement and other things, catching bad behavior, but then as soon as people see the price tag, we just run away from it. So, a lot of the security services themselves, actually, the costs, kind of like, goes—skyrockets tremendously when you start trying to use it across a large organization. And oftentimes, the organization isn't actually that large.Corey: Yeah, it gets to this point where, especially in small environments, you have to spend more energy and money chasing down what the cost is than you're actually spending on the thing. There were blog posts early on that, “Oh, here's how you analyze your bill with Redshift,” and that was a minimum 750 bucks a month. It's, well, I'm guessing that that's not really for my $50 a month account.Evelyn: Yeah. No, precisely. I remember seeing that, like, entire ETL process is just, you know, analyze your invoice. Cost [unintelligible 00:10:33], you know, is fantastic, but at the end of the day, like, what you're actually looking at [laugh], is infinitesimally small compared to all the data in that report. Like, I think oftentimes, it's simply, you know, like, I just want to look at my resources and allocate them in a multidimensional way. Which actually isn't really that multidimensional, when you think about it [laugh].Corey: Increasingly, Cost Explorer has gotten better. It's not a new service, but every iteration seems to improve it to a point now where I'm talking to folks, and they're having a hard time justifying most of the tools in the cost optimization space, just because, okay, they want a percentage of my spend on AWS to basically be a slightly better version of a thing that's already improving and works for free. That doesn't necessarily make sense. And I feel like that's what you get trapped into when you start going down the VC path in the cost optimization space. You've got to wind up having a revenue model and an offering that scales through software… and I thought, originally, I was going to be doing something like that. At this point, I'm unconvinced that anything like that is really tenable.Evelyn: Yeah. When you're a small organization you're trying to optimize, you might not have the expertise and the knowledge to do so, so when one of these small consultancies comes along, saying, “Hey, we're going to charge you a really small percentage of your invoice,” like, okay, great. That's, like, you know, like, a few $100 a month to make sure I'm fully optimized, and I'm saving, you know, far more than that. But as soon as your invoice turns into, you know, it's like $100,000, or $300,000 or more, that percentage becomes rather significant. And I've had vendors come to me and, like, talk to me and is like, “Hey, we can, you know, for a small percentage, you know, we're going to do this machine learning, you know, AI optimization for you. You know, you don't have to do anything. We guaranteed buybacks your RIs.” And as soon as you look at the price tag with it, we just have to walk away. Or oftentimes we look at it, and there are truly very simple ways to do it on your own, if you just kind of put some thought into it.Corey: While we want to talking a bit before this show, you taught me something new about GameLift, which I think is a different problem that AWS has been dealing with lately. I've never paid much attention to it because it is the—as I assume from what it says on the tin, oh, it's a service for just running a whole bunch of games at scale, and I'm not generally doing that. My favorite computer game remains to be Twitter at this point, but that's okay. What is GameLift, though, because you want to shining a different light on it, which makes me annoyed that Amazon Marketing has not pointed this out.Evelyn: Yeah, so I'll preface this by saying, like, I'm not an expert on GameLift. I haven't even spun it up myself because there's quite a bit of price. I learned this fall while chatting with an SA who works in the gaming space, and it kind of like, I went, like, “Back up a second.” If you think about, like, I'm, you know, like, World of Warcraft, all you have are thousands of game clients all over the world, playing the same game, you know, on the same server, in the same instance, and you need to make sure, you know, that when I'm running, and you're running, that we know that we're going to reach the same point the same time, or if there's one object in that room, that only one of us can get it. So, all these servers are doing is tracking state across thousands of clients.And GameLift, when you think about your dedicated game service, it really is just multi-region distributed state management. Like, at the basic, that's really what it is. Now, there's, you know, quite a bit more happening within GameLift, but that's what I was going to explain is, like, it's just state management. And there are far more use cases for it than just for video games.Corey: That's maddening to me because having a global session state store, for lack of a better term, is something that so many customers have built themselves repeatedly. They can build it on top of primitives like DynamoDB global tables, or alternately, you have a dedicated region where that thing has to live and everything far away takes forever to round-trip. If they've solved some of those things, why on earth would they bury it under a gaming-branded service? Like, offer that primitive to the rest of us because that's useful.Evelyn: No, absolutely. And honestly, I wouldn't be surprised if you peeled back the curtain with GameLift, you'll find a lot of—like, several other you know, AWS services that it's just built on top of. I kind of mentioned earlier is, like, what I see now with innovation, it's like we just see other services packaged together and releases a new product.Corey: Yeah, IoT had the same problem going on for years where there was a lot of really good stuff buried in there, like IOT events. People were talking about using that for things like browser extensions and whatnot, but you need to be explicitly told that that's a thing that exists and is handy, but otherwise you'd never know it was there because, “Well, I'm not building anything that's IoT-related. Why would I bother?” It feels like that was one direction that they tended to go in.And now they take existing services that are, mmm, kind of milquetoast, if I'm being honest, and then saying, “Oh, like, we have Comprehend that does, effectively detection of themes, keywords, and whatnot, from text. We're going to wind up re-releasing that as Comprehend Medical.” Same type of thing, but now focused on a particular vertical. Seems to me that instead of being a specific service for that vertical, just improve the baseline the service and offer HIPAA compliance if it didn't exist already, and you're mostly there. But what do I know? I'm not a product manager trying to get promoted.Evelyn: Yeah, that's true. Well, I was going to mention that maybe it's the HIPAA compliance, but actually, a lot of their services already have HIPAA compliance. And I've stared far too long at that compliance section on AWS's site to know this, but you know, a lot of them actually are HIPAA-compliant, they're PCI-compliant, and ISO-compliant, and you know, and everything. So, I'm actually pretty intrigued to know why they [wouldn't 00:16:04] take that advantage.Corey: I just checked. Amazon Comprehend is itself HIPAA-compliant and is qualified and certified to hold Personal Health Information—PHI—Private Health Information, whatever the acronym stands for. Now, what's the difference, then, between that and Medical? In fact, the HIPAA section says for Comprehend Medical, “For guidance, see the previous section on Amazon Comprehend.” So, there's no difference from a regulatory point of view.Evelyn: That's fascinating. I am intrigued because I do know that, like, within AWS, you know, they have different segments, you know? There's, like, Digital Native Business, there's Enterprise, there's Startup. So, I am curious how things look over the engineering side. I'm going to talk to somebody about this now [laugh].Corey: Yeah, it's the—like, I almost wonder, on some level, it feels like, “Well, we wound to building this thing in the hopes that someone would use it for something. And well, if we just use different words, it checks a box in some analyst's chart somewhere.” I don't know. I mean, I hate to sound that negative about it, but it's… increasingly when I talk to customers who are active in these spaces around the industry vertical targeted stuff aimed at their industry, they're like, “Yeah, we took a look at it. It was adorable, but we're not using it that way. We're going to use either the baseline version or we're going to work with someone who actively gets our industry.” And I've heard that repeated about three or four different releases that they've put out across the board of what they've been doing. It feels like it is a misunderstanding between what the world needs and what they're able to or willing to build for us.Evelyn: Not sure. I wouldn't be surprised, if we go far enough, it could probably be that it's just a product manager saying, like, “We have to advertise directly to the industry.” And if you look at it, you know, in the backend, you know, it's an engineer, you know, kicking off a build and just changing the name from Comprehend to Comprehend Medical.Corey: And, on some level, too, they're moving a lot more slowly than they used to. There was a time where they were, in many cases, if not the first mover, the first one to do it well. Take Code Whisperer, their AI powered coding assistant. That would have been a transformative thing if GitHub Copilot hadn't beaten them every punch, come out with new features, and frankly, in head-to-head experiments that I've run, came out way better as a product than what Code Whisperer is. And while I'd like to say that this is great, but it's too little too late. And when I talk to engineers, they're very excited about what Copilot can do, and the only people I see who are even talking about Code Whisperer work at AWS.Evelyn: No, that's true. And so, I think what's happening—and this is my opinion—is that first you had AWS, like, launching a really innovative new services, you know, that kind of like, it's like, “Ah, it's a whole new way of running your workloads in the cloud.” Instead of you know, basically, hiring a whole team, I just click a button, you have your instance, you use it, sell software, blah, blah, blah, blah. And then they went towards serverless, and then IoT, and then it started targeting large data lakes, and then eventually that kind of run backwards towards security, after the umpteenth S3 data leak.Corey: Oh, yeah. And especially now, like, so they had a hit in some corners with SageMaker, so now there are 40 services all starting with the word SageMaker. That's always pleasant.Evelyn: Yeah, precisely. And what I kind of notice is… now they're actually having to run it even further back because they caught all the corporations that could pivot to the cloud, they caught all the startups who started in the cloud, and now they're going for the larger behemoths who have massive data centers, and they don't want to innovate. They just want to reduce this massive sysadmin team. And I always like to use the example of a Bare Metal. When that came out in 2019, everybody—we've all kind of scratched your head. I'm like, really [laugh]?Corey: Yeah, I could see where it makes some sense just for very specific workloads that involve things like specific capabilities of processors that don't work under emulation in some weird way, but it's also such a weird niche that I'm sure it's there for someone. My default assumption, just given the breadth of AWS's customer base, is that whenever I see something that they just announced, well, okay, it's clearly not for me; that doesn't mean it's not meeting the needs of someone who looks nothing like me. But increasingly as I start exploring the industry in these services have time to percolate in the popular imagination and I still don't see anything interesting coming out with it, it really makes you start to wonder.Evelyn: Yeah. But then, like, I think, like, roughly a year or something, right after Bare Metal came out, they announced Outposts. So, then it was like, another way to just stay within your data center and be in the cloud.Corey: Yeah. There's a bunch of different ways they have that, okay, here's ways you can run AWS services on-prem, but still pay us by the hour for the privilege of running things that you have living in your facility. And that doesn't seem like it's quite fair.Evelyn: That's exactly it. So, I feel like now it's sort of in diminishing returns and sort of doing more cloud-native work compared to, you know, these huge opportunities, which is everybody who still has a data center for various reasons, or they're cloud-native, and they grow so big, that they actually start running their own data centers.Corey: I want to call out as well before we wind up being accused of being oblivious, that we're recording this before re:Invent. So, it's entirely possible—I hope this happens—that they announce something or several some things that make this look ridiculous, and we're embarrassed to have had this conversation. And yeah, they're totally getting it now, and they have completely surprised us with stuff that's going to be transformative for almost every customer. I've been expecting and hoping for that for the last three or four re:Invents now, and I haven't gotten it.Evelyn: Yeah, that's right. And I think there's even a new service launches that actually are missing fairly obvious things in a way. Like, mine is the Managed Workflow for Amazon—it's Managed Airflow, sorry. So, we were using Data Pipeline for, you know, big ETL processing, so it was an in-house tool we kind of built at Autoscout, we do platform engineering.And it was deprecated, so we looked at a new—what to replace it with. And so, we looked at Airflow, and we decided this is the way to go, we want to use managed because we don't want to maintain our own infrastructure. And the problem we ran into is that it doesn't have support for shared VPCs. And we actually talked to our account team, and they were confused. Because they said, like, “Well, every new service should support it natively.” But it just didn't have it. And that's, kind of, what, I kind of found is, like, there's—it feels—sometimes it's—there's a—it's getting rushed out the door, and it'll actually have a new managed service or new service launched out, but they're also sort of cutting some corners just to actually make sure it's packaged up and ready to go.Corey: When I'm looking at this, and seeing how this stuff gets packaged, and how it's built out, I start to understand a pattern that I've been relatively down on across the board. I'm curious to get your take because you work at a fairly sizable company as an engineering manager, running teams of people who do this sort of thing. Where do you land on the idea of companies building internal platforms to wrap around the offerings that the cloud service providers that they use make available to them?Evelyn: So, my opinion is that you need to build out some form of standardized tool set in order to actually be able to innovate quickly. Now, this sounds counterintuitive because everyone is like, “Oh, you know, if I want to innovate, I should be able to do this experiment, and try out everything, and use what works, and just release it.” And that greatness [unintelligible 00:23:14] mentality, you know, it's like five talented engineers working to build something. But when you have, instead of five engineers, you have five teams of five engineers each, and every single team does something totally different. You know, one uses Scala, and other on TypeScript, another one, you know .NET, and then there could have been a [last 00:23:30] one, you know, comes in, you know, saying they're still using Ruby.And then next thing you know, you know, you have, like, incredibly diverse platforms for services. And if you want to do any sort of like hiring or cross-training, it becomes incredibly difficult. And actually, as the organization grows, you want to hire talent, and so you're going to have to hire, you know, a developer for this team, you going to have to hire, you know, Ruby developer for this one, a Scala guy here, a Node.js guy over there.And so, this is where we say, “Okay, let's agree. We're going to be a Scala shop. Great. All right, are we running serverless? Are we running containerized?” And you agree on those things. So, that's already, like, the formation of it. And oftentimes, you start with DevOps. You'll say, like, “I'm a DevOps team,” you know, or doing a DevOps culture, if you do it properly, but you always hit this scaling issue where you start growing, and then how do you maintain that common tool set? And that's where we start looking at, you know, having a platform… approach, but I'm going to say it's Platform-as-a-Product. That's the key.Corey: Yeah, that's a good way of framing it because originally, the entire world needed that. That's what RightScale was when EC2 first came out. It was a reimagining of the EC2 console that was actually usable. And in time, AWS improved that to the point where RightScale didn't really have a place anymore in a way that it had previously, and that became a business challenge for them. But you have, what is it now, 2, 300 services that AWS has put out, and out, and okay, great. Most companies are really only actively working with a handful of those. How do you make those available in a reasonable way to your teams, in ways that aren't distracting, dangerous, et cetera? I don't know the answer on that one.Evelyn: Yeah. No, that's true. So, full disclosure. At AutoScout, we do platform engineering. So, I'm part of, like, the platform engineering group, and we built a platform for our product teams. It's kind of like, you need to decide to [follow 00:25:24] those answers, you know? Like, are we going to be fully containerized? Okay, then, great, we're going to use Fargate. All right, how do we do it so that developers don't actually—don't need to think that they're running Fargate workloads?And that's, like, you know, where it's really important to have those standardized abstractions that developers actually enjoy using. And I'd even say that, before you start saying, “Ah, we're going to do platform,” you say, “We should probably think about developer experience.” Because you can do a developer experience without a platform. You can do that, you know, in a DevOps approach, you know? It's basically build tools that makes it easy for developers to write code. That's the first step for anything. It's just, like, you have people writing the code; make sure that they can do the things easily, and then look at how to operate it.Corey: That sure would be nice. There's a lack of focus on usability, especially when it comes to a number of developer tools that we see out there in the wild, in that, they're clearly built by people who understand the problem space super well, but they're designing these things to be used by people who just want to make the website work. They don't have the insight, the knowledge, the approach, any of it, nor should they necessarily be expected to.Evelyn: No, that's true. And what I see is, a lot of the times, it's a couple really talented engineers who are just getting shit done, and they get shit done however they can. So, it's basically like, if they're just trying to run the website, they're just going to write the code to get things out there and call it a day. And then somebody else comes along, has a heart attack when see what's been done, and they're kind of stuck with it because there is no guardrails or paved path or however you want to call it.Corey: I really hope—truly—that this is going to be something that we look back and laugh when this episode airs, that, “Oh, yeah, we just got it so wrong. Look at all the amazing stuff that came out of re:Invent.” Are you going to be there this year?Evelyn: I am going to be there this year.Corey: My condolences. I keep hoping people get to escape.Evelyn: This is actually my first one in, I think, five years. So, I mean, the last time I was there was when everybody's going crazy over pins. And I still have a bag of them [laugh].Corey: Yeah, that did seem like a hot-second collectable moment, didn't it?Evelyn: Yeah. And then at the—I think, what, the very last day, as everybody's heading to re:Play, you could just go into the registration area, and they just had, like, bags of them lying around to take. So, all the competing, you know, to get the requirements for a pin was kind of moot [laugh].Corey: Don't you hate it at some point where it's like, you feel like I'm going to finally get this crowning achievement, it's like or just show up at the buffet at the end and grab one of everything, and wow, that would have saved me a lot of pain and trouble.Evelyn: Yeah.Corey: Ugh, scavenger hunts are hard, as I'm about to learn to my own detriment.Evelyn: Yeah. No, true. Yeah. But I am really hoping that re:Invent proves me wrong. Embarrassingly wrong, and then all my colleagues can proceed to mock me for this ridiculous podcast that I made with you. But I am a fierce skeptic. Optimistic nihilist, but still a nihilist, so we'll see how re:Invent turns out.Corey: So, I am curious, given your experience at more large companies than I tend to be embedded with for any period of time, how have you found that these large organizations tend to pick up new technologies? What does the adoption process look like? And honestly, if you feel like throwing some shade, how do they tend to get it wrong?Evelyn: In most cases, I've seen it go… terrible. Like, it just blows up in their face. And I say that is because a lot of the time, an organization will say, “Hey, we're going to adopt this new way of organizing teams or developing products,” and they look at all the practices. They say, “Okay, great. Product management is going to bring it in, they're going to structure things, how we do the planning, here's some great charts and diagrams,” but they don't really look at the culture aspect.And that's always where I've seen things fall apart. I've been in a room where, you know, our VP was really excited about team topologies and say, “Hey, we're going to adopt it.” And then an engineering manager proceeded to say, “Okay, you're responsible for this team, you're responsible for that team, you're responsible for this team talking to, like, a team of, like, five engineers,” which doesn't really work at all. Or, like, I think the best example is DevOps, you know, where you say, “Ah, we're going to adopt DevOps, we're going to have a DevOps team, or have a DevOps engineer.”Corey: Step one: we're going to rebadge everyone with existing job titles to have the new fancy job titles that reflect it. It turns out that's not necessarily sufficient in and of itself.Evelyn: Not really. The Spotify model. People say, like, “Oh, we're going to do the Spotify model. We're going to do skills, tribes, you know, and everything. It's going to be awesome, it's going to be great, you know, and nice, cross-functional.”The reason I say it bails on us every single time is because somebody wants to be in control of the process, and if the process is meant to encourage collaboration and innovation, that person actually becomes a chokehold for it. And it could be somebody that says, like, “Ah, I need to be involved in every single team, and listen to know what's happening, just so I'm aware of it.” What ends up happening is that everybody differs to them. So, there is no collaboration, there is no innovation. DevOps, you say, like, “Hey, we're going to have a team to do everything, so your developers don't need to worry about it.” What ends up happening is you're still an ops team, you still have your silos.And that's always a challenge is you actually have to say, “Okay, what are the cultural values around this process?” You know, what is SRE? What is DevOps, you know? Is it seen as processes, is it a series of principles, platform, maybe, you know? We have to say, like—that's why I say, Platform-as-a-Product because you need to have that product mindset, that culture of product thinking, to really build a platform that works because it's all about the user journey.It's not about building a common set of tools. It's the user journey of how a person interacts with their code to get it into a production environment. And so, you need to understand how that person sits down at their desk, starts the laptop up, logs in, opens the IDE, what they're actually trying to get done. And once you understand that, then you know your requirements, and you build something to fill those things so that they are happy to use it, as opposed to saying, “This is our platform, and you're going to use it.” And they're probably going to say, “No.” And the next thing, you know, they're just doing their own thing on the side.Corey: Yeah, the rise of Shadow IT has never gone away. It's just, on some level, it's the natural expression, I think it's an immune reaction that companies tend to have when process gets in the way. Great, we have an outcome that we need to drive towards; we don't have a choice. Cloud empowered a lot of that and also has given tools to help rein it in, and as with everything, the arms race continues.Evelyn: Yeah. And so, what I'm going to continue now, kind of like, toot the platform horn. So, Gregor Hohpe, he's a [solutions architect 00:31:56]—I always f- up his name. I'm so sorry, Gregor. He has a great book, and even a talk, called The Magic of Platforms, that if somebody is actually curious about understanding of why platforms are nice, they should really watch that talk.If you see him at re:Invent, or a summit or somewhere giving a talk, go listen to that, and just pick his brain. Because that's—for me, I really kind of strongly agree with his approach because that's really how, like, you know, as he says, like, boost innovation is, you know, where you're actually building a platform that really works.Corey: Yeah, it's a hard problem, but it's also one of those things where you're trying to focus on—at least ideally—an outcome or a better situation than you currently find yourselves in. It's hard to turn down things that might very well get you there sooner, faster, but it's like trying to effectively cargo-cult the leadership principles from your last employer into your new one. It just doesn't work. I mean, you see more startups from Amazonians who try that, and it just goes horribly because without the cultural understanding and the supporting structures, it doesn't work.Evelyn: Exactly. So, I've worked with, like, organizations, like, 4000-plus people, I've worked for, like, small startups, consulted, and this is why I say, almost every single transformation, it fails the first time because somebody needs to be in control and track things and basically be really, really certain that people are doing it right. And as soon as it blows up in their face, that's when they realize they should actually take a step back. And so, even for building out a platform, you know, doing Platform-as-a-Product, I always reiterate that you have to really be willing to just invest upfront, and not get very much back. Because you have to figure out the whole user journey, and what you're actually building, before you actually build it.Corey: I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place for them to find you?Evelyn: So, I used to be on Twitter, but I've actually got off there after it kind of turned a bit toxic and crazy.Corey: Feels like that was years ago, but that's beside the point.Evelyn: Yeah, precisely. So, I would even just say because this feels like a corporate show, but find me on LinkedIn of all places because I will be sharing whatever I find on there, you know? So, just look me up on my name, Evelyn Osman, and give me a follow, and I'll probably be screaming into the cloud like you are.Corey: And we will, of course, put links to that in the show notes. Thank you so much for taking the time to speak with me. I appreciate it.Evelyn: Thank you, Corey.Corey: Evelyn Osman, engineering manager at AutoScout24. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, and I will read it once I finish building an internal platform to normalize all of those platforms together into one.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business, and we get to the point. Visit duckbillgroup.com to get started.

Nun sieht auch die evangelisch-reformierte Kirche der Schweiz Handlungsbedarf und will ihre Fälle von sexualisierter Gewalt aufarbeiten. Rita Famos, Präsidentin der EKS, ist überzeugt, dass es auch in ihrer Kirche viele Fälle gibt und zu lange weggeschaut wurde. Die Missbrauchsfälle in der römisch-katholischen Kirche haben auch die evangelisch-reformierte Kirche aufgewühlt. Noch vor zwei Jahren sah die Präsidentin der EKS, Rita Famos, keinen Bedarf, für die reformierte Kirche eine schweizweite Untersuchung zu sexualisierter Gewalt zu machen. Diese Haltung hat nun geändert. Auch aufgrund einer Studie, der deutschen evangelischen Kirche. Diese Ergebnisse werden im Januar präsentiert. «Ich bin immer noch der Meinung, dass das Ausmass kleiner ist als in der römisch-katholischen Kirche, aber auch bei uns gibt es viele Fälle und wir haben viel zu lange weggeschaut.» Die evangelisch-reformierte Kirche sucht nach Wegen, die Vergangenheit aufzuarbeiten, richtet landesweit Meldestellen ein. Sie will aber auch Zahlungen an Opfer leisten und in die Prävention und Sensibilisierung investieren.

Not Escaping Containers but escaping Clusters - Managed Kubernetes distributions such as Amazon EKS, Google Kubernetes Engine (GKE) and Azure Kubernetes Service (AKS) attack vectors can allow you to reach the underlying AWS Account etc. In conversation with Christophe Tafani-Dereeper & Nick Frichette, from Datadog on how this is possible in Amazon EKS and achieving potentially the same in GKE & AKS too. Thank you to our episode sponsor Sagetap Guest Socials: Nick's and Christophe's Linkedin (⁠⁠⁠⁠⁠⁠⁠⁠⁠Nick Frichette + Christophe Tafani-Dereeper) Podcast Twitter - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠@CloudSecPod⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ If you want to watch videos of this LIVE STREAMED episode and past episodes - Check out our other Cloud Security Social Channels: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security Newsletter ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security BootCamp Questions asked: (00:00) Introduction (04:11) A bit about Christophe (04:37) A bit about Nick (05:03) What is managed Kubernetes? (06:26) Security of managed Kubernetes (09:02) Comparison between different managed Kubernetes (10:41) Service accounts and managed Kubernetes (14:22) What is container escape? (18:20) IMDSv2 for EKS (19:51) IMDSv2 in EKS vs AKES and GKE (22:01) Benchmark compliance for Kubernetes architecture (24:49) Low hanging fruits for container escape (27:17) Shared responsibility for managed Kubernetes (29:34) Fargate for Managed Kubernetes (32:00) Different ways to run containers (33:37) Escaping Managed Kubernetes cluster (38:39) Find more about this attack path (42:38) Escalation priviledge in EKS cluster (44:19) Reducing the Kubernetes attack service (44:58) MKAT for Kubernetes Security (48:23) Preventing AWS AuthConfig (50:11) Propagation Security (54:55) The fun section (57:47) Resources for latest Kubernetes updates Resources spoken about during the episode Nick Frichette's Blog - Hacking the Cloud Christophe Tafani-Dereeper' Blog Corey Quinn's - 17 ways to run containers on AWS MKAT cloudseclist newsletter

John Wynkoop, Cloud Economist & Platypus Herder at The Duckbill Group, joins Corey on Screaming in the Cloud to discuss why he decided to make a career move and become an AWS billing consultant. Corey and John discuss how once you're deeply familiar with one cloud provider, those skills become transferable to other cloud providers as well. John also shares the trends he has seen post-pandemic in the world of cloud, including the increased adoption of a multi-cloud strategy and the need for costs control even for VC-funded start-ups. About JohnWith over 25 years in IT, John's done almost every job in the industry, from running cable and answering helpdesk calls to leading engineering teams and advising the C-suite. Before joining The Duckbill Group, he worked across multiple industries including private sector, higher education, and national defense. Most recently he helped IGNW, an industry leading systems integration partner, get acquired by industry powerhouse CDW. When he's not helping customers spend smarter on their cloud bill, you can find him enjoying time with his family in the beautiful Smoky Mountains near his home in Knoxville, TN.Links Referenced: The Duckbill Group: https://duckbillgroup.com LinkedIn: https://www.linkedin.com/in/jlwynkoop/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. And the times, they are changing. My guest today is John Wynkoop. John, how are you?John: Hey, Corey, I'm doing great. Thanks for having me.Corey: So, big changes are afoot for you. You've taken a new job recently. What are you doing now?John: Well [laugh], so I'm happy to say I have joined The Duckbill Group as a cloud economist. So, came out of the big company world, and have dived back in—or dove back into the startup world.Corey: It's interesting because when we talk to those big companies, they always identify us as oh, you're a startup, which is hilarious on some level because our AWS account hangs out in AWS's startup group, but if you look at the spend being remarkably level from month to month to month to year to year to year, they almost certainly view us as they're a startup, but they suck at it. They completely failed. And so, many of the email stuff that you get from them presupposes that you're venture-backed, that you're trying to conquer the entire world. We don't do that here. We have this old-timey business model that our forebears would have understood of, we make more money than we spend every month and we continue that trend for a long time. So first, thanks for joining us, both on the show and at the company. We like having you around.John: Well, thanks. And yeah, I guess that's—maybe a startup isn't the right word to describe what we do here at The Duckbill Group, but as you said, it seems to fit into the industry classification. But that was one of the things I actually really liked about the—that was appealing about joining the team was, we do spend less than we make and we're not after hyper-growth and we're not trying to consume everything.Corey: So, it's interesting when you put a job description out into the world and you see who applies—and let's be clear, for those who are unaware, job descriptions are inherently aspirational shopping lists. If you look at a job description and you check every box on the thing and you've done all the things they want, the odds are terrific you're going to be bored out of your mind when you wind up showing up to do these… whatever that job is. You should be learning stuff and growing. At least that's always been my philosophy to it. One of the interesting things about you is that you checked an awful lot of boxes, but there is one that I think would cause people to raise an eyebrow, which is, you're relatively new to the fun world of AWS.John: Yeah. So, obviously I, you know, have been around the block a few times when it comes to cloud. I've used AWS, built some things in AWS, but I wouldn't have classified myself as an AWS guru by any stretch of the imagination. I spent the last probably three years working in Google Cloud, helping customers build and deploy solutions there, but I do at least understand the fundamentals of cloud, and more importantly—at least for our customers—cloud costs because at the end of the day, they're not all that different.Corey: I do want to call out that you have a certain humility to you which I find endearing. But you're not allowed to do that here; I will sing your praises for you. Before they deprecated it like they do almost everything else, you were one of the relatively few Google Cloud Certified Fellows, which was sort of like their Heroes program only, you know, they killed it in favor of something else like there's a Champion program or whatnot. You are very deep in the world of both Kubernetes and Google Cloud.John: Yeah. So, there was a few of us that were invited to come out and help Google pilot that program in, I believe it was 2019, and give feedback to help them build the Cloud Fellows Program. And thankfully, I was selected based on some of our early experience with Anthos, and specifically, it was around Certified Fellow in what they call hybrid multi-cloud, so it was experience around Anthos. Or at the time, they hadn't called it Anthos; they were calling it CSP or Cloud Services Platform because that's not an overloaded acronym. So yeah, definitely, was very humbled to be part of that early on.I think the program, as you said, grew to about 70 or so maybe 100 certified individuals before they transitioned—not killed—transitioned to that program into the Cloud Champions program. So, those folks are all still around, myself included. They've just now changed the moniker. But we all get to use the old title still as well, so that's kind of cool.Corey: I have to ask, what would possess you to go from being one of the best in the world at using Google Cloud over here to our corner of the AWS universe? Because the inverse, if I were to somehow get ejected from here—which would be a neat trick, but I'm sure it's theoretically possible—like, “What am I going to do now?” I would almost certainly wind up doing something in the AWS ecosystem, just due to inertia, if nothing else. You clearly didn't see things quite that way. Why make the switch?John: Well, a couple of different reasons. So, being at a Google partner presents a lot of challenges and one of the things that was supremely interesting about coming to Duckbill is that we're independent. So, we're not an AWS partner. We are an independent company that is beholden only to our customers. And there isn't anything like that in the Google ecosystem today.There's, you know, there's Google partners and then there's Google customers and then there's Google. So, that was part of the appeal. And the other thing was, I enjoy learning new things, and honestly, learning, you know, into the depths of AWS cost hell is interesting. There's a lot to learn there and there's a lot of things that we can extract and use to help customers spend less. So, that to me was super interesting.And then also, I want to help build an organization. So, you know, I think what we're doing here at The Duckbill Group is cool and I think that there's an opportunity to grow our services portfolio, and so I'm excited to work with the leadership team to see what else we can bring to market that's going to help our customers, you know, not just with cost optimization, not just with contract negotiation, but you know, through the lifecycle of their AWS… journey, I guess we'll call it.Corey: It's one of those things where I always have believed, on some level, that once you're deep in a particular cloud provider, if there's reason for it, you can rescale relatively quickly to a different provider. There are nuances—deep nuances—that differ from provider to provider, but the underlying concepts generally all work the same way. There's only so many ways you can have data go from point A to point B. There's only so many ways to spin up a bunch of VMs and whatnot. And you're proof-positive that theory was correct.You'd been here less than a week before I started learning nuances about AWS billing from you. I think it was something to do with the way that late fees are assessed when companies don't pay Amazon as quickly as Amazon desires. So, we're all learning new things constantly and no one stuffs this stuff all into their head. But that, if nothing else, definitely cemented that yeah, we've got the right person in the seat.John: Yeah, well, thanks. And certainly, the deeper you go on a specific cloud provider, things become fresh in your memory, you know, other cached so to speak. So, coming up to speed on AWS has been a little bit more documentation reading than it would have been, if I were, say, jumping right into a GCP engagement. But as he said, at the end of the day, there's a lot of similarities. Obviously understanding the nuances of, for example, account organization versus, you know, GCP's Project and Folders. Well, that's a substantial difference and so there's a lot of learning that has to happen.Thankfully, you know, all these companies, maybe with the exception of Oracle, have done a really good job of documenting all of the concepts in their publicly available documentation. And then obviously, having a team of experts here at The Duckbill Group to ask stupid questions of doesn't hurt. But definitely, it's not as hard to come up to speed as one may think, once you've got it understood in one provider.Corey: I took a look recently and was kind of surprised to discover that I've been doing this—as an independent consultant prior to the formation of The Duckbill Group—for seven years now. And it's weird, but I've gone through multiple industry cycles and changes as a part of this. And it feels like I haven't been doing it all that long, but I guess I have. One thing that's definitely changed is that it used to be that companies would basically pick one provider and almost everything would live there. At any reasonable point of scale, everyone is using multiple things.I see Google in effectively every client that we have. It used to be that going to Google Cloud Next was a great place to hang out with AWS customers. But these days, it's just as true to say that a great reason to go to re:Invent is to hang out with Google Cloud customers. Everyone uses everything, and that has become much more clear over the last few years. What have you seen change over the… I guess, since the start of the pandemic, just in terms of broad cycles?John: Yeah. So, I think there's a couple of different trends that we're seeing. Obviously, one is that as you said, especially as large enterprises make moves to the cloud, you see independent teams or divisions within a given organization leveraging… maybe not the right tool for the job because I think that there's a case to be made for swapping out a specific set of tools and having your team learn it, but we do see what I like to refer to as tool fetishism where you get a team that's super, super deep into BigQuery and they're not interested in moving to Redshift, or Snowflake, or a competitor. So, you see, those start to crop up within large organizations where the distributed—the purchasing power, rather—is distributed. So, that's one of the trends is the multi-cloud adoption.And I think the big trend that I like to emphasize around multi-cloud is, just because you can run it anywhere doesn't mean you should run it everywhere. So Kubernetes, as you know, right, as it took off 2019 timeframe, 2020, we started to see a lot of people using that as an excuse to try to run their production application in two, three public cloud providers and on-prem. And unless you're a SaaS customer—or SaaS company with customers in every cloud, there's very little reason to do that. But having that flexibility—that's the other one, is we've seen that AWS has gotten a little difficult to negotiate with, or maybe Google and Microsoft have gotten a little bit more aggressive. So obviously, having that flexibility and being able to move your workloads, that was another big trend.Corey: I'm seeing a change in things that I had taken as givens, back when I started. And that's part of the reason, incidentally, I write the Last Week in AWS newsletter because once you learn a thing, it is very easy not to keep current with that thing, and things that are not possible today will be possible tomorrow. How do you keep abreast of all of those changes? And the answer is to write a deeply sarcastic newsletter that gathers in everything from the world of AWS. But I don't recommend that for most people. One thing that I've seen in more prosaic terms that you have a bit of background in is that HPC on cloud was, five, six years ago, met with, “Oh, that's a good one; now pull the other one, it has bells on it,” into something that, these days, is extremely viable. How'd that happen?John: So, [sigh] I think that's just a—again, back to trends—I think that's just a trend that we're seeing from cloud providers and listening to their customers and continuing to improve the service. So, one of the reasons that HPC was—especially we'll call it capacity-level HPC or large HPC, right—you've always been able to run high throughput; the cloud is a high throughput machine, right? You can run a thousand disconnected VMs no problem, auto-scaling, anybody who runs a massive web front-end can attest to that. But what we saw with HPC—and we used to call those [grid 00:12:45] jobs, right, the small, decoupled computing jobs—but what we've seen is a huge increase in the quality of the underlying fabric—things like RDMA being made available, things like improved network locality, where you now have predictive latency between your nodes or between your VMs—and I think those, combined with the huge investment that companies like AWS have made in their file systems, the huge investment companies like Google have made in their data storage systems have made HPC viable, especially at a small-scale—for cloud-based HPC specifically—viable for organizations.And for a small engineering team, who's looking to run say, computer-aided engineering simulation or who's looking to prototype some new way of testing or doing some kind of simulation, it's a huge, huge improvement in speed because now they don't have to order a dozen or two dozen or five dozen nodes, have them shipped, rack them, stack them, cool them, power them, right? They can just spin up the resource in the cloud, test it out, try their simulation, try out the new—the software that they want, and then spin it all down if it doesn't work. So, that elasticity has also been huge. And again, I think the big—to kind of summarize, I think the big driver there is the improvement in this the service itself, right? We're seeing cloud providers taking that discipline a little bit more seriously.Corey: I still see that there are cases where the raw math doesn't necessarily add up for sustained, long-term use cases. But I also see increasingly that with HPC, that's usually not what the workload looks like. With, you know, the exception of we're going to spend the next 18 months training some new LLM thing, but even then the pricing is ridiculous. What is it their new P6 or whatever it is—P5—the instances that have those giant half-rack Nvidia cards that are $800,000 and so a year each if you were to just rent them straight out, and then people running fleets of these things, it's… wow that's more commas in that training job than I would have expected. But I can see just now the availability for driving some of that, but the economics of that once you can get them in your data center doesn't strike me as being particularly favoring the cloud.John: Yeah, there's a couple of different reasons. So, it's almost like an inverse curve, right? There's a crossover point or a breakeven point at which—you know, and you can make this argument with almost any level of infrastructure—if you can keep it sufficiently full, whether it's AI training, AI inference, or even traditional HPC if you can keep the machine or the group of machines sufficiently full, it's probably cheaper to buy it and put it in your facility. But if you don't have a facility or if you don't need to use it a hundred percent of the time, the dividends aren't always there, right? It's not always worth, you know, buying a $250,000 compute system, you know, like say, an Nvidia, as you—you know, like, a DGX, right, is a good example.The DGX H100, I think those are a couple $100,000. If you can't keep that thing full and you just need it for training jobs or for development and you have a small team of developers that are only going to use it six hours a day, it may make sense to spin that up in the cloud and pay for a fractional use, right? It's no different than what HPC has been doing for probably the past 50 years with national supercomputing centers, which is where my background came from before cloud, right? It's just a different model, right? One is public economies of, you know, insert your credit card and spend as much as you want and the other is grant-funded and supporting academic research, but the economy of scales is kind of the same on both fronts.Corey: I'm also seeing a trend that this is something that is sort of disturbing when you realize what I've been doing and how I've been going about things, that for the last couple of years, people actually started to care about the AWS bill. And I have to say, I felt like I was severely out of sync with a lot of the world the first few years because there's giant savings lurking in your AWS bill, and the company answer in many cases was, “We don't care. We'd rather focus our energies on shipping faster, building something new, expanding, capturing market.” And that is logical. But suddenly those chickens are coming home to roost in a big way. Our phone is ringing off the hook, as I'm sure you've noticed and your time here, and suddenly money means something again. What do you think drove it?John: So, I think there's a couple of driving factors. The first is obviously the broader economic conditions, you know, with the economic growth in the US, especially slowing down post-pandemic, we're seeing organizations looking for opportunities to spend less to be able to deliver—you know, recoup that money and deliver additional value. But beyond that, right—because, okay, but startups are probably still lighting giant piles of VC money on fire, and that's okay, but what's happening, I think, is that the first wave of CIOs that said cloud-first, cloud-only basically got their comeuppance. And, you know, these enterprises saw their explosive cloud bills and they saw that, oh, you know, we moved 5000 servers to AWS or GCP or Azure and we got the bill, and that's not sustainable. And so, we see a lot of cloud repatriation, cloud optimization, right, a lot of second-gen… cloud, I'll call them second-gen cloud-native CIOs coming into these large organizations where their predecessor made some bad financial decisions and either left or got asked to leave, and now they're trying to stop from lighting their giant piles of cash on fire, they're trying to stop spending 3X what they were spending on-prem.Corey: I think an easy mistake for folks to make is to get lost in the raw infrastructure cost. I'm not saying it's not important. Obviously not, but you could save a giant pile of money on your RDS instances by running your own database software on top of EC2, but I don't generally recommend folks do it because you also need engineering time to be focusing on getting those things up, care and feeding, et cetera. And what people lose sight of is the fact that the payroll expense is almost universally more than the cloud bill at every company I've ever talked to.So, there's a consistent series of, “Well, we're just trying to get to be the absolute lowest dollar figure total.” It's the wrong thing to emphasize on, otherwise, “Cool, turn everything off and your bill drops to zero.” Or, “Migrate to another cloud provider. AWS bill becomes zero. Our job is done.” It doesn't actually solve the problem at all. It's about what's right for the business, not about getting the absolute lowest possible score like it's some kind of code golf tournament.John: Right. So, I think that there's a couple of different ways to look at that. One is obviously looking at making your workloads more cloud-native. I know that's a stupid buzzword to some people, but—Corey: The problem I have with the term is that it means so many different things to different people.John: Right. But I think the gist of that is taking advantage of what the cloud is good at. And so, what we saw was that excess capacity on-prem was effectively free once you bought it, right? There were there was no accountability for burning through extra V CPUs or extra RAM. And then you had—Corey: Right. You spin something up in your data center and the question is, “Is the physical capacity there?” And very few companies had a reaping process until they were suddenly seeing capacity issues and suddenly everyone starts asking you a whole bunch of questions about it. But that was a natural forcing function that existed. Now, S3 has infinite storage, or it might as well. They can add capacity faster than you can fill it—I know this; I've tried—and the problem that you have then is that it's always just a couple more cents per gigabyte and it keeps on going forever. There's no, we need to make an investment decision because the SAN is at 80% capacity. Do you need all those 16 copies of the production data that you haven't touched since 2012? No, I probably don't.John: Yeah, there's definitely a forcing function when you're doing your own capacity planning. And the cloud, for the most part, as you've alluded to, for most organizations is infinite capacity. So, when they're looking at AWS or they're looking at any of the public cloud providers, it's a potentially infinite bill. Now, that scares a lot of organizations, and so because they didn't have the forcing function of, hey, we're out of CPUs, or we're out of hard disk space, or we're out of network ports, I think that because the cloud was a buzzword that a lot of shareholders and boards wanted to see in IT status reports and IT strategic plans, I think we grew a little bit further than we should have, from an enterprise perspective. And I think a lot of that's now being clawed back as organizations are maturing and looking to manage cost. Obviously, the huge growth of just the term FinOps from a search perspective over the last three years has cemented that, right? We're seeing a much more cost-conscious consumer—cloud consumer—than we saw three years ago.Corey: I think that the baseline level of understanding has also risen. It used to be that I would go into a client environment, prepared to deploy all kinds of radical stuff that these days look like context-aware architecture and things that would automatically turn down developer environments when developers were done for the day or whatnot. And I would discover that, oh, you haven't bought Reserved Instances in three years. Maybe start there with the easy thing. And now you don't see those, the big misconfigurations or the big oversights the way that you once did.People are getting better at this, which is a good thing. I'm certainly not having a problem with this. It means that we get to focus on things that are more architecturally nuanced, which I love. And I think that it forces us to continue innovating rather than just doing something that basically any random software stack could provide.John: Yeah, I think to your point, the easy wins are being exhausted or have been exhausted already, right? Very rarely do we walk into a customer and see that they haven't bought a, you know, Reserved Instance, or a Savings Plan. That's just not a thing. And the proliferation of software tools to help with those things, of course, in some cases, dubious proposition of, “We'll fix your cloud bill automatically for a small percentage of the savings,” that some of those software tools have, I think those have kind of run their course. And now you've got a smarter populace or smarter consumer and it does come into the more nuanced stuff, right.All right, do you really need to replicate data across AZs? Well, not if your workloads aren't stateful. Well, so some of the old things—and Kubernetes is a great example of this, right—the age old adage of, if I'm going to spin up an EKS cluster, I need to put it in three AZs, okay, why? That's going to cost you money [laugh], the cross-AZ traffic. And I know cross-AZ traffic is a simple one, but we still see that. We still see, “Well, I don't know why I put it across all three AZs.”And so, the service-to-service communication inside that cluster, the control plane traffic inside that cluster, is costing you money. Now, it might be minimal, but as you grow and as you scale your product or the services that you're providing internally, that may grow to a non-trivial sum of money.Corey: I think that there's a tipping point where an unbounded growth problem is always going to emerge as something that needs attention and needs to be focused on. But I should ask you this because you have a skill set that is, as you know, extremely in demand. You also have that rare gift that I wish wasn't as rare as it is where you can be thrown into the deep end knowing next to nothing about a particular technology stack, and in a remarkably short period of time, develop what can only be called subject matter expertise around it. I've seen you do this years past with Kubernetes, which is something I'm still trying to wrap my head around. You have a natural gift for it which meant that, from many respects, the world was your oyster. Why this? Why now?John: So, I think there's a couple of things that are unique at this thing, at this time point, right? So obviously, helping customers has always been something that's fun and exciting for me, right? Going to an organization and solving the same problem I've solved 20 different times, for example, spinning up a Kubernetes cluster, I guess I have a little bit of a little bit of squirrel syndrome, so to speak, and that gets—it gets boring. I'd rather just automate that or build some tooling and disseminate that to the customers and let them do that. So, the thing with cost management is, it's always a different problem.Yeah, we're solving fundamentally the same problem, which is, I'm spending too much, but it's always a different root cause, you know? In one customer, it could be data transfer fees. In another customer, it could be errant development growth where they're not controlling the spend on their development environments. In yet another customer, it could be excessive object storage growth. So, being able to hunt and look for those and play detective is really fun, and I think that's one of the things that drew me to this particular area.The other is just from a timing perspective, this is a problem a lot of organizations have, and I think it's underserved. I think that there are not enough companies—service providers, whatever—focusing on the hard problem of cost optimization. There's too many people who think it's a finance problem and not enough people who think it's an engineering problem. And so, I wanted to do work on a place where we think it's an engineering problem.Corey: It's been a very… long road. And I think that engineering problems and people problems are both fascinating to me, and the AWS bill is both. It's often misunderstood as a finance problem, and finance needs to be consulted absolutely, but they can't drive an optimization project, and they don't know what the context is behind an awful lot of decisions that get made. It really is breaking down bridges. But also, there's a lot of engineering in here, too. It scratches my itch in that direction, anyway.John: Yeah, it's one of the few business problems that I think touches multiple areas. As you said, it's obviously a people problem because we want to make sure that we are supporting and educating our staff. It's a process problem. Are we making costs visible to the organization? Are we making sure that there's proper chargeback and showback methodologies, et cetera? But it's also a technology problem. Did we build this thing to take advantage of the architecture or did we shoehorn it in a way that's going to cost us a small fortune? And I think it touches all three, which I think is unique.Corey: John, I really want to thank you for taking the time to speak with me. If people want to learn more about what you're up to in a given day, where's the best place for them to find you?John: Well, thanks, Corey, and thanks for having me. And, of course obviously, our website duckbillgroup.com is a great place to find out what we're working on, what we have coming. I also, I'm pretty active on LinkedIn. I know that's [laugh]—I'm not a huge Twitter guy, but I am pretty active on LinkedIn, so you can always drop me a follow on LinkedIn. And I'll try to post interesting and useful content there for our listeners.Corey: And we will, of course, put links to that in the [show notes 00:28:37], which in my case, is of course extremely self-aggrandizing. But that's all right. We're here to do self-promotion. Thank you so much for taking the time to chat with me, John. I appreciate it. Now, get back to work.John: [laugh]. All right, thanks, Corey. Have a good one.Corey: John Wynkoop, cloud economist at The Duckbill Group. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice while also taking pains to note how you're using multiple podcast platforms these days because that just seems to be the way the world went.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Organizations use their data to make better decisions and build innovative experiences for their customers. With the exponential growth in data, and the rapid pace of innovation in machine learning (ML), there is a growing need to build modern data applications that are agile and scalable. In this episode, Jillian is joined by Vara Bonthu, Principal Solutions Architect, and Alex Lines, Sr. Containers Specialist, to talk through why Kubernetes is becoming a popular choice for modernizing data applications, like batch processing and ML. They also discuss how AWS's open-source project, Data on EKS, helps customers build and test common use cases, like batch processing with Apache Spark or training an ML language model, to decrease the time it takes to get to production. Data on EKS website: https://awslabs.github.io/data-on-eks/ Data on EKS GitHub repository: https://github.com/awslabs/data-on-eks Data on EKS blog: https://go.aws/46bD1b8

In this episode we talk about Containers in the cloud with AWS Solutions Architects Jamal Arif and Nikita Patel. We start with the basics - What Are Containers? When does it make sense to use Containers? We then dive into Docker, Docker Swarm, Kubernetes and how container orchestration can be implemented through specific AWS services such as EKS and Fargate. If you're new to Containers, this episode will help you decide the right container strategy in the cloud.AWS Hosts: Nolan Chen & Malini Chatterjee

Kendall Miller is the Co-Founder and COO of CTO Lunches, a network of engineering leaders to get trusted advice and connections. The first half of the conversation with host Victoria Guido and special guest host, Joe Ferris, CTO of thoughtbot revolves around the use, adoption, and growth of Kubernetes within the technology industry. The discussion explores Kubernetes' history, influence, and its comparison with other platforms like Heroku and WordPress, emphasizing its adaptability and potential. The second half focuses on more practical aspects of Kubernetes, including its adoption and scalability. It centers on the appropriateness of adopting Kubernetes for different projects and how it can future-proof infrastructure. The importance of translating technical language into business speak is emphasized to influence executives and others in the decision-making process and Kendall also discuss communication and empathy in tech, particularly the skill of framing questions and understanding others' emotional states. __ CTO Lunches (http://ctolunches.com/) Follow CTO Lunches on LinkedIn (https://www.linkedin.com/company/ctolunches/) or Twitter (https://twitter.com/cto_lunches). Follow Kendall Miller on LinkedIn (https://www.linkedin.com/in/kendallamiller/). Follow thoughtbot on Twitter (https://twitter.com/thoughtbot) or LinkedIn (https://www.linkedin.com/company/150727/). Become a Sponsor (https://thoughtbot.com/sponsorship) of Giant Robots! Transcript: VICTORIA: This is the Giant Robots Smashing Into Other Giant Robots Podcast, where we explore the design, development, and business of great products. I'm your host, Victoria Guido. And with me today is Kendall Miller, Co-Founder and COO of CTO Lunches, a network of engineering leaders to get trusted advice and connections. Kendall, thank you for joining me. KENDALL: Thanks for having me. I'm excited. VICTORIA: And today, we have a special guest host, Joe Ferris, CTO of thoughtbot. Joe, thank you for joining us. JOE: Hello there. Thank you for having me. KENDALL: Hi, Joe. Thanks for being here. It's exciting. VICTORIA: Yes. It's so exciting. I think this is going to be a great episode. So, Kendall, I met you at a San Diego CTO lunch recently, and I know that's not the only thing that you do. So, you're also an advisor, a board member, and CXO. So, maybe tell us a little bit more about your background. KENDALL: Gosh, my background is complicated. I've been involved in tech for a very long time. In college, I worked for a company that started Twitter about five years too soon, and then worked in the nonprofit space in China for ten years, then came back, got back involved in tech. Today, I'm usually the business guy. So, when technical founders start technical products and want help turning them into successful technical businesses, that's when they call me. So, I have the technical background. I have never been paid to write code, which is probably a good thing. But I can hang in the technical conversations for the most part, but I'm much more interested in the business side and the people leadership side of business. So that tends to be where I play. Every organization hires me to do something different. VICTORIA: Thank you for that. And I'm just curious about the CTO Lunches. Just tell me a little bit more about that. And what's the idea behind it that led you to co-found it? KENDALL: CTO Lunches has actually been around for about eight years. And I didn't start the initial incarnation of it. It was two people that got us started, and I was trying to hire one of them; one thing led to another. Actually, originally, they did not want me to join. I think, at the time, my title was COO at a company that I was working with. About six months later, I took over engineering as VP of engineering, and then they're like, you can join the group now. We're less strict about that [laughs] now. Although it is highly focused on senior engineering leaders, it's not exclusively CTOs. But the group's been in place for a very long time, just intended as a place to network, have conversation with people who are in that senior-most technical position at technical organization. So, the CTO role is a lonely role. CTOs get fired all the time. There's not a technical person at the company that doesn't think they can do the job better than them. So, the CTO is always getting feedback. You're doing this wrong. The trade-offs you're making are wrong. This isn't going where it should be going. We should automate that. Why haven't we automated that? We should switch to this other tool. I've used it before; it's 100 times better. Joe, let me know if I'm getting any of this wrong. But that's the experience that I've had. Having a place where people can get together and, you know, half the time just complain to each other, hey, this is hard, is really why the networking group exists. So, it's a listserv. And there are local lunches that started in Boulder, Colorado. It's gotten pretty global. About a year ago, a little over a year ago, I was talking with one of the people who'd gotten it started. I've been involved in the Denver chapter for most of those eight years. And I was suggesting to him that he change a few things about it, to monetize it so that he could invest in it further. And he came back a few months later and said, "I want to take your advice and do this, but I want you to come do it with me." So, we founded the company officially...I think in December is when paperwork went into place. And we started investing in it a little bit more heavily. I was living in Europe last year, so we went and put on lunches in Paris, and Lisbon, and London, and, gosh, all over the place. I'm sure I'm missing some, Amsterdam. But there's been chapters all over the U.S. and a couple of other parts of the world for a long time. VICTORIA: That reflects my experience attending a CTO lunch. It's just very casual, like, just get together and eat food and talk about what you've worked on recently, issues you're having, just get ideas and make some friends. So, I really appreciated the group, and I'm going to personally plug the San Diego Chapter has picked up again. And we're meeting next Friday down in Del Mar. And we're going to be meeting on the last Friday of every month through October. So, I'm super excited to be a part of the group. And Joe, yeah, I'm curious about your perspective. As a CTO with thoughtbot, just what are your thoughts about that kind of thing? KENDALL: Yeah. How right am I about how lonely you are, Joe? JOE: [laughs] You know, I've been lonelier since we went remote. I used to work in the office, and I was a CTO, but also, I had lunch with people, which was nice. So, I'm lonelier. But yeah, I think everybody needs a group like that, like, senior developer therapy just to talk about your woes together, drown your sorrows. KENDALL: Well, I think years ago, I heard that CTOs are the most fired C-level executive. JOE: You're making me nervous now. KENDALL: [laughs] You've been there a long time, Joe. I know you've been there a long time. If you haven't been fired yet, you probably got a little while longer in you. This will be really awkward if it's published and you've already been fired. VICTORIA: We can always edit that out afterwards. [laughter] KENDALL: Yeah, no, I think it is a particularly lonely position. And, again, I think a lot of it is the average engineer in a technical company doesn't look at the COO or the CFO or even the CEO and think I could do that. But they're all looking at the CTO and thinking, what does that person do that I can't do? It's ridiculous because most of them would make terrible CTOs because it does require some of the business sense. Or, you know, right out of the gate, they might make terrible CTOs. It actually is quite a skill to be the most technical person and speak the business language. I mean, am I right about that, Joe? Like, was that hard for you to learn? JOE: Yeah, I definitely think...so, my background is also technical. I have a background in consulting. So, I always did a lot of metaprogramming, if you will. But making that transition to thinking about organizations that way, thinking about how all the other pieces play into it, was a pretty big step for me, even before I became a CTO as a consultant. KENDALL: Well, because you can't just chase the newest, hottest technology. You have to make business trade-offs. And not everything can be resume-driven development, right? Even if that technology over there is newer or hotter, it doesn't mean you have a business model that supports it. And it doesn't mean that migrating to it can be done, right? JOE: Yeah. I mean, even beyond choosing technologies, just choosing where to invest in your software stack, like, what needs to be reworked, what doesn't, and trying to explain those trade-offs, I think, is a rare skill. Being able to explain why something would be harder than something else when you're working with the leadership to prioritize a backlog it's a puzzle. KENDALL: Well, and I think when I'm in an executive conversation, and the CTO says, "Here's the thing that I think is the best decision technically, and I think it's the wrong decision for the business because of X, Y, or Z," I'm always super impressed, right? Like, this is the right technical solution for what we want. However, we shouldn't pursue that for business reasons right now. Maybe we can in six months, but right now, we need to prioritize this other thing. I don't know, that's always when I feel like, oh, this person knows what they're doing. JOE: There's nothing more dangerous to software than a bored developer. [laughs] One nice thing about being a consultant is that I don't have to invent problems to solve with technology at my company because sooner or later, I'll run across a company that has those problems, and I'll get to use that technology. But I think a lot of people are mostly happy...they might be happy in their role. They might be happy with our team. But they're very interested in whatever is hot right now, like machine learning, AI. And so, suddenly, that surreptitiously makes its way into the tech stack. And then, years later, it's somebody's problem to maintain. KENDALL: [laughs] Well, I have a specific memory of a firm in New York City that was, you know, this is relevant to y'all as thoughtbot is that, you know, at least historically, it was, to me, the premier Ruby on Rails consulting shop. I think that's still largely y'alls focus. Am I right about that? JOE: We still do a ton of Rails, yeah. KENDALL: Okay. Well, so this organization was all Ruby on Rails. It was a big organization. They had a very large customer base. And they hired a new CTO who came in, told everybody in the company they were stupid, laid off 70% of the engineering organization, and told the CEO he was going to completely rewrite the product from scratch in .NET, and he could do it in three weeks. And I'm pretty sure the business went under about three months later [laughs] because that was just so outrageously nuts to me. JOE: It's too bad he laid everybody off beforehand. I've been in that situation where somebody tells me, "I'm going to rewrite this. It'll be ready in three weeks." And I could fight with them and try and convince them they're wrong. But I feel like somebody who's approaching that with that attitude they're missing all of the nuance and context that would make it possible to explain to them why it's not going to work. And so, it's easier to just say, "You know, take the three weeks. I'll talk to you in three weeks." But if you've already laid off your development team, that's hard [laughs] to recover from. KENDALL: That's exactly right. VICTORIA: There's got to be a name for that kind of CTO who just wants to come in and blow everything up [laughs]. Yeah, so you spend a lot of time talking to different CTOs and doing this social networking aspect. I'm wondering if there's, like, patterns that you see. You've mentioned already one about just, like, the most often getting fired. [laughs] But what are the patterns you see, like, in challenges, and then what makes someone successful in that CTO role? KENDALL: Well, oh gosh, I have so many thoughts about this. First of all, I run into a couple of different categories of CTOs. There's a lot of people who come to CTO Lunches who are small company CTOs. I mean, it makes sense that there's a lot more small company CTOs than there are big company CTOs. But the small company CTO who maybe it's their first gig in the role or they're a serial CTO. There's the fractional CTOs that come that are doing it across several different organizations at the same time, and then there's the big company CTO who shows up. And honestly, all of their problems are very different. The thing that they have in common is even at a very large organization, in that position, they can make a decision that causes the company to go under. So, there is a significant amount of volatility in the amount of power that they wield. So, what's interesting about that is not everybody understands that. And so, first of all, there's the kind of CTO that just doesn't get that, and that doesn't matter if they're fractional, or a small company CTO, or a big company CTO. If they don't understand that, they're going to cause significant problems, right? Like the person I just mentioned who said, "I can just re-platform this in three weeks in .NET." There's that. I mean, I think, as with any senior leadership position, the comfort with volatility, the ability to know what to communicate down versus across and versus up, and then the ability to speak the business language. For everybody, the CFO's job is to communicate the financial needs alongside of the business leads, right? If the CFO's sole goal is to cut costs or make sure we're running as lean as possible, they're a bad CFO. But they're not as good of a CFO as the CFO who can say, "Hey, we're underspending right here. And I can look at the numbers and know we should invest more there. How can we invest more there and invest it well?" And it's the same thing for a technology executive to be able to look at the business context and communicate it back. And there are so many CTOs that I've worked with who they're the most technical person in the room, and they know it. And as a result, they're just a jerk to everyone around them, like, everything you did here was wrong. You know, that's where they fail. And so, if they can communicate the business needs, navigate the volatility, and support a team that's going to make decisions that aren't always the same decision they're going to make, they're going to be successful. Honestly, there's very, very few CTOs that I've met like that. People who are excited to meet you at work, excited to see you succeed, excited to see that you went and built a thing is great. I mean, the reason I was VP of engineering is the CTO that I was working with at the time...it's a terrible story. There was an engineer who had seen something that we were doing on repeat all the time and, in his spare time, spent about 40 hours outside of work, not during work hours, automating this task that we were doing regularly. And it was related to standing up a whole bunch of things in our standard infrastructure. He brings it to the CTO and says, "Look what I built." And the CTO, instead of saying, "Hey, this is incredible. Thank you. This is going to save us a bunch of time. Let's iterate on it. Here's some things I'd like to tweak. Can we bring it in this direction? Can we..." you know, whatever, said, "Why is this in Python? It should be in Ansible," something like that. I can't remember. And the engineer literally burst into tears. [laughs] JOE: Oh my God. KENDALL: [laughs] Well, I mean, yeah, it was like; literally, that's why the CTO stopped managing people that day. There's a lot of examples that I have like that. Joe, I appreciate that your response is, "Oh my God." Because I think there's a lot of people who'd be like, wait, what was wrong with that? Shouldn't it have been in Ansible? JOE: [laughs] Yeah, I've seen CTOs come into primarily two groups. One is the CTO who just tells, you know, like, they make the decisions, and they tell everybody what to do. They obviously don't have all of the information because you can't be in every room all the time. And the other is the CTO, who just wants to be one of the team members and doesn't make any decisions and tries to get people to make decisions collectively on their own without any particular guidance or structure. And finding that middle spot of, like, not just saying, "Hey, everything's in Ansible," allowing for the creativity and initiative, but also coalescing the group into a single direction, I think, is what makes a good CTO. KENDALL: Well, yeah, because the CTO does have to say no, sometimes, right? Like, the best product, people say, "No." Good CTOs say, "No." There is some amount of, hey, I need you to come to me with trade-offs about this. Why are you going to make that decision? And I'm sorry, you still didn't convince me, right? Like, I mean, those are appropriate things to say. But yeah, I'm with you on that. You said they fall into two categories. But you really mean the third and that middle ground. Is it easy for you to walk that middle ground, Joe? JOE: I wouldn't say it's easy. [laughs] KENDALL: Yeah. Well, I'm always nervous to say something. I'm doing well because I know there's a report out there that can point at every time I failed at it, right? So... MID-ROLL AD: Are you an entrepreneur or start-up founder looking to gain confidence in the way forward for your idea? At thoughtbot, we know you're tight on time and investment, which is why we've created targeted 1-hour remote workshops to help you develop a concrete plan for your product's next steps. Over four interactive sessions, we work with you on research, product design sprint, critical path, and presentation prep so that you and your team are better equipped with the skills and knowledge for success. Find out how we can help you move the needle at: tbot.io/entrepreneurs. VICTORIA: Yeah, what I'm getting from what you're saying, too, is this communication ability and not just, like, to communicate clearly but with a high level of empathy. So, if you say like, "Well, why is it in Python and not Ansible?" is different than being like, "What makes Python the best solution here?" Like, it's a different way to frame the question that could put someone on the defensive that just really requires, like, a high level of emotional intelligence. And also, if they've just worked, like, an 80-hour week, [laughs] I probably would maybe choose a different time to bring those questions up and notice that they have been burning the candle at both ends and prioritize getting them some rest. So, speaking of, like, communication and getting prioritization for [inaudible 15:34], especially on, like, infrastructure teams, maybe we could talk a little bit about Kubernetes, like, when that comes up as an appropriate solution, and how you talk about it with the business. KENDALL: My background with Kubernetes is long because a company that I still work with, Fairwinds, used to be called ReactiveOps, has been in the Kubernetes space for a very long time. I think we were one of the very first companies working with Kubernetes. It was coming up that people were running into the limits of something like Heroku, right? And I think it's Kelsey Hightower who said every company wants a PaaS. They just want the Paas that they built themselves. And that's really accurate. And I think Kubernetes isn't quite a framework for building your own PaaS or isn't quite a foundation where I think of a foundation for a house. Instead, it's more like rebar and cement and somebody saying, "Good luck, buddy." You know, you still have to know how to put the rebar and cement together to even make the foundation, but it is the building blocks that help get you to a custom-built PaaS. And it's become something that a lot of people have landed on as, you know, the broadly accepted way to build cloud-native infrastructure. The reason I've been in the Kubernetes space and the space that I see Kubernetes still filling is we need to standardize on something. We can choose a cloud provider's PaaS. We can choose a third-party PaaS, or we can standardize on something like Kubernetes. And even though we're not going to migrate from AWS to Azure, the flexibility that Kubernetes gives us as a broadly adopted pattern is going to give us some ability to be future-proofed in our infrastructure in a way that previous stacks were not, you know, it was Puppet, and it was Ansible. And it was SaltStack. And it was all Terraform all the time. I'm not saying those things don't exist anymore. I'm saying Kubernetes kind of has won that battle. Joe, since you're here and I know y'all are doing some Kubernetes work now at thoughtbot, I'm curious if you agree with that characterization. JOE: Yeah, I think that's true. I think it's the center for people to coalesce around. Like, for an effort in the industry to move forward, there needs to be some common language, some common ground. And I think Kubernetes struck the right balance of being abstract. So, you can use it in different environments but still making some decisions, so you don't have to make them all. And so, like, all of the things you had to do with containers like figuring out what your data solution is going to be, what your networking solution is going to be, Kubernetes didn't even really make those decisions. [laughs] They just made a platform where those decisions can be made in a common way. And that allowed the community and the ecosystem to grow. KENDALL: I mean, I think of it a lot like WordPress; you know, WordPress is hated by many. When WordPress came out, it was hot, right? And it was PHP, which everybody was super excited about at the time. Kubernetes is going to reach a point where it's as long in the tooth and terrible as people think WordPress is, but it has become the standard. And the advantage of the standard is you can use the not standard. You can go build a website in Jekyll instead of WordPress, and there's going to be some things that are nicer about Jekyll. But because WordPress is so broadly adopted, there's a plugin for everything. And I think that's where Kubernetes sits is because it's become so widely adopted everybody's building for it. Everybody's adapting for it. If you run into a problem, you're going to find somebody else out there who has that problem. In fact, I think of one organization that I know that was on HashiCorp's Nomad. And they said, "Actually, we think Nomad has better technology through and through. But we think we're the only company at this size and scale using Nomad. And so, when we run into a problem, we can't Google for it. There's no such thing as a plugin that exists to solve this. Nobody has ever run into this before on Nomad. But there's 100 companies dealing with the same problem in Kubernetes, and there's ten solutions." And I think that's the power that it brings. VICTORIA: So, it's not just a trend that CTOs are moving towards, you think. KENDALL: I mean, I think it's already won the battle and the hockey stick of adoption. We're still right at the very bottom of that tick-up because it takes people a long time to adapt new technology like this, especially in their infrastructure. It's a big migration, to move. So, I don't think it's the widely adopted infrastructure technology even yet. I think a lot of the biggest organizations are still running on things that predate Kubernetes. But I think it has won the battle, and it is winning the battle and is going to be the thing going forward, so yeah. JOE: I think it also has a lot of room to grow still. Like, there are other technologies that I used previously, like Docker, and they were a big step up from some of the things I was doing at the time. But you quickly hit the ceiling, or it was, like, I don't know where to go with this next. I don't know what else is going to happen. Whereas with Kubernetes, there are so many directions it can go in. Like, the serverless Kubernetes offerings that are starting to pop up are extremely interesting, where, you know, you don't actually maintain a cluster or anything. You just deploy things to this ethereal cluster that always exists. And so, that sort of combination of platform as a service, function as a service, Kubernetes, as that evolves, I think there are a lot of exciting things that have yet to come in the Kubernetes space. KENDALL: Well, so say more about that, Joe, because I've been going to KubeCon for a very long time, maybe...I don't know if it's 2016 or so when I first went. And it felt for a number of years...maybe those first four-ish years it was always the people at KubeCon were the, like, big dreamers and thinkers and, like, we're here to change the future of cloud infrastructure. And this is going places, and we're excited to be here and be a part of it. And here's what I'm going to do that changes the next thing. And I feel like now if I go to KubeCon, it's a lot of people from, you know, IBM and some big bank that are, like, deep sigh, well, I have to adopt Kubernetes. I need to know what the vendors are. What do you guys do, and how does this work? Can you please teach it to me? Because I'm being told by my boss, I have to do it. I don't see that excitement around Kubernetes anymore. The excitement I see is all around further up the stack, you know, things like Wasm, WebAssembly, or eBPF, the networking things and tracing things that are possible. Maybe that's further down the stack. I guess it depends on how you think about it, but different part of the stack. So, I'm curious, touching on the serverless components of Kubernetes; sure, I get that. And I do think, increasingly, the PaaSs of the future are all going to be Kubernetes-based, whether that's exposed or not. But where are the places that you think it's still going to go? Because I feel like it's already gotten boring, maybe in a positive way. But I don't see the excitement around it like I saw a few years back. And I'm curious what else you think is going to happen. JOE: Yeah, I mean, I don't think I disagree. I think Kubernetes itself, the core concept, is, like, it's still changing. But you're right that the excitement about Kubernetes existing has gone down because it's been there for a while. But I feel like the ecosystem is still growing pretty rapidly. Like, the things you mentioned, like Wasm and Istio, and all the tools in that ecosystem that continue to grow, is where I think the interesting things will happen. Like, it's created this new lower-level layer of abstraction that makes it possible to build concepts and technology that could not have existed before. KENDALL: Yeah, well, and I'm, you know, talking to people who are working really hard at making short-run ephemeral workloads work better on things like GPUs for the sake of AI, right? Like, I mean, there is some really interesting things happening, and people are doing this in Kubernetes. So, I get that. I agree with that. It is interesting that Kubernetes has become sort of the stable thing, and now it's about who can build the interesting add-ons. It's almost like, okay, we've built Half-Life. What is Counter-Strike going to look like? You know. That's a terrible (I'm aging myself.) example. But still. VICTORIA: I think it's interesting, I mean, to look at the size of the market for platform engineering right now. In 2022, was 4.8 billion, and it's estimated to be in 10 years $41 billion. So, there is this emerging trend of different platform engineering products, different abstractions on top of Kubernetes. And I wonder what advice you would have for a technical founder who's looking to build and solve some of these interesting issues in Kubernetes and create a business around it. KENDALL: Well, okay, let me clarify that question. Are you thinking, I'm a startup, and I need to build my infrastructure, and I'm going to choose Kubernetes. What advice do I need? Or are you thinking, I am founder, and I want to go build on the Kubernetes ecosystem. What advice do you have? VICTORIA: Now I want to know the answer to both. But my question was the second one to start. KENDALL: One of the things that is hard about the Kubernetes ecosystem is there's not a ton of companies that have made a whole bunch of money in Kubernetes because, as I said, I still think we're actually really early in the adoption curve. The kinds of companies that have adopted Kubernetes are the kinds of companies that don't spend lots and lots of money on an infrastructure. [laughs] They're the kinds of companies that are fast-moving, early adopters, or, you know, those first followers, and so they're under $100 million companies for the most part. Where the JP Morgans and Chase are running Kubernetes somewhere in their stack, but they haven't adopted it across the stack to need the biggest, best tools about it. So, the first piece of advice that I'd give is, be a little wary. It's still very early to the market. Maybe now is the time to build the thing. When ReactiveOps pivoted to Kubernetes, I think it was six months of having conversations with companies who were just, like, so excited about it, and this is definitely what we want to do. But nobody was doing it yet. You know, it was, we have, like, six solid months of just excitement and nobody actually pulling the trigger. And, you know, we were a little too early to that market. And that was just the people adopting it. So, I think there is some nervousness that cloud-native solutions the only people who are really making money in Kubernetes are named Amazon, Google, and Microsoft because it's the cloud providers that are making a ton off of it. Now, there's Rancher. There is StackPointCloud. There's a few others that have had big exits in this space. But I don't think it's actually as big of a booming economy as a lot of people think, in part because EKS is an incredibly amazing product. Like, eight years ago, the thing people paid us the most to do at ReactiveOps was just stand up Kubernetes because it was so stinking hard to just get it up and working. And now you click some buttons. Anybody can go do that. So, it's changed a lot, right? And I think be wary when you're entering that ecosystem. And then, my advice to the founder that's not building on the ecosystem but just looking to adopt a technology that's going to be a future-proofed infrastructure is just adopt one of the cloud-native platforms. And there are a whole bunch of sort of default best-in-class add-ons out there that you need to throw in. Don't adopt too many because then you have to maintain them forever. That's the easiest way to get started. You can figure out all the rest of it later. But if you go use EKS, or GKE, AKS, you can get started pretty easily and build something that is going to be future-proofed. I don't know, Joe; I'm curious if you disagree with any of that. JOE: Well, I think it's interesting to think about who's making money in Kubernetes. Like, I think there might not be as many companies who are doing only Kubernetes and Kubernetes-focused products that are massively successful. But I think because it has had a good amount of adoption and because it's easier to work with something that's standardized, it has helped companies sell things that they wanted to sell anyway. Like, all the Datadog, all the Scalas, the logging companies, they all have Kubernetes add-ons. And now everybody is paying Datadog [laughs] to have a dashboard for their Kubernetes cluster. I think they're making more money than they would have been without targeting the market. And so, I think that's really...if you want to get into the market, it's not, like, I'm going to build a Kubernetes product. It's if I'm building operations and an infrastructure product, I should definitely have it work with Kubernetes, and people will want to click and install it. KENDALL: So, to be clear, you know, one of the companies that I work with is called Axiom, and they play in the same, you know, monitoring, observability space as Datadog does. And part of what makes Kubernetes interesting in that space is in a microservice environment; there's so much happening. Where are problems being caused? We don't live in a day where I can just run my code, and it tells me that there's an unexpected semicolon on line 23, right? Like, that still happens. You're still doing those things. But this microservice talking to that microservice is where things tend to break down. Did I communicate this correctly? What was sent? What was received? Where did it break down? What was the latency? And if you were doing things in the old way back when you were standing it up with, say, Ansible, or Puppet, or something like that, and you were orchestrating all of these cloud virtual machines, you had to really work hard to instrument the tracing and logging and everything involved in order to track what was going on. Whereas that's one of the magic things about Kubernetes is with a few of the add-ons or some of the things out of the box with Kubernetes, it's a couple of clicks to get so, so much of the data and have insight into where things are going and what's going wrong. And so, I 100% agree with that. Kubernetes is generating a tremendous amount of data. And if you're a data company, it's really nice to have all that come in, and it helps them make money, helps the user of Kubernetes in that situation understand where problems are happening and breaking down. Yeah, there's definitely some network effects of what Kubernetes is doing in that. I completely agree. JOE: I think there are also some interesting companies, like, where they make...Emissary, Ambassador, and they have that sort of dual -- KENDALL: Komodor, is that -- JOE: Yeah, maybe. They have open source, but then they have a product. KENDALL: You're thinking of Ambassador Labs. JOE: Yeah. Ambassador Labs, yeah. I guess I don't really know how much money they're making. But I think that's a really interesting concept as people who make open-source things then make a well-supported product built around it. KENDALL: Sure. What's interesting is, I think in the VC world, at least right now, and it may pick up again, but post-Silicon Valley Bank nearly caving in, I think that the VC tolerance for, yeah, just go get a billion open-source adopters, and we'll figure out how to monetize later I think that the tolerance for that is a lot lower than it was even six months ago. JOE: Yeah, I think you have to have a dual model right from the beginning now. KENDALL: Yeah. Agreed. VICTORIA: You got to figure out how to make money on Kubernetes before you can. [laughs] KENDALL: You know, minor detail. That's why I think services companies in this space still have a lot going for it. Because in order to even be able to sell software to a company using Kubernetes, you half the time have to go stand up Kubernetes for them because it is still that hard for so many people to really adopt it. VICTORIA: Yeah. And maybe, like, talking more about, like, when it is the right decision to start on Kubernetes because I think the question I get sometimes is just, is it overkill? Is it too much for what we're building? Especially, like, if you're building a brand-new product, you're not even sure if it's going to get adopted that widely. KENDALL: I mean, and I'm [laughs] curious your thought on this, Joe, but there's a good argument to be made that Heroku was enough for the vast majority of founders early on. But the thing is, Kubernetes isn't as hard as it used to be. Going and clicking a couple of buttons on GKE and deploying something into Kubernetes with GKE Autopilot running it's not as easy as Heroku, but it's not wildly far off. And it does substantially future-proof you. So, when is it too early? I'm not sure it's ever too early if you have an intention of scaling if you're planning on running some kind of legacy workload, like, things that are going to be stateful. Or maybe WordPress, for example, you don't probably need to deploy your WordPress blog onto Kubernetes. You can do that in your cPanel on Bluehost. I don't actually know if Bluehost even exists anymore, but I assume it's still a thing. I don't know, what would you say, Joe? JOE: I agree with that. I think it's a hard first pill to swallow. But I think the reality is that it's very easy to underestimate the infrastructure needs of even an early product. Like, it doesn't really matter what you're building. You're still going to have things like secrets management. You're still going to have to worry about networking. They just don't go away. There's no way you have a product without them. And so, rather than slowly solving all those problems from scratch on a platform that isn't designed for it, I think it's easier to just bite the bullet and use one of the managed solutions, especially, as you said, I think it's getting easier and easier. The activation energy from going from credit card to Kubernetes cluster is just getting lower. KENDALL: And so, the role of the CTO is just getting easier and easier because they can just adopt the one technology, and it's obviously Kubernetes. And it's obviously Rust, right? [laughter] Yeah, no, I'm with you. And I think if you find somebody who knows Kubernetes inside and out, it's really not going to take them long to get started. VICTORIA: Yeah, once again, change management is the biggest challenge for any new innovation coming into adoption. So, I'm curious to talk more about the influence that you need and how you influence others to come around to these types of ideas, like, in the executive suite and with the leadership of a company, especially on these types of topics, which can feel maybe a little abstract for people. KENDALL: How you influence them specifically to use Kubernetes, or just how you talk with them about technology adoption in general? Or what are you asking? VICTORIA: Yeah, like, how do I get people to not just turn their ears off when I say the word Kubernetes? [laughs] KENDALL: Yeah, I mean, I think...so I think that's where it's the technologist's job and the role of the CTO to translate these things into business speak. And that's why I'm using words like future-proofing your infrastructure is because there are companies that...I know one company that made a conscious decision that they were going to try to re-platform every single year, and that is not a good idea or sustainable for the vast majority [laughs] of companies. In fact, I can't think of a single situation where that makes sense. But if you can say to the CFO, "Hey, it's going to cost us a little bit more right now. It's going to save us substantially in the long term because this is the thing that's winning. And if we go standardize on Heroku right now, every company does eventually have to migrate off of Heroku. They either go out of business, or they get too big for it." That's the kind of thing that needs to be communicated in order to get people to adopt it. They don't care what the word is. They don't care if you're saying Kubernetes; you know, most CFOs understand it about as well as my mom does. My mom tries to bring it up in conversation because she's heard me use it. And she thinks it makes her sound smart, which maybe it does in the right climate. VICTORIA: My partner does the same thing. He says DevOps and Kubernetes all the time. I'm like; you don't know what you're talking about. [laughter] JOE: Those words do not come up in my house. KENDALL: One of my kids asked me to explain Kubernetes. And I do a whole talk, particularly at organizations where understanding Kubernetes is essential to the salespeople's role. And I give a whole talk about the background of how we got here from deploying on some servers in our back room. And, you know, what's different about the cloud, what containerization did, et cetera. And I have this long explanation. And I remember taking a deep breath and saying to my kids, "Do you really want to hear this?" And I had one son say, "Yes, absolutely." And my wife and three of the other kids all stood up and said, "No way," and left the room. So, when somebody asks me, "What do you do?" Actually, one of the key relationships I built with some of the early people at GCP when we were partnering closely with them was a person that I met, and I asked, "What do you do for a living?" And he said, "I can tell you, but it's not going to mean anything to you." And I was like, "That's what I say to people." And it turned out he was in charge of, you know, Kubernetes partnerships for Google. I can explain to you what it means and why it's important. But you're not going to be happy that I spent that time explaining it to you. VICTORIA: [laughs] That sounds awesome, though. It sounds like you built a server rack just to demo to your children what it was. KENDALL: No, no. I just talked back through the history of...that company that I mentioned that built Twitter about five years too early; we had a, you know, we had a server rack in the...literally physically in our closet that was serving up our product at the time. VICTORIA: Probably the best demo I ever saw was at Google headquarters in Herndon, and someone had built...They had 3D-printed a little mini server rack that they had put Raspberry Pis onto, and then they had Kubernetes deployed on it. And they did an automatic failover of a node to just demo how it works and had little lights that went with it. It was pretty fun. So maybe you should get one for yourself. [laughter] It's a fun project. KENDALL: They remember the things that it enables. They don't remember what it does. And so, when I say so, and so is a client that's using this technology, then they get real excited because they're like, "My dad makes that work." And I'm like, well, okay, that's kind of a stretch, but you get the idea. VICTORIA: Yeah, you got to lean into that kind of reputation in your house. KENDALL: That's right. VICTORIA: And you're like, yes, that's correct. KENDALL: That's right. [laughs] VICTORIA: I do make Kubernetes. I make all the clouds work, yeah. KENDALL: Actually, my most common explanation is Kubernetes is the plumbing of the internet. Unless you're a plumber, you don't care about the pipes. You just want your shit to flush when you use the toilet. You want the things to load when you click your buttons. You don't actually care what's going on behind the scenes, but this is what's orchestrating it increasingly across the internet. VICTORIA: So far, we've called Kubernetes WordPress or the toilet. [laughs] KENDALL: The plumbing. [laughter] VICTORIA: You are really good at selling it. [laughter] KENDALL: Hey, if you want to build a nice, clean city, you need good plumbing. You might not care what the pipes are made of, but you need good plumbing. [laughs] VICTORIA: Works for me. On that note -- [laughs] KENDALL: Yeah. Right? Right? VICTORIA: That's [inaudible 36:41] on a high note. Is there anything else that you'd like to promote? KENDALL: With regards to CTO Lunches, we have a free listserv. There are local lunches. If there isn't a local lunch where you are, it's very lightweight to start up a chapter. We often have folks who are willing to sponsor that first lunch to get you going. We do have a paid tier of CTO Lunches. If you want a small back room Slack channel of people to discuss, I think it's $99 a month. Yeah, if you're a CTO and/or a senior engineering leader and you want a community of people to process with, be it our free tier or our paid tier, we've got something for you. We're trying to invest in this to build community around it. And it's something we enjoy doing more than almost anything. Come take part. VICTORIA: You can subscribe to the show and find notes along with a complete transcript for this episode at giantrobots.fm. If you have questions or comments, email us at hosts@giantrobots.fm. And you can find me on Twitter @victori_ousg. This podcast is brought to you by thoughtbot and produced and edited by Mandy Moore. Thanks for listening. See you next time. ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com. Special Guest: Kendall Miller.

Welcome episode 220 of The Cloud Pod podcast - where the forecast is always cloudy! This week your hosts, Justin, Jonathan, Ryan, and Matthew discuss all things cloud, including virtual machines, an AI partnership between Microsoft and Meta for Llama 2, Lambda functions, Fargate, and lots of security updates including the Outlook breach and WORM protections. This and much more in our newest episode.  Titles we almost went with this week: Too Many Bees died for Honeycode Microsoft announces that AI will only cost you 3 arms and a leg.   The Cloud Pod also detects Recursive Loops in cloud news The cloud pod disables health checks bc who needs them A big thanks to this week's sponsor: Foghorn Consulting, provides top-notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you have trouble hiring?  Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week.

Rich Mogull, SVP of Cloud Security at FireMon, joins Corey on Screaming in the Cloud to discuss his career in cybersecurity going back to the early days of cloud. Rich describes how he identified that cloud security would become a huge opportunity in the early days of cloud, as well as how cybersecurity parallels his other jobs in aviation and emergency medicine. Rich and Corey also delve into the history of Rich's involvement in the TidBITS newsletter, and Rich unveils some of his insights into the world of cloud security as a Gartner analyst. About RichRich is the SVP of Cloud Security at FireMon where he focuses on leading-edge cloud security research and implementation. Rich joined FireMon through the acquisition of DisruptOps, a cloud security automation platform based on his research while as CEO of Securosis. He has over 25 years of security experience and currently specializes in cloud security and DevSecOps, having starting working hands-on in cloud over 12 years ago. He is also the principle course designer of the Cloud Security Alliance training class, primary author of the latest version of the CSA Security Guidance, and actively works on developing hands-on cloud security techniques. Prior to founding Securosis and DisruptOps, Rich was a Research Vice President at Gartner on the security team. Prior to his seven years at Gartner, Rich worked as an independent consultant, web application developer, software development manager at the University of Colorado, and systems and network administrator.Rich is the Security Editor of TidBITS and a frequent contributor to industry publications. He is a frequent industry speaker at events including the RSA Security Conference, Black Hat, and DefCon, and has spoken on every continent except Antarctica (where he's happy to speak for free -- assuming travel is covered).Links Referenced: FireMon: https://www.firemon.com/. Twitter: https://twitter.com/rmogull Mastodon: [https://defcon.social/@rmogull](https://defcon.social/@rmogull) FireMon Blogs: https://www.firemon.com/blogs/ Securosis Blogs: https://securosis.com/blog TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is Rich Mogull, SVP of Cloud Security over at FireMon now that I'm a bit too old to be super into Pokémon, so I forget which one that is. Rich, thanks for joining me. I appreciate it.Rich: Thank you. Although I think we need to be talking more Digimon than Pokémon. Not that I want to start a flame war on the internet in the first two minutes of the conversation.Corey: I don't even have the level of insight into that. But I will say one of the first areas where you came to my notice, which I'm sure you'll blame yourself for later, is that you are the security editor behind TidBITS, which is, more or less, an ongoing newsletter longer than I've been in the space, to my understanding. What is that, exactly?Rich: So, TidBITS is possibly the longest-running—one of the longest-running newsletters on the internet these days and it's focused on all things Apple. So, TidBITS started back in the very early days as kind of more of an email, I think like, 30 years ago or something close to that. And we just write a lot about Apple and I've been reading about Apple security there.Corey: That's got to be a bit of an interesting experience compared to my writing about AWS because people have opinions about AWS, particularly, you know, folks who work there, but let's be clear, there is nothing approaching the zealotry, I think I want to call it, of certain elements of the Apple ecosystem whenever there is the perception of criticism about the company that they favor. And I want to be clear here to make sure I don't get letters myself for saying this: if there's an Apple logo on a product, I will probably buy it. I have more or less surrounded myself with these things throughout the course of the last ten years. So, I say this from a place of love, but I also don't wind up with people threatening me whenever I say unkind things about AWS unless they're on the executive team.Rich: So, it's been a fascinating experience. So, I would say that I'm on the tail end of being involved with kind of the Mac journalist community. But I've been doing this for over 15 years is kind of what I first started to get involved over there. And for a time, I wrote most of the security articles for Macworld, or a big chunk of those, I obviously was writing over a TidBITS. I've been very lucky that I've never been on the end of the death threats and the vitriol in my coverage, even though it was balanced, but I've also had to work a lot—or have a lot of conversations with Apple over the years.And what will fascinate you is at what point in time, there were two companies in the world where I had an assigned handler on the PR team, and one was Apple and then the other was AWS. I will say Apple is much better at PR than [laugh] AWS, especially their keynotes, but we can talk about re:Invent later.Corey: Absolutely. I have similar handlers at a number of companies, myself, including of course, AWS. Someone has an impossible job over there. But it's been a fun and exciting world. You're dealing with the security side of things a lot more than I am, so there's that additional sensitivity that's tied to it.And I want to deviate for a second here, just because I'm curious to get your take on this given that you are not directly representing one of the companies that I tend to, more or less, spend my time needling. It seems like there's a lot of expectation on companies when people report security issues to them, that you're somehow going to dance to their tune and play their games the entire time. It's like, for a company that doesn't even have a public bug bounties process, that feels like it's a fairly impressively high bar. On some level, I could just report this via Twitter, so what's going on over there? That feels like it's very much an enterprise world expectation that probably means I'm out of step with it. But I'm curious to get your take.Rich: Out of step with which part of it? Having the bug bounty programs or the nature of—Corey: Oh, no. That's beside the point. But having to deal with the idea of oh, an independent security researcher shows up. Well, now they have to follow our policies and procedures. It's in my world if you want me to follow your policies and procedures, we need a contract in place or I need to work for you.Rich: Yeah, there is a long history about this and it is so far beyond what we likely have time to get into that goes into my history before I even got involved with dealing with any of the cloud pieces of it. But a lot about responsible disclosure, coordinated disclosure, no more free bugs, there's, like, this huge history around, kind of, how to handle these pieces. I would say that the core of it comes from, particularly in some of the earlier days, there were researchers who wanted to make their products better, often as you criticize various things, to speak on behalf of the customer. And with security, that is going to trigger emotional responses, even among vendors who are a little bit more mature. Give you an example, let's talk about Apple.When I first started covering them, they were horrific. I actually, some of the first writing I did that was public about Apple was all around security and their failures on security disclosures and their inability to work with security researchers. And they may struggle still, but they've improved dramatically with researcher programs, and—but it was iterative; it really did take a cultural change. But if you really want to know the bad stories, we have to go back to when I was writing about Oracle when I was a Gartner analyst.Corey: Oh, dear. I can only imagine how that played out. They have been very aggressive when it comes to smacking down what they perceive to be negative coverage of anything that they decide they like.Rich: Yeah, you know, if I would look at how culturally some of these companies deal with these things when I was first writing about some of the Oracle stuff—and remember, I was a Gartner analyst, not a vulnerability researcher—but I'm a hacker; I go to Blackhat and DEF CON. I'm friends with the people who are smarter than me at that or have become friends with them over the years. And I wrote a Gartner research note saying, “You probably shouldn't buy any more Oracle until they fix their vulnerability management process.” That got published under the Gartner name, which that may have gotten some attention and created some headaches and borderline legal threats and shade and all those kinds of things. That's an organization that looks at security as a PR problem. Even though they say they're more secure, they look at security as a PR problem. There are people in there who are good at security, but that's different. Apple used to be like that but has switched. And then Amazon is… learning.Corey: There is a lot of challenge around basically every aspect of communication because again, to me, a big company is one that has 200 people. I think that as soon as you wind up getting into the trillion-dollar company scale, everything you say gets you in trouble with someone, somehow, somewhere, so the easiest thing to do is to say nothing. The counterpoint is that on some point of scale, you hit a level where you need a fair bit of scrutiny; it's deserved at this point because you are systemically important, and them's the breaks.Rich: Yeah, and they have improved. A lot of the some of the larger companies have definitely improved. Microsoft learned a bunch of those lessons early on. [unintelligible 00:07:33] the product in Azure, maybe we'll get there at some point. But you have to—I look at it both sides a little bit.On the vendor side, there are researchers who are unreasonable because now that I'm on the vendor side for the first time in my career, if something gets reported, like, it can really screw up plans and timing and you got to move developer resources. So, you have outside influences controlling you, so I get that piece of it. But the reality is if some researcher discovered it, some China, Russia, random criminals are going to discover it. So, you need to deal with those issues. So, it's a bit of control. You lose control of your messaging and everything; if marketing gets their hands in this, then it becomes ugly.On the other hand, you have to, as a vendor, always realize that these are people frequently trying to make your products better. Some may be out just to extort you a little bit, whatever. That's life. Get used to it. And in the end, it's about putting the customers first, not necessarily putting your ego first and your marketing first.Corey: Changing gears slightly because believe it or not, neither you nor I have our primary day jobs focused on, you know, journalism or analyst work or anything like that these days, we focus on these—basically cloud, for lack of a better term—through slightly different lenses. I look at it through cost—which is of course architecture—and you look at it through the lens of security. And I will point out that only one of us gets called at three in the morning when things get horrible because of the bill is a strictly business-hours problem. Don't think that's an accident as far as what I decided to focus on. What do you do these days?Rich: You mean, what do I do in my day-to-day job?Corey: Well, it feels like a fair question to ask. Like, what do you do as far as day job, personal life et cetera. Who is Rich Mogull? You've been a name on the internet for a long time; I figured we'd add some color and context to it.Rich: Well, let's see. I just got back from a flying lesson. I'm honing in on my getting ready for my first solo. My side gig is as a disaster response paramedic. I dressed up as a stormtrooper for the 501st Legion. I've got a few kids and then I have a job. I technically have two jobs. So—Corey: I'm envious of some of those things. I was looking into getting into flying but that path's not open to me, given that I have ADHD. And there are ways around it in different ways. It's like no, no, you don't understand. With my given expression of it, I am exactly the kind of person that should not be flying a plane, let's be very clear here. This is not a regulatory thing so much as it is a, “I'm choosing life.”Rich: Yeah. It's a really fascinating thing because it's this combination of a physical and a mental challenge. And I'm still very early in the process. But you know, I cracked 50, it had always been a life goal to do this, and I said, “You know what? I'm going to go do it.”So, first thing, I get my medical to make sure I can actually pass that because I'm over 50, and then from there, I can kind of jump into lessons. Protip though: don't start taking lessons right as summer is kicking in in Phoenix, Arizona, with winds and heat that messes up your density altitude, and all sorts of fun things like that because it's making it a little more challenging. But I'm glad I'm doing it.Corey: I have to imagine. That's got to be an interesting skill set that probably doesn't have a huge amount of overlap with the ins and outs of the cloud business. But maybe I'm wrong.Rich: Oh God, Corey. The correlations between information security—my specialty, and cloud security as a subset of that—aviation, and emergency medicine are incredible. These are three areas with very similar skill sets required in terms of thought processes. And in the case of both the paramedic and aviation, there's physical skills and mental skills at the same time. But how you look at incidents, how you process things algorithmically, how you—your response times, checklists, the correlations.And I've been talking about two of those three things for years. I did a talk a couple years ago, during Covid, my Blackhat talk on the “Paramedics Guide to Surviving Cybersecurity,” where I talked a lot about these kinds of pieces. And now aviation is becoming another part of that. Amazing parallels between all three. Very similar mindsets are required.Corey: When you take a look at the overall sweep of the industry, you've been involved in cloud for a fairly long time. I have, too, but I start off as a cynic. I started originally when I got into the space, 2006, 2007, thinking virtualization was a flash in the pan because of the security potential impact of this. Then cloud was really starting to be a thing and pfff, that's not likely to take off. I mean, who's going to trust someone else to run all of their computing stuff?And at this point, I've learned to stop trying to predict the future because I generally get it 180 degrees wrong, which you know, I can own that. But I'm curious what you saw back when you got into this that made you decide, yeah, cloud has legs. What was that?Rich: I was giving a presentation with this guy, Chris Hoff, a good friend of mine. And Chris and I joined together are individual kind of research threads and were talking about, kind of, “Disruptive Innovation and the Future of Security.” I think that was the title. And we get that at RSA, we gave that at SOURCE Boston, start kind of doing a few sessions on this, and we talked about grid computing.And we were looking at, kind of, the economics of where things were going. And very early, we also realized that on the SaaS side, everybody was already using cloud; they just didn't necessarily know it and they called them Application Service Providers. And then the concepts of cloud in the very early days were becoming compelling. It really hit me the first time I used it.And to give you perspective, I'd spent years, you know, seven years as a Gartner analyst getting hammered with vendors all the time. You can't really test those technologies out because you can never test them in a way that an enterprise would use them. Even if I had a lab, the lab would be garbage; and we know this. I don't trust things coming out of labs because that does not reflect operational realities at enterprise scale. Coming out of Gartner, they train me to be an enterprise guy. You talk about a large company being 200? Large companies start at 3000 to 5000 employees.Corey: Does that map to cloud services the way that AWS expresses? Because EKS, you're going to manage that differently in an enterprise environment—or any other random AWS service; I'm just picking EKS as an example on this. But I can spin up a cluster and see what it's like in 15 minutes, you know, assuming the cluster gets with the program. And it's the same type of thing I would use in an enterprise, but I'm also not experiencing it in the enterprise-like way with the processes and the gating and the large team et cetera, et cetera, et cetera. Do you think it's still a fair comparison at that point?Rich: Yeah, I think it absolutely is. And this is what really blew my mind. 11 or 12 years ago, when I got my first cloud account setup. I realized, oh, my God. And that was, there was no VPC, there was no IAM. It was ephemeral—and—no, we just had EBS was relatively new, and IAM was API only, it wasn't in the console yet.Corey: And the network latency was, we'll charitably call it non-deterministic.Rich: That was the advantage of not running anything at scale, wasn't an issue at the time. But getting the hands-on and being able to build what I could build so quickly and easily and with so little friction, that was mind-blowing. And then for me, the first time I've used security groups I'm like, “Oh, my God, I have the granularity of a host firewall with the manageability of a network firewall?” And then years later, getting much deeper into how AWS networking and all the other pieces were—Corey: And doesn't let it hit the host, which I always thought a firewall that lets—Rich: Yes.Corey: —traffic touch the host is like a seatbelt that lets your face touch the dashboard.Rich: Yeah. The first thing they do, they go in, they're going to change the rules. But you can't do that. It's those layers of defense. And then I'm finding companies in the early days who wanted to put virtual appliances in front of everything. And still do. I had calls last week about that.But those are the things that really changed my mind because all of a sudden, this was what the key was, that I didn't fully realize—and it's kind of something that's evolved into something I call the ‘Grand Unified Theory of Cloud Governance,' these days—but what I realized was those barriers are gone. And there is no way to stop this as people want to build and test and deploy applications because the benefits are going to be too strong. So, grab onto the reins, hold on to the back of the horse, you're going to get dragged away, and it's your choice if your arm gets ripped off in the process or if you're going to be able to ride that thing and at least steer it in the general direction that you need it to go in.Corey: One of the things that really struck me when I started playing around with cloud for more than ten minutes was everything you say is true, but I can also get started today to test out an idea. And most of them don't work, but if something hits, suddenly I don't have the data center constraints, whereas today, I guess you'd call it, I built my experiment MVP on top of a Raspberry Pi and now I have to wait six weeks for Dell to send me something that isn't a piece of crap that I can actually take production traffic on. There's no okay, and I'll throw out the junky hardware and get the good stuff in once you start hitting a point of scale because you're already building on that stuff without the corresponding massive investment of capital to get there.Rich: Yeah well, I mean, look, I lived this, I did a startup that was based on demos at a Blackhat—sorry, at a Blackhat. Blackhat. Did some demos on stage, people were like, “We want your code.” It was about cloud security automation. That led to doing your startup, the thing called DisruptOps, which got acquired, and that's how I ended up at FireMon. So, that's the day job route where I ended up.And what was amazing for that is, to add on to what you said, first of all, the friction was low; once we get the architecture right, scalability is not something we are hugely concerned with, especially because we're CI/CD. Oh, no, we hit limits. Boom, let's just stand up a new version and redirect people over there. Problem solved. And then the ability to, say, run multiple versions of our platform simultaneously? We're doing that right now. We just had to release an entirely free version of it.To do that. It required back-end architectural changes for cost, not for scalability so much, but for a lot around cost and scheduling because our thing was event-driven, we're able to run that and run our other platform fully in parallel, all shared data structures, shared messaging structures. I can't even imagine how hard that would have not been to do in a traditional data center. So, we have a lot of freedom, still have those cost constraints because that's [laugh] your thing, but the experimentation, the ability to integrate things, it's just oh, my God, it's just exciting.Corey: And let's be clear, I, having spent a lot of time as a rat myself in these data centers, I don't regret handing a lot of that responsibility off, just because, let's not kid ourselves, they are better at replacing failed or failing hardware than I will ever be. That's part of the benefit you get from the law of large numbers.Rich: Yeah. I don't want to do all of that stuff, but we're hovering around something that is kind of—all right, so former Gartner analyst means I have a massive ego, and because of that, I like to come up with my own terms for things, so roll with me here. And it's something I'm calling the ‘Grand Unified Theory of Cloud Governance' because you cannot possibly get more egotistical than referring to something as your solution to the biggest problem in all of physics. The idea is, is that cloud, as we have just been discussing, it drops friction and it decentralizes because you don't have to go ask somebody for the network, you don't have to ask somebody for the server. So, all of a sudden, you can build a full application stack without having to call somebody for help. We've just never had that in IT before.And all of our governance structures—and this includes your own costs, as well as security—are built around scarcity. Scarcity of resources, natural choke points that evolved from the data center. Not because it was bad. It wasn't bad. We built these things because that's what we needed for that environment at the data center.Now, we've got cloud and it's this whole new alien technology and it decentralizes. That said, particularly for us on security, you can build your whole application stack, of course, we have completely unified the management interfaces in one place and then we stuck them on the internet, protected with nothing more than a username and password. And if you can put those three things together in your head, you can realize why these are such dramatic changes and so challenging for enterprises, why my kids get to go to Disney a fair bit because we're in demand as security professionals.Corey: What does FireMon do exactly? That's something that I'm not entirely up to speed on, just because please don't take this the wrong way, but I was at RSA this year, and it feels like all the companies sort of blend together as you walk between the different booths. Like, “This is what you should be terrified of today.” And it always turns into a weird sales pitch. Not that that's what you do, but it at some point just blinds me and overloads me as far as dealing with any of the cloud security space.Rich: Oh, I've been going to RSA for 20 years. One of our SEs, I was briefly at our booth—I'm usually in outside meetings—and he goes, “Do you see any fun and interesting?” I go—I just looked at him like I was depressed and I'm like, “I've been to RSA for 20 years. I will never see anything interesting here again. Those days are over.” There's just too much noise and cacophony on that show floor.What do we do? So—Corey: It makes re:Invent's Expo Hall look small.Rich: Yeah. I mean, it's, it's the show over at RSA. And it wasn't always. I mean, it was—it's always been big as long as I've been there, but yeah, it's huge, everyone is there, and they're all saying exactly the same thing. This year, I think the only reason it wasn't all about AI is because they couldn't get the printers to reprint the banners fast enough. Not that anybody has any products that would do anything there. So—you look like you want to say something there.Corey: No, no. I like the approach quite a bit. It's the, everything was about AI this year. It was a hard pivot from trying to sell me a firewall, which it seems like everyone was doing in the previous year. It's kind of wild. I keep saying that there's about a dozen companies that exhibit at RSA. A guess, there are hundreds and hundreds of booths, but it all distills down to the same 12 things. They have different logos and different marketing stories, but it does seem like a lot of stuff is very much just like the booth next to it on both sides.Rich: Yeah. I mean, that's—it's just the nature. And part of—there's a lot of reasons for this. We used to, when I was—so prior to doing the startup thing and then ending up at FireMon, I did Securosis, which was an analyst firm, and we used to do the Securosis guide to RSA every year where we would try and pick the big themes. And the reality is, there's a reason for that.I wrote something once the vendors lied to you because you want them to. It's the most dysfunctional relationship because as customers, you're always asking, “Well, what are you doing for [unintelligible 00:22:16]? What are you doing for zero trust? What are you doing for AI?” When those same customers are still just working on fundamental patch management and firewall management. But it doesn't stop them from asking the questions and the vendors have to have answers because that's just the nature of that part of the world.Corey: I will ask you, over are past 12 years—I have my own thoughts on this, but I want to hear your take on it—what's changed in the world of cloud security?Rich: Everything. I mean, I was one of the first to be doing this.Corey: Oh, is that all?Rich: Yeah. So, there's more people. When I first started, very few people doing it, nobody knew much about it outside AWS, we all knew each other. Now, we've got a community that's developed and there's people that know what they're doing. There's still a shortage of skills, absolutely still a shortage of skills, but we're getting a handle on that, you know? We're getting a bit of a pipeline.And I'd say that's still probably the biggest challenge faced. But what's improved? Well, it's a give-and-take. On one hand, we now have strategies, we have tools that are more helpful, unfortunately—I'll tell you the biggest mistake I made and it ties to the FireMon stuff in my career, in a minute; relates directly to this question, but we're kind of getting there on some of the tool pieces.On the other hand, that complexity is increasing faster. And that's what's made it hard. So, as much as we're getting more skilled people, better at tooling, for example, we kind of know—and we didn't have CloudTrail when I started. We didn't have the fundamental things you need to actually implement security at the start of cloud. Most of those are there; they may not be working the way we wish they always worked, but we've got the pieces to assemble it, depending on which platform you're on. That's probably the biggest change. Now, we need to get into the maturity phase of cloud, and that's going to be much more difficult and time-consuming to kind of get over that hump.Corey: It's easy to wind up saying, “Oh, I saw the future so clearly back then,” but I have to ask, going back 12 years, the path the world would take was far from certain. Did you have doubts?Rich: Like, I had presented with Chris Hoff. We—we're still friends—presented stuff together, and he got a job that was kind of clouding ancillary. And I remember calling him up once and going, “Chris, I don't know what to do.” I was running my little analyst firm—little. We were doing very, very well—I could not get paid to do any work around cloud.People wanted me to write shitty papers on DLP and take customer inquiries on DLP because I had covered that at the Gartner days, and data encryption and those pieces. That was hard. And fortunately, a few things started trickling in. And then it was a flood. It completely changed our business and led to me, you know, eventually going down into the vendor path. But that was a tough day when I hit that point. So, absolutely I knew it was the future. I didn't know if I was going to be able to make a living at it.Corey: It would seem that you did.Rich: Yeah. Worked out pretty well [laugh].Corey: You seem sprightly to me. Good work. You're not on death's door.Rich: No. You know, in fact, the analyst side of it exploded over the years because it turns out, there weren't people who had this experience. So, I could write code to the APIs, but they'll still talk with CEOs and boards of directors around these cloud security issues and frame them in ways that made sense to them. So, that was wonderful. We partnered up with the Cloud Security Alliance, I actually built a bunch of the CSA training, I wrote the current version of the CSA guidance, we're writing the next version of that, did a lot of research with them. They've been a wonderful partner.So, all that went well. Then I got diverted down onto the vendor path. I had this research idea and then it came out, we ended up founding that as a startup and then it got, as I mentioned, acquired by FireMon, which is interesting because FireMon, you asked what we did, it's firewall policy management is the core of the company. Yet the investors realize the company was not going in the right direction necessarily, to deal with the future of cloud. They went to their former CEO and said, “Hey, can you come back”—the founder of the company—“And take this over and start moving us in the right direction?”Well, he happened to be my co-founder at the startup. And so, we kind of came in and took over there. And so, now it's a very interesting position because we have this one cloud-native thing we built for all these years. We made one mistake with that, which I'll talk about which ties back to your predicting the future piece if you want to go into it, but then we have the network firewall piece now extending into hybrid, and we have an asset management moving into the attack surface management space as well. And both of those products have been around for, like, 15-plus years.Corey: No, I'm curious to your thoughts on it because it's been one of those weird areas where there's been so much change and so much evolution, but you also look at today's “OWASP Top 10” list of vulnerabilities, and yeah, they updated a year or so ago, but it still looks basically like things that—from 2008—would have made sense to me when I'm looking at this. Well, insomuch as they do now. I didn't know then, nor do I now what a cross-site scripting attack might be, but other than that, I find that there's, “Oh, you misconfigured something and it winds up causing a problem.” Well, no kidding. Imagine that.Rich: Yeah. Look, the fundamentals don't change, but it's still really easy to screw up.Corey: Oh, having done so a lot, I believe you.Rich: There's a couple of principles, and I'll break it into two sides. One is, a lot of security sounds simple. There's nothing simple at scale. Nothing simple scales. The moment you get up to even 200 employees, everything just becomes ridiculously harder. That's the nature of reality. Simplicity doesn't scale.The other part is even though it's always the same, it's still easy to think you're going to be different this time and you're not going to screw it up, and then you do. For example, so cloud, we were talking about the maturity. I assumed CSPM just wasn't going to be a thing. For real. The Cloud Security Posture Management. Because why would the cloud providers not just make that problem go away and then all the vulnerability assessment vendors and everybody else? It seemed like it was an uninteresting problem.And yet, we were building a cloud security automation thing and we missed the boat because we had everything we needed to be one of the very first CSPM vendors on the market and we're like, “No, no. That problem is going to go away. We'll go there.” And it ties back to what you said, which is it's the same stuff and we just outsmarted ourselves. We thought that people would go further faster. And they don't and they aren't.And that's kind of where we are today. We are dramatically maturing. At the same time, the complexity is increasing dramatically. It's just a huge challenge for skills and staffing to adjust governance programs. Like I think we've got another 10 to 20 years to go on this cloud security thing before we even get close. And then maybe we'll get down to the being bored by the problems. But probably not because AI will ruin us.Corey: I'd like to imagine, on some level, that AI could be that good. I mean, don't get me wrong. It has value and it is transformative for a bunch of things, but I also think a lot of the fear-mongering is more than a little overblown.Rich: No, I agree with you. I'm trying to keep a very close eye on it because—I can't remember if you and I talked about this when we met face-to-face, or… it was somebody at that event—AI is just not just AI. There's different. There's the LLMs, there's the different kinds of technologies that are involved. I mean, we use AI all over the place already.I mean my phone's got it built in to take better pictures. It's a matter of figuring out what the use cases and the, honestly, some of the regulatory structure around it in terms of copyright and everything else. I'm not worried about Clippy turning into Skynet, even though I might make jokes about that on Mastodon, maybe someday there will be some challenges, but no, it's just going to be another tech that we're going to figure out over time. It is disruptive, so we can't ignore that part of it.Corey: I really want to thank you for taking the time to speak with me. If people want to learn more, where's the best place to find you that isn't one of the Disney parks?Rich: That really is kind of the best place to find—no. So, these days, I do technically still have a Twitter presence at @rmogull. I'm not on there much, but I will get DMs if people send those over. I'm more on Mastodon. It's at @rmogull defcon.social. I write over at FireMon these days, as well as occasionally still over Securosis, on those blogs. And I'm in the [Cloud Security Slack community 00:30:49] that is now under the banner for CloudSec. That's probably the best place if you want to hit me up and get quick answers on anything.Corey: And I will, of course, include links to all of that in the show notes. Thank you so much for taking the time to speak with me today. I really appreciate it.Rich: Thanks, Corey. I was so happy to be here.Corey: Rich Mogull, SVP of Cloud Security at FireMon. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment talking about how at Dell these days, it does not take six weeks to ship a server. And then I will get back to you in six to eight weeks.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Eswar Bala, Director of Amazon EKS at AWS, joins Corey on Screaming in the Cloud to discuss how and why AWS built a Kubernetes solution, and what customers are looking for out of Amazon EKS. Eswar reveals the concerns he sees from customers about the cost of Kubernetes, as well as the reasons customers adopt EKS over ECS. Eswar gives his reasoning on why he feels Kubernetes is here to stay and not just hype, as well as how AWS is working to reduce the complexity of Kubernetes. Corey and Eswar also explore the competitive landscape of Amazon EKS, and the new product offering from Amazon called Karpenter.About EswarEswar Bala is a Director of Engineering at Amazon and is responsible for Engineering, Operations, and Product strategy for Amazon Elastic Kubernetes Service (EKS). Eswar leads the Amazon EKS and EKS Anywhere teams that build, operate, and contribute to the services customers and partners use to deploy and operate Kubernetes and Kubernetes applications securely and at scale. With a 20+ year career in software , spanning multimedia, networking and container domains, he has built greenfield teams and launched new products multiple times.Links Referenced: Amazon EKS: https://aws.amazon.com/eks/ kubernetesthemuchharderway.com: https://kubernetesthemuchharderway.com kubernetestheeasyway.com: https://kubernetestheeasyway.com EKS documentation: https://docs.aws.amazon.com/eks/ EKS newsletter: https://eks.news/ EKS GitHub: https://github.com/aws/eks-distro TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It's easy to **BEEP** up on AWS. Especially when you're managing your cloud environment on your own!Mission Cloud un **BEEP**s your apps and servers. Whatever you need in AWS, we can do it. Head to missioncloud.com for the AWS expertise you need. Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. Today's promoted guest episode is brought to us by our friends at Amazon. Now, Amazon is many things: they sell underpants, they sell books, they sell books about underpants, and underpants featuring pictures of books, but they also have a minor cloud computing problem. In fact, some people would call them a cloud computing company with a gift shop that's attached. Now, the problem with wanting to work at a cloud company is that their interviews are super challenging to pass.If you want to work there, but can't pass the technical interview for a long time, the way to solve that has been, “Ah, we're going to run Kubernetes so we get to LARP as if we worked at a cloud company but don't.” Eswar Bala is the Director of Engineering for Amazon EKS and is going to basically suffer my slings and arrows about one of the most complicated, and I would say overwrought, best practices that we're seeing industry-wide. Eswar, thank you for agreeing to subject yourself to this nonsense.Eswar: Hey, Corey, thanks for having me here.Corey: [laugh]. So, I'm a little bit unfair to Kubernetes because I wanted to make fun of it and ignore it. But then I started seeing it in every company that I deal with in one form or another. So yes, I can still sit here and shake my fist at the tide, but it's turned into, “Old Man Yells at Cloud,” which I'm thrilled to embrace, but everyone's using it. So, EKS has recently crossed, I believe, the five-year mark since it was initially launched. What is EKS other than Amazon's own flavor of Kubernetes?Eswar: You know, the best way I can define EKS is, EKS is just Kubernetes. Not Amazon's version of Kubernetes. It's just Kubernetes that we get from the community and offer it to customers to make it easier for them to consume. So, EKS. I've been with EKS from the very beginning when we thought about offering a managed Kubernetes service in 2017.And at that point, the goal was to bring Kubernetes to enterprise customers. So, we have many customers telling us that they want us to make their life easier by offering a managed version of Kubernetes that they've actually beginning to [erupt 00:02:42] at that time period, right? So, my goal was to figure it out, what does that service look like and which customer base should be targeting service towards.Corey: Kelsey Hightower has a fantastic learning tool out there in a GitHub repo called, “Kubernetes the Hard Way,” where he talks you through building the entire thing, start to finish. I wound up forking it and doing that on top of AWS, and you can find that at kubernetesthemuchharderway.com. And that was fun.And I went through the process and my response at the end was, “Why on earth would anyone ever do this more than once?” And we got that sorted out, but now it's—customers aren't really running these things from scratch. It's like the Linux from Scratch project. Great learning tool; probably don't run this in production in the same way that you might otherwise because there are better ways to solve for the problems that you will have to solve yourself when you're building these things from scratch. So, as I look across the ecosystem, it feels like EKS stands in the place of the heavy, undifferentiated lifting of running the Kubernetes control plane so customers functionally don't have to. Is that an effective summation of this?Eswar: That is precisely right. And I'm glad you mentioned, “Kubernetes the Hard Way,” I'm a big fan of that when it came out. And if anyone who did that tutorial, and also your tutorial, “Kubernetes the Harder Way,” would walk away thinking, “Why would I pick this technology when it's super complicated to setup?” But then you see that customers love Kubernetes and you see that reflected in the adoption, even in 2016, 2017 timeframes.And the reason is, it made life easier for application developers in terms of offering web services that they wanted to offer to their customer base. And because of all the features that Kubernetes brought on, application lifecycle management, service discoveries, and then it evolved to support various application architectures, right, in terms of stateless services, stateful applications, and even daemon sets, right, like for running your logging and metrics agents. And these are powerful features, at the end of the day, and that's what drove Kubernetes. And because it's super hard to get going to begin with and then to operate, the day-two operator experience is super complicated.Corey: And the day one experience is super hard and the day two experience of, “Okay, now I'm running it and something isn't working the way it used to. Where do I start,” has been just tremendously overwrought. And frankly, more than a little intimidating.Eswar: Exactly. Right? And that exactly was our opportunity when we started in 2017. And when we started, there was question on, okay, should we really build a service when you have an existing service like ECS in place? And by the way, like, I did work in ECS before I started working in EKS from the beginning.So, the answer then was, it was about giving what customers want. And their space for many container orchestration systems, right, ECS was the AWS service at that point in time. And our thinking was, how do we give customers what they wanted? They wanted a Kubernetes solution. Let's go build that. But we built it in a way that we remove the undifferentiated heavy lifting of managing Kubernetes.Corey: One of the weird things that I find is that everyone's using Kubernetes, but I don't see it in the way that I contextualize the AWS universe, which of course, is on the bill. That's right. If you don't charge for something in AWS Lambda, and preferably a fair bit, I don't tend to know it exists. Like, “What's an IAM and what might that possibly do?” Always have reassuring thing to hear from someone who's often called an expert in this space. But you know, if it doesn't cost money, why do I pay attention to it?The control plane is what EKS charges for, unless you're running a bunch of Fargate-managed pods and containers to wind up handling those things. So, it mostly just shows up as an addenda to the actual big, meaty portions of the belt. It just looks like a bunch of EC2 instances with some really weird behavior patterns, particularly with regard to auto-scaling and crosstalk between all of those various nodes. So, it's a little bit of a murder mystery, figuring out, “So, what's going on in this environment? Do you folks use containers at all?” And the entire Kubernetes shop is looking at me like, “Are you simple?”No, it's just I tend to disregard the lies that customers say, mostly to themselves because everyone has this idea of what's going on in their environment, but the bill speaks. It's always been a little bit of an investigation to get to the bottom of anything that involves Kubernetes at significant points of scale.Eswar: Yeah, you're right. Like if you look at EKS, right, like, we started with managing the control plane to begin with. And managing the control plane is a drop in the bucket when you actually look at the costs in terms of operating a Kubernetes cluster or running a Kubernetes cluster. When you look at how our customers use and where they spend most of their cost, it's about where their applications run; it's actually the Kubernetes data plane and the amount of compute and memory that the applications end of using end up driving 90% of the cost. And beyond that is the storage, beyond that as a networking costs, right, and then after that is the actual control plane costs. So, the problem right now is figuring out, how do we optimize our costs for the application to run on?Corey: On some level, it requires a little bit of understanding of what's going on under the hood. There have been a number of cost optimization efforts that have been made in the Kubernetes space, but they tend to focus around stuff that I find relatively, well, I call it banal because it basically is. You're looking at the idea of, okay, what size instances should you be running, and how well can you fill them and make sure that all the resources per node wind up being taken advantage of? But that's also something that, I guess from my perspective, isn't really the interesting architectural point of view. Whether or not you're running a bunch of small instances or a few big ones or some combination of the two, that doesn't really move the needle on any architectural shift, whereas ingesting a petabyte a month of data and passing 50 petabytes back and forth between availability zones, that's where it starts to get really interesting as far as tracking that stuff down.But what I don't see is a whole lot of energy or effort being put into that. And I mean, industry-wide, to be clear. I'm not attempting to call out Amazon specifically on this. That's [laugh] not the direction I'm taking this in. For once. I know, I'm still me. But it seems to be just an industry-wide issue, where zone affinity for Kubernetes has been a very low priority item, even on project roadmaps on the Kubernetes project.Eswar: Yeah, the Kubernetes does provide ability for customers to restrict their workloads within as particular [unintelligible 00:09:20], right? Like, there is constraints that you can place on your pod specs that end up driving applications towards a particular AZ if they want, right? You're right, it's still left to the customers to configure. Just because there's a configuration available doesn't mean the customers use it. If it's not defaulted, most of the time, it's not picked up.That's where it's important for service providers—like EKS—to offer ability to not only provide the visibility by means of reporting that it's available using tools like [Cue Cards 00:09:50] and Amazon Billing Explorer but also provide insights and recommendations on what customers can do. I agree that there's a gap today. For example in EKS, in terms of that. Like, we're slowly closing that gap and it's something that we're actively exploring. How do we provide insights across all the resources customers end up using from within a cluster? That includes not just compute and memory, but also storage and networking, right? And that's where we are actually moving towards at this point.Corey: That's part of the weird problem I've found is that, on some level, you get to play almost data center archaeologists when you start exploring what's going on in these environments. I found one of the only reliable ways to get answers to some of this stuff has been oral tradition of, “Okay, this Kubernetes cluster just starts hurling massive data quantities at 3 a.m. every day. What's causing that?” And it leads to, “Oh, no no, have you talked to the data science team,” like, “Oh, you have a data science team. A common AWS billing mistake.” And exploring down that particular path sometimes pays dividends. But there's no holistic way to solve that globally. Today. I'm optimistic about tomorrow, though.Eswar: Correct. And that's where we are spending our efforts right now. For example, we recently launched our partnership with Cue Cards, and Cue Cards is now available as an add-on from the Marketplace that you can easily install and provision on Kubernetes EKS clusters, for example. And that is a start. And Cue Cards is amazing in terms of features, in terms of insight it offers, right, it looking into computer, the memory, and the optimizations and insights it provides you.And we are also working with the AWS Cost and Usage Reporting team to provide a native AWS solution for the cost reporting and the insights aspect as well in EKS. And it's something that we are going to be working really closely to solve the networking gaps in the near future.Corey: What are you seeing as far as customer concerns go, with regard to cost and Kubernetes? I see some things, but let's be very clear here, I have a certain subset of the market that I spend an inordinate amount of time speaking to and I always worry that what I'm seeing is not holistically what's going on in the broader market. What are you seeing customers concerned about?Eswar: Well, let's start from the fundamentals here, right? Customers really want to get to market faster, whatever services and applications that they want to offer. And they want to have it cheaper to operate. And if they're adopting EKS, they want it cheaper to operate in Kubernetes in the cloud. They also want a high performance, they also want scalability, and they want security and isolation.There's so many parameters that they have to deal with before they put their service on the market and continue to operate. And there's a fundamental tension here, right? Like they want cost efficiency, but they also want to be available in the market quicker and they want performance and availability. Developers have uptime, SLOs, and SLAs is to consider and they want the maximum possible resources that they want. And on the other side, you've got financial leaders and the business leaders who want to look at the spending and worry about, like, okay, are we allocating our capital wisely? And are we allocating where it makes sense? And are we doing it in a manner that there's very little wastage and aligned with our customer use, for example? And this is where the actual problems arise from [unintelligible 00:13:00].Corey: I want to be very clear that for a long time, one of the most expensive parts about running Kubernetes has not been the infrastructure itself. It's been the people to run this responsibly, where it's the day two, day three experience where for an awful lot of companies like, oh, we're moving to Kubernetes because I don't know we read it in an in-flight magazine or something and all the cool kids are doing it, which honestly during the pandemic is why suddenly everyone started making better IT choices because they're execs were not being exposed to airport ads. I digress. The point, though, is that as customers are figuring this stuff out and playing around with it, it's not sustainable that every company that wants to run Kubernetes can afford a crack SRE team that is individually incredibly expensive and collectively staggeringly so. That it seems to be the real cost is the complexity tied to it.And EKS has been great in that it abstracts an awful lot of the control plane complexity away. But I still can't shake the feeling that running Kubernetes is mind-bogglingly complicated. Please argue with me and tell me I'm wrong.Eswar: No, you're right. It's still complicated. And it's a journey towards reducing the complexity. When we launched EKS, we launched only with managing the control plane to begin with. And that's where we started, but customers had the complexity of managing the worker nodes.And then we evolved to manage the Kubernetes worker nodes in terms two products: we've got Managed Node Groups and Fargate. And then customers moved on to installing more agents in their clusters before they actually installed their business applications, things like Cluster Autoscaler, things like Metric Server, critical components that they have come to rely on, but doesn't drive their business logic directly. They are supporting aspects of driving core business logic.And that's how we evolved into managing the add-ons to make life easier for our customers. And it's a journey where we continue to reduce the complexity of making it easier for customers to adopt Kubernetes. And once you cross that chasm—and we are still trying to cross it—once you cross it, you have the problem of, okay so, adopting Kubernetes is easy. Now, we have to operate it, right, which means that we need to provide better reporting tools, not just for costs, but also for operations. Like, how easy it is for customers to get to the application level metrics and how easy it is for customers to troubleshoot issues, how easy for customers to actually upgrade to newer versions of Kubernetes. All of these challenges come out beyond day one, right? And those are initiatives that we have in flight to make it easier for customers [unintelligible 00:15:39].Corey: So, one of the things I see when I start going deep into the Kubernetes ecosystem is, well, Kubernetes will go ahead and run the containers for me, but now I need to know what's going on in various areas around it. One of the big booms in the observability space, in many cases, has come from the fact that you now need to diagnose something in a container you can't log into and incidentally stopped existing 20 minutes for you got the alert about the issue, so you'd better hope your telemetry is up to snuff. Now, yes, that does act as a bit of a complexity burden, but on the other side of it, we don't have to worry about things like failed hard drives taking systems down anymore. That has successfully been abstracted away by Kubernetes, or you know, your cloud provider, but that's neither here nor there these days. What are you seeing as far as, effectively, the sidecar pattern, for example of, “Oh, you have too many containers and need to manage them? Have you considered running more containers?” Sounds like something a container salesman might say.Eswar: So, running containers demands that you have really solid observability tooling, things that you're able to troubleshoot—successfully—debug without the need to log into the containers itself. In fact, that's an anti-pattern, right? You really don't want a container to have the ability to SSH into a particular container, for example. And to be successful at it demands that you publish your metrics and you publish your logs. All of these are things that a developer needs to worry about today in order to adopt containers, for example.And it's on the service providers to actually make it easier for the developers not to worry about these. And all of these are available automatically when you adopt a Kubernetes service. For example, in EKS, we are working with our managed Prometheus service teams inside Amazon, right—and also CloudWatch teams—to easily enable metrics and logging for customers without having to do a lot of heavy lifting.Corey: Let's talk a little bit about the competitive landscape here. One of my biggest competitors in optimizing AWS bills is Microsoft Excel, specifically, people are going to go ahead and run it themselves because, “Eh, hiring someone who's really good at this, that sounds expensive. We can screw it up for half the cost.” Which is great. It seems to me that one of your biggest competitors is people running their own control plane, on some level.I don't tend to accept the narrative that, “Oh, EKS is expensive that winds up being what 35 bucks or 70 bucks or whatever it is per control plane per cluster on a monthly basis.” Okay, yes, that's expensive if you're trying to stay completely within a free tier perhaps, but if you're running anything that's even slightly revenue-generating or a for-profit company, you will spend far more than that just on people's time. I have no problems—for once—with the EKS pricing model, start to finish. Good work on that. You've successfully nailed it. But are you seeing significant pushback from the industry of, “Nope, we're going to run our own Kubernetes management system instead because we enjoy pain, corporately speaking.”Eswar: Actually, we are in a good spot there, right? Like, at this point, customers who choose to run Kubernetes on AWS by themselves and not adopt EKS just fall into one main category, so—or two main categories: number one, they have existing technical stack built on running Kubernetes on themselves and they'd rather maintain that and not moving to EKS. Or they demand certain custom configurations of the Kubernetes control plane that EKS doesn't support. And those are the only two reasons why we see customers not moving into EKS and prefer to run their own Kubernetes on AWS clusters.[midroll 00:19:46]Corey: It really does seem, on some level, like there's going to be a… I don't want to say reckoning because that makes it sound vaguely ominous and that's not the direction that I intend for things to go in, but there has to be some form of collapsing of the complexity that is inherent to all of this because the entire industry has always done that. An analogy that I fall back on because I've seen this enough times to have the scars to show for it is that in the '90s, running a web server took about a week of spare time and an in-depth knowledge of GCC compiler flags. And then it evolved to ah, I could just unzip a tarball of precompiled stuff, and then RPM or Deb became a thing. And then Yum, or something else, or I guess apt over in the Debian land to wind up wrapping around that. And then you had things like Puppet where it was it was ensure installed. And now it's Docker Run.And today, it's a checkbox in the S3 console that proceeds to yell at you because you're making a website public. But that's neither here nor there. Things don't get harder with time. But I've been surprised by how I haven't yet seen that sort of geometric complexity collapsing of around Kubernetes to make it easier to work with. Is that coming or are we going to have to wait for the next cycle of things?Eswar: Let me think. I actually don't have a good answer to that, Corey.Corey: That's good, at least because if you did, I'd worried that I was just missing something obvious. That's kind of the entire reason I ask. Like, “Oh, good. I get to talk to smart people and see what they're picking up on that I'm absolutely missing.” I was hoping you had an answer, but I guess it's cold comfort that you don't have one off the top of your head. But man, is it confusing.Eswar: Yeah. So, there are some discussions in the community out there, right? Like, it's Kubernetes the right layer to do interact? And there are some tooling that's built on top of Kubernetes, for example, Knative that tries to provide a serverless layer on top of Kubernetes, for example. There are also attempts at abstracting Kubernetes completely and providing tooling that just completely removes any sort of Kubernetes API out of the picture and maybe a specific CI/CD-based solution that takes it from the source and deploys the service without even showing you that there's Kubernetes underneath, right?All of these are evolutions that are being tested out there in the community. Time will tell whether these end up sticking. But what's clear here is the gravity around Kubernetes. All sorts of tooling that gets built on top of Kubernetes, all the operators, all sorts of open-source initiatives that are built to run on Kubernetes. For example, Spark, for example, Cassandra, so many of these big, large-scale, open-source solutions are now built to run really well on Kubernetes. And that is the gravity that's pushing Kubernetes at this point.Corey: I'm curious to get your take on one other, I would consider interestingly competitive spaces. Now, because I have a domain problem, if you go to kubernetestheeasyway.com, you'll wind up on the ECS marketing page. That's right, the worst competition in the world: the people who work down the hall from you.If someone's considering using ECS, Elastic Container Service versus EKS, Elastic Kubernetes Service, what is the deciding factor when a customer's making that determination? And to be clear, I'm not convinced there's a right or wrong answer. But I am curious to get your take, given that you have a vested interest, but also presumably don't want to talk complete smack about your colleagues. But feel free to surprise me.Eswar: Hey, I love ECS, by the way. Like I said, I started my life in the AWS in ECS. So look, ECS is a hugely successful container orchestration service. I know we talk a lot about Kubernetes, I know there's a lot of discussions around Kubernetes, but I wouldn't make it a point that, like, ECS is a hugely successful service. Now, what determines how customers go to?If customers are… if the customers tech stack is entirely on AWS, right, they use a lot of AWS services and they want an easy way to get started in the container world that has really tight integration with other AWS services without them having to configure a lot, ECS is the way, right? And customers have actually seen terrific success adopting ECS for that particular use case. Whereas EKS customers, they start with, “Okay, I want an open-source solution. I really love Kubernetes. I lo—or, I have a tooling that I really like in the open-source land that really works well with Kubernetes. I'm going to go that way.” And those kind of customers end up picking EKS.Corey: I feel like, on some level, Kubernetes has become the most the default API across a wide variety of environments. AWS obviously, but on-prem other providers. It seems like even the traditional VPS companies out there that offer just rent-a-server in the cloud somewhere are all also offering, “Oh, and we have a Kubernetes service as well.” I wound up backing a Kickstarter project that runs a Kubernetes cluster with a shared backplane across a variety of Raspberries Pi, for example. And it seems to be almost everywhere you look.Do you think that there's some validity to that approach of effectively whatever it is that we're going to wind up running in the future, it's going to be done on top of Kubernetes or do you think that that's mostly hype-driven these days?Eswar: It's definitely not hype. Like we see the proof in the kind of adoption we see. It's becoming the de facto container orchestration API. And with all the tooling, open-source tooling that's continuing to build on top of Kubernetes, CNCF tooling ecosystem that's actually spawned to actually support Kubernetes at option, all of this is solid proof that Kubernetes is here to stay and is a really strong, powerful API for customers to adopt.Corey: So, four years ago, I had a prediction on Twitter, and I said, “In five years, nobody will care about Kubernetes.” And it was in February, I believe, and every year, I wind up updating an incrementing a link to it, like, “Four years to go,” “Three years to go,” and I believe it expires next year. And I have to say, I didn't really expect when I made that prediction for it to outlive Twitter, but yet, here we are, which is neither here nor there. But I'm curious to get your take on this. But before I wind up just letting you savage the naive interpretation of that, my impression has been that it will not be that Kubernetes has gone away. That is ridiculous. It is clearly in enough places that even if they decided to rip it out now, it would take them ten years, but rather than it's going to slip below the surface level of awareness.Once upon a time, there was a whole bunch of energy and drama and debate around the Linux virtual memory management subsystem. And today, there's, like, a dozen people on the planet who really have to care about that, but for the rest of us, it doesn't matter anymore. We are so far past having to care about that having any meaningful impact in our day-to-day work that it's just, it's the part of the iceberg that's below the waterline. I think that's where Kubernetes is heading. Do you agree or disagree? And what do you think about the timeline?Eswar: I agree with you; that's a perfect analogy. It's going to go the way of Linux, right? It's here to stay; it just going to get abstracted out if any of the abstraction efforts are going to stick around. And that's where we're testing the waters there. There are many, many open-source initiatives there trying to abstract Kubernetes. All of these are yet to gain ground, but there's some reasonable efforts being made.And if they are successful, they just end up being a layer on top of Kubernetes. Many of the customers, many of the developers, don't have to worry about Kubernetes at that point, but a certain subset of us in the tech world will need to do a deal with Kubernetes, and most likely teams like mine that end up managing and operating their Kubernetes clusters.Corey: So, one last question I have for you is that if there's one thing that AWS loves, it's misspelling things. And you have an open-source offering called Karpenter spelled with a K that is an extending of that tradition. What does Karpenter do and why would someone use it?Eswar: Thank you for that. Karpenter is one of my favorite launches in the last one year.Corey: Presumably because you're terrible at the spelling bee back when you were a kid. But please tell me more.Eswar: [laugh]. So Karpenter, is an open-source flexible and high performance cluster auto-scaling solution. So basically, when your cluster needs more capacity to support your workloads, Karpenter automatically scales the capacity as needed. For people that know the Kubernetes space well, there's an existing component called Cluster Autoscaler that fills this space today. And it's our take on okay, so what if we could reimagine the capacity management solution available in Kubernetes? And can we do something better? Especially for cases where we expect terrific performance at scale to enable cost efficiency and optimization use cases for our customers, and most importantly, provide a way for customers not to pre-plan a lot of capacity to begin with.Corey: This is something we see a lot, in the sense of very bursty workloads where, okay, you're going to steady state load. Cool. Buy a bunch of savings plans, get things set up the way you want them, and call it a day. But when it's bursty, there are challenges with it. Folks love using Spot, but in the event of a sudden capacity shortfall, the question is, is can we spin up capacity to backfill it within those two minutes that we have a warning on that on? And if the answer is no, then it becomes a bit of a non-starter.Customers have had to build an awful lot of those things around EC2 instances that handle a lot of that logic for them in ways that are tuned specifically for their use cases. I'm encouraged to see there's a Kubernetes story around this that starts to remove some of that challenge from the customer side.Eswar: Yeah. So, the burstiness is where complexity comes [here 00:29:42], right? Like many customers for steady state, they know what their capacity requirements are, they set up the capacity, they can also reason out what is the effective capacity needed for good utilization for economical reasons and they can actually pre plan that and set it up. But once burstiness comes in, which inevitably does it at [unintelligible 00:30:05] applications, customers worry about, “Okay, am I going to get the capacity that I need in time that I need to be able to service my customers? And am I confident at it?”If I'm not confident, I'm going to actually allocate capacity beforehand, assuming that I'm going to actually get the burst that I needed. Which means, you're paying for resources that you're not using at the moment. And the burstiness might happen and then you're on the hook to actually reduce the capacity for it once the peak subsides at the end of the [day 00:30:36]. And this is a challenging situation. And this is one of the use cases that we targeted Karpenter towards.Corey: I find that the idea that you're open-sourcing this is fascinating because of two reasons. One, it does show a willingness to engage with the community that… again, it's difficult. When you're a big company, people love to wind up taking issue with almost anything that you do. But for another, it also puts it out in the open, on some level, where, especially when you're talking about cost optimization and decisions that affect cost, it's all out in public. So, people can look at this and think, “Wait a minute, it's not—what is this line of code that means if it's toward the end of the month, crank it up because we might need to hit our numbers.” Like, there's nothing like that in there. At least I'm assuming. I'm trusting that other people have read this code because honestly, that seems like a job for people who are better at that than I am. But that does tend to breed a certain element of trust.Eswar: Right. It's one of the first things that we thought about when we said okay, so we have some ideas here to actually improve the capacity management solution for Kubernetes. Okay, should we do it out in the open? And the answer was a resounding yes, right? I think there's a good story here that actually enables not just AWS to offer these ideas out there, right, and we want to bring it to all sorts of Kubernetes customers.And one of the first things we did is to architecturally figure out all the core business logic of Karpenter, which is, okay, how to schedule better, how quickly to scale, what is the best instance types to pick for this workload. All of that business logic was abstracted out from the actual cloud provider implementation. And the cloud provider implementation is super simple. It's just creating instances, deleting instances, and describing instances. And it's something that we bake from the get-go so it's easier for other cloud providers to come in and to add their support to it. And we as a community actually can take these ideas forward in a much faster way than just AWS doing it.Corey: I really want to thank you for taking the time to speak with me today about all these things. If people want to learn more, where's the best place for them to find you?Eswar: The best place to learn about EKS, right, as EKS evolves, is using our documentation, we have an EKS newsletter that you can go subscribe, and you can also find us on GitHub where we share our product roadmap. So, it's a great places to learn about how EKS is evolving and also sharing your feedback.Corey: Which is always great to hear, as opposed to, you know, in the AWS Console, where we live, waiting for you to stumble upon us, which, yeah. No it's good does have a lot of different places for people to engage with you. And we'll put links to that, of course, in the [show notes 00:33:17]. Thank you so much for being so generous with your time. I appreciate it.Eswar: Corey, really appreciate you having me.Corey: Eswar Bala, Director of Engineering for Amazon EKS. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice telling me why, when it comes to tracking Kubernetes costs, Microsoft Excel is in fact the superior experience.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

John Mille, Principal Cloud Engineer at Sainsbury's UK joins Corey on Screaming in the Cloud to discuss how retail companies are using cloud services. John describes the lessons he's learned since joining the Sainsbury's UK team, including why it's important to share knowledge across your team if you don't want to be on call 24/7,  as well as why he doesn't subscribe to the idea that every developer needs access to production. Corey and John also discuss an open-source project John created called ECS Compose-X.About JohnJohn is an AWS Community Builder (devtools), Open Source enthusiast, SysAdmin born in the cloud, and has worked with AWS since his very first job. He enjoys writing code and creating projects. John likes to focus on automation & architecture that delivers business value, and has been dabbling with data & the wonderful world of Kafka for the past 3 years.Links Referenced: AWS Open-Source Roundup newsletter blog post about ECS Compose-X: https://aws.amazon.com/blogs/opensource/automating-your-ecs-container-architecture-deployments-with-ecs-composex/ ECS Compose-X: https://docs.compose-x.io/ LinkedIn: https://www.linkedin.com/in/john-mille/ Twitter: https://twitter.com/JohnPre32286850 TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It's easy to **BEEP** up on AWS. Especially when you're managing your cloud environment on your own!Mission Cloud un **BEEP**s your apps and servers. Whatever you need in AWS, we can do it. Head to missioncloud.com for the AWS expertise you need. Corey: Do you wish your developers had less permanent access to AWS? Has the complexity of Amazon's reference architecture for temporary elevated access caused you to sob uncontrollably? With Sym, you can protect your cloud infrastructure with customizable, just-in-time access workflows that can be setup in minutes. By automating the access request lifecycle, Sym helps you reduce the scope of default access while keeping your developers moving quickly. Say goodbye to your cloud access woes with Sym. Go to symops.com/corey to learn more. That's S-Y-M-O-P-S.com/coreyCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today my guest is a long-time listener, first-time caller. John Mille is a Principal Cloud Engineer at Sainsbury's, which is UK-speak for ‘grocery store.' John, thank you for joining me.John: Hi, Corey. Thanks for having me.Corey: So, I have to begin with, I guess, the big question that I used to run into people in San Francisco with all the time. They would work at Walmart Labs and they would mention in conversation that they work at Walmart, and people who weren't aware that there was a labs out here figured they were a greeter at the grocery store. Do you ever wind up with people making that sort of fundamental assumption around the fact, oh, you work at Sainsbury's as a checker or whatnot?John: No. But it actually is one of the—if you look at one of the job descriptions from Sainsbury's, the first thing is, why would you join a retail company to do tech? And as it turns out, tech—I mean, I think retail companies, as any other companies in the world, rely on Cloud more and more and more. And I think that one of the things that is interesting today is, if you look at the landscape of retailers, I've heard many times people saying, “We don't want to go for AWS because we're giving money to the competition.” And actually, I think AWS does a fantastic job overall giving you all the tools to actually beat them as your competition. And as it turns out, we've had really, really great success running a lot of our workloads on AWS for many, many years now.Corey: On some level, if you can't come to terms with the idea of Amazon as competition, you shouldn't be using AWS, regardless of what industry you're in, because their entire company strategy is yes. It's very hard to start to even come up with industries that they don't have some form of presence within. On some level, that's a problem. In fact a lot of levels, that's something of a problem.Everyone tends to wind up viewing the world in a bunch of different ways. I like to divide companies into two groups. More or less it's, is the AWS bill one of the top three line items at the company? And if the answer's no, on some level, you know, that usually is an indicator that there's a sustainable business there that, you know, both our grandparents and our grandchildren will be able to recognize, in the fullness of time. You absolutely have a business that winds up falling into that category, whereas, “Oh yeah, I fix the AWS bill,” yeah, my parents would have no idea what I do and my kids don't have much of a better one. It feels like it's very point-in-time type of problem. At least I hope.Technology is not the core of what grocery stores tend to do, but I also don't get the sense that what you're doing is sitting there doing the back office corporate IT style of work, either. How do you use technology in the overall context of the business?John: Well, so we use it in a very wide variety of sense. So, you obviously have everything that has to do with online shopping, orders and all of those sort of things, which obviously, especially with the drive of Covid and being everybody from home, has been a huge driver to improve our ability to deliver to customers. But certainly, I think that Sainsbury's sees AWS as a key partner to be able to go and say we want to deliver more value. And so, there's been a number of transformation over the years to—and one of the reasons I was hired is actually to be part of one of those transformation, where we're going to take existing infrastructure servers that literally—I usually say to people, “Oh, are we doing an upgrade this month? Has somebody gotten their little brush to go and brush onto the hard drives to make sure that nothing is going to die?” And actually do that transformation and move over to the cloud in order to never have to really worry about whether or not they have to manage hardware and infrastructure.Corey: It's strange in that I never got very deep into containers until I was no longer hands-on hardware, managing things. I was more or less doing advisory work and then messing around with them. And you'd think given my proclivities historically, of being very unlucky when it comes to data, you would think that this would be great because, oh yeah, you blow away an ephemeral container? Well, that's kind of the point. We'll all laugh and it'll re-instantiate itself and life goes on.But no. Making fun of them was more or less how I tended to do approach them for the longest time until I started to see them a little bit… well I guess less as a culture, less as a religion, and more as an incredibly versatile packaging format, which is probably going to annoy the people I know who are the packaging [unintelligible 00:04:58] for Linux distributions. How do you tend to view them? And how did you start using them?John: Right. So, that's a great question. So historically, I was a student at, I think the school were one of the original creators of Docker were. And one of the things that you learn when you do development at the school is that, you know, containers [unintelligible 00:05:18] new invention. Docker, I think, came on the platform as the way to, you know, give everybody a great framework, a great API, to drive the deployment of containers in the world and bundle them and ship them around the world, on your laptop and somebody else's, and help a little bit with, you know, solving the problem of it works on my laptop, but not just on the laptop properly. Maybe.It's obviously gone viral over the years and I really enjoy containers; I quite like containers. What I find interesting is what people are going to do with. And I think that over the last few years, we've seen a number of technologies such as Kubernetes and others come into the scene and say—and trying to solve people's problem, but everybody seems to be doing, sort of, things on their own way. And historically, I started off using ECS, when it was terrible and you didn't have security groups per containers and all of this. But over the years, you know, you learn, and AWS has improved the service quite significantly with more and more features.And I think we are today in the place where there's this landscape, I think, where a lot of workloads are going to be extremely ephemeral and you can go [unintelligible 00:06:28], you know, wherever you want and you have a bit—if you have a platform or workflow that you need to have working in different places, maybe Kubernetes could be an easy way to have a different sort of sets of features that allows you to move around in maybe an easier way. But that also comes with a set of drawbacks. Again, I look at using EKS, for example, and I see okay, I have to manage IAM in our back now, whereas if I used something like ECS, for the whatever the [unintelligible 00:06:56] cloud vendor of choice, I don't have to deal with any of this. So, I think it's finding the fine balance between how you do orchestration of containers now and what works for you and is any sustainable over the time, more than about are you going to use containers? Because the chances are, somebody is using containers.Corey: My experiences and workflows and constraints are radically different than that of other folks because for a lot of the things I'm building, these are accounts that are I'm the only person that has access to them. It is me. So, the idea of fine-grained permissions for users from an ARBAC perspective doesn't really factor into it. Yes, yes, in theory, I should have a lot of the systems themselves with incidents roles being managed in safe and secure ways, but in many cases, the AWS account boundary is sufficient for that, depending on what it is we're talking about. But that changes when you start having a small team of people working with you and having to collaborate on these things.And we do a little bit of that with some of our consulting stuff that isn't just the shitpost stuff I build for fun. But there's multiple levels beyond that. You are clearly in a full-blown enterprise at this point where there are a bunch of different teams working on different things, all ideally going in the same direction. And it's easy to get stuck in the weeds of having to either go through central IT for these things, which gives rise to shadow IT every time you find a corporate credit card in the wild, or it winds up being everyone can do what they want, but then there's no consensus, there's no control, there's no architectural similarity. And I'm not sure which path is worse in some respects. How do you land on it?John: Right. So, what I've seen done in companies that works very well—and again, to the credit of my current company—is one of the things they've done really well is build a hub of people who are going to manage solely everything that has to do with accounts access, right? So, the control, IAM, Security Hub, all of those sorts of things, for you. There's things that are mandatory that you can't deal without, you have permissions boundary, that's it, you have to use those things, end of story. But beyond that point, once you have access to your accounts, you've been given all of the access that is necessary for you to deliver application and deploy them all the way up to production without asking permission for anybody else apart from your delivery managers, potentially.And I think from there, because there is the room to do all of this, one of the things that we've done within my business unit is that we've put in place a framework that enables developers—and when I say that it really is a question of allowing them to do everything they have to do, focus on the code, and I know it's a little catchy [unintelligible 00:09:33] a phrase that you hear these days, but the developers really are the customers that we have. And all that we do is to try to make sure that they have a framework in place that allows them to do what they need and deploy the applications in a secure fashion. And the only way to do that for us was to build the tools for them that allows them to do all of that. And I honestly haven't checked a single service IAM policies in a very are longtime because I know that by providing the tools to developers, they don't have this [will 00:10:05] to go and mess with the permissions because their application suddenly doesn't have the permissions. They just know that with the automation we've providing them, the application gets the access it needs and no more.Corey: On some level, it feels like there's a story around graduated development approach where in a dev environment you can do basically whatever you want with a big asterisk next to it. That's the same asterisk, by the way, next to the AWS free tier. But as you start elevating things into higher environments, you start to see gating around things like who has access to what, security reviews, et cetera, et cetera, and ideally, by the time you wind up getting into production, almost no one should have access and that access that people do have winds up being heavily gated. That is, of course, the vision that folks have. In practice, reality is what happens instead of what we plan on. The idea of it works in theory, but not in production is of course, why I call my staging environment ‘theory.' Does that tend to resonate as far as what you've seen in the wild?John: Yeah. Very much so. And when I joined the company, and we put together our [standard 00:11:11] pipelines for developers to be able to do everything, the rule that I would give to my team—so I manage a small team of cloud engineers—the one rule I would say is, “We have access to prod because we need to provision resources, but when we're going to build the pipelines for the developers, you have to build everything in such a way that the developers will only have read-only access to the production environment, and that is only to go and see their logs.” And at least try to foster this notion that developers do not need access to production, as much as possible because that avoids people going and do something they shouldn't be doing in those production environments.Now, as the pipeline progresses and applications get deployed to production, there are some operational capabilities that people need to have, and so in that case, what we do is we try to fine-tune what do people need to do and grant those people access to the accounts so that they can perform the jobs and I don't have to be woken up at two in the morning. The developers are.Corey: One thing that I think is going to be a cause of some consternation for folks—because I didn't really think about this in any meaningful sense until I started acting as a consultant, which means you're getting three years of experience for every year that you're in the wild, just by virtue of the variety of environments you encounter—on some level, there's a reasonable expectation you can have when you're at a small, scrappy startup, that everyone involved knows where all the moving parts live. That tends to break down with scale. So, the idea of a Cloud Center of Excellence has been bandied around a lot. And personally, I hate the term because it implies the ‘Data Center of Mediocrity,' which is a little on the nose for some people at times. So, the idea of having a sort of as a centralized tiger team that has the expertise and has the ability to go on deep dives and sort of loan themselves out to different teams seems to be a compromise between nobody knows what they're doing and, every person involved should have an in-depth knowledge of the following list of disciplines.For example, most folks do not need an in-depth primer on AWS billing constructs. They need about as much information fits on an index card. Do you find that having the centralized concentration of cloud knowledge on a particular team works out or do you find that effectively doing a rotating embedding story is the better answer?John: It varies a lot, I think, because it depends on the level of curiosity of the developers quite a lot. So, I have a huge developer background. People in my team are probably more coming from ex-IT environments or this sort of operation and then it just naturally went into the cloud. And in my opinion, is fairly rare to find somebody that is actually good at doing both AWS and coding. I am by no means really, really great at coding. I code pretty much every day but I wouldn't call myself a professional developer.However, it does bring to my knowledge the fact that there are some good patterns and good practices that you can bring into building your applications in the cloud and some really bad ones. However, I think it's really down to making sure that the knowledge is here within the team. If there's a specialized team, those really need to be specialists. And I think the important thing then is to make sure that the developers and the people around you that are curious and want to ask questions know that you're available to them to share that knowledge. Because at the end of the day, if I'm the only one with the knowledge, I'm going to be the one who is always going to be on call for this or doing that and this is no responsibility that I want. I am happy with a number of responsibilities, but not to be the only person to ever do this. I want to go on holidays from time to time.So, at the end of the day, I suppose it really is up to what people want or expect out of their careers. I do a job that it was a passion for me since I was about 14 years old. And I've always been extremely curious to understand how things work, but I do draw the line that I don't write anything else than Python these days. And if you ask me to write Java, I'll probably change job in the flip of a second. But that's the end of it. But I enjoy understanding how Java things work so that I can help my developers make better choices with what services in AWS to use.Corey: On some level, it feels like there's a, I guess, lack of the same kind of socialization that startups have sort of been somewhat guided by as far as core ethos goes, where, oh whatever I'm working on, I want to reach out to other people, and, “Hey, I'm trying to solve this problem. What is it that you have been working on that's germane to this and how can we collaborate together?” It has nothing to do, incidentally, with the idea that, oh, big company people aren't friendly or are dedicated or aren't good or aren't well-connected; none of that. But there are so many people internally that you're spending your time focusing on and there's so much more internal context that doesn't necessarily map to anything outside of the company that the idea of someone off the street who just solved a particular problem in a weird way could apply to what a larger company with, you know, regulatory burdens, starts to have in mind, it becomes a little bit further afield. Do you think that that's accurate? Do you think that there's still a strong sense of enterprise community that I'm just potentially not seeing in various ways because I don't work at big companies?John: It's a very fine line to walk. So, when I joined the company, I was made aware that there's a lot of Terraform and Kubernetes, which I went [unintelligible 00:16:28] all the way with CloudFormation is yes. So, that was one of the changes I knew I would have. But I can move an open mind and when I looked around at, okay, what are the Terraform modules—because I used Terraform with anger for an entire year of suffering—and I thought, “Okay, well, maybe people have actually got to a point where they've built great modules that I can just pick up off the shelf and reuse or customize only a tiny little bit, add maybe a couple of features and that's, it move on; it's good enough for me.” But as it turns out, there is I think, a lot of the time a case where the need for standardization goes against the need for business to move on.So, I think this is where you start to see silos start to being built within the company and people do their own thing and the other ones do their own. And I think it's always a really big challenge for a large company with extremely opinionated individuals to say, “All right, we're going to standardize on this way.” And it definitely was one of the biggest challenge that I had when I joined the company because again, big communities and Terraform place, we're going to need to do something else. So, then it was the case of saying, “Hey, I don't think we need Kubernetes and I definitely don't think we need Terraform for any the things—for any of those reasons, so how about we do something a little different?”Corey: Speaking of doing things a little bit different, you were recently featured in an AWS Open-Source Roundup newsletter that was just where you, I think, came across my desk one of the first times, has specifically around an open-source project that you built: ECS Compose-X.So, I assume it's like, oh, it's like Docker Compose for ECS and also the ‘X' implies that it is extreme, just, like, you know, snack foods at the convenience store. What does it do and where'd it come from?John: Right. So, you said most of it, right? It literally is a question where you take a Docker Compose file and you want to deploy your services that you worked on and all of that together, and you want to deploy it to AWS. So, ECS Compose-X is a CLI tool very much like the Copilot. I think it was released about four months just before Copilots came out—so, sorry, I beat you to the ball there—but with the Docker Compose specification supported.And again, it was really out of I needed to find a neat way to take my services and deploy them in AWS. So, Compose-X is just a CLI tool that is going to parse your Docker Compose file and create CloudFormation templates out of it. Now, the X is not very extreme or anything like that, but it's actually coming from the [finite 00:18:59] extension fields, which is something supported in Docker Compose. And so, you can do things like x-RDS, or x-DynamoDB, which Docker Compose on your laptop will totally ignore, but ECS Compose-X however will take that into account.And what it will do is if you need a database or a DynamoDB table, for example, in your Docker Compose file, you do [x-RDS, my database, some properties, 00:19:22]—exactly the same properties as CloudFormation, actually—and then you say, “I want this service to have access to it in read-only fashion.” And what ECS Compose-X is going to do is just understand what it has to do when—meaning creating IAM policies, opening security groups, all of that stuff, and make all of that available to the containers in one way or another.Corey: It feels like it's a bit of a miss for Copilot not to do this. It feels like they wanted to go off in their own direction with the way that they viewed the world—which I get; I'm not saying there's anything inherently wrong with that. There's a reason that I point kubernetestheeasyway.com to the ECS marketing site—but there's so much stuff out there that is shipped or made available in other ways with a Docker Compose file, and the question of okay, how do I take this and run it in Fargate or something because I don't want to run it locally for whatever reason, and the answer is, “That's the neat part. You don't.”And it just becomes such a clear miss. There have been questions about this Since Copilot launched. There's a GitHub issue tracking getting support for this that was last updated in September—we are currently recording this at the end of March—it just doesn't seem to be something that's a priority. I mean, I will say the couple of times that I've used Copilot myself, it was always for greenfield experiments, never for adopting something else that already existed. And that was… it just felt like a bit of a heavy lift to me of oh, you need to know from the beginning that this is the tool you're going to use for the thing. Docker Compose is what the ecosystem has settled on a long time ago and I really am disheartened by the fact that there's no direct ECS support for it today.John: Yeah, and it was definitely a motivation for me because I knew that ECS CLI version 1 was going into the sunset, and there wasn't going to be anything supporting it. And so, I just wanted to have Docker Compose because it's familiar to developers and again, if you want to have adoption and have people use your thing, it has to be easy. And when I looked at Copilot the first time around, I was extremely excited because I thought, “Yes, thank you, Amazon for making my life easy. I don't have to maintain this project anymore and I'm going to be able to just lift and shift, move over, and be happy about it.” But when the specification for Copilot was out and I could go for the documentation, I was equally disheartened because I was like, “Okay, not for me.”And something very similar happened when they announced Proton. I was extremely excited by Proton. I opened a GitHub issue on the roadmap immediately to say, “Hey, are you going to support to have some of those things together or not?” And the fact that the Proton templates—I mean, again, it was, what, two, three years ago now—and I haven't looked at Proton since, so it was a very long time now.Corey: The beta splasher was announced in 2020 and I really haven't seen much from it since.John: Well, and I haven't done anything [unintelligible 00:22:07] with it. And literally, one of the first thing did when the project came out. Because obviously, this is an open-source project that we use in Sainsbury's, right because we deploy everything in [ECS 00:22:17] so why would I reinvent the wheel the third time? It's been done, I might as well leverage it. But every time something on it came out, I was seeing it as the way out of nobody's going to need me anymore—which is great—and that doesn't create a huge potential dependency on the company for me, oh, well, we need this to, you know, keep working.Now, it's open-source, it's on the license you can fork it and do whatever you want with it, so from that point of view, nobody's going to ask me anything in the future, but from the point of view where I need to, as much as possible, use AWS native tools, or AWS-built tools, I differently wanted every time to move over to something different. But every time I tried and tiptoed with those alternative offerings, I just went back and said, “No, this [laugh] either is too new and not mature enough yet, or my tool is just better.” Right? And one of the things I've been doing for the past three years is look at the Docker ECS plugin, all of the issues, and I see all of the feature requests that people are asking for and just do that in my project. And some with Copilots. The only thing that Copilot does that I don't do is tell people how to do CI/CD pipelines.Corey: One thing you said a second ago just sort of, I guess, sent me spiraling for a second because I distinctly remember this particular painful part. You're right, there was an ECS CLI for a long time that has since been deprecated. But we had internal tooling built around that. When there was an issue with a particular task that failed, getting logs out of it was non-trivial, so great. Here's the magic incantation that does it.I still haven't found a great way to do that with the AWS v2 CLI and that feels like it's a gap where yes, I understand, old tools go away and new ones show up, but, “Hey, I [unintelligible 00:24:05] task. Can you tell me what the logs are?” “No. Well, Copilot's the new answer.” “Okay. Can I use this to get logs from something that isn't Copilot?” “Oh, absolutely not.” And the future is inherently terrible as a direct result.John: Yeah. Well, I mean, again, the [unintelligible 00:24:20]—the only thing that ECS Compose-X does is create all the templates for you so you can, you know, then just query it and know where everything has been created. And one of the things it definitely does create is all of the log groups. Because again, least-privileged permissions being something that is very dear to me, I create the log groups and just allow the services to only write in those log groups and that's it.Now, typically this is not a thing that I've thought Compose-X was going to do because that's not its purpose. It's not going to be an operational tool to troubleshoot all the things and this is where I think that other projects are much better suited and I would rather use them as an extension or library of the project as opposed to reinvent them. So, if you're trying to find a tool for yourself to look at logs, I highly recommend something called ‘AWS logs,' which is fantastic. You just say, “Hey, can you list the groups?” “Okay.” “Can you get me the groups and can I tell them on a terminal?”And that's it. Job done. So, as much as I enjoy building new features into the project, for example, I think that there's a clear definition between what the project is for and what it's not. And what it's for is giving people CloudFormation templates they can reuse in any region and deploy their services and not necessarily deal with their operations; that's up to them. At the end of the day, it's really up to the user to know what they want to do with it. I'm not trying to force anybody into doing something specific.Corey: I would agree. I think that there's value to there's more than one way to do it. The problem is, at some point, there's a tipping point where you have this proliferation of different options to the point where you end up in this analysis paralysis model where you're too busy trying to figure out what is the next clear step. And yes, that flexibility is incredibly valuable, especially when you get into, you know, large, sophisticated enterprises—ahem, ahem—but when you're just trying to kick the tires on something new, I feel like there's a certain lack of golden path where in the event of not having an opinion on any of these things, this is what you should do just to keep things moving forward, as opposed to here are two equal options that you can check with radio boxes and it's not at all clear what you which does what or what the longer-term implications are. We've all gotten caught with the one-way doors we didn't realize we were passing through at the time and then had to do significant technical debt repayment efforts to wind up making it right again.I just wish that those questions would be called out, but everything else just, it doesn't matter. If you don't like the name of the service that you're creating, you can change it later. Or if you can't, maybe you should know now, so you don't have—in my case—a DynamoDB table that is named ‘test' running in production forever.John: Yeah. You're absolutely right. And again, I think it goes back to one of the biggest challenges that I had when I joined the company, which was when I said, “I think we should be using CloudFormation, I think we should be using ECS and Terraforming Kubernetes for those reasons.” And one of the reasons was, the people. Meaning we were a very small team, only five cloud engineers at the time.And as I joined the company, they were already was three different teams using four different CI/CD tools. And they all wanted to use Kubernetes, for example, and they were all using different CI/CD—like I said, just now—different CI/CD tools. And so, the real big challenge for me was how do I pitch that simplicity is what's going to allow us to deliver value for the business? Because at the end of the day, like you said many, many times before, the AWS bill is a question of architecture, right? And there's a link and intricacy between the two things.So, the only thing that really mattered for me and the team was to find a way, find the service that was going to allow to do a number of things, A, delivering value quickly, being supported over time. Because one of the things that I think people forget these days—well, one of the things I'm allergic to and one of the things that makes me spiral is what I call CV-driven tech choices where people say, “Hey, I love this great thing I read about and I think that we should use that in production. How great idea.” But really, I don't know anything about it and is then up to somebody else to maintain it long-term.And that goes to the other point, which is, turnover-proof is what I call it. So, making tech choices that are going to be something that people will be able to use for many, many years, there is going to be a company behind the scenes that he's going to be able to support you as well as you go and use the service for the many, many years to go.Corey: I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place for them to find you?John: So, people can find me on LinkedIn. I'm also around on Twitter these days, although I probably about have nine followers. Well, probably shouldn't say that [laugh] and that doesn't matter.Corey: It's fine. We'll put a link into it—we'll put a link to that in the [show notes 00:29:02] and maybe we'll come up with number ten. You never know. Thanks again for your time. I really appreciate it.John: Thanks so much, Corey, for having me.Corey: John Mille, Principal Cloud Engineer at Sainsbury's. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that you go to great pains to type out but then fails to post because the version of the tool you use to submit it has been deprecated without a viable replacement.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Welcome to the newest episode of The Cloud Pod podcast! Justin, Ryan and Matthew are your hosts this week as we discuss all the latest news and announcements in the world of the cloud and AI. Do people really love Matt's Azure know-how? Can Google make Bard fit into literally everything they make? What's the latest with Azure AI and their space collaborations? Let's find out! Titles we almost went with this week: Clouds in Space, Fictional Realms of Oracles, Oh My.  The cloudpod streams lambda to the cloud A big thanks to this week's sponsor:  Foghorn Consulting, provides top-notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you have trouble hiring?  Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week.

AWS Puts Up a New VPC Lattice to Ease the Growth of Your Connectivity AKA Welcome to April (how is it April already?) This week, Justin, Jonathan, and Matt are your guides through all the latest and greatest in Cloud news; including VPC Lattice from AWS, the one and only time we'll talk about Service Catalog, and an ultra premium DDoS experience. All this week on The Cloud Pod.  This week's alternate title(s): AWS Finally makes service catalogs good with Terraform Amazon continues to believe retailers with supply chain will give all their data to them Azure copies your data from S3… AWS copies your data from Azure Blobs… or how I set money on fire with data egress charges

Waldemar Hummer, Co-Founder & CTO of LocalStack, joins Corey on Screaming in the Cloud to discuss how LocalStack changed Corey's mind on the futility of mocking clouds locally. Waldemar reveals why LocalStack appeals to both enterprise companies and digital nomads, and explains how both see improvements in their cost predictability as a result. Waldemar also discusses how LocalStack is an open-source company first and foremost, and how they're working with their community to evolve their licensing model. Corey and Waldemar chat about the rising demand for esoteric services, and Waldemar explains how accommodating that has led to an increase of adoption from the big data space. About WaldemarWaldemar is Co-Founder and CTO of LocalStack, where he and his team are building the world-leading platform for local cloud development, based on the hugely popular open source framework with 45k+ stars on Github. Prior to founding LocalStack, Waldemar has held several engineering and management roles at startups as well as large international companies, including Atlassian (Sydney), IBM (New York), and Zurich Insurance. He holds a PhD in Computer Science from TU Vienna.Links Referenced: LocalStack website: https://localstack.cloud/ LocalStack Slack channel: https://slack.localstack.cloud LocalStack Discourse forum: https://discuss.localstack.cloud LocalStack GitHub repository: https://github.com/localstack/localstack TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Until a bit over a year ago or so, I had a loud and some would say fairly obnoxious opinion around the futility of mocking cloud services locally. This is not to be confused with mocking cloud services on the internet, which is what I do in lieu of having a real personality. And then one day I stopped espousing that opinion, or frankly, any opinion at all. And I'm glad to be able to talk at long last about why that is. My guest today is Waldemar Hummer, CTO and co-founder at LocalStack. Waldemar, it is great to talk to you.Waldemar: Hey, Corey. It's so great to be on the show. Thank you so much for having me. We're big fans of what you do at The Duckbill Group and Last Week in AWS. So really, you know, glad to be here with you today and have this conversation.Corey: It is not uncommon for me to have strong opinions that I espouse—politely to be clear; I'll make fun of companies and not people as a general rule—but sometimes I find that I've not seen the full picture and I no longer stand by an opinion I once held. And you're one of my favorite examples of this because, over the course of a 45-minute call with you and one of your business partners, I went from, “What you're doing is a hilarious misstep and will never work,” to, “Okay, and do you have room for another investor?” And in the interest of full disclosure, the answer to that was yes, and I became one of your angel investors. It's not exactly common for me to do that kind of a hard pivot. And I kind of suspect I'm not the only person who currently holds the opinion that I used to hold, so let's talk a little bit about that. At the very beginning, what is LocalStack and what does it you would say that you folks do?Waldemar: So LocalStack, in a nutshell, is a cloud emulator that runs on your local machine. It's basically like a sandbox environment where you can develop your applications locally. We have currently a range of around 60, 70 services that we provide, things like Lambda Functions, DynamoDB, SQS, like, all the major AWS services. And to your point, it is indeed a pretty large undertaking to actually implement the cloud and run it locally, but with the right approach, it actually turns out that it is feasible and possible, and we've demonstrated this with LocalStack. And I'm glad that we've convinced you to think of it that way as well.Corey: A couple of points that you made during that early conversation really stuck with me. The first is, “Yeah, AWS has two, no three no four-hundred different service offerings. But look at your customer base. How many of those services are customers using in any real depth? And of those services, yeah, the APIs are vast, and very much a sprawling pile of nonsense, but how many of those esoteric features are those folks actually using?” That was half of the argument that won me over.The other half was, “Imagine that you're an enormous company that's an insurance company or a bank. And this year, you're hiring 5000 brand new developers, fresh out of school. Two to 3000 of those developers will still be working here in about a year as they wind up either progressing in other directions, not winding up completing internships, or going back to school after internships, or for a variety of reasons. So, you have that many people that you need to teach how to use cloud in the context that we use cloud, combined with the question of how do you make sure that one of them doesn't make a fun mistake that winds up bankrupting the entire company with a surprise AWS bill?” And those two things combined turned me from, “What you're doing is ridiculous,” to, “Oh, my God. You're absolutely right.”And since then, I've encountered you in a number of my client environments. You were absolutely right. This is something that resonates deeply and profoundly with larger enterprise customers in particular, but also folks who just don't want to wind up being beholden to every time they do a deploy to anything to test something out, yay, I get to spend more money on AWS services.Waldemar: Yeah, totally. That's spot on. So, to your first point, so definitely we have a core set of services that most people are using. So, things like Lambda, DynamoDB, SQS, like, the core serverless, kind of, APIs. And then there's kind of a long tail of more exotic services that we support these days, things like, even like QLDB, the quantum ledger database, or, you know, managed streaming for Kafka.But like, certainly, like, the core 15, 20 services are the ones that are really most used by the majority of people. And then we also, you know, pro offering have some very, sort of, advanced services for different use cases. So, that's to your first point.And second point is, yeah, totally spot on. So LocalStack, like, really enables you to experiment in the sandbox. So, we both see it as an experimentation, also development environment, where you don't need to think about cloud costs. And this, I guess, will be very close to your heart in the work that you're doing, the costs are becoming really predictable as well, right? Because in the cloud, you know, work to different companies before doing LocalStack where we were using AWS resources, and you can end up in a situation where overnight, you accumulate, you know, hundreds of thousands of dollars of AWS bill because you've turned on a certain feature, or some, you know, connectivity into some VPC or networking configuration that just turns out to be costly.Also, one more thing that is worth mentioning, like, we want to encourage, like, frequent testing, and a lot of the cloud's billing and cost structure is focused around, for example, hourly billing of resources, right? And if you have a test that just spins up resources that run for a couple of minutes, you still end up paying the entire hour. And we LocalStack, really, that brings down the cloud builds significantly because you can really test frequently, the cycles become much faster, and it's also again, more efficient, more cost-effective.Corey: There's something useful to be said for, “Well, how do I make sure that I turn off resources when I'm done?” In cloud, it's a bit of a game of guess-and-check. And you turn off things you think are there and you wait a few days and you check the bill again, and you go and turn more things off, and the cycle repeats. Or alternately, wait for the end of the month and wonder in perpetuity why you're being billed 48 cents a month, and not be clear on why. Restarting the laptop is a lot more straightforward.I also want to call out some of my own bias on this where I used to be a big believer in being able to build and deploy and iterate on things locally because well, what happens when I'm in a plane with terrible WiFi? Well, in the before times, I flew an awful lot and was writing a fair bit of, well, cloudy nonsense and I still never found that to be a particular blocker on most of what I was doing. So, it always felt a little bit precious to me when people were talking about, well, what if I can't access the internet to wind up building and deploying these things? It's now 2023. How often does that really happen? But is that a use case that you see a lot of?Waldemar: It's definitely a fair point. And probably, like, 95% of cloud development these days is done in a high internet bandwidth environment, maybe some corporate network where you have really fast internet access. But that's only a subset, I guess, of the world out there, right? So, there might be situations where, you know, you may have bad connectivity. Also, maybe you live in a region—or maybe you're traveling even, right? So, there's a lot more and more people who are just, “Digital nomads,” quote-unquote, right, who just like to work in remote places.Corey: You're absolutely right. My bias is that I live in San Francisco. I have symmetric gigabit internet at home. There's not a lot of scenarios in my day-to-day life—except when I'm, you know, on the train or the bus traveling through the city—because thank you, Verizon—where I have impeded connectivity.Waldemar: Right. Yeah, totally. And I think the other aspect of this is kind of the developers just like to have things locally, right, because it gives them the feeling of you know, better control over the code, like, being able to integrate into their IDEs, setting breakpoints, having these quick cycles of iterations. And again, this is something that there's more and more tooling coming up in the cloud ecosystem, but it's still inherently a remote execution that just, you know, takes the round trip of uploading your code, deploying, and so on, and that's just basically the pain point that we're addressing with LocalStack.Corey: One thing that did surprise me as well was discovering that there was a lot more appetite for this sort of thing in enterprise-scale environments. I mean, some of the reference customers that you have on your website include divisions of the UK Government and 3M—you know, the Post-It note people—as well as a number of other very large environments. And at first, that didn't make a whole lot of sense to me, but then it suddenly made an awful lot of sense because it seems—and please correct me if I'm wrong—that in order to use something like this at scale and use it in a way that isn't, more or less getting it into a point where the administration of it is more trouble than it's worth, you need to progress past a certain point of scale. An individual developer on their side project is likely just going to iterate against AWS itself, whereas a team of thousands of developers might not want to be doing that because they almost certainly have their own workflows that make that process high friction.Waldemar: Yeah, totally. So, what we see a lot is, especially in larger enterprises, dedicated teams, like, developer experience teams, whose main job is to really set up a workflow and environment where developers can be productive, most productive, and this can be, you know, on one side, like, setting up automated pipelines, provisioning maybe AWS sandbox and test accounts. And like some of these teams, when we introduce LocalStack, it's really a game-changer because it becomes much more decoupled and like, you know, distributed. You can basically configure your CI pipeline, just, you know, spin up the container, run your tests, tear down again afterwards. So, you know, it's less dependencies.And also, one aspect to consider is the aspect of cloud approvals. A lot of companies that we work with have, you know, very stringent processes around, even getting access to the clouds. Some SRE team needs to enable their IAM permissions and so on. With LocalStack, you can just get started from day one and just get productive and start testing from the local machine. So, I think those are patterns that we see a lot, in especially larger enterprise environments as well, where, you know, there might be some regulatory barriers and just, you know, process-wise steps as well.Corey: When I started playing with LocalStack myself, one of the things that I found disturbingly irritating is, there's a lot that AWS gets largely right with its AWS command-line utility. You can stuff a whole bunch of different options into the config for different profiles, and all the other tools that I use mostly wind up respecting that config. The few that extend it add custom lines to it, but everything else is mostly well-behaved and ignores the things it doesn't understand. But there is no facility that lets you say, “For this particular profile, use this endpoint for AWS service calls instead of the normal ones in public regions.” In fact, to do that, you effectively have to pass specific endpoint URLs to arguments, and I believe the syntax on that is not globally consistent between different services.It just feels like a living nightmare. At first, I was annoyed that you folks wound up having to ship your own command-line utility to wind up interfacing with this. Like, why don't you just add a profile? And then I tried it myself and, oh, I'm not the only person who knows how this stuff works that has ever looked at this and had that idea. No, it's because AWS is just unfortunate in that respect.Waldemar: That is a very good point. And you're touching upon one of the major pain points that we have, frankly, with the ecosystem. So, there are some pull requests against the AWS open-source repositories for the SDKs and various other tools, where folks—not only LocalStack, but other folks in the community have asked for introducing, for example, an AWS endpoint URL environment variable. These [protocols 00:12:32], unfortunately, were never merged. So, it would definitely make our lives a whole lot easier, but so far, we basically have to maintain these, you know, these wrapper scripts, basically, AWS local, CDK local, which basically just, you know, points the client to local endpoints. It's a good workaround for now, but I would assume and hope that the world's going to change in the upcoming years.Corey: I really hope so because everything else I can think of is just bad. The idea of building a custom wrapper around the AWS command-line utility that winds up checking the profile section, and oh, if this profile is that one, call out to this tool, otherwise it just becomes a pass-through. That has security implications that aren't necessarily terrific, you know, in large enterprise companies that care a lot about security. Yeah, pretend to be a binary you're not is usually the kind of thing that makes people sad when security politely kicks their door in.Waldemar: Yeah, we actually have pretty, like, big hopes for the v3 wave of the SDKs, AWS, because there is some restructuring happening with the endpoint resolution. And also, you can, in your profile, by now have, you know, special resolvers for endpoints. But still the case of just pointing all the SDKs and CLI to a custom endpoint is just not yet resolved. And this is, frankly, quite disappointing, actually.Corey: While we're complaining about the CLI, I'll throw one of my recurring issues with it in. I would love for it to adopt the Linux slash Unix paradigm of having a config.d directory that you can reference from within the primary config file, and then any file within that directory in the proper syntax winds up getting adopted into what becomes a giant composable config file, generated dynamically. The reason being is, I can have entire lists of profiles in separate files that I could then wind up dropping in and out on a client-by-client basis. So, I don't inadvertently expose who some of my clients are, in the event that winds up being part of the way that they have named their AWS accounts.That is one of those things I would love but it feels like it's not a common enough use case for there to be a whole lot of traction around it. And I guess some people would make a fair point if they were to say that the AWS CLI is the most widely deployed AWS open-source project, even though all it does is give money to AWS more efficiently.Waldemar: Yeah. Great point. Yeah, I think, like, how and some way to customize and, like, mingle or mangle your configurations in a more easy fashion would be super useful. I guess it might be a slippery slope to getting, you know, into something like I don't know, Helm for EKS and, like, really, you know, having to maintain a whole templating language for these configs. But certainly agree with you, to just you know, at least having [plug 00:15:18] points for being able to customize the behavior of the SDKs and CLIs would be extremely helpful and valuable.Corey: This is not—unfortunately—my first outing with the idea of trying to have AWS APIs done locally. In fact, almost a decade ago now, I did a build-out at a very large company of a… well, I would say that the build-out was not itself very large—it was about 300 nodes—that were all running Eucalyptus, which before it died on the vine, was imagined as a way of just emulating AWS APIs locally—done in Java, as I recall—and exposing local resources in ways that comported with how AWS did things. So, the idea being that you could write configuration to deploy any infrastructure you wanted in AWS, but also treat your local data center the same way. That idea unfortunately did not survive in the marketplace, which is kind of a shame, on some level. What was it that inspired you folks to wind up building this with an eye towards local development rather than run this as a private cloud in your data center instead?Waldemar: Yeah, very interesting. And I do also have some experience [unintelligible 00:16:29] from my past university days with Eucalyptus and OpenStack also, you know, running some workloads in an on-prem cluster. I think the main difference, first of all, these systems were extremely hard, notoriously hard to set up and maintain, right? So, lots of moving parts: you had your image server, your compute system, and then your messaging subsystems. Lots of moving parts, and wanting to have everything basically much more monolithic and in a single container.And Docker really sort of provides a great platform for us, which is create everything in a single container, spin up locally, make it very lightweight and easy to use. But I think really the first days of LocalStack, the idea was really, was actually with the use case of somebody from our team. Back then, I was working at Atlassian in the data engineering team and we had folks in the team were commuting to work on the train. And it was literally this use case that you mentioned before about being able to work basically offline on your commute. And this is kind of were the first lines of code were written and then kind of the idea evolves from there.We put it into the open-source, and then, kind of, it was growing over the years. But it really started as not having it as an on-prem, like, heavyweight server, but really as a lightweight system that you can easily—that is easily portable across different systems as well.Corey: That is a good question. Very often, when I'm using various tools that are aimed at development use cases, it is very clear that one particular operating system is invariably going to be the first-class citizen and everything else is a best effort. Ehh, it might work; it might not. Does LocalStack feel that way? And if so, what's the operating system that you want to be on?Waldemar: I would say we definitely work best on Mac OS and Linux. It also works really well on Windows, but I think given that some of our tooling in the ecosystem also pretty much geared towards Unix systems, I think those are the platforms it will work well with. Again, on the other hand, Docker is really a platform that helps us a lot being compatible across operating systems and also CPU architectures. We have a multi-arch build now for AMD and ARM64. So, I think in that sense, we're pretty broad in terms of the compatibility spectrum.Corey: I do not have any insight into how the experience goes on Windows, given that I don't use that operating system in anger for, wow, 15 years now, but I will say that it's been top-flight on Mac OS, which is what I spend most of my time. Depressed that I'm using, but for desktop experiences, it seems to work out fairly well. That said, having a focus on Windows seems like it would absolutely be a hard requirement, given that so many developer workstations in very large enterprises tend to skew very Windows-heavy. My hat is off to people who work with Linux and Linux-like systems in environments like that where even line endings becomes psychotically challenging. I don't envy them their problems. And I have nothing but respect for people who can power through it. I never had the patience.Waldemar: Yeah. Same here and definitely, I think everybody has their favorite operating system. For me, it's also been mostly Linux and Mac in the last couple of years. But certainly, we definitely want to be broad in terms of the adoption, and working with large enterprises often you have—you know, we want to fit into the existing landscape and environment that people work in. And we solve this by platform abstractions like Docker, for example, as I mentioned, and also, for example, Python, which is some more toolings within Python is also pretty nicely supported across platforms. But I do feel the same way as you, like, having been working with Windows for quite some time, especially for development purposes.Corey: What have you noticed that your customer usage patterns slash requests has been saying about AWS service adoption? I have to imagine that everyone cares whether you can mock S3 effectively. EC2, DynamoDB, probably. SQS, of course. But beyond the very small baseline level of offering, what have you seen surprising demand for, as I guess, customer implementation of more esoteric services continues to climb?Waldemar: Mm-hm. Yeah, so these days it's actually pretty [laugh] pretty insane the level of coverage we already have for different services, including some very exotic ones, like QLDB as I mentioned, Kafka. We even have Managed Airflow, for example. I mean, a lot of these services are essentially mostly, like, wrappers around the API. This is essentially also what AWS is doing, right? So, they're providing an API that basically provisions some underlying resources, some infrastructure.Some of the more interesting parts, I guess, we've seen is the data or big data ecosystem. So, things like Athena, Glue, we've invested quite a lot of time in, you know, making that available also in LocalStack so you can have your maybe CSV files or JSON files in an S3 bucket and you can query them from Athena with a SQL language, basically, right? And that makes it very—especially these big data-heavy jobs that are very heavyweight on AWS, you can iterate very quickly in LocalStack. So, this is where we're seeing a lot of adoption recently. And then also, obviously, things like, you know, Lambda and ECS, like, all the serverless and containerized applications, but I guess those are the more mainstream ones.Corey: I imagine you probably get your fair share of requests for things like CloudFormation or CloudFront, where, this is great, but can you go ahead and add a very lengthy sleep right here, just because it returns way too fast and we don't want people to get their hopes up when they use the real thing. On some level, it feels like exact replication of the AWS customer experience isn't quite in line with what makes sense from a developer productivity point of view.Waldemar: Yeah, that's a great point. And I'm sure that, like, a lot of code out there is probably littered with sleep statements that is just tailored to the specific timing in AWS. In fact, we recently opened an issue in the AWS Terraform provider repository to add a configuration option to configure the timings that Terraform is using for the resource deployment. So, just as an example, an S3 bucket creation takes 60 seconds, like, more than a minute against [unintelligible 00:22:37] AWS. I guess LocalStack, it's a second basically, right?And AWS Terraform provider has these, like, relatively slow cycles of checking whether the packet has already been created. And we want to get that configurable to actually reduce the time it takes for local development, right? So, we have an open, sort of, feature request, and we're probably going to contribute to a Terraform repository. But definitely, I share the sentiment that a lot of the tooling ecosystem is built and tailored and optimized towards the experience against the cloud, which often is just slow and, you know, that's what it is, right?Corey: One thing that I didn't expect, though, in hindsight, is blindingly obvious, is your support for a variety of different frameworks and deployment methodologies. I've found that it's relatively straightforward to get up and running with the CDK deploying to LocalStack, for instance. And in hindsight, of course; that's obvious. When you start out down that path, though it's well, you tend to think—at least I don't tend to think in that particular way. It's, “Well, yeah, it's just going to be a console-like experience, or I wind up doing CloudFormation or Terraform.” But yeah, that the world is advancing relatively quickly and it's nice to see that you are very comfortably keeping pace with that advancement.Waldemar: Yeah, true. And I guess for us, it's really, like, the level of abstraction is sort of increasing, so you know, once you have a solid foundation, with, you know, CloudFormation implementation, you can leverage a lot of tools that are sitting on top of it, CDK, serverless frameworks. So, CloudFormation is almost becoming, like, the assembly language of the AWS cloud, right, and if you have very solid support for that, a lot of, sort of, tools in the ecosystem will natively be supported on LocalStack. And then, you know, you have things like Terraform, and in the Terraform CDK, you know, some of these derived versions of Terraform which also are very straightforward because you just need to point, you know, the target endpoint to localhost and then the rest of the deployment loop just works out of the box, essentially.So, I guess for us, it's really mostly being able to focus on, like, the core emulation, making sure that we have very high parity with the real services. We spend a lot of time and effort into what we call parity testing and snapshot testing. We make sure that our API responses are identical and really the same as they are in AWS. And this really gives us, you know, a very strong confidence that a lot of tools in the ecosystem are working out-of-the-box against LocalStack as well.Corey: I would also like to point out that I'm also a proud LocalStack contributor at this point because at the start of this year, I noticed, ah, in one of the pages, the copyright year was still saying 2022 and not 2023. So, a single-character pull request? Oh, yes, I am on the board now because that is how you ingratiate yourself with an open-source project.Waldemar: Yeah. Eternal fame to you and kudos for your contribution. But, [laugh] you know, in all seriousness, we do have a quite an active community of contributors. We are an open-source first project; like, we were born in the open-source. We actually—maybe just touching upon this for a second, we use GitHub for our repository, we use a lot of automation around, you know, doing pull requests, and you know, service owners.We also participate in things like the Hacktoberfest, which we participated in last year to really encourage contributions from the community, and also host regular meetups with folks in the community to really make sure that there's an active ecosystem where people can contribute and make contributions like the one that you did with documentation and all that, but also, like, actual features, testing and you know, contributions of different levels. So really, kudos and shout out to the entire community out there.Corey: Do you feel that there's an inherent tension between being an open-source product as well as being a commercial product that is available for sale? I find that a lot of companies feel vaguely uncomfortable with the various trade-offs that they make going down that particular path, but I haven't seen anyone in the community upset with you folks, and it certainly hasn't seemed to act as a brake on your enterprise adoption, either.Waldemar: That is a very good point. So, we certainly are—so we're following an open-source-first model that we—you know, the core of the codebase is available in the community version. And then we have pro extensions, which are commercial and you basically, you know, setup—you sign up for a license. We are certainly having a lot of discussions on how to evolve this licensing model going forward, you know, which part to feed back into the community version of LocalStack. And it's certainly an ongoing evolving model as well, but certainly, so far, the support from the community has been great.And we definitely focus to, kind of, get a lot of the innovation that we're doing back into our open-source repo and make sure that it's, like, really not only open-source but also open contribution for folks to contribute their contributions. We also integrate with other third-party libraries. We're built on the shoulders of giants, if I may say so, other open-source projects that are doing great work with emulators. To name just a few, it's like, [unintelligible 00:27:33] which is a great project that we sort of use and depend upon. We have certain mocks and emulations, for Kinesis, for example, Kinesis mock and a bunch of other tools that we've been leveraging over the years, which are really great community efforts out there. And it's great to see such an active community that's really making this vision possible have a truly local emulated clouds that gives the best experience to developers out there.Corey: So, as of, well, now, when people are listening to this and the episode gets released, v2 of LocalStack is coming out. What are the big differences between LocalStack and now LocalStack 2: Electric Boogaloo, or whatever it is you're calling the release?Waldemar: Right. So, we're super excited to release our v2 version of LocalStack. Planned release date is end of March 2023, so hopefully, we will make that timeline. We did release our first version of OpenStack in July 2022, so it's been roughly seven months since then and we try to have a cadence of roughly six to nine months for the major releases. And what you can expect is we've invested a lot of time and effort in last couple of months and in last year to really make it a very rock-solid experience with enhancements in the current services, a lot of performance optimizations, we've invested a lot in parity testing.So, as I mentioned before, parity is really important for us to make sure that we have a high coverage of the different services and how they behave the same way as AWS. And we're also putting out an enhanced version and a completely polished version of our Cloud Pods experience. So, Cloud Pods is a state management mechanism in LocalStack. So, by default, the state in LocalStack is ephemeral, so when you restart the instance, you basically have a fresh state. But with Cloud Pods, we enable our users to take persistent snapshot of the states, save it to disk or to a server and easily share it with team members.And we have very polished experience with Community Cloud Pods that makes it very easy to share the state among team members and with the community. So, those are just some of the highlights of things that we're going to be putting out in the tool. And we're super excited to have it done by, you know, end of March. So, stay tuned for the v2 release.Corey: I am looking forward to seeing how the experience shifts and evolves. I really want to thank you for taking time out of your day to wind up basically humoring me and effectively re-covering ground that you and I covered about a year and a half ago now. If people want to learn more, where should they go?Waldemar: Yeah. So definitely, our Slack channel is a great way to get in touch with the community, also with the LocalStack team, if you have any technical questions. So, you can find it on our website, I think it's slack.localstack.cloud.We also host a Discourse forum. It's discuss.localstack.cloud, where you can just, you know, make feature requests and participate in the general conversation.And we do host monthly community meetups. Those are also available on our website. If you sign up, for example, for a newsletter, you will be notified where we have, you know, these webinars. Take about an hour or so where we often have guest speakers from different companies, people who are using, you know, cloud development, local cloud development, and just sharing the experiences of how the space is evolving. And we're always super happy to accept contributions from the community in these meetups as well. And last but not least, our GitHub repository is a great way to file any issues you may have, feature requests, and just getting involved with the project itself.Corey: And we will, of course, put links to that in the [show notes 00:31:09]. Thank you so much for taking the time to speak with me today. I appreciate it.Waldemar: Thank you so much, Corey. It's been a pleasure. Thanks for having me.Corey: Waldemar Hummer, CTO and co-founder at LocalStack. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment, presumably because your compensation structure requires people to spend ever-increasing amounts of money on AWS services.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Lex Neva, Staff Site Reliability Engineer at Honeycomb and Curator of SRE Weekly, joins Corey on Screaming in the Cloud to discuss reliability and the life of a newsletter curator. Lex shares some interesting insights on how he keeps his hobbies and side projects separate, as well as the intrusion that open-source projects can have on your time. Lex and Corey also discuss the phenomenon of newsletter curators being much more demanding of themselves than their audience typically is. Lex also shares his views on how far reliability has come, as well as how far we have to go, and the critical implications reliability has on our day-to-day lives. About LexLex Neva is interested in all things related to running large, massively multiuser online services.  He has years of SRE,  Systems Engineering, tinkering, and troubleshooting experience and perhaps loves incident response more than he ought to.  He's previously worked for Linden Lab, DeviantArt, Heroku, and Fastly, and currently works as an SRE at Honeycomb while also curating the SRE Weekly newsletter on the side.Lex lives in Massachusetts with his family including 3 adorable children, 3 ridiculous cats, and assorted other awesome humans and animals.  In his copious spare time he likes to garden, play tournament poker, tinker with machine embroidery, and mess around with Arduinos.Links Referenced: SRE Weekly: https://sreweekly.com/ Honeycomb: https://www.honeycomb.io/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Chronosphere. Tired of observability costs going up every year without getting additional value? Or being locked into a vendor due to proprietary data collection, querying, and visualization? Modern-day, containerized environments require a new kind of observability technology that accounts for the massive increase in scale and attendant cost of data. With Chronosphere, choose where and how your data is routed and stored, query it easily, and get better context and control. 100% open-source compatibility means that no matter what your setup is, they can help. Learn how Chronosphere provides complete and real-time insight into ECS, EKS, and your microservices, wherever they may be at snark.cloud/chronosphere that's snark.cloud/chronosphere.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Once upon a time, I decided to start writing an email newsletter, and well, many things happened afterwards, some of them quite quickly. But before that, I was reading a number of email newsletters in the space. One that I'd been reading for a year at the time, was called SRE Weekly. It still comes out. I still wind up reading it most weeks.And it's written by Lex Neva, who is not only my guest today but also a staff site reliability engineer at Honeycomb. Lex, it is so good to finally talk to you, other than reading emails that we send to the entire world that pass each other like ships in the night.Lex: Yeah. I feel like we should have had some kind of meeting before now. But yeah, it's really good to [laugh] finally meet you.Corey: It was one of the inspirations that I had. And to be clear, when I signed up for your newsletter originally—I was there for issue 15, which is many, many years ago—I was also running a small-scale SRE team at the time. It was, I found as useful as a part of doing my job and keeping abreast of what was going on in the ecosystem. And I found myself, once I went independent, wishing that your newsletter and a few others had a whole bunch more AWS content. Well, why doesn't it?And the answer is because you are, you know, a reasonable person who understands that mental health is important and boundaries exist for a reason. No one sensible is going to care that much about one cloud provider all the time [sigh]. If only we were all that wise.Lex: Right? Well, [laugh] well, first of all, I love your newsletter, and also the content that you write that—I mean, I would be nowhere without content to link to. And I'm glad you took on the AWS thing because, much like how I haven't written Security Weekly, I also didn't write any kind of AWS Weekly because there's just too much. So, thanks for falling on that sword.Corey: I fell on another one about two years ago and started the Thursdays, which are Last Week in AWS Security. But I took a different bent on it because there are a whole bunch of security newsletters that litter the landscape and most of them are very good—except for the ones that seem to be entirely too vendor-captured—but the problem is, is that they lacked both a significant cloud focus, as well as an understanding that there's a universe of people out here who care about security—or at least should—but don't have the word security baked into their job title. So, it was very insular, using acronyms they assume that everyone knows, or it's totally vendor-captured and it's trying to the whole fear, uncertainty, and doubt thing, “And that's why you should buy this widget.” “Will it solve problems?” “Well, it'll solve our revenue problems at our company that sells the widgets, but other than that, not really.” And it just became such an almost incestuous ecosystem. I wanted something different.Lex: Yeah. And the snark is also very useful [laugh] in order to show us that you're not in their pocket. So yeah, nice work.Corey: Well, I'll let you in on a secret, now that we are—what, I'm somewhat like 300 and change issues in, which means I've been doing this for far too long, the snark is a byproduct of what I needed to do to write it myself. Because let's face it, this stuff is incredibly boring. I needed to keep myself interested as I started down that path. And how can I continually keep it fresh and funny and interesting, but not go too far? That's a fun game, whereas copying and pasting some announcement was never fun.Lex: Yeah, that's not—I hear you on trying to make it interesting.Corey: One regret that I've had, and I'm curious if you've ever encountered this yourself because most people don't get to see any of this. They see the finished product that lands in their inbox every Monday, and—in my case, Monday; I forget the exact day that yours comes out. I collect them and read through them for them all at once—but I find that I have often had caused a look back and regret the implicit commitment in Last Week in AWS as a name because it would be nice to skip a week here and there, just because either I don't particularly feel like it, or wow, there was not a lot of news worth talking about that came out last week. But it feels like I've forced myself onto a very particular treadmill schedule.Lex: Yeah. Yeah, it comes with, like, calling it SRE Weekly. I just followed suit for some of the other weeklies. But yeah, that can be hard. And I do give myself permission to take a week off here and there, but you know, I'll let you in on a secret.What I do is I try to target eight to ten articles a week. And if I have more than that, I save some of them. And then when it comes time to put out an issue, I'll go look at what's in that ready queue and swap some of those in and swap some of the current ones out just so I keep things fresh. And then if I need a week off, I'll just fill it from that queue, you know, if it's got enough in it. So, that lets me take vacations and whatnot. Without that, I think I would have had a lot harder of a time sticking with this, or there just would have been more gaps. So yeah.Corey: You're fortunate in that you have what appears to be a single category of content when you construct your newsletter, whereas I have three that are distinct: AWS releases and announcements and news and things to make fun of for the past week; the things from the larger community folks who do not work there, but are talking about interesting approaches or news that is germane; and then ideally a tip or a tool of the week. And I found, at least lately, that I've been able to build out the tools portion of it significantly far in advance. Because a tool that makes working with AWS easier this week is probably still going to be fairly helpful a month from now.Lex: Yeah, that's fair. Definitely.Corey: But putting some of the news out late has been something of a challenge. I've also learned—by getting it wrong—that I'm holding myself to a tighter expectation of turnaround time than any part of the audience is. The Thursday news is all written the week before, almost a full week beforehand and no one complains about that. I have put out the newsletter a couple of times an hour or two after its usual 7:30 pacific time slot that it goes out in; not a single person has complained. In one case, I moved it by a day to accommodate an announcement but didn't explain why; not a single person emailed in. So, okay. That's good to know.Lex: Yeah, I've definitely gotten to, like, Monday morning, like, a couple of times. Not much, not many times, but a couple of times, I've gotten a Monday morning be like, “Oh, hey. I didn't do that thing yesterday.” And then I just release it in the morning. And I've never had a complaint.I've cancelled last minute because life interfered. The most I've ever had was somebody emailing me and be like, you know, “Hope you feel better soon,” like when I had Covid, and stuff like that. So, [laugh] yeah, sometimes maybe we do hold ourselves to a little bit of a higher standard than is necessary. I mean, there was a point where I got—I had major eye surgery and I had to take a month off of everything and took a month off the newsletter. And yeah, I didn't lose any subscribers. I didn't have any complaints. So people, I think, appreciate it when it's there. And, you know, if it's not there, just wait till it comes out.Corey: I think that there is an additional challenge that I started feeling as soon as I started picking up sponsors for it because it's well, but at this point, I have a contractual obligation to put things out. And again, life happens, but you also don't want to have to reach out on apology tours every third week or whatnot. And I think that's in part due to the fact that I have multiple sponsors per issue and that becomes a bit of a juggling dance logistically on this end.Lex: Yeah. When I started, I really didn't think I necessarily wanted to have sponsors because, you know, it's like, I have a job. This is just for fun. It got to the point where it's like, you know, I'll probably stop this if there's not some kind of monetary advantage [laugh]. And having a sponsor has been really helpful.But I have been really careful. Like, I have always had only a single sponsor because I don't want that many people to apologize to. And that meant I took in maybe less money than I then I could have, but that's okay. And I also was very clear, you know, even from the start having a contract that I may miss a week without notice. And yes, they're paying in advance, but it's not for a specific range of time, it's for a specific number of issues, whenever those come out. That definitely helped to reduce the stress a little bit. And I think without that, you know, having that much over my head would make it hard to do this, you know? It has to stay fun, right?Corey: That's part of the things that kept me from, honestly, getting into tech for the first part of my 20s. It was the fear that I would be taking a hobby, something that I love, and turning it into something that I hated.Lex: Yeah, there is that.Corey: It's almost 20 years now and I'm still wondering whether I actually succeeded or not in avoiding hating this.Lex: Well, okay. But I mean, are you, you know, are you depressed [unintelligible 00:09:16] so there's this other thing, there's this thing that people like to say, which is like, “You should only do a job that you really love.” And I used to think that. And I don't actually think that anymore. I think that it is important to have a job that you can do and not hate day-to-day, but there's no shame in not being passionate about your work and I don't think that we should require passion from anyone when we're hiring. And I think to do so is even, like, privilege. So, you know, I think that it's totally fine to just do something because it pays the bills.Corey: Oh, absolutely. I find it annoying as hell when I'm talking to folks who are looking to hire for roles and, “Well, include a link to your GitHub profile,” is a mandatory field. It's, well, great. What about people who work in places where they're not working on open-source projects as a result, and they can't really disclose what they're doing? And the expectation that oh, well outside of work, you should be doing public stuff, too.It's, I used to do a lot of public open-source style work on GitHub, but I got yelled at all the time for random, unrelated reasons and it's, I don't want to put something out there that I have to support and people start to ask me questions about. It feels like impromptu unasked-for code review. No, thanks. So, my GitHub profile looks fairly barren.Lex: You mean like yelling at you, like, “Oh, you're not contributing enough.” Or, you know, “We need this free thing you're doing, like, immediately,” or that kind of thing?Corey: Worse than that. The worst example I've ever had for this was when I was giving a talk called “Terrible Ideas in Git,” and because I wanted to give some hilariously contrived demos that took a fair bit of work to set up, I got them ready to go inside of a Docker container because I didn't trust that my laptop would always work, I'm might have to borrow someone else's, I pushed that image called “Terrible Ideas” up to Docker Hub. And I wound up with people asking questions about it. Like, “Is this vulnerable to ShellCheck.” And it's, “You do realize that this is intentionally designed to be awful? It is only for giving a very specific version of a very specific talk. It's in public, just because I didn't bother to make it private. What are you doing? Please tell me you're not running this in production at a bank?” “No comment.” Right. I don't want that responsibility of people yelling at me for things I didn't do on purpose. I want to get yelled at for the things I did intentionally.Lex: Exactly. It's funny that sometimes people expect more out of you when you're giving them something free versus when they're paying you for it. It's an interesting quirk of psychology that I'm sure that professionals could tell me all about. Maybe there's been research on it, I don't know. But yeah, that can be difficult.Corey: Oh, absolutely. I used to work at a web hosting company and the customer spending thousands a month with us were uniformly great. But there was always the lowest tier customer of the cheapest thing that we offered that seemed to expect that that entitle them to 80 hours a month of support from engineering problems and whatnot. And it was not profitable to service some of those folks. I've also found that there's a real transitive barrier that begins as soon as you find a way to charge someone a dollar for something.There's a bit of a litmus test of can you transfer a dollar from your bank account to mine? And suddenly, the entire tenor of the conversations with people who have crossed that boundary change. I have toyed, on some level, with the idea of launching a version of this newsletter—or wondering if I retcon the whole thing—do I charge people to subscribe to this? And the answer I keep coming away with is not at all because it started in many respects is marketing for AWS bill consulting and I want the audience as fast as possible. Artificially limiting its distribution via a pay-for model just seemed a little on the strange side.Lex: Yeah. And then you're beholden to a very many people and there's that disproportionality. So, years ago, before I even started in my career in I guess, you know, things that were SRE before SRE was cool, I worked for a living in Second Life. Are you familiar with Second Life?Corey: Oh, yes. I'm very familiar with that. Linden Labs.Lex: Yep. So, I worked for Linden Lab years later, but before I worked for them, I sort of spent a lot of my time living in Second Life. And I had a product that I sold for two or three dollars. And actually, it's still in there; you could still buy it. It's interesting. I don't know if it's because the purchase price was 800 Linden dollars, which equates to, like, $2.16, or something like that, but—Corey: The original cryptocurrency.Lex: Right, exactly. Except there's no crypto involved.Corey: [laugh].Lex: But people seem to have a disproportionate amount of, like, how much of my time they expected for support. You know, I'm going to support them a little bit. You have to recognize at some point, I actually can't come give you a tutorial on using this product because you're one of 500 customers for this month. And you give me two dollars and I don't have ten hours to give you. You know, like, sorry [laugh]. Yeah, so that can be really tough.Corey: And on some level, you need to find a way to either charge more or charge for support on top of it, or ideally—it I wish more open-source projects would take this approach—“Huh. We've had 500 people asking us the exact same question. Should we improve our docs? No, of course not. They're the ones who are wrong. It's the children who are getting it wrong.”I don't find that approach [laugh] to be particularly useful, but it bothers me to no end when I keep running into the same problem onboarding with something new and I ask about it, and, “Oh, yeah, everyone runs into that problem. Here's how you get around it.” This would have been useful to mention in the documentation. I try not to ask questions without reading the manual first.Lex: Well, so there's a couple different directions. I could go with this. First of all, there's a really interesting thing that happened with the core-js project that I recommend people check out. Another thing that I think the direction I'll go at the moment—we can bookmark that other one, but I have an open-source project on the side that I kind of did for my own fun, which is a program for creating designs that can be processed by computer-controlled embroidery machines. So, this is sewing machines that can plot stitches in the x-y plane based on a program that you give it.And there really wasn't much in the way of open-source software available that could help you create these designs and so I just sort of hack something together and started hacking with Python for my own fun, and then put it out there and open-sourced. And it's kind of taken off, kind of like gotten a life of its own. But of course, I've got a newsletter, I've got three kids, I've got a family, and a day job, and I definitely hear you on the, like, you know, yeah, we should put this FAQ in the docs, but there can be so little time to even do that. And I'm finding that there's, like—you know, people talk about work-life balance, there's, like, work slash life slash open-source balance that you really—you know, you have to, like, balance all three of them.And a lot of weeks, I don't have any time to spend on the project. But you know what, it's still kicks along and people just kind of, they use my terrible little project [laugh] as best they can, even though it has a ton of rough edges. I'm sorry, everyone, I'm so sorry. I know it has a t—the UI is terrible. But yeah, it's interesting how these things sometimes take on a life of their own and you can feel dragged along by your own open-source work, you know?Corey: It always bothers me—I think this might tie back to the core-js issue you talked about a second ago—where there are people who are building and supporting open-source tools or libraries that they originally constructed to scratch an itch and now they are core dependencies of basically half the internet. And these people are still wondering on some level, how do I put food on the table this month? It's wild to me. If there were justice in the world, you'd start to think these people would wind up in never-have-to-work-again-if-they-don't-want-to positions. But in many cases, it's exactly the opposite.Lex: Well, that's the really interesting thing. So, first of all, I'm hugely privileged to have any time to get to work on open-source. There's plenty of people that don't, and yeah, so requiring people to have a GitHub link to show their open-source contributions is inherently unfair and biased and discriminatory. That aside, people have asked all along, like, “Lex, this is decent software, you could sell this. You could charge money for this thing and you could probably make a, you know, a decent living at this.”And I categorically refuse to accept money for that project because I don't want to have to support it on a commercial level like that. If I take your money, then you have an expectation that—especially if I charge what one would expect—so this software, part of the reason I decided to write my own is because it starts at two-hundred-some-off dollars for the competitors that are commercial and goes up into the five, ten-thousand dollars. For a software package. Mine is free. If I started charging money, then yeah, I'm going to have to build a support department and we're going to have a knowledge base, I'm going to have to incorporate. I don't want to do that for something I'm doing for fun, you know? So yeah, I'm going to keep it free and terrible [laugh].Corey: It becomes something you love, turns into something you hate without even noticing that it happens. Or at least something that you start to resent.Lex: Yeah. I don't think I would necessarily hate machine embroidery because I love it. It's an amazingly fun little quirky hobby, but I think it would definitely take away some of the magic for me. Where there's no stress at all, I can spend months noodling on an algorithm getting it right, whereas it'd be, you know, if I start having to have deliverables, it changes it entirely. Yeah.Corey: It's odd, it seems, on some level too, that the open-source world that I got started with has evolved in a whole bunch of different ways. Whereas it used to be write a quick fix for something and it would get merged, in many cases by the time you got back from lunch. And these days, it seems like it takes multiple weeks, especially with a corporate-controlled open-source project, and there's so much back and forth. And even getting the boilerplate, like the CLI—the Contributor License Agreement—aside and winding up getting other people to sign off on it, then there's back and forth, in some cases for weeks about, well, the right kind of test coverage and how to look at this and the right holistic framework. And I appreciate that there is validity and value to these things, but is that the bulk of the effort should be going when there's a pull request ready to go that solves a breaking customer problem?But the test coverage isn't right so we're going to delay it for two or three releases. It's what are you doing there? Someone lost the plot somewhere. And I'm sure there are reasons that makes sense, given the framework people are operating within. I just find it maddening from the side of having to [laugh] deal with this as a human.Lex: Yeah, I hear you. And it sometimes can go even beyond test coverage to something like code style, you know? It's like, “Oh, that's not really in the style of this project,” or, “You know, I would have written it this way.” And one thing I've had to really work on, on this project is to make it as inviting to developers as possible. I have to sometimes look at things and be like, yeah, I might do that a different way. But does that actually matter? Like, do I have a reason for that that really matters or is it just my style? And maybe because it's a group project I should just be like, no, that's good as it is.[midroll 00:20:23]Corey: So, you've had an interesting career. And clearly you have opinions about SRE as a result. When I started seeing that you were the author of SRE Weekly, years ago, I just assumed something that I don't believe is true. Is it possible that you have been contributing to the community around SRE, but somehow have never worked at Google?Lex: I have never worked at Google. I have never worked at Netflix. I've never worked at any of those big companies. The biggest company I've worked for is Salesforce. Although I worked for Heroku who had been bought by Salesforce a couple of years prior, and so it was kind of like working for a startup inside a big company. And here's the other thing. I created that newsletter two months after starting my first job where I had a—like, the first job in which I was titled ‘SRE.' So, that's possibly contentious right there.Corey: You know, I hadn't thought of it this way, but you're right. I did almost the exact same thing. I was no expert in AWS when I started these things. It came out of an effort that I needed to do of keeping touch with everything that came out that had potential economic impact, which it turns out are most things when you understand architecture and cost are the same thing when it comes to cloud. But I was more or less gathering what smart people were saying.And somehow there's been this osmotic effect, where people start to view me as the wise old sage of the mountain when it comes to AWS. And no, no, no, I'm just old and grumpy. That looks alike. Don't mistake it for wisdom. But people will now seek me out to get my opinion on things and I have no idea what the answer looks like for most of the stuff.But that's the old SRE model—or sysadmin model that I've followed, which is when you don't know the answer, well, how do you get to a place where you can find the answer? How do you troubleshoot this? Click the button. It doesn't work? Well, time to start taking the button apart to figure out why.Lex: Yeah, definitely. I hear you on people. So, first of all, thanks to everyone who writes the articles that I include. I would be nothing without—I mean—literally, that I could not have a newsletter without content creators. I also kind of started the newsletter as an exploration of this new career title.I mean, I've been doing things that basically fit along with SRE for a long time, but also, I think my view of SRE might be not really the same as a lot of folks, or, like, that Google passed down from the [Google Book Model 00:22:46]. I don't—I'm going to be a little heretical here—I don't necessarily a hundred percent believe in the SLI SLO SLA error budget model. I don't think that that necessarily fits everyone, I'm not sure even suits the bigger companies as well as they think it does. I think that there's a certain point to which you can't actually predict failure and just slowing down on your deploys. And it likes to cause there to be fewer incidents so that you can get—your you know, you can go back to passing in your error budget, to passing your SLO, I'm not sure that actually makes sense or is realistic and works in the real world.Corey: I've been left with the distinct impression that it's something of a framework for how to think about a lot of those things. And it's for folks on a certain point of their development along whatever maturity model or maturity curve you want to talk about, it becomes extraordinarily useful. And at some point, it feels like the path that a given company is on will deviate from that. And, on some level, if you don't wind up addressing it, it turns into what it seems like Agile did, where you wind up with the Cult of Agile around it and the entire purpose of it is to perpetuate the Cult of Agile.And I don't know that I'm necessarily willing to go so far as to say that's where SLOs are headed right now, but I'm starting to get the same sort of feeling around the early days of the formalization of frameworks like that, and the ex cathedra proclamation that this is right for everyone. So, I'm starting to wonder whether there's a reckoning, in that sense, coming down the road. I'm fortunate that I don't run anything that's production-facing, so for me, it's, I don't have to care about these things. Mostly.Lex: Yeah. I mean, we are in… we're in 2023. Things have come so much further than when I was a kid. I have a little computer in my pocket. Yeah, you know, “Hey, math teacher, turns out yeah, we do carry calculators around with us wherever we go.” We've built all these huge, complicated systems online and built our entire society around them.We're still in our infancy. We still don't know what we're doing. We're still feeling out what SRE even is, if it even makes sense, and I think there's—yeah, there's going to be more evolution. I mean, there's been the, like, what is DevOps and people coining the term DevOps and then getting, you know, almost immediately subsumed or turned into whatever other people want. Same thing for observability.I think same thing for SRE. So honestly, I'm feeling it out as I go and I think we all are. And I don't think anyone really knows what we're doing. And I think that the moment we feel like we do is probably where we're in trouble. Because this is all just so new. Look where we were even 40 years, 30, even 20 years ago. We've come really far.Corey: For me, one of the things that concerns slash scares me has been that once someone learns something and it becomes rote, it sort of crystallizes in amber within their worldview, and they don't go back and figure out, “Okay, is this still the right approach?” Or, “Has the thing that I know changed?” And I see this on a constant basis just because I'm working with AWS so often. And there are restrictions and things you cannot do and constraints that the cloud provider imposes on you. Until one day, that thing that was impossible is now possible and supported.But people don't keep up with that so they still operate under the model of what used to be. I still remember a year or so after they raised the global per-resource tag limit to 50, I was seeing references to only ten tags being allowed per resource in the AWS console because not even internal service teams are allowed to talk to each other over there, apparently. And if they can't keep it straight internally, what hope to the rest of us have? It's the same problem of once you get this knowledge solidified, it's hard to keep current and adapt to things that are progressing. Especially in tech where things are advancing so rapidly and so quickly.Lex: Yeah, I gather things are a little feudalistic over inside AWS, although I've never worked there, so I don't know. But it's also just so big. I mean, there's just—like, do you even know all of the—like, I challenge you to go through the list of services. I bet you're going to find when you don't know about. You know, the AWS services. Maybe that's a challenge I would lose, but it's so hard to keep track of all this stuff with how fast it's changing that I don't blame people for not getting that.Corey: I would agree. We've long since passed the point where I can talk incredibly convincingly about AWS services that do not exist and not get called out on it by AWS employees. Because who would just go and make something up like that? That would be psychotic. No one in the right mind would do it.“Hi, I'm Corey, we haven't met yet. But you're going to remember this, whether I want you to or not because I make an impression on people. Oops.”Lex: Yeah. Mr. AWS Snark. You're exactly who I would expect to do that. And then there was Hunter, what's his name? The guy who made the—[singing] these are the many services of AWS—song. That was pretty great, too.Corey: Oh, yeah. Forrest Brazeal. He was great. I loved having him in the AWS community. And then he took a job, head of content over at Google Cloud. It's, well, suddenly, you can't very well make fun of AWS anymore, not without it taking a very different tone. So, I feel like that's our collective loss.Lex: Yeah, definitely. But yeah, I feel like we've done amazing things as a society, but the problem is that we're still, like, at the level of, we don't know how to program the VCR as far as, like, trying to run reliable services. It's really hard to build a complex system that, by its nature of being useful for customers, it must increase in complexity. Trying to run that reliably is hugely difficult and trying to do so profitably is almost impossible.And then I look at how hard that is and then I look at people trying to make self-driving cars. And I think that I will never set foot in one of those things until I see us getting good at running reliable services. Because if we can't do this with all of these people involved, how do I expect that a little car is going to be—that they're going to be able to produce a car that can drive and understand the complexities of navigating around and all the hazards that are involved to keep me safe.Corey: It's wild to me. The more I learned about the internet, the more surprised I am that any of it works at all. It's like, “Well, at least you're only using it for ridiculous things like cat pictures, right?” “Oh, no, no, no. We do emergency services and banking and insurance on top of that, too.” “Oh, good. I'm sure that won't end horribly one day.”Lex: Right? Yeah. I mean, you look at, like—you look at how much of a concerted effort towards safety they've had to put in, in the aviation industry to go from where they were in the '70s and '80s to where we are now where it's so incredibly safe. We haven't made that kind of full industry push toward reliability and safety. And it's going to have to happen soon as more and more of the services we're building are, exactly as you say, life-critical.Corey: Yeah, the idea of having this stuff be life-critical means you have to take a very different approach to it than you do when you're running, I don't know, Twitter for Pets. Though, I probably need a new fake reference startup now that Twitter for reality is becoming more bizarre than anything I can make up. But the idea that, “Well, our ad network needs to have the same rigor and discipline applied to it as the life support system,” maybe that's the wrong framing.Lex: Or maybe it's not. I keep finding instances of situations—maybe not necessarily ad networks, although I wouldn't put it past them—but situations where a system that we're dealing with becomes life-critical when we had no idea that it could possibly do. So, for example, a couple companies back, there was this billing situation where a vendor of ours accidentally nilled our customers incorrectly and wiped bank accounts, and real people were unable to make their mortgage payments and unable to, like, their bank accounts were empty, so they couldn't buy food. Like, that's starting to become life-critical and it all came down to a single, like, this could have been any outage at any company. And that's going to happen more and more, I think.Corey: I really want to thank you for taking time to speak with me. If people want to learn more, where's the best place for them to find you?Lex: sreweekly.com. You can subscribe there. Thank you so much for having me on. It has been a real treat.Corey: It really has. You'll have to come back and we'll find other topics to talk about, I'm sure, in the very near future. Thank you so much for your time. I appreciate it.Lex: Thanks.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.