Podcasts about amazon eks

AWS Morning Brief for the week of June 2nd, with Corey Quinn. Links:Amazon ECS increases container exit reason message to 1024 charactersAmazon GameLift Servers SDKs are now on GitHubAWS Cost Explorer now offers new Cost Comparison featureAWS DataSync simplifies and accelerates cross-cloud data transfersAWS Secrets Manager announces support for cost allocation tags for secretsCloudTrail Lake now supports event enrichment and expanded event sizeCost Optimization Hub now supports Savings Plans and reservations preferencesAmazon Aurora DSQL is now generally availableEnhance AI-assisted development with Amazon ECS, Amazon EKS and AWS Serverless MCP serverOpenSearch UI: Six months in review5 steps for building a VMware transition strategy for public sector customersCloud Repatriation is Getting ComplicatedSponsorThe Duckbill Group: https://www.duckbillgroup.com/Join us for Office Hours!https://www.duckbillgroup.com/officehours/

New Relic's Chief Customer Officer Arnaldo (Arnie) Lopez details how their observability platform helps 70,000+ customers monitor cloud performance through AWS infrastructure while introducing AI capabilities that simplify operations.Topics Include:Arnie Lopez is SVP, Chief Customer Officer at New Relic.Oversees pre-sales, post-sales, technical support, and enablement teams.New Relic University offers customer certifications.Founded in 2008, pioneered application performance monitoring (APM).Now offers "Observability 3.0" for full-stack visibility.Prevents interruptions during cloud migration and operations.Serves 70,000+ customers across various industries.16,000 enterprise-level paying customers.Platform consolidates multiple monitoring tools into one solution.Helps detect issues before customers experience performance problems.Market challenge: customers using disparate observability solutions.Reduces TCO by eliminating multiple monitoring tools.Targets VPs, CTOs, CIOs, and sometimes CEOs.Decade-long partnership with AWS.Platform built on largest unified telemetry data cloud.Uses AWS Graviton instances and Amazon EKS.AWS partnership enables innovation and customer trust.Three AI approaches: user assistance, LLM monitoring, faster insights.New Relic AI helps write query language (NURCLs).Monitors LLMs in customer environments.Uses AI to accelerate incident resolution.Lesson learned: should have started AI implementation sooner.Many customers still cautiously adopting AI technologies.Goal: continue growth with AWS partnership.Offers compute-based pricing model.Customers only pay for what they use.Announced one-step AWS monitoring for enterprise scale.Amazon Q Business and New Relic AI integration.Agent-to-agent AI eliminates data silos.Embeds performance insights into business application workflows.Participants:Arnie Lopez – SVP Chief Customer Officer, New RelicSee how Amazon Web Services gives you the freedom to migrate, innovate, and scale your software company at https://aws.amazon/isv/

Che cos'è Amazon EKS? Quali sono le novità di Amazon EKS presentate al re:Invent 2024? Cos'è Amazon EKS Auto Mode? Cosa sono gli Hybrid Nodes? Oggi ne parliamo con Alberto Crescini, Solutions Architect di Amazon Web Services.Link utili:- Amazon EKS Hybrid Nodes- Amazon EKS Auto Mode

Justin serves as Head of Product at Sidero Labs. His career includes contributions to Oscar-winning films, the Disney+ streaming platform, and Amazon EKS. In his free time, Justin enjoys building modern-retro computers and watching Moana. He is the co-host of the FAFO.FM podcast with Autumn Nash. In this Episode he talks to Scott about his love for Linux and the Linux Desktop

AWS Morning Brief for the week of February 24, with Corey Quinn. Links:Amazon ECS increases the CPU limit for ECS tasks to 192 vCPUsAmazon Q Developer now supports upgrade to Java 21AWS announces Backup Payment Methods for invoicesAWS CodePipeline adds native Amazon EKS deployment supportAWS Price List API supports AWS PrivateLinkAWS CloudFormation: 2024 Year in ReviewCost optimize your Minecraft Java EC2 ServerImproving Security in Amazon WorkMail with MFAUpdate on Support for Amazon ChimeBest practices to respond to security risks across your AWS OrganizationsReduce IT costs by implementing automatic shutdown for Amazon EC2 instancesHow to restrict Amazon S3 bucket access to a specific IAM roleIntroducing the AWS Trust CenterIs AWS Delivering on Its 3-Layer Approach to AI?

AWS re:Invent 2024 included big updates for Kubernetes on AWS. Host Daniel Newman is with Amazon Web Services' Barry Cooks, Vice President of AWS Kubernetes, for an insightful conversation on the future of Kubernetes on AWS on this episode of the Six Five On The Road at AWS re:Invent. They explore the latest innovations and the strategic vision shaping the growth of Kubernetes services within the AWS ecosystem. Tune in for more on ⬇️ The announcements of Amazon EKS Hybrid Nodes, Amazon EKS Auto Mode, and other cutting-edge developments Open-source collaboration: How AWS champions the Kubernetes community and contributes to its growth The roadmap for Kubernetes on AWS: A glimpse into the exciting future of this powerful technology  

AWS Morning Brief for the week of November 18, with Corey Quinn. Links:Buy a shirt benefiting 826 National!Amazon DataZone updates pricing and removes the user-level subscription feeAmazon DynamoDB reduces prices for on-demand throughput and global tablesAmazon DynamoDB introduces warm throughput for tables and indexesAmazon EBS now supports detailed performance statistics on EBS volume healthAmazon Q Developer plugins for Datadog and Wiz now generally availableAmazon S3 now supports up to 1 million buckets per AWS accountAWS Backup now supports copying Amazon S3 backups across Regions and accounts in opt-in RegionsAWS CloudTrail Lake announces enhanced event filteringHow and why you should move to Cost and Usage Report (CUR) 2.0?AWS BuilderCards second edition at re:Invent 2024Accelerate your third-party Amazon EKS add-on onboarding using ConformitronPython 3.13 runtime now available in AWS LambdaDeploy the Cost Optimizer for Amazon WorkSpaces in a highly-regulated environment.Introducing the Live Event Framework: Live Streaming with Ad Insertion on AWSIntroducing kro: Kube Resource OrchestratorAWS Snow device updates

This week, we discuss IBM's intent to acquire HashiCorp, the state of Open Source Businesses, and the (slow) adoption of Continuous Integration. Plus, some thoughts on the end of non-compete agreements. Watch the YouTube Live Recording of Episode (https://www.youtube.com/watch?v=JYXl62_VMX0) 465 (https://www.youtube.com/watch?v=JYXl62_VMX0) Runner-up Titles Extra Innings Later Innings Customer is always right, except for pricing Leave the party crying Put a price on it Rundown Hashi Introducing The Infrastructure Cloud (https://www.hashicorp.com/blog/introducing-the-infrastructure-cloud) HashiCorp unveils The Infrastructure Cloud, a unified platform for cloud Infrastructure and Security Lifecycle Management (https://www.globenewswire.com/news-release/2024/04/22/2866901/0/en/HashiCorp-unveils-The-Infrastructure-Cloud-a-unified-platform-for-cloud-Infrastructure-and-Security-Lifecycle-Management.html) IBM to Acquire HashiCorp, Inc. Creating a Comprehensive End-to-End Hybrid Cloud Platform (https://www.prnewswire.com/news-releases/ibm-to-acquire-hashicorp-inc-creating-a-comprehensive-end-to-end-hybrid-cloud-platform-302126646.html) IBM to acquire Hashi for $6.4 billion, seeks software boost (https://www.theregister.com/2024/04/25/ibm_q1_2024/) IBM falls as enterprise-spending constraints choke consulting demand (https://finance.yahoo.com/news/ibm-falls-enterprise-spending-constraints-115210911.html) IBM Is Buying HashiCorp. What Comes Next? (https://www.forbes.com/sites/justinwarren/2024/04/26/ibm-is-buying-hashicorp-what-comes-next/) The threat to open source comes from within (https://newsletter.goodtechthings.com/p/the-threat-to-open-source-comes-from) You should automate your builds and tests - 71% of people do not “use continuous integration to automatically build and test my code changes.” (https://newsletter.cote.io/p/you-should-automate-your-builds-and) FTC Announces Rule Banning Noncompetes (https://www.ftc.gov/news-events/news/press-releases/2024/04/ftc-announces-rule-banning-noncompetes) Relevant to your Interests AlmaLinux 9.4 beta: RHEL compatible, but a little different (https://www.theregister.com/2024/04/17/almalinux_94_ciq_lts_kernels/) Building for our AI future (https://blog.google/inside-google/company-announcements/building-ai-future-april-2024/) Lacework, last valued at $8.3B, is in talks to sell for just $150M to $300M, say sources (https://techcrunch.com/2024/04/18/wiz-is-in-talks-to-buy-lacework-for-150-200m-security-firm-was-last-valued-at-8-3b/) Amazon recently bought a $650 million nuclear-powered data center (https://x.com/finmoorhouse/status/1781022862482059444) Lacework, last valued at $8.3B, is in talks to sell for just $150M to $300M, say sources (https://techcrunch.com/2024/04/18/wiz-is-in-talks-to-buy-lacework-for-150-200m-security-firm-was-last-valued-at-8-3b/) Netflix Dealt With the Freeloaders. Its Next Act Will Be Tougher. (https://www.wsj.com/business/media/netflix-earnings-analysis-23b12db8) Tesla recalls the Cybertruck for faulty accelerator pedals that can get stuck (https://techcrunch.com/2024/04/19/tesla-cybertruck-throttle-accelerator-pedal-stuck/) Do software companies actually have good margins? (https://benn.substack.com/p/do-software-companies-actually-have) Women Who Code: Influential tech network shuts down unexpectedly (https://www.bbc.com/news/articles/cw0769446nyo) The invisible seafaring industry that keeps the internet afloat (https://www.theverge.com/c/24070570/internet-cables-undersea-deep-repair-ships) Tesla's Q&A with investors rips open Musk anguish: ‘Will you please at least appear to make Tesla your top priority?' (https://fortune.com/2024/04/19/teslas-investors-musk-anguish-conduct-x-erratic/) The SignalFire State of Talent Report: 2023 tech employee trends (https://www.signalfire.com/blog/state-of-talent-tech-trends) Appeals court rules that cops can physically make you unlock your phone (https://reason.com/2024/04/19/appeals-court-rules-that-cops-can-physically-make-you-unlock-your-phone/) Improve cost visibility of Amazon EKS with AWS Split Cost Allocation Data | Amazon Web Services (https://aws.amazon.com/blogs/aws-cloud-financial-management/improve-cost-visibility-of-amazon-eks-with-aws-split-cost-allocation-data/) Introducing Our Open Mixed Reality Ecosystem | Meta (https://about.fb.com/news/2024/04/introducing-our-open-mixed-reality-ecosystem/) US is reviewing risks of China's use of RISC-V chip technology (https://www.reuters.com/technology/us-is-reviewing-risks-chinas-use-risc-v-chip-technology-2024-04-23/) Google's First Tensor Processing Unit : Origins (https://open.substack.com/pub/thechipletter/p/googles-first-tensor-processing-unit?utm_campaign=post&utm_medium=web) UnitedHealth says Change hackers stole health data on 'substantial proportion of people in America' | TechCrunch (https://techcrunch.com/2024/04/22/unitedhealth-change-healthcare-hackers-substantial-proportion-americans/) Broadcom Tells Partner Negotiating For Charity ‘VMware Is Not For Everybody' (https://www.crn.com/news/virtualization/2024/broadcom-tells-partner-negotiating-for-charity-vmware-is-not-for-everybody) Oracle is moving its world headquarters to Nashville to be closer to health-care industry (https://www.cnbc.com/2024/04/23/oracle-is-moving-its-world-hq-to-nashville.html) Congress Passed a Bill That Could Ban TikTok. Now Comes the Hard Part. (https://www.nytimes.com/2024/04/23/technology/tiktok-ban-bill-congress.html) The Coca-Cola Company and Microsoft announce five-year strategic partnership to accelerate cloud and generative AI initiatives - Stories (https://news.microsoft.com/2024/04/23/the-coca-cola-company-and-microsoft-announce-five-year-strategic-partnership-to-accelerate-cloud-and-generative-ai-initiatives/) Atlassian Co-CEO Scott Farquhar Resigns, Leaving Mike Cannon-Brookes as Sole Chief (https://www.bloomberg.com/news/articles/2024-04-25/atlassian-team-co-ceo-farquhar-resigns-leaving-cannon-brookes-as-sole-chief) Thoma Bravo to buy UK-listed Darktrace for £4.3bn (https://www.ft.com/content/44b9884b-0b7b-4cb7-b372-0b390ed96947) All we have to fear is FUD itself — Oxide and Friends (https://overcast.fm/+4jBHj8QDI) The walls of Apple's garden are tumbling down (https://www.theverge.com/24141929/apple-iphone-imessage-antitrust-dma-lock-in) We're moving continuous integration back to developer machines (https://world.hey.com/dhh/we-re-moving-continuous-integration-back-to-developer-machines-3ac6c611) Enterprise Browser Island Receives Capital at $3 Billion Value (https://www.bloomberg.com/news/articles/2024-04-30/enterprise-browser-island-receives-capital-at-3-billion-value) Kubernetes Market Sizing Windmills (https://newsletter.cote.io/p/kubernetes-market-sizing-windmills?utm_source=post-email-title&publication_id=50&post_id=144035534&utm_campaign=email-post-title&isFreemail=true&r=2l9&triedRedirect=true&utm_medium=email) FTC Announces Rule Banning Noncompetes (https://www.ftc.gov/news-events/news/press-releases/2024/04/ftc-announces-rule-banning-noncompetes) Amazon cloud unit kills Snowmobile data transfer truck eight years after driving 18-wheeler onstage (https://www.cnbc.com/2024/04/17/aws-stops-selling-snowmobile-truck-for-cloud-migrations.html) The Port State of Platform Engineering in two surveys (https://newsletter.cote.io/p/the-port-state-of-platform-engineering?utm_source=post-email-title&publication_id=50&post_id=143633003&utm_campaign=email-post-title&isFreemail=true&r=2l9&triedRedirect=true&utm_medium=email) Cloud native platforms: To build or to buy? (https://www.cio.com/article/2091709/cloud-native-platforms-to-build-or-to-buy.html) State of DevSecOps | Datadog (https://www.datadoghq.com/state-of-devsecops/) Letter from Edward Norton, Founder of Zeck (https://www.zeck.app/letter-from-edward-norton-founder-of-zeck), Why Zeck (https://www.zeck.app/why-zeck) The Man Who Killed Google Search (https://www.wheresyoured.at/the-men-who-killed-google/) Alphabet earnings are out — here are the numbers (https://www.cnbc.com/2024/04/25/alphabet-set-to-report-first-quarter-results-after-market-close.html) Alphabet stock surges on earnings beat, dividend announcement (https://finance.yahoo.com/news/alphabet-stock-surges-on-earnings-beat-dividend-announcement-142011040.html) Amazon earnings are out — here are the numbers (https://www.cnbc.com/2024/04/30/amazon-amzn-q1-earnings-report-2024.html) Microsoft earnings are out – here are the numbers (https://www.cnbc.com/2024/04/25/microsoft-msft-q3-earnings-2024.html) Nonsense Adam Neumann moves to buy back WeWork as it seeks funds to exit bankruptcy (https://finance.yahoo.com/news/adam-neumann-moves-buy-back-045708441.html?utm_source=newsletter&utm_medium=email&utm_campaign=newsletter_axiosprorata&stream=top) A Mansion, Two Dogs and a Wall: Inside The Conflict Between a Utah Billionaire And His Neighbors (https://www.bloomberg.com/news/articles/2024-04-17/cloudflare-billionaire-matthew-prince-fights-utah-locals-over-house-dogs-wall) Red Lobster Is Reportedly Heading For Bankruptcy After Losing Millions On Endless Shrimp (https://www.msn.com/en-us/money/companies/red-lobster-is-reportedly-heading-for-bankruptcy-after-losing-millions-on-endless-shrimp/ar-AA1nnPt7) Conferences Executive Dinner in Atlanta, May 22nd (https://sincusa.com/events/tanzu-atlanta-ga-dinner/) NDC Oslo (https://substack.com/redirect/8de3819c-db2b-47c8-bd7a-f0a40103de9e?j=eyJ1IjoiMmQ0byJ9.QKaKsDzwnXK5ipYhX0mLOvRP3vpk_3o2b5dd3FXmAkw), Coté speaking (https://substack.com/redirect/41e821af-36ba-4dbb-993c-20755d5f040a?j=eyJ1IjoiMmQ0byJ9.QKaKsDzwnXK5ipYhX0mLOvRP3vpk_3o2b5dd3FXmAkw), June 12th. DevOpsDays Amsterdam (https://devopsdays.org/events/2024-amsterdam/welcome/), June 19-21, 2024, Coté speaking. DevOpsDays Birmingham, August 19–21, 2024 (https://devopsdays.org/events/2024-birmingham-al/welcome/). SDT news & hype Join us in Slack (http://www.softwaredefinedtalk.com/slack). Get a SDT Sticker! Send your postal address to stickers@softwaredefinedtalk.com (mailto:stickers@softwaredefinedtalk.com) and we will send you free laptop stickers! Follow us: Twitch (https://www.twitch.tv/sdtpodcast), Twitter (https://twitter.com/softwaredeftalk), Instagram (https://www.instagram.com/softwaredefinedtalk/), Mastodon (https://hachyderm.io/@softwaredefinedtalk), BlueSky (https://bsky.app/profile/softwaredefinedtalk.com), LinkedIn (https://www.linkedin.com/company/software-defined-talk/), TikTok (https://www.tiktok.com/@softwaredefinedtalk), Threads (https://www.threads.net/@softwaredefinedtalk) and YouTube (https://www.youtube.com/channel/UCi3OJPV6h9tp-hbsGBLGsDQ/featured). Use the code SDT to get $20 off Coté's book, Digital WTF (https://leanpub.com/digitalwtf/c/sdt), so $5 total. Become a sponsor of Software Defined Talk (https://www.softwaredefinedtalk.com/ads)! Recommendations Brandon: MarkdownDown (https://markdowndown.vercel.app/) BBEdit's Extract and new feature blindness (https://leancrew.com/all-this/2016/05/bbedits-extract-and-new-feature-blindness/) Coté: Adam Savage Q&A (https://www.youtube.com/playlist?list=PLJtitKU0CAeg88RBY08TZkB7dcVmJLJLJ) Photo Credits Header (https://unsplash.com/photos/a-building-with-a-sign-on-the-side-of-it-4EnA55QfxKo) Artwork (https://unsplash.com/photos/two-burger-with-lettuce-and-tomato-ndNw_6QGR_c)

A virtual cluster, described by Loft Labs CEO Lukas Gentele at Kubecon+ CloudNativeCon Paris, is a Kubernetes control plane running inside a container within another Kubernetes cluster. In this New Stack Makers episode, Gentele explained that this approach eliminates the need for numerous separate control planes, allowing VMs to run in lightweight, quickly deployable containers. Loft Labs' open-sourced vcluster technology enables virtual clusters to spin up in about six seconds, significantly faster than traditional Kubernetes clusters that can take over 30 minutes to start in services like Amazon EKS or Google GKE.The integration of vCluster into Rancher at KubeCon Paris enables users to manage virtual clusters alongside real clusters seamlessly. This innovation addresses challenges faced by companies managing multiple applications and clusters, advocating for a multi-tenant cluster approach for improved sharing and security, contrary to the trend of isolated single-tenant clusters that emerged due to complexities in cluster sharing within Kubernetes. Learn more from The New Stack about virtual clusters: Vcluster to the Rescue Navigating the Trade-Offs of Scaling Kubernetes Dev Environments Managing Kubernetes Clusters for Platform Engineers Join our community of newsletter subscribers to stay on top of the news and at the top of your game. https://thenewstack.io/newsletter/

AWS Morning Brief for the week of April 22, 2024, with Corey Quinn. Links:AWS IAM Identity Center adds independent 90-days session duration for Amazon CodeWhisperer Deloitte and AWS Strategic Collaboration to Accelerate Cloud Adoption in Growth MarketsImprove cost visibility of Amazon EKS with AWS Split Cost Allocation Data Congratulations to the PartyRock generative AI hackathon winners Access Amazon RDS across AWS accounts using AWS PrivateLink, Network Load Balancer, and Amazon RDS ProxyProgrammatic approach to optimize the cost of Amazon RDS snapshots Reduce cost and improve performance by migrating to Amazon DocumentDB 5.0A secure approach to generative AI with AWS AWS celebrates big technology wins at NAB 2024 New AWS survey reveals the link between AI fluency and the next education revolutionCVE-2024-28056Creating shortcut links to AWS Management Console destinations - AWS IAM Identity Center 

Discover how athenahealth, a healthcare technology leader, accelerates hybrid cloud deployment using Amazon EKS, AWS Local Zones, and AWS Outposts. Explore their innovative digital transformation strategy empowering developers to leverage cloud benefits across diverse environments.

AWS Morning Brief for the week of April 8, 2024, with Corey Quinn. Links:Amazon GuardDuty EC2 Runtime Monitoring is now generally availableAmazon SageMaker Canvas announces new pricing for training tabular models Introducing AWS CodeConnections, formerly known as AWS CodeStar Connections -Amazon CloudWatch now supports cross-account anomaly detection Amazon EKS extended support for Kubernetes versions now generallyAnnouncing AWS Deadline Cloud AWS Console Mobile Application adds support for CloudWatch custom dashboardsAWS Lambda adds support for Ruby 3.3 Announcing general availability of Amazon EC2 G6 instances Announcing per-second billing for EC2 Red Hat Enterprise Linux (RHEL)-based instances Why AWS Supports ValkeyAWS Activate credits now accepted for third-party models on Amazon Bedrock DoorDash saves millions annually using Amazon S3 Storage Lens 

In this episode, find out how Mobileye migrated their massive scale Spark workloads for autonomous vehicle mapping to Amazon Elastic Kubernetes Service (EKS), the challenges they overcame, the cultural shifts required, and the impressive performance gains they achieved by adopting Kubernetes.

En este nuevo episodio sobre Contenedores en AWS conversamos con David Ugarte sobre los lanzamientos más destacados del año 2023 para los servicios de contenedores: Amazon EKS, Amazon ECS, Amazon ECR, Fargate y mucho más ¡No se lo pierdan! Material Adicional: https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-guardduty-ecs-runtime-monitoring-fargate/

Covering how Karpenter extends Kubernetes' native autoscaling functionality, why it is so useful for Kubernetes users, and where it makes sense to use it.

In the first episode of Season 4 for the Kubernetes Bytes podcast, Bhavin sits down with Ahmed Bebars, Staff Software Engineer at NYTimes to talk about how the times uses Kubernetes and Platform Engineering to accelerate their developer productivity and improve developer experience. They talk about what the technology stack at NYTimes looks like, how the platform team has built a resilient platform on Amazon EKS and share some best practices for anyone starting their journey with Platform Engineering.Check out our website at https://kubernetesbytes.com/Timestamps: 01:20 Cloud Native News 09:55 Interview with Ahmed 50:04 Key takeawaysCloud Native News: https://www.aquasec.com/news/60m-additional-funding/ https://devclass.com/2023/12/12/docker-buys-atomicjar-to-integrate-container-based-test-automation/ https://www.businessinsider.com/armory-acquired-startup-harness-7-million-2023-12 https://techcrunch.com/2023/12/19/scaleops-looks-to-cut-cloud-bills-by-automating-kubernetes-configurations https://wraltechwire.com/2023/12/21/ciscos-latest-cloud-play-exec-explains-the-deal-for-tech-startup-isovalent/ https://www.kubernetes.dev/resources/release/Show links: https://open.nytimes.com/ https://tickets.kcdnewyork.com/ https://github.com/abebars https://www.linkedin.com/in/ahmedbebars/ https://github.com/nytimes

Robert Ross, CEO and Co-Founder at FireHydrant, joins Corey on Screaming in the Cloud to discuss how being an on-call engineer fighting incidents inspired him to start his own company. Robert explains how FireHydrant does more than just notify engineers of an incident, but also helps them to be able to effectively put out the fire. Robert tells the story of how he “accidentally” started a company as a result of a particularly critical late-night incident, and why his end goal at FireHydrant has been and will continue to be solving the problem, not simply choosing an exit strategy. Corey and Robert also discuss the value and pricing models of other incident-reporting solutions and Robert shares why he feels surprised that nobody else has taken the same approach FireHydrant has. About RobertRobert Ross is a recovering on-call engineer, and the CEO and co-founder at FireHydrant. As the co-founder of FireHydrant, Robert plays a central role in optimizing incident response and ensuring software system reliability for customers. Prior to founding FireHydrant, Robert previously contributed his expertise to renowned companies like Namely and Digital Ocean. Links Referenced: FireHydrant: https://firehydrant.com/ Twitter: https://twitter.com/bobbytables TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Developers are responsible for more than ever these days. Not just the code they write, but also the containers and cloud infrastructure their apps run on. And a big part of that responsibility is app security — from code to cloud. That's where Snyk comes in. Snyk is a frictionless security platform that meets teams where they are, automating application security controls across their existing tools, workflows, and the AWS application stack — including seamless integrations with AWS CodePipeline, Amazon EKS, Amazon Inspector and several others. I'm a customer myself. Deploy on AWS. Secure with Snyk. Learn more at snyk.co/scream. That's S-N-Y-K-dot-C-O/scream.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. And this featured guest episode is brought to us by our friends at FireHydrant and for better or worse, they've also brought us their CEO and co-founder, Robert Ross, better known online as Bobby Tables. Robert, thank you for joining us.Robert: Super happy to be here. Thanks for having me.Corey: Now, this is the problem that I tend to have when I've been tracking companies for a while, where you were one of the only people that I knew of at FireHydrant. And you kind of still are, so it's easy for me to imagine that, oh, it's basically your own side project that turned into a real job, sort of, side hustle that's basically you and maybe a virtual assistant or someone. I have it on good authority—and it was also signaled by your Series B—that there might be more than just you over there now.Robert: Yes, that's true. There's a little over 60 people now at the company, which is a little mind-boggling for me, starting from side projects, building this in Starbucks to actually having people using the thing and being on payroll. So, a little bit of a crazy thing for me. But yes, over 60.Corey: So, I have to ask, what is it you folks do? When you say ‘fire hydrant,' the first thing that I think I was when I was a kid getting yelled at by the firefighter for messing around with something I probably shouldn't have been messing around with.Robert: So, it's actually very similar where I started it because I was messing around with software in ways I probably shouldn't have and needed a fire hydrant to help put out all the fires that I was fighting as an on-call engineer. So, the name kind of comes from what do you need when you're putting out a fire? A fire hydrant. So, what we do is we help people respond to incidents really quickly, manage them from ring to retro. So, the moment you declare an incident, we'll do all the timeline tracking and eventually help you create a retrospective at the very end. And it's been a labor of love because all of that was really painful for me as an engineer.Corey: One of the things that I used to believe was that every company did something like this—and maybe they do, maybe they don't—I'm noticing these days an increasing number of public companies will never admit to an incident that very clearly ruined things for their customers. I'm not sure if they're going to talk privately to customers under NDAs and whatnot, but it feels like we're leaving an era where it was an expectation that when you had a big issue, you would do an entire public postmortem explaining what had happened. Is that just because I'm not paying attention to the right folks anymore, or are you seeing a downturn in that?Robert: I think that people are skittish of talking about how much reliability they—or issues they may have because we're having this weird moment where people want to open more incidents like the engineers actually want to say we have more incidents and officially declare those, and in the past, we had these, like, shadow incidents that we weren't officially going to say it was an incident, but was a pretty big deal, but we're not going to have a retro on it so it's like it didn't happen. And kind of splitting the line between what's a SEV1, when should we actually talk about this publicly, I think companies are still trying to figure that out. And then I think there's also opposing forces. We talk to folks and it's, you know, public relations will sometimes get involved. My general advice is, like, you should be probably talking about it no matter what. That's how you build trust.It's trust, with incidences, lost in buckets and gained back in drops, so you should be more public about it. And I think my favorite example is a major CDN had a major incident and it took down, like, the UK government website. And folks can probably figure out who I'm talking about, but their stock went up the next day. You would think that a major incident taking down a large portion of the internet would cause your stock to go down. Not the case. They were on it like crazy, they communicated about it like crazy, and lo and behold, you know, people were actually pretty okay with it as far as they could be at the end of the day.Corey: The honest thing that really struck me about that was I didn't realize that CDN that you're referencing was as broadly deployed as it was. Amazon.com took some downtime as a result of this.Robert: Yeah.Corey: It's, “Oh, wow. If they're in that many places, I should be taking them more seriously,” was my takeaway. And again, I don't tend to shame folks for incidents because as soon as you do that, they stopped talking about them. They still have them, but then we all lose the ability to learn from them. I couldn't help but notice that the week that we're recording this, so there was an incident report put out by AWS for a Lambda service event in Northern Virginia.It happened back in June, we're recording this late in October. So, it took them a little bit of time to wind up getting it out the door, but it's very thorough, very interesting as far as what it talks about as far as their own approach to things. Because otherwise, I have to say, it is easy as a spectator slash frustrated customer to assume the absolute worst. Like, you're sitting around there and like, “Well, we have a 15-minute SLA on this, so I'm going to sit around for 12 minutes and finish my game of solitaire before I answer the phone.” No, it does not work that way. People are scrambling behind the scenes because as systems get more complicated, understanding the interdependencies of your own system becomes monstrous.I still remember some of the very early production engineering jobs that I had where—to what you said a few minutes ago—oh, yeah, we'll just open an incident for every alert that goes off. Then we dropped a [core switch 00:05:47] and Nagio sent something like 8000 messages inside of two minutes. And we would still, 15 years later, not be done working through that incident backlog had we done such a thing. All of this stuff gets way harder than you would expect as soon as your application or environment becomes somewhat complicated. And that happens before you realize it.Robert: Yeah, much faster. I think that, in my experience, there's a moment that happens for companies where maybe it's the number of customers you have, number of servers you're running in production, that you have this, like, “Oh, we're running a big workload right now in a very complex system that impacts people's lives, frankly.” And the moment that companies realize that is when you start to see, like, oh, process change, you build it, you own it, now we have an SRE team. Like, there's this catalyst that happens in all of these companies that triggers this. And it's—I don't know, from my perspective, it's coming at a faster rate than people probably realize.Corey: From my perspective, I have to ask you this question, and my apologies in advance if it's one of those irreverent ones, but do you consider yourself to be an observability company?Robert: Oh, great question. No. No, actually. We think that we are the baton handoff between an observability tool and our platform. So, for example, we think that that's a good way to kind of, you know, as they say, monitor the system, give reports on that system, and we are the tool that based on that monitor may be going off, you need to do something about it.So, for example, I think of it as like a smoke detector in some cases. Like, in our world, like that's—the smoke detector is the thing that's kind of watching the system and if something's wrong, it's going to tell you. But at that point, it doesn't really do anything that's going to help you in the next phase, which is managing the incident, calling 911, driving to the scene of the fire, whatever analogies you want to use. But I think the value-add for the observability tools and what they're delivering for businesses is different than ours, but we touch each other, like, very much so.Corey: Managing an incident when something happens and diagnosing what is the actual root cause of it, so to speak—quote-unquote, “Root cause.” I know people have very strong opinions on—Robert: Yeah, say the word [laugh].Corey: —that phrase—exactly—it just doesn't sound that hard. It is not that complicated. It's, more or less, a bunch of engineers who don't know what they're actually doing, and why are they running around chasing this stuff down is often the philosophy of a lot of folks who have never been in the trenches dealing with these incidents themselves. I know this because before I was exposed to scale, that's what I thought and then, oh, this is way harder than you would believe. Now, for better or worse, an awful lot of your customers and the executives at those customers did, for some strange reason, not come up through production engineering as the thing that they've done. They are executives, so it feels like it would be a challenging conversation to have with them, but one thing that you've got in your back pocket, which I always love talking to folks about, is before this, you were an engineer and then you became a CEO of a reasonably-sized company. That is a very difficult transition. Tell me about it.Robert: Yeah. Yeah, so a little of that background. I mean, I started writing code—I've been writing code for two-thirds of my life. So, I'm 32 now; I'm relatively young. And my first job out of high school—skipping college entirely—was writing code. I was 18, I was working in a web dev shop, I was making good enough money and I said, you know what? I don't want to go to college. That sounds—I'm making money. Why would I go to college?And I think it was a good decision because I got to be able—I was right kind of in the centerpiece of when a lot of really cool software things were happening. Like, DevOps was becoming a really cool term and we were seeing the cloud kind of emerge at this time and become much more popular. And it was a good opportunity to see all this confluence of technology and people and processes emerge into what is, kind of like, the base plate for a lot of how we build software today, starting in 2008 and 2009. And because I was an on-call engineer during a lot of that, and building the systems as well, that I was on call for, it meant that I had a front-row seat to being an engineer that was building things that was then breaking, and then literally merging on GitHub and then five minutes later [laugh], seeing my phone light up with an alert from our alerting tool. Like, I got to feel the entire process.And I think that that was nice because eventually one day, I snapped. And it was after a major incident, I snapped and I said, “There's no tool that helps me during this incident. There's no tool that kind of helps me run a process for me.” Because the only thing I care about in the middle of the night is going back to bed. I don't have any other priority [laugh] at 2 a.m.So, I wanted to solve the problem of getting to the fire faster and extinguishing it by automating as much as I possibly could. The process that was given to me in an outdated Confluence page or Google Doc, whatever it was, I wanted to automate that part so I could do the thing that I was good at as an engineer: put out the fire, take some notes, and then go back to bed, and then do a retrospective sometime next day or in that week. And it was a good way to kind of feel the problem, try to build a solution for it, tweak a little bit, and then it kind of became a company. I joke and I say on accident, actually.Corey: I'll never forget one of the first big, hairy incidents that I had to deal with in 2009, where my coworker had just finished migrating the production environment over to LDAP on a Thursday afternoon and then stepped out for a three-day weekend, and half an hour later, everything started exploding because LDAP will do that. And I only had the vaguest idea of how LDAP worked at all. This was a year into my first Linux admin job; I'd been a Unix admin before that. And I suddenly have the literal CEO of the company breathing down my neck behind me trying to figure out what's going on and I have no freaking idea of myself. And it was… feels like there's got to be a better way to handle these things.We got through. We wound up getting it back online, no one lost their job over it, but it was definitely a touch-and-go series of hours there. And that was a painful thing. And you and I went in very different directions based upon experiences like that. I took a few more jobs where I had even worse on-call schedules than I would have believed possible until I started this place, which very intentionally is centered around a business problem that only exists during business hours. There is no 2 a.m. AWS billing emergency.There might be a security issue masquerading as one of those, but you don't need to reach me out of business hours because anything that is a billing problem will be solved in Seattle's timeline over a period of weeks. You leaned into it and decided, oh, I'm going to start a company to fix all of this. And okay, on some level, some wit that used to work here, wound up once remarking that when an SRE doesn't have a better idea, they start a monitoring company.Robert: [laugh].Corey: And, on some level, there's some validity to it because this is the problem that I know, and I want to fix it. But you've differentiated yourself in a few key ways. As you said earlier, you're not an observability company. Good for you.Robert: Yeah. That's a funny quote.Corey: Pete Cheslock. He has a certain way with words.Robert: Yeah [laugh]. I think that when we started the company, it was—we kind of accidentally secured funding five years ago. And it was because this genuinely was something I just, I bought a laptop for because I wanted to own the IP. I always made sure I was on a different network, if I was going to work on the company and the tool. And I was just writing code because I just wanted to solve the problem.And then some crazy situation happened where, like, an investor somehow found FireHydrant because they were like, “Oh, this SRE thing is a big space and incidents is a big part of it.” And we got to talking and they were like, “Hey, we think what you're building is valuable and we think you should build a company here.” And I was—like, you know, the Jim Carrey movie, Yes Man? Like, that was kind of me in that moment. I was like, “Sure.” And here we are five years later. But I think the way that we approached the problem was let's just solve our own problem and let's just build a company that we want to work at.And you know, I had two co-founders join me in late 2018 and that's what we told ourselves. We said, like, “Let's build a company that we want to work for, that solves problems that we have had, that we care about solving.” And I think it's worked out, you know? We work with amazing companies that use our tool—much to their chagrin [laugh]—multiple times a day. It's kind of a problem when you build an incident response tool is that it's a good thing when people are using it, but a bad thing for them.Corey: I have to ask of all of the different angles to approach this from, you went with incident management as opposed to focusing on something that is more purely technical. And I don't say that in any way that is intended to be sounding insulting, but it's easier from an engineering mind to—having been one myself—to come up with, “Here's how I make one computer talk to his other computer when the following event happens.” That's a much easier problem by orders of magnitude than here's how I corral the humans interacting with that computer's failure to talk to another computer in just the right way. How did you get onto this path?Robert: Yeah. The problem that we were trying to solve for it was the getting the right people in the room problem. We think that building services that people own is the right way to build applications that are reliable and stable and easier to iterate on. Put the right people that build that software, give them, like, the skin in the game of also being on call. And what that meant for us is that we could build a tool that allowed people to do that a lot easier where allowing people to corral the right people by saying, “This service is broken, which powers this functionality, which means that these are the people that should get involved in this incident as fast as possible.”And the way we approached that is we just built up part of our functionality called Runbooks, where you can say, “When this happens, do this.” And it's catered for incidents. So, there's other tools out there, you can kind of think of as, like, we're a workflow tool, like Zapier, or just things that, like, fire webhooks at services you build and that ends up being your incident process. But for us, we wanted to make it, like, a really easy way that a project manager could help define the process in our tool. And when you click the button and say, “Declare Incident: LDAP is Broken,” and I have a CEO standing behind me, our tool just would corral the people for you.It was kind of like a bat signal in the air, where it was like, “Hey, there's this issue. I've run all the other process. I just need you to arrive at and help solve this problem.” And we think of it as, like, how can FireHydrant be a mech suit for the team that owns incidents and is responsible for resolving them?Corey: There are a few easier ways to make a product sound absolutely ridiculous than to try and pitch it to a problem that it is not designed to scale to. What is the ‘you must be at least this tall to ride' envisioning for FireHydrant? How large slash complex of an organization do you need to be before this starts to make sense? Because I promise, as one person with a single website that gets no hits, that is probably not the best place for—Robert: Probably not.Corey: To imagine your ideal user persona.Robert: Well, I'm sure you get way more hits than that. Come on [laugh].Corey: It depends on how controversial I'm being in a given week.Robert: Yeah [laugh].Corey: Also, I have several ridiculous, nonsense apps out there, but honestly, those are for fun. I don't charge people for them, so they can deal with my downtime till I get around to it. That's the way it works.Robert: Or, like, spite-visiting your website. No it's—for us, we think that the ‘must be this tall' is when do you have, like, sufficiently complicated incidents? We tell folks, like, if you're a ten-person shop and you have incidents, you know, just use our free tier. Like, you need something that opens a Slack channel? Fine. Use our free tier or build something that hits the Slack API [unintelligible 00:18:18] channel. That's fine.But when you start to have a lot of people in the room and multiple pieces of functionality that can break and multiple people on call, that's when you probably need to start to invest in incident management. Because it is a return on investment, but there is, like, a minimum amount of incidents and process challenges that you need to have before that return on investment actually, I would say, comes to fruition. Because if you do think of, like, an incident that takes downtime, or you know, you're a retail company and you go down for, let's say, ten minutes, and your number of sales per hour is X, it's actually relatively simple for that type of company to understand, okay, this is how much impact we would need to have from an incident management tool for it to be valuable. And that waterline is actually way—it's way lower than I think a lot of people realize, but like you said, you know, if you have a few 100 visitors a day, it's probably not worth it. And I'll be honest there, you can use our free tier. That's fine.Corey: Which makes sense. It's challenging to wind up-sizing things appropriately. Whenever I look at a pricing page, there are two things that I look for. And incidentally, when I pull up someone's website, I first make a beeline for pricing because that is the best way I found for a lot of the marketing nonsense words to drop away and it get down to brass tacks. And the two things I want are free tier or zero-dollar trial that I can get started with right now because often it's two in the morning and I'm trying to see if this might solve a problem that I'm having.And I also look for the enterprise tier ‘contact us' because there are big companies that do not do anything that is not custom nor do they know how to sign a check that doesn't have two commas in it. And whatever is between those two, okay, that's good to look at to figure out what dimensions I'm expected to grow on and how to think about it, but those are the two tent poles. And you've got that, but pricing is always going to be a dark art. What I've been seeing across the industry. And if we put it under the broad realm of things that watch your site and alert you and help manage those things, there are an increasing number of, I guess what I want to call component vendors, where you'll wind up bolting together a couple dozen of these things together into an observability pipeline-style thing, and each component seems to be getting extortionately expensive.Most of the wake-up-in-the-middle-of-the-night services that will page you—and there are a number of them out there—at a spot check of these, they all cost more per month per user than Slack, the thing that most of us to end up living within. This stuff gets fiendishly expensive, fiendishly quickly, and at some point, you're looking at this going, “The outage is cheaper than avoiding the outage through all of these things. What are we doing here?” What's going on in the industry, other than ‘money printing machine stopped going brrr' in quite the same way?Robert: Yeah, I think that for alerting specifically, this is a big part of, like, the journey that we wanted to have in FireHydrant was like, we also want to help folks with the alerting piece. So, I'll focus on that, which is, I think that the industry around notifying people for incidents—texts, call, push notifications, emails, there's a bunch of different ways to do it—I think where it gets really crazy expensive as in this per-seat model that most of them seem to have landed on. And we're per-seat for, like, the core platform of FireHydrant—so you know, before people spite-visit FireHydrant, look at our pricing pitch—but we're per-seat there because the value there is, like, we're the full platform for the service catalog retrospectives, Runbooks, like, there's a whole other component of FireHydrant—status pages—but when it comes to alerting, like, in my opinion, that should be active user for a few reasons. I think that if you're going to have people responding to incidents and the value from us is making sure they get to that incident very quickly because we wake them up in the middle of the night, we text them, we call them we make their Hue lights turn red, whatever it is, then that's, like, the value that we're delivering at that moment in time, so that's how we should probably invoice you.And I think that what's happened is that the pricing for these companies, they haven't innovated on the product in a way that allows them to package that any differently. So, what's happened, I think, is that the packaging of these products has been almost restrictive in the way that they could change their pricing models because there's nothing much more to package on. It's like, cool there's an alerting aspect to this, but that's what people want to buy those tools for. They want to buy the tool so it wakes them up. But that tool is getting more expensive.There was even a price increase announced today for a big one [laugh] that I've been publicly critical of. That is crazy expensive for a tool that texts you and call you. And what peo—what's going on now are people are looking, they're looking at the pricing sheet for Twilio and going, “What the heck is going on?” Like, I—to send a text on Twilio in the United States is fractions of a penny and here we are paying $40 a user for that person to receive six texts that month because of a webhook that hit an HCP server and, like, it's supposed to call that person? That's kind of a crazy model if you think about it. Like, engineers are kind of going, “Wait a minute. What's up here?” Like, and when engineers start thinking, “I could build this on a weekend,” like, something's wrong, like, with that model. And I think that people are starting to think that way.Corey: Well engineers, to be fair, will think that about an awful lot of stuff.Robert: Anything. Yeah, they [laugh]—Corey: I've heard it said about Dropbox, Facebook, the internet—Robert: Oh, Dropbox is such a good one.Corey: BGP. Yeah okay, great. Let me know how that works out for you.Robert: What was that Dropbox comment on Hacker News years ago? Like, “Just set up NFS and host it that way and it's easy.” Right?Corey: Or rsync. Yeah—Robert: Yeah, it was rsync.Corey: What are you going to make with that? Like, who's going to buy that? Like, basically everyone for at least a time.Robert: And whether or not the engineers are right, I think is a different point.Corey: It's the condescension dismissal of everything that isn't writing the code that really galls, on some level.Robert: But I think when engineers are thinking about, like, “I could build this on a weekend,” like, that's a moment that you have an opportunity to provide the value in an innovative, maybe consolidated way. We want to be a tool that's your incident management ring to retro, right? You get paged in the middle of the night, we're going to wake you up, and when you open up your laptop, groggy-eyed, and like, you're about to start fighting this fire, FireHydrant's already done a lot of work. That's what we think is, like, the right model do this. And candidly, I have no idea why the other alerting tools in this space haven't done this. I've said that and people tend to nod in agreement and say like, “Yeah, it's been—it's kind of crazy how they haven't approached this problem yet.” And… I don't know, I want to solve that problem for folks.Corey: So, one thing that I have to ask, you've been teasing on the internet for a little bit now is something called Signals where you are expanding your product into the component that wakes people up in the middle of the night, which in isolation, fine, great, awesome. But there was a company whose sole stated purpose was to wake people up in the middle of the night, and then once they started doing some business things such as, oh I don't know, going public, they needed to expand beyond that to do a whole bunch of other things. But as a customer, no, no, no, you are the thing that wakes me up in the middle of the night. I don't want you to sprawl and grow into everything else because if you're going to have to pick a vendor that claims to do everything, well, I'll just stay with AWS because they already do that and it's one less throat to choke. What is that pressure that is driving companies that are spectacular at the one thing to expand into things that frankly, they don't have the chops to pull off? And why is this not you doing the same thing?Robert: Oh, man. The end of that question is such a good one and I like that. I'm not an economist. I'm not—like, that's… I don't know if I have a great comment on, like, why are people expanding into things that they don't know how to do. It seems to be, like, a common thing across the industry at a certain point—Corey: Especially particularly generative AI. “Oh, we've been experts in this for a long time.” “Yeah, I'm not that great at dodgeball, but you also don't see me mouthing off about how I've been great at it and doing it for 30 years, either.”Robert: Yeah. I mean, there was a couple ads during football games I watched. I'm like, “What is this AI thing that you just, like, tacked on the letter X to the end of your product line and now all of a sudden, it's AI?” I have plenty of rants that are good for a cocktail at some point, but as for us, I mean, we knew that we wanted to do alerting a long time ago, but it does have complications. Like, the problem with alerting is that it does have to be able to take a brutal punch to the face the moment that AWS us-east-2 goes down.Because at that moment in time, a lot of webhooks are coming your way to wake somebody up, right, for thousands of different companies. So, you do have to be able to take a very, very sufficient amount of volume instantaneously. So, that was one thing that kind of stopped us. In 2019 even, we wrote a product document about building an alerting tool and we kind of paused. And then we got really deep into incident management, and the thing that makes us feel very qualified now is that people are actually already integrating their alerting tools into FireHydrant today. This is a very common thing.In fact, most people are paying for a FireHydrant and an alerting tool. So, you can imagine that gets a little expensive when you have both. So, we said, well, let's help folks consolidate, let's help folks have a modern version of alerting, and let's build on top of something we've been doing very well already, which is incident management. And we ended up calling it Signals because we think that we should be able to receive a lot of signals in, do something correct with them, and then put a signal out and then transfer you into incident management. And yeah, we're are excited for it actually. It's been really cool to see it come together.Corey: There's something to be said for keeping it in a certain area of expertise. And people find it very strange when they reach out to my business partner and me asking, okay, so are you going to expand into Google Cloud or Azure or—increasingly, lately—Datadog—which has become a Fortune 500 board-level expense concern, which is kind of wild to me, but here we are—and asking if we're going to focus on that, and our answer is no because it's very… well, not very, but it is relatively easy to be the subject matter expert in a very specific, expensive, painful problem, but as soon as you start expanding that your messaging loses focus and it doesn't take long—since we do you view this as an inherent architectural problem—where we're saying, “We're the best cloud engineers and cloud architects in the world,” and then we're competing against basically everyone out there. And it costs more money a year for Accenture or Deloitte's marketing budget than we'll ever earn as a company in our entire lifetime, just because we are not externally boosted, we're not putting hundreds of people into the field. It's a lifestyle business that solves an expensive, painful problem for our customers. And that focus lends clarity. I don't like the current market pressure toward expansion and consolidation at the cost of everything, including it seems, customer trust.Robert: Yeah. That's a good point. I mean, I agree. I mean, when you see a company—and it's almost getting hard to think about what a company does based on their name as well. Like, names don't even mean anything for companies anymore. Like Datadog has expanded into a whole lot of things beyond data and if you think about some of the alerting tools out there that have names of, like, old devices that used to attach to our hips, that's just a different company name than what represents what they do.And I think for us, like, incidents, that's what we care about. That's what I know. I know how to help people manage incidents. I built software that broke—sometimes I was an arsonist—sometimes I was a firefighter, it really depends, but that's the thing that we're going to be good at and we're just going to keep building in that sphere.Corey: I think that there's a tipping point that starts to become pretty clear when companies focus away from innovating and growing and serving customers into revenue protection mode. And I think this is a cyclical force that is very hard to resist. But I can tell even having conversations like this with folks, when the way that a company goes about setting up one of these conversations with me, you came by yourself, not with a squadron of PR people, not with a whole giant list of talking points you wanted to go to, just, “Let's talk about this stuff. I'm interested in it.”As a company grows, that becomes more and more uncommon. Often, I'll see it at companies a third the size of yours, just because there's so much fear around everything we say must be spoken in such a way that it could never be taken in a negative way against us. That's not the failure mode. The failure mode is that no one listens to you or cares what you have to say. At some point, yeah, I get the shift, but damned if it doesn't always feel like it's depressing.Robert: Yeah. This is such great questions because I think that the way I think about it is, I care about the problem and if we solve the problem and we solve it well and people agree with us on our solution being a good way to solve that problem, then the revenue, like, happens because of that. I've gotten asked from, like, from VCs and customers, like, “What's your end goal with FireHydrant as the CEO of the company?” And what they're really asking is, like, “Do you want to IPO or be acquired?” That's always a question every single time.And my answer is, maybe, I don't know, philosophical, but it's, I think if we solve the problem, like, one of those will happen, but that's not the end goal. Because if I aim at that, we're going to come up short. It's like how they tell you to throw a ball, right? Like they don't say, aim at the glove. They say, like, aim behind the person.And that's what we want to do. We just want to aim at solving a problem and then the revenue will come. You have to be smart about it, right? It's not a field of dreams, like, if you build it, like, revenue arrives, but—so you do have to be conscious of the business and the operations and the model that you work within, but it should all be in service of building something that's valuable.Corey: I really want to thank you for taking the time to speak with me. If people want to learn more, where should they go to find you, other than, you know, to their most recent incident page?Robert: [laugh]. No, thanks for having me. So, to learn more about me, I mean, you can find me on Twitter on—or X. What do we call it now?Corey: I call it Twitter because I don't believe in deadnaming except when it's companies.Robert: Yeah [laugh]. twitter.com/bobbytables if you want to find me there. If you want to learn more about FireHydrant and what we're doing to help folks with incidents and incident response and all the fun things in there, it's firehydrant.com or firehydrant.io, but we'll redirect you to dot com.Corey: And we will, of course, put a link to all of that in the [show notes 00:33:10]. Thank you so much for taking the time to speak with me. It's deeply appreciated.Robert: Thank you for having me.Corey: Robert Ross, CEO and co-founder of FireHydrant. This featured guest episode has been brought to us by our friends at FireHydrant, and I'm Corey Quinn. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment that will never see the light of day because that crappy platform you're using is having an incident that they absolutely do not know how to manage effectively.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

This week, we review the major announcements from AWS re:Invent and discuss how the hyperscalers are embracing A.I. Plus, a few thoughts on children's chores. Watch the YouTube Live Recording of Episode (https://www.youtube.com/watch?v=q0xwqUis6xA) 443 (https://www.youtube.com/watch?v=q0xwqUis6xA) Runner-up Titles No Slack The Corporate Podcast. Quality of life stop Our roads diverge Eats a bag of llama Nobody wants to do a bake-off AI all the time Rundown AWS re:Invent Top announcements of AWS re:Invent 2023 | Amazon Web Services (https://aws.amazon.com/blogs/aws/top-announcements-of-aws-reinvent-2023/) Salesforce Inks Deal to Sell on Amazon Web Services' Marketplace (https://www.bloomberg.com/news/articles/2023-11-27/salesforce-to-sell-software-on-aws-marketplace-in-self-service-purchase-push#xj4y7vzkg) AWS Unveils Next Generation AWS-Designed Chips (https://press.aboutamazon.com/2023/11/aws-unveils-next-generation-aws-designed-chips) Join the preview for new memory-optimized, AWS Graviton4-powered Amazon EC2 instances (R8g) (https://aws.amazon.com/blogs/aws/join-the-preview-for-new-memory-optimized-aws-graviton4-powered-amazon-ec2-instances-r8g/) Announcing the new Amazon S3 Express One Zone high performance storage class (https://aws.amazon.com/blogs/aws/new-amazon-s3-express-one-zone-high-performance-storage-class/) AWS unveils new Trainium AI chip and Graviton 4, extends Nvidia partnership (https://www.zdnet.com/article/aws-unveils-new-trainium-ai-chip-and-graviton-4-extends-nvidia-partnership/) AI Chip - AWS Inferentia - AWS (https://aws.amazon.com/machine-learning/inferentia/) DGX Platform (https://www.nvidia.com/en-au/data-center/dgx-platform/) Foundational Models - Amazon Bedrock - AWS (https://aws.amazon.com/bedrock/) Supported models in Amazon Bedrock - Amazon Bedrock (https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html#models-supported-meta) Agents for Amazon Bedrock is now available with improved control of orchestration and visibility into reasoning (https://aws.amazon.com/blogs/aws/agents-for-amazon-bedrock-is-now-available-with-improved-control-of-orchestration-and-visibility-into-reasoning/) Knowledge Bases now delivers fully managed RAG experience in Amazon Bedrock (https://aws.amazon.com/blogs/aws/knowledge-bases-now-delivers-fully-managed-rag-experience-in-amazon-bedrock/) Customize models in Amazon Bedrock with your own data using fine-tuning and continued pre-training (https://aws.amazon.com/blogs/aws/customize-models-in-amazon-bedrock-with-your-own-data-using-fine-tuning-and-continued-pre-training/) Amazon Q brings generative AI-powered assistance to IT pros and developers (https://aws.amazon.com/blogs/aws/amazon-q-brings-generative-ai-powered-assistance-to-it-pros-and-developers-preview/) Improve developer productivity with generative-AI powered Amazon Q in Amazon CodeCatalyst (https://aws.amazon.com/blogs/aws/improve-developer-productivity-with-generative-ai-powered-amazon-q-in-amazon-codecatalyst-preview/) Upgrade your Java applications with Amazon Q Code Transformation (https://aws.amazon.com/blogs/aws/upgrade-your-java-applications-with-amazon-q-code-transformation-preview/) Introducing Amazon Q, a new generative AI-powered assistant (https://aws.amazon.com/blogs/aws/introducing-amazon-q-a-new-generative-ai-powered-assistant-preview/) New Amazon Q in QuickSight uses generative AI assistance for quicker, easier data insights (https://aws.amazon.com/blogs/aws/new-amazon-q-in-quicksight-uses-generative-ai-assistance-for-quicker-easier-data-insights-preview/) Amazon Managed Service for Prometheus collector provides agentless metric collection for Amazon EKS (https://aws.amazon.com/blogs/aws/amazon-managed-service-for-prometheus-collector-provides-agentless-metric-collection-for-amazon-eks/) Amazon CloudWatch Logs now offers automated pattern analytics and anomaly detection (https://aws.amazon.com/blogs/aws/amazon-cloudwatch-logs-now-offers-automated-pattern-analytics-and-anomaly-detection/) Use Amazon CloudWatch to consolidate hybrid, multicloud, and on-premises metrics (https://aws.amazon.com/blogs/aws/new-use-amazon-cloudwatch-to-consolidate-hybrid-multi-cloud-and-on-premises-metrics/) Amazon EKS Pod Identity simplifies IAM permissions for applications on Amazon EKS clusters (https://aws.amazon.com/blogs/aws/amazon-eks-pod-identity-simplifies-iam-permissions-for-applications-on-amazon-eks-clusters/) Amazon DynamoDB zero-ETL integration with Amazon OpenSearch Service is now available (https://aws.amazon.com/blogs/aws/amazon-dynamodb-zero-etl-integration-with-amazon-opensearch-service-is-now-generally-available/) Amazon says its first Project Kuiper internet satellites were fully successful in testing (https://www.cnbc.com/2023/11/16/amazon-kuiper-internet-satellites-fully-successful-in-testing.html) AWS takes the cheap shots (https://techcrunch.com/2023/11/28/aws-takes-the-cheap-shots/) Here's everything Amazon Web Services announced at AWS re:Invent (https://techcrunch.com/2023/11/28/heres-everything-aws-reinvent-2023-so-far/) Relevant to your Interests Oracle Cloud Made All The Right Moves In 2022 (https://moorinsightsstrategy.com/oracle-cloud-made-all-the-right-moves-in-2022/) Ransomware gang files SEC complaint over victim's undisclosed breach (https://www.bleepingcomputer.com/news/security/ransomware-gang-files-sec-complaint-over-victims-undisclosed-breach/) Keynote Highlights: Satya Nadella at Microsoft Ignite 2023 (https://www.youtube.com/watch?v=QMlUJqxhdoY) Thoma Bravo to sell about $500 million in Dynatrace stock (https://www.marketwatch.com/story/thoma-bravo-to-sell-about-500-million-in-dynatrace-stock-9d7bd0e6) FinOps Open Cost and Usage Specification 1.0-preview Released to Demystify Cloud Billing Data (https://www.prnewswire.com/news-releases/finops-open-cost-and-usage-specification-1-0-preview-released-to-demystify-cloud-billing-data-301990559.html?tc=eml_cleartime) AWS, Microsoft, Google and Oracle partner to make cloud spend more transparent | TechCrunch (https://techcrunch.com/2023/11/16/aws-microsoft-google-and-oracle-partner-to-make-cloud-spend-more-transparent/) Privacy is Priceless, but Signal is Expensive (https://signal.org/blog/signal-is-expensive/) Several popular AI products flagged as unsafe for kids by Common Sense Media | TechCrunch (https://techcrunch.com/2023/11/16/several-popular-ai-products-flagged-as-unsafe-for-kids-by-common-sense-media/) Amazon to sell Hyundai vehicles online starting in 2024 (https://finance.yahoo.com/news/amazon-sell-hyundai-vehicles-online-180500951.html) Amazon to launch car sales next year with Hyundai (https://news.google.com/articles/CBMiP2h0dHBzOi8vd3d3LmF4aW9zLmNvbS8yMDIzLzExLzE2L2FtYXpvbi1oeXVuZGFpLWNhcnMtc2FsZS1hbGV4YdIBAA?hl=en-US&gl=US&ceid=US%3Aen) Canonical Microcloud: Simple, free, on-prem Linux clustering (https://www.theregister.com/2023/11/16/canonical_microcloud/) Introducing the Functional Source License: Freedom without Free-riding (https://blog.sentry.io/introducing-the-functional-source-license-freedom-without-free-riding/) The Problems with Money In (Open Source) Software | Aneel Lakhani | Monktoberfest 2023 (https://www.youtube.com/watch?v=LTCuLyv6SHo) DXC Technology and AWS Take Their Strategic Partnership to the Next Level to Deliver the Future of Cloud for Customers (https://dxc.com/us/en/about-us/newsroom/press-releases/11202023) Broadcom and VMware Intend to Close Transaction on November 22, 2023 (https://www.businesswire.com/news/home/20231121379706/en/Broadcom-and-VMware-Intend-to-Close-Transaction-on-November-22-2023) Broadcom announces successful acquisition of VMware | Hock Tan (https://www.broadcom.com/blog/broadcom-announces-successful-acquisition-of-vmware) Broadcom closes $69 billion VMware deal after China approval (https://finance.yahoo.com/news/broadcom-closes-69-billion-vmware-133704461.html) VMware is now part of Broadcom | VMware by Broadcom (https://www.broadcom.com/info/vmware) Binance CEO Changpeng Zhao Reportedly Quits and Pleads Guilty to Breaking US Law (https://www.wired.com/story/binance-cz-ceo-quits-pleads-guilty-breaking-law/) Congrats To Elon Musk: I Didn't Think You Had It In You To File A Lawsuit This Stupid. But, You Crazy Bastard, You Did It! (https://www.techdirt.com/2023/11/21/congrats-to-elon-musk-i-didnt-think-you-had-it-in-you-to-file-a-lawsuit-this-stupid-but-you-crazy-bastard-you-did-it/) Hackers spent 2+ years looting secrets of chipmaker NXP before being detected (https://arstechnica.com/security/2023/11/hackers-spent-2-years-looting-secrets-of-chipmaker-nxp-before-being-detected/) Meet ‘Anna Boyko': How a Fake Speaker Blew up DevTernity (https://thenewstack.io/meet-anna-boyko-how-a-fake-speaker-blew-up-devternity/) IBM's Db2 database dinosaur comes to AWS (https://go.theregister.com/feed/www.theregister.com/2023/11/29/aws_launch_ibms_db2_database/) Reports of AI ending human labour may be greatly exaggerated (https://www.ecb.europa.eu/pub/economic-research/resbull/2023/html/ecb.rb231128~0a16e73d87.es.html) New Google geothermal electricity project could be a milestone for clean energy (https://apnews.com/article/geothermal-energy-heat-renewable-power-climate-5c97f86e62263d3a63d7c92c40f1330d) VMware's $92bn sale showers cash on Michael Dell and Silver Lake (https://www.ft.com/content/d01901a2-db4b-45df-8ce5-f57ff46d463e) Gartner Says Cloud Will Become a Business Necessity by 2028 (https://www.gartner.com/en/newsroom/press-releases/2023-11-29-gartner-says-cloud-will-become-a-business-necessity-by-2028) IRS starts the bidding for $1.9B IT services recompete (https://www.nextgov.com/acquisition/2023/11/irs-starts-bidding-19b-it-services-recompete/392303/) WSJ News Exclusive | Apple Pulls Plug on Goldman Credit-Card Partnership (https://www.wsj.com/finance/banking/apple-pulls-plug-on-goldman-credit-card-partnership-ca1dfb45) Apple employees most likely to leave to join Google shows LinkedIn (https://9to5mac.com/2023/11/23/apple-employees-next-jobs/) Ranked: Worst Companies for Employee Retention (U.S. and UK) (https://www.visualcapitalist.com/cp/ranked-worst-companies-for-employee-retention-u-s-and-uk/) Apple announces RCS support for iMessage (https://arstechnica.com/gadgets/2023/11/apple-announces-rcs-support-for-imessage/) Apple says iPhones will support RCS in 2024 (https://www.theverge.com/2023/11/16/23964171/apple-iphone-rcs-support) Today on The Vergecast: what Apple really means when it talks about RCS. (https://www.theverge.com/2023/11/17/23965656/today-on-the-vergecast-what-apple-really-means-when-it-talks-about-rcs) **## Nonsense Ikea debuts a trio of affordable smart home sensors (https://www.theverge.com/2023/11/28/23977693/ikea-sensors-door-window-water-motion-price-date-specs) Apple and Spotify have revealed their top podcasts of 2023 (https://www.theverge.com/2023/11/29/23981468/apple-replay-spotify-wrapped-podcasts-rogan-crime-junkie-alex-cooper) Listener Feedback Matt's Trackball: Amazon.com: Kensington Expert Trackball Mouse (K64325), Black Silver, 5"W x 5-3/4"D x 2-1/2"H : Electronics (https://amzn.to/3ujm7ct) Conferences Jan 29, 2024 to Feb 1, 2024 That Conference Texas (https://that.us/events/tx/2024/schedule/) If you want your conference mentioned, let's talk media sponsorships. SDT news & hype Join us in Slack (http://www.softwaredefinedtalk.com/slack). Get a SDT Sticker! Send your postal address to stickers@softwaredefinedtalk.com (mailto:stickers@softwaredefinedtalk.com) and we will send you free laptop stickers! Follow us: Twitch (https://www.twitch.tv/sdtpodcast), Twitter (https://twitter.com/softwaredeftalk), Instagram (https://www.instagram.com/softwaredefinedtalk/), Mastodon (https://hachyderm.io/@softwaredefinedtalk), BlueSky (https://bsky.app/profile/softwaredefinedtalk.com), LinkedIn (https://www.linkedin.com/company/software-defined-talk/), TikTok (https://www.tiktok.com/@softwaredefinedtalk), Threads (https://www.threads.net/@softwaredefinedtalk) and YouTube (https://www.youtube.com/channel/UCi3OJPV6h9tp-hbsGBLGsDQ/featured). Use the code SDT to get $20 off Coté's book, Digital WTF (https://leanpub.com/digitalwtf/c/sdt), so $5 total. Become a sponsor of Software Defined Talk (https://www.softwaredefinedtalk.com/ads)! Recommendations Brandon: The Complete History & Strategy of Visa (https://www.acquired.fm/episodes/visa) Matt: Markdown in Google Docs (https://support.google.com/docs/answer/12014036) Google Docs to Markdown (https://workspace.google.com/marketplace/app/docs_to_markdown/700168918607) Coté: pork chops, preferably thin sliced. Photo Credits Header (https://unsplash.com/photos/bike-on-concrete-floor-j0zlzt40J-0) Artwork (https://unsplash.com/photos/person-holding-black-amazon-echo-dot-qQRrhMIpxPw)

AWS Morning Brief for the week of November 20, 2023 with Corey Quinn. Links: re:Quinnvent Wednesday night drinkup at Atomic Liquors Nature Walk Amazon CloudWatch Logs announces regular expression filter pattern support for Live Tail  Amazon EBS announces Snapshot Lock to protect snapshots from inadvertent or malicious deletions  Amazon MSK Serverless now supports all programming languages Amazon Time Sync Service now supports microsecond-accurate time  AWS CloudTrail Lake announces new pricing option optimized for flexible retention AWS Cost Explorer now provides more historical and granular data AWS announces IPv6 tiered VPCs and subnets AWS Lambda console now features a single pane view of metrics, logs, and traces Announcing Research and Engineering Studio on AWS  Announcing PartyRock, an Amazon Bedrock Playground Amazon Bedrock now provides access to Meta's Llama 2 Chat 13B model  Happy anniversary, Amazon CloudFront: 15 years of evolution and internet advancements New – Multi-account search in AWS Resource Explorer Introducing instance maintenance policy for Amazon EC2 Auto Scaling  The serverless attendee's guide to AWS re:Invent 2023  Amazon EKS and Kubernetes sessions at AWS re:Invent 2023  Optimize AZ traffic costs using Amazon EKS, Karpenter, and Istio Editorial Join us for a week of AWS Amplify launches 

Not Escaping Containers but escaping Clusters - Managed Kubernetes distributions such as Amazon EKS, Google Kubernetes Engine (GKE) and Azure Kubernetes Service (AKS) attack vectors can allow you to reach the underlying AWS Account etc. In conversation with Christophe Tafani-Dereeper & Nick Frichette, from Datadog on how this is possible in Amazon EKS and achieving potentially the same in GKE & AKS too. Thank you to our episode sponsor Sagetap Guest Socials: Nick's and Christophe's Linkedin (⁠⁠⁠⁠⁠⁠⁠⁠⁠Nick Frichette + Christophe Tafani-Dereeper) Podcast Twitter - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠@CloudSecPod⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ If you want to watch videos of this LIVE STREAMED episode and past episodes - Check out our other Cloud Security Social Channels: - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security Newsletter ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ - ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Cloud Security BootCamp Questions asked: (00:00) Introduction (04:11) A bit about Christophe (04:37) A bit about Nick (05:03) What is managed Kubernetes? (06:26) Security of managed Kubernetes (09:02) Comparison between different managed Kubernetes (10:41) Service accounts and managed Kubernetes (14:22) What is container escape? (18:20) IMDSv2 for EKS (19:51) IMDSv2 in EKS vs AKES and GKE (22:01) Benchmark compliance for Kubernetes architecture (24:49) Low hanging fruits for container escape (27:17) Shared responsibility for managed Kubernetes (29:34) Fargate for Managed Kubernetes (32:00) Different ways to run containers (33:37) Escaping Managed Kubernetes cluster (38:39) Find more about this attack path (42:38) Escalation priviledge in EKS cluster (44:19) Reducing the Kubernetes attack service (44:58) MKAT for Kubernetes Security (48:23) Preventing AWS AuthConfig (50:11) Propagation Security (54:55) The fun section (57:47) Resources for latest Kubernetes updates Resources spoken about during the episode Nick Frichette's Blog - Hacking the Cloud Christophe Tafani-Dereeper' Blog Corey Quinn's - 17 ways to run containers on AWS MKAT cloudseclist newsletter

AWS Morning Brief for the week of October 30, 2023 with Corey Quinn. Links: Amazon Aurora MySQL includes optimizations that reduce the database restart time by up to 65% Amazon EKS adds support for customer managed IAM policies AMI Block Public Access now enabled for all new accounts and existing accounts with no public AMIs AWS Config now supports 19 new resource types AWS Marketplace announces enhanced private offer user experience for sellers AWS re:Post introduces Selections EC2 Hibernate now supports 20 additional instance families on EC2 Spot  Announcing general availability of Amazon EC2 M2 Mac instances for macOS  Comparing AWS Lambda Arm vs. x86 Performance, Cost, and Analysis How Infosys Built an Enterprise Knowledge Management Assistant Using Generative AI on AWS Rotate Your SSL/TLS Certificates Now – Amazon RDS and Amazon Aurora Expire in 2024 Build ROSA Clusters with Terraform Build a web-based cryptocurrency wallet tracker with Amazon Managed Blockchain Access and Query  Why AWS is the Best Place to Run Rust What's top of mind for Chief Data Officers going into 2024? EFA: how fixing one thing, lead to an improvement for … everyone | AWS HPC Blog Is Generative AI the Answer to All Questions? How to download your AWS Resilience Hub assessment results  Gain practical experience building with Amazon CodeWhisperer through AWS Jam Boost your AWS proficiency with Solution-Focused Immersion Days Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region AWS Disallowing Resale of Reserved Instances that were purchased at a discount

In this episode of Kubernetes Bytes, Bhavin Shah and Ryan Wallner interview Brian Chambers, Chief Architect at Chick-fil-A. Brian walks through some of the design decisions, challenges and architecture of how Chick-fil-A uses Kubernetes at the edge in their restaurants.Join the Kubernetes Bytes slack using: https://bit.ly/k8sbytesReady to shop better hydration, use "kubernetesbytes" to save 20% off anything you order.Try Nom Nom today, go to https://trynom.com/kubernetesbytes and get 50% off your first order plus free shipping. 01:05 Introduction 06:22 Cloud Native News 19:13 Interview with Madhuri 01:13:20 TakeawaysCloud Native News: K8s 1.28 https://kubernetes.io/blog/2023/08/15/kubernetes-v1-28-release/ SC assignment stable- https://kubernetes.io/blog/2023/08/15/kubernetes-v1-28-release/#automatic-retroactive-assignment-of-a-default-storageclass-graduates-to-stable Non graceful shutdown stable - https://kubernetes.io/blog/2023/08/15/kubernetes-v1-28-release/#generally-available-recovery-from-non-graceful-node-shutdown Ceph RBD and FS in tree deprecated Control plan and node supported version go from n-2 to n-3 Redhat Openstack services on OpenShift - https://www.redhat.com/en/blog/red-hat-openstack-services-openshift-next-generation-red-hat-openstack-platform Alcion 21 Million funding round: https://techcrunch.com/2023/09/19/alcion-which-provides-backup-and-security-services-to-enterprises-raises-21m/ Veeam was major funder: https://www.techtarget.com/searchdatabackup/news/366552363/Veeam-leads-funding-round-for-SaaS-backup-provider-Alcion Kubescape 3.0 - https://kubescape.io/blog/2023/09/19/introducing-kubescape-3/ GPU sharing on Amazon EKS with NVIDIA time-slicing and accelerated EC2 instances or MIG based sharing https://aws.amazon.com/blogs/containers/gpu-sharing-on-amazon-eks-with-nvidia-time-slicing-and-accelerated-ec2-instances https://aws.amazon.com/blogs/containers/maximizing-gpu-utilization-with-nvidias-multi-instance-gpu-mig-on-amazon-eks-running-more-pods-per-gpu-for-enhanced-performance/ Akuity launches Kargo - New Open Source project to automate declarative promotion of changes across multiple app environments - https://www.businesswire.com/news/home/20230918552920/en/Akuity-Launches-Kargo---a-New-Open-Source-Project-to-Automate-the-Declarative-Promotion-of-Changes-Across-Multiple-Application-Environments OpenTofu - Linux Foundations alternative to Terraform - loads of community support https://www.linuxfoundation.org/press/announcing-opentofu?hss_channel=lcp-208777 CFP already open for Paris!!!! https://events.linuxfoundation.org/kubecon-cloudnativecon-europe/program/cfp/ Spectro Cloud funding round - last was Series B - so this should be series C, but doesnt say. No amount disclosed, no valuation change disclosed. https://www.businesswire.com/news/home/20230914570380/en/Spectro-Cloud-Announces-Qualcomm-Ventures-Investment-to-Accelerate-Edge-and-AI-Innovation-at-Scale?hss_channel=lcp-36114581

AWS Morning Brief for the week of September 11, 2023, with Corey Quinn. Links: Amazon Aurora and Amazon RDS announces Extended Support for MySQL and PostgreSQL databases Amazon CloudWatch adds Amazon EKS control plane logs as Vended Logs Amazon CloudWatch Logs announces regular expression filter pattern syntax support As SwiftOnSecurity pointed out a week or two ago, a lot of folks can now discover firsthand just how many of their rules allow all 10* traffic Introducing Amazon EC2 R7iz instances  AWS Marketplace now supports AWS CloudTrail to improve procurement activity monitoring  AWS Step Functions launches enhanced error handling AWS Trusted Advisor adds 1 new fault tolerance check Announcing daily disbursements for AWS Marketplace sellers  Embracing FinOps to Maximize Cloud Value and Control Costs with the Deloitte FinOps Framework  Transforming Aviation Maintenance with the Infosys Generative AI Solution Built on Amazon Bedrock  How Vercel Shipped Cron Jobs in 2 Months Using Amazon EventBridge Scheduler How contact center leaders can prepare for generative AI  A Culture of Resilience  How generative AI is energizing the beauty industry Migrating AWS Direct Connect to a new location Reduce the security and compliance risks of messaging apps with AWS Wickr  AWS Guild Tournament builds cloud skills and innovative customer solutions From chocolate sales to a career in cloud with training from AWS re/Start Amazon to Discontinue Honeycode App-Building Service 

Nickolas Means, VP Engineering at Sym, joins Corey on Screaming in the Cloud to discuss how Sym is looking to solve the most common and most frustrating elements of compliance. Nick reveals why he finds it valuable to focus on making it easy for people to do the right thing over preventing them from doing the wrong thing, and why he feels the true spirit of compliance involves helping teams collaboratively come up with mutually beneficial solutions. Corey and Nick also dive into the common problems that engineers experience as a result of traditional compliance methods, and why historically the compliance industry has gotten a bad rap. About NickolasNickolas Means loves nothing more than a story of engineering triumph (except maybe a story of engineering disaster). When he's not stuck in a Wikipedia loop reading about plane crashes, he leads the engineering team at Sym, helping create the building blocks engineering teams need to build delightful developer access and approval workflows.Nick has been leading software engineering teams for more than a decade in the healthtech and devtools spaces. His focus is on building distributed organizations defined by their cultures of high trust and autonomy. He's also an international keynote speaker, having shared his unique brand of storytelling with audiences around the world. He works remotely from Austin, TX, and spends his spare time going on adventures with his wife and kids, running very slowly, and trying to brew the perfect cup of coffee.Links Referenced: symops.com: https://symops.com Twitter: https://twitter.com/nmeans TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Developers are responsible for more than ever these days. Not just the code they write, but also the containers and cloud infrastructure their apps run on. And probably the billing on top of that - which is neither here nor there. And a big part of that responsibility is app security — from code to cloud.That's where Snyk comes in. Snyk is a frictionless security platform that meets teams where they are, automating application security controls across their existing tools, workflows, and the AWS application stack — including seamless integrations with AWS CodePipeline, Amazon EKS, Amazon Inspector and several others.Deploy on AWS. Secure with Snyk. Learn more at snyk.co/scream. That's S-N-Y-K-dot-C-O/scream. And my thanks to them for sponsoring this ridiculous nonsense!Corey:  LANs of the late 90's and early 2000's were a magical place to learn about computers, hang out with your friends, and do cool stuff like share files, run websites & game servers, and occasionally bring the whole thing down with some ill-conceived software or network configuration. That's not how things are done anymore, but what if we could have a 90's style LAN experience along with the best parts of the 21st century internet? (Most of which are very hard to find these days.) Tailscale thinks we can, and I'm inclined to agree. With Tailscale I can use trusted identity providers like Google, or Okta, or GitHub to authenticate users, and automatically generate & rotate keys to authenticate devices I've added to my network. I can also share access to those devices with friends and teammates, or tag devices to give my team broader access. And that's the magic of it, your data is protected by the simple yet powerful social dynamics of small groups that you trust.Try now - it's free forever for personal use. I've been using it for almost two years personally, and am moderately annoyed that they haven't attempted to charge me for what's become an essential-to-my-workflow service.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted guest episode is brought to us by our friends over at Sym, and into my verbal grist mill, they have thrown their VP of Engineering, Nickolas Means. Nickolas, thank you for joining me.Nickolas: Thank you so much for having me, Corey. And feel free to call me Nick.Corey: I certainly shall. So, let's begin at a high level. When you're starting a company and trying to, sort of, bootstrap and raise initial rounds of funding and the rest, you're trying to save money in a bunch of places. And one of the most expensive things you can buy when starting a company is, of course, a vowel. You wound up not naming the company—or the vowel, really—the y is sometimes a vowel, sometimes not. It's S-Y-M. What is it you folks do exactly? What do you folks start? Where do you stop?Nickolas: So, the name of the company comes from the idea of helping humans and machines work together more effectively. And that's really nice and high level; it doesn't tell you any information about what we do.Corey: It feels like we're—we'd assume that most startups pivot at some point; we're just going to set—Nickolas: [laugh].Corey: —[crosstalk 00:01:33] seeds for that nice and early on, and dive on in.Nickolas: So, what we actually do, the two co-founders and myself all have a background in highly compliant industries. I've done VPN stints at a couple of health tech startups; they've done similarly. And all three of us ended up building sort of a certain set of things every time we were at one of these companies. Because you have to be compliant with things, and in order to be compliant with things, you have to have a set of controls, you have to restrict certain things: how people get to production, how people access customer data. And those controls, by and large, all suck. They're all painful and every company ends up building something from scratch at some point to make them not suck quite so bad. And it seemed like there was a product opportunity there.Corey: I would argue there absolutely is. One of the big problems that I've found throughout the time that I've been fixing AWS bills on a consultancy basis has been, we're really talking about cloud governance. But even now, by using the phrase cloud governance, three-quarters of the audience immediately wound up skipping to the next podcast over on their playlist because it sounds like it is one of those incredibly boring things. And to be fair, usually, when it comes to compliance, you want some of the most boring, least creative people in the world overseeing that. Like, when you wind up talking to someone at a company and they have a great sense of humor and they are constantly cracking jokes constantly, it's like, “What do you do?” Like, “Oh, I'm the CFO.” All you hear from that is, “Oh, I'm about to go to prison. Awesome.”Like, you want the wild, cutting-loose CEO to have three drinks and then confide, “I really like typing the number six.” You want them [laugh] to be predictable in a whole bunch of ways. And it always feels like compliance takes that entire mindset of, it's always about risk management, it's about wanting to make sure that people don't go off script in a bunch of weird ways, but as an engineer, what I always heard from that is slow down, don't be creative, go ahead and do things in very predictable ways. Only release things once a quarter, et cetera, et cetera. And yes, that's one way to meet compliance goals, but it's a crappy way, in my experience. I'm going to guess, though, that you have a lot more experience with the compliance world than I do because having worked a few times now, for big regulated finance companies, I wanted to get the hell out of the compliance universe.Nickolas: Yeah, I mean, you used an interesting turn of phrase there. You used the phrase, “Avoid going off script,” and I think there's a subtle turn there that actually makes all of this work a lot better. Instead of focusing on keeping people from going off script, you focus on keeping them on script. You focus on making it easier to do the right thing than to do the wrong thing. And that takes away a significant amount of the pain involved in compliance stuff.You look at implementing controls—and everybody has the exact same reaction you just brought up about governance—because there's so much FUD around this stuff. Everybody has been slowed down by one of these silly rules that makes no sense, that's checking a box and not actually meeting the spirit of any kind of meaningful improvement.Corey: Oh, cloud has absolutely doubled our speed of iteration because it used to take six weeks to get a server racked in the data center and we moved our processes to cloud and now to spin up an EC2 instance, it only takes three weeks of approvals. And at that point, it's what are you really doing? You wind up with people building on shadow IT. It's part of what contributed to the rise of cloud in the first place. Well, I can go through the annoying thing that this company wants me to do, or I have a corporate credit card and by the time it raises the level of spend to a point where it gets scrutiny, it's in production serving customers and what are they going to do?Some of the very early AWS sales conversations with customers started off as, “Well, why should we build on top of your cloud?” asks the exec, and they say, “Oh, sorry, you have 87 different accounts throughout your organization currently with us. We're just trying to give you some unified view into it and possibly some discounting if you want.” Yeah, these days, that's a fast track to getting yourself fired in some companies, if you wind up deviating from that story. But also, people are not doing this out of malfeasance; they're trying to get their job done.And as soon as guardrails start increasing friction, making it harder to do things the right way than to go around it, people will not comply. I strongly believe that, whether it's cost—which is my universe, and frankly, only a business hours problem—or actual governance issues with some compliance regimes, which get those wrong and hope you enjoy some time in prison.Nickolas: Yeah, exactly. I mean, you know, if you look at SOC 2, for example, there's a lot of companies out there that are willing to sell you a program that will help you become SOC 2 compliant. They show you all the steps you need to take, all the programs you need to put in place. The thing they don't do is help you establish the controls that are required. They'll tell you that you have to have somebody formally approving before software goes out to production. They won't give you any guidance whatsoever on how to put that control in place. And so, it's really easy for a compliance person that's not looking to collaborate with engineering just to go, “Okay, I need you to put a button in the deploy process and I need the CTO to click that button.”Corey: Yes. We've always seen that as reactions to different things. I was at a company once where there were some outages caused by bad deploys, so they decided that a VP had to sign off on every deploy. Now, I come from the sysadmin ops world, which explains so much about my cynical perspective on life, so the way we got that overturned within two days is we did the malicious compliance thing, where oh, we need to deploy this. Great, we are walking into the middle of a senior leadership team meeting to get them to—with a tablet or comput—laptop—“I need you to click the button right now.”And doing that out of hours and all kinds of other things, it's oh. Yeah. How about we wind up only doing that for significant large changes? How about that? Maybe you don't need to wake someone up at home in the middle of the night when there's a deploy going out that fixes a typo on the marketing page; little things like that.And at some point, you're always felt like the goal of governance was either ossified scar tissue around all the ways that things have failed before, or through a, frankly, misguided belief that if we wind up distilling everything down to processes and procedures, eventually, someday, we can have a bunch of trained monkeys doing this job instead of people who are expensive and, you know, cynical, and difficult to please. I feel like that is not the right way to think about these things.Nickolas: Well, I mean, the thing about those controls, you know, it's exactly what you just said. Nowhere in SOC 2 does it say that your VPN [unintelligible 00:07:56] or CTO has to approve all code deploys, that's not in there. But that's the reality of life at a bunch of companies. In reality, if you just follow a software development life cycle that has multiple people looking at code before it gets deployed, multiple people signing off on that code being okay to deploy and you have a staging environment before you hit production, you've met the control. And SOC 2 gives you so much flexibility in how you write the control.So, I think the thing that I've seen that makes compliance so much less painful, is when you have somebody that is 95% the boring persona like you're talking about, but 5% creative. 5% willing to kind of get their hands dirty, empathize with the engineering team, collaborate with the engineering team, and find a way to put some of these controls in place that doesn't just bring things to a grinding halt.Corey: I have to assume that, given that you've built an entire product slash company around this idea, that you have some opinions other than doing what I do, which is sitting in my lofty ivory tower and oh, you should, in this idealized case, do things a little bit differently. But it's going to be bespoke and the answer to any complex question, the more senior you get is, “It depends.” You, of course, have built something that scales out in a bunch of different ways. How do you view that in a way that makes it not either completely useless or overly prescriptive?Nickolas: We focus on giving the power to engineering teams and giving the security complexity [unintelligible 00:09:23] the power to oversee those things. You know, it would be easy to give somebody, like, a clickbox UI, let them design controls for SOC 2 or whatever, end-user interface, but that's not how engineers think; engineers think and express ideas and code. So, we've made the rather controversial decision in the face of a bunch of no-code tools to go low-code instead. So, to build a compliance workflow in Sym, you're going to write some Terraform, you're going to write a little bit of Python—a lot less than if you were building it from scratch—but you're going to end up with something that perfectly fits the way that you already work versus having to shift your work practices around to fit the tool.Corey: If you have inadvertently stumbled upon one of my hot buttons. There's a lot of people that take a perspective around low code. And I just want to say that that perspective is often garbage. Like, oh, that's not a real program—great. Hypothetically, if you have an idea for a business or a product or something, and involve software as most things seem to these days, maybe having to go to a boot camp for six months first as a prerequisite is not the best path forward.“Well, you're never going to build something hyperscale in a low-code environment.” Great, how many things that we built that actually need to be hyperscale that don't go through 16 different architectural iterations between ridiculous idea one day and thing that is actually hyperscale? It's an early optimization. I have an entire production pipeline in Retool that I built using low code. I think that that is a very powerful thing. And this idea that, “Oh, that's not real code.” Cool. What's your point?Nickolas: Well, and for us, one of the things that we're trying to enable is for software engineering teams, ops teams, whoever is building these controls, to interact with a security person or a compliance person, for them to be able to read the code, understand what it does, understand the way that the control has been implemented. And so, we provide a bunch of frameworks around that and a bunch of things. Like, you don't have to go and build a Slack workflow from scratch and nobody has to understand that code because it's buried in the platform. The only thing that the security or compliance person has to understand is the business logic that's been put into place. Who can approve it? Who can't approve it? How does that change after hours? How does that change if there's an incident? All of that is in very simple Python that you don't have to be an experienced programmer to be able to read.Corey: One of the big powerful things behind that is it really reduces the interrupt volume of someone coming by to an engineer who is deep in the middle of something else, and, “Hey, guess what I have? A surprise context switch for something that's going to take you probably 30 seconds, but then you're going to be distracted by all of this.” If you give people the ability to self-serve, everything tends to work a lot more smoothly.Nickolas: Yeah, absolutely. And, you know, that's one of the ways we use Sym at Sym: we've got it in front of our AWS production environment, so if you need to go and do anything in production, you just have to get approval from any other engineer that happens to be in the approval channel, sort of a two-keys-to-launch-a-missile model. And that works fine for our compliance needs and it avoids there being a single point of failure that every time you need to go and get into production, you have to go and say, “Mother, may I?”Corey: Exactly. It's one of those things where every time you wind up with something that injects friction, people are going to find ways around it. And in some cases, this leads to positive outcomes where, when you're subject to PCI, which is a lot more prescriptive than a number of other compliance regimes, it's, great; this is a lot of things that don't necessarily reflect how we work, how we want to work, et cetera. We can ignore it, which is not a great plan, we can wind up having to slow everything down, which is the common case, or the right answer is, we're going to build the PCI environment that is very self-contained, just the critical stuff that needs to be in there is going to be in there, and then we can build everything that touches it around it in ways that are a lot more aligned with how we believe software should be built.Nickolas: Yeah, absolutely. I mean, you silo off those high-control places, but there are controls that have to extend into the rest of the business. And one of the things that I'm a very firm believer in is, if you're going to impose a control upon somebody, they need to have the agency to shape and to change that control so that it lets them work the way that they want to work.Corey: I just want to call out how wonderful that is because I had a belief that looked borderline heretical, 12 years ago, when I said that, “Okay, simple rule. If you want me on call, I am empowered to change the thing that wakes me up.” Whether that is the code itself, the system itself, the paging threshold and frequency, or ultimately, I'm turning the physical pager off. It's one of those things where I decide what's an emergency outside of hours on that point. If it's going to wake me up, I need the power to make sure it never does. Otherwise, you have no agency. It just feels like you're being victimized by the stuff.Nickolas: Yeah, absolutely. I mean, there was a wave of on-call regimes that ran through large companies for a while where there would be a centralized on-call team that would be responsible for responding to hundreds of services. And thankfully, we are maturing past that; we're distributing on-call rotations so that teams that actually build services are responsible for them. And it's the same mindset, right? If you're going to be participating, if you're going to be working with a system or working with a control, then you need to be able to change it, you need to be able to make it work the way that you think that it ought to work.And in the context of compliance, you need to bring somebody along with you. You need to bring the person that's responsible for the controls that actually has to sign on the dotted line at the end of the audit period, saying that we do all of these things. So, you have to be able to explain what you're doing to them. But you have to be able to iterate.Corey: I have to ask, given that what you are building is going to have heavy involvement from engineering, how do you respond to the probably most common engineering objection I imagine you get, which is, “Well, this doesn't look hard. I could build this in a weekend.”Nickolas: You know, it's funny. We joke that our biggest competitor is build in-house, right? It's pretty easy to start looking at what it takes to build a from scratch workflow in Slack to build a Slack app, to understand the cost of building it in-house. Because nothing about building an elegant user interface in Slack is easy or cheap. That API is difficult to work with and hard to get good user experience out of.And we've spent a lot of time polishing a lot of places in the platform: we've got good documentation, we've got a good SDK, we've got good integration with third-party services that make all of this stuff easy to do. And it does look easy on the surface, it does look like ‘I can build it,' but we've had customers that have had that objection gone and tried to build it and come back. Because it's not as easy as anybody thinks.Corey: My biggest competitor for fixing AWS bills has always been Microsoft Excel. It's the, we're going to do it ourselves—badly—internally. Okay, great. If that works for you, terrific—Nickolas: Yeah.Corey: —but very often it doesn't. I mean, I think a classic case study of this is, in the terms of something that is well designed but is almost mind-bogglingly complex—and we're getting a case study in it this year—is Twitter because it looks from the outside, very simple. I wind up writing a thing and I hit the post button and it shows up in a timeline. And then other people can subscribe to it or not, and they see it themselves. That sounds like something you can build on a weekend. And we look at all the ways it's now exploding and collapsing and having weird bugs that no one anticipated, to realize, oh, this is a very challenging, very sophisticated application. But because it was well designed at one point, it looks easy.Nickolas: Yeah. Yeah, it continues to run despite the fact that it's having less than a quarter of the staff that originally maintained it, maintaining it because the services were well designed in the first place. They're resilient on their own and they're self-healing in a lot of cases. It's the same thing with Sym. You can build these tools in-house, you can build them yourself, but then you've got more software to maintain. Because once you build something, you own it, forever. And the cheapest code is no code; the cheapest code is code that you don't have to write.It's easy to look at a simple use case and understand a little bit of the cost of this. If you want a Slack workflow that gives you access to production in AWS, you can wire that up fairly quickly. Those APIs are not all that difficult. Now, let's say you want to add an integration where if you're on-call in PagerDuty, you can get to production without having to get an approval. Okay, well, now you've got a new API that you need to wire in.And let's say that every time that happens, you want to open a Jira ticket so that you can record that that's happened. Well, there's another API that you've got to wire in.j, whereas with Sym, it's just, it's right there. It's a few lines of code to wire it all together. And it deploys in Terraform alongside the rest of your infrastructure, so you manage it the same way you're used to managing things.Corey: It reminds me of my earlier career when I was deep in the configuration management weeds with Puppet and SaltStack, where the biggest competitor we had any of those projects was always someone writing a bash script to do it themselves. And yes, you can do that, but then the requirements change, or you're going to hit a point of scale that was surprising. And one of the valuable parts of it is that when the future is uncertain, as it always is—Nickolas: Always.Corey: Having folks who work in environments that aren't just yours who encounter a lot of those edge cases you're going to stumble into and can build things in is incredibly valuable. I don't think I've ever met anyone who ran an infrastructure that said, “I would build it the same way if I had to start over again.” They always want to, “I would fix these annoying things.” Well, by having a product focused on a space like this, it's yeah, today, you can have that VP click the approve button inside the GitHub Actions workflow. Good for you.But when you get just a little bit further down the path, you aren't going to want to do that anymore. There needs to be some decision-making it builds into it, and for certain high-risk changes, maybe a second person and so on. How do you build that logic engine? How do you build that workflow approach? How do you have a break glass thing for middle of the night when the site is down? Et cetera, et cetera, et cetera.And that's exactly the sort of thing that I would expect something like Sym to get very right, just because there's always a bigger fish. You've seen this [unintelligible 00:19:17] before in other shops. And more to the point, if there's something I want to do as a part of this that Sym doesn't support and you are looking at me strangely if I asked how to do it, that's usually a good early warning sign that maybe there's something I'm not thinking about here. Because whatever the problem space is, I'm probably not the only person that has to do this. How are other companies solving for this? And it turns out that all my copy of our SOC 2 report has a typo on it. That would explain a lot. That's a ‘can' instead of ‘can't.' Nevermind. Or something like that.Nickolas: Well, and the flip side of that is also true. I mean, the interesting thing about working on something that is sort of wide open with what you can wire up and build with it is we're always learning from our customers. We're always learning from the things that they're doing. And so, you know, when somebody approaches us of, “Hey, we need to solve this particular problem,” if we don't have a ready answer, we brainstorm and help figure that out. And to your point, that always extrapolates to other customers finding the same sort of thing useful.The other bit of this that's really interesting beyond the durability and the ability to kind of rapidly evolve these workflows is the audibility. It's helpful in a lot of these compliance regimes to have a third-party tracking this data for you. So, when somebody accesses AWS production, who approved that access? When somebody deploys code, who approved those deploys? Well, we sit there as kind of a third party on the side, observing all of this, taking all these notes for you, and piping them into whatever audit tool that you want.So, you've got that data long-term and when it comes time to audit, you've got all the evidence you need; it's already there, already collected. You don't have to go through and write a regex to parse a bunch of logs to get the information you need.Corey: And invariably, that regex is always going to be different, depending upon the log stuff. It's great having a unified central approach that is the trusted repository for this stuff. As you've been going to market and talking to your earlier customers and seeing the problems that you folks solve, what have you learned about the market space since you've gone into this direction? Because I feel like this is one of those products where you start designing and thinking you know a lot about the space, and you learn so much more just from the customer conversations and seeing that you can build the most finely crafted torque wrench in the world and the customer complains because it turns out, you built a crappy hammer.Nickolas: So, I think what's been really interesting to me is how much use our Lambda integration gets. We have a lot of first-party integrations with things like IAM and IAM identity center and Aptible and a bunch of tools that you can interact with, but a lot of our customers have wanted to do very specific things inside their infrastructure and put those things behind an approval. And the Lambda integration turns out to be a great Swiss army knife to do that because you can wire it up—it runs inside your firewall—to take essentially whatever action that you need it to. And that gets a ton of use. Probably more than half of our customers have at least one Lambda workflow in production, and I would not have expected that going in.Corey: It's wild to me just how pervasive Lambda has become. And even from a compliance perspective, it's great because unlike, “Well, it's a script that runs on a server somewhere,” yeah, it's immutable. It's versioned. There's a way to conclusively prove that at invocation, this is the code that ran, the end, with the following parameters. Done.There's no, “Well, looking at the timestamp on the file”—like, no. None of that nonsense. It's arguable that something that I have seen has been that Lambda is one of those rare technologies where you're seeing faster adoption in the enterprise and you are in startup land.Nickolas: Yeah, I would say that's true. I mean, it's so great for running undifferentiated workloads. I just need this one thing to happen really quickly and I don't want to mess with standing up a server to run this thing that runs once a week. Okay, well, here's a computer that will run just long enough for you to run this thing and then go away. It tracks exactly what ran, exactly when it ran, exactly how it got kicked off.And in our case, it has access to all of the internal AWS APIs that we wall off in our platform because we obviously don't want you using those things in the Sym runtime. But you can do anything that you want to your AWS environment from your own Lambda and we will gladly provide the approval step ahead of kicking that job off.Corey: Are you seeing people use Lambda-based workflows to manage on-premises things or is it more heavily in environments that are already within the AWS boundary?Nickolas: The Lambda stuff that we see is almost entirely—I think it is entirely for things that are within the AWS boundary. I can't think of an instance when somebody is managing something on-prem with it.Corey: I am increasingly discovering, through the magic of Tailscale—among a few other things—that I can use that for things on-premises that talk directly and interface with my Raspberry Pi in the spare room, et cetera. Which is—I think some people call it hybrid, which is the business enterprise term for ‘horrifying—Nickolas: Yep.Corey: —because it's a terrible pattern in some ways. But it's so convenient and it's so nice not to have to worry about some of these things, just an infrastructure point of view. One thing that I think that AWS has done very well at, as they've evolved, has been with AWS Artifact, which ties directly to their own compliance reports, where in the early days when I was responsible for SOC 2 controls at a company, I found myself answering security questionnaires from vendors as if I was running in a data center. And sure enough, they wanted to tour us-east-1. And it turns out, you can't really do that.So now, just pointing them to the stuff that comes out of Artifact, it's written by auditors for auditors and they go away and leave you alone without having to explain your bespoke artisanal nonsense to them. There's something very pleasant about being able to throw the lion's share of the work over to someone who already knows how to do it.Nickolas: Our audit period is ending here shortly and I have recently been and spending time in Artifact. So yes, a hundred percent.Corey: It used to be that you would only be able to get those things under explicit NDAs, you'd have to talk to your account manager for every one, it was a back-and-forth process, and you didn't really know if what you were going to get was going to answer the questions that they had. Now it's, you show up, you click things three times, and you're done. The hardest part is sorting out which ones you need from the hundreds of things available within Artifact.It's like, okay, that's great, but this one is in Spanish for some reason. And that's awesome, but on some level, it feels like that should be an easy filter option. But yeah, no one ever accused AWS of building a good user interface. But once you get the thing you need and can pass it off, great. Job over. It's one of my favorite services that most people who are what we know as ‘happy' don't know exist.Nickolas: Yeah well, and that, it points to a larger industry trend, right, that companies are getting SOC 2 specifically earlier and earlier because it is becoming table stakes to be able to sell into other companies. They want to see your SOC 2 report before they're willing to work with you before they're willing to let your software touch their infrastructure. And there is a lot of value in these compliance programs as essentially a stamp of approval that you're taking these things seriously, even with as much flexibility as SOC 2 has, just the stamp that we've thought about these things and we have serious answers to them is a pretty important signal to be able to send to somebody that's wanting to buy your software.Corey: We've toyed with the idea of going through the process ourselves because we get asked about it all the time, but it feels like the procurement processes that ask us for it expect us to come in with a whole software suite and the rest. And yeah, if that's the world we're operating in, it makes a lot of sense. We're a services-based consultancy; we come in as individuals, we have conversations with people, and we talk about this and we have no write access to anything in your environment and give you scoped-down permissions for what we talk to because we don't want the responsibility of that stuff.And a lot of companies get that intrinsically, but there's occasionally a few you have to go round and round and round with. It just it feels like it's one of those, okay, you're not quite there yet. You're trying to view everything through this very specific worldview. Maybe it works for your constraints and requirements, but I've never understood it. And I've learned the older I get, the more time I spend around this, I used to have such a negative perspective on compliance.And now it's, you know, everything's nuanced. There's a reason that these things are there. It's not just a make-work project for an industry that wants to slow everyone else down. It's, there are risks here; these things exist for a reason. There's a reason that you can go start Twitter for Pets tonight and not be regulated, but the same is not true of First Bank of Twitter Pets.It's okay, yeah, one of those things is going to require a fair bit of regulatory scrutiny, and as a society, we want that. Now, the counterargument that I don't necessarily want to get too far into is, should Twitter for Pets be regulated?Nickolas: [laugh].Corey: And that's a can of worms that I think we'll leave for another episode.Nickolas: Yeah, I mean, that's—you know, the people that hate compliance the most are the people that are on the sharp end of compliance, people that are having to actually deal with the controls that are imposed upon them by these compliance regimes and by somebody who's taking a very literal view in interpreting the things that some of these compliance programs say that you'd have to put in place. And I think, you know, that's—kind of bring the conversation full circle—that's the thing that we want to change more than anything. If we can wave a magic wand and change the compliance universe, the thing that I most want is to help compliance and security people collaborate with their engineering teams and come up with mutually beneficial solutions. Things that actually—the spirit of compliance.Corey: Oh, yeah. My first PCI audit was a little bit of a challenge, just because the auditor wasn't really conversant with anything that wasn't a large company. So, they show up at our twelve-person start off, and, “Okay, where's the Active Directory?” It's like, “We don't have one of those.” “Okay, well how do you authenticate to the WiFi?” It's like, “Oh, the password's on the wall.”It's, “Well, what happens if I get on that WiFi?” It's, “What can I do that I couldn't do from anywhere else?” Like, “Use that printer over there. That's it.” Because everything else was the idea of the security boundary was built on identity, not on what blessed network you happened to be on; there was no special permissioning that didn't apply to the Starbucks WiFi next town over.But that was one of those things where at first they thought this was a horrifying problem and they were not going to be able to certify us, and it turned into no, we had significantly advanced culture of security compliance, oversight, separation of duties, all the things you really care about. We just didn't have the trappings that usually came across with when you're thinking about this or starting—or having the temerity to start a company, you know, longer than 18 months ago at a place that wasn't San Francisco on the latest version of a MacBook Pro running the bleeding edge version of Chrome. It turns out that there's a big universe out there. And not that there's anything wrong with either side of it, until they start forgetting that not everyone operates the way that they do.Nickolas: Yeah. I mean, you know, we talked about checkbox compliance a lot and I think that's probably the biggest problem is there is a lot of checkbox compliance out there. And people have seen it not actually solve anything and just make everything harder. And so, compliance gets a bad rap.Corey: Oh, for me, the one that I've been picking fights on social media about for a few years now is encryption-at-rest in the cloud. Like, yes, you want full-disk encryption turned on your laptops, your phones, your tablets, et cetera. Someone steals it from the coffee shop, you want to be out the cost the hardware. The end. But if you can get a hard drive intact out of an AWS facility and then reassemble it with the right number of drives in the right places, without… and hasn't been encrypted. Congratulations, you earned it. As far as I'm concerned, that's yours. You can keep it.Because AWS employees aren't able to do that, let alone third parties. But it is easier by far to click the box to enable encryption-at-rest and not spend half an hour arguing with the auditor… and just get on with your day. And recently in S3, for example, they wound up making that a default. Good for them. It's just, can we please focus on the part of the story that's relevant and germane to our business? Because that is not the threat model of modern attacks.Nickolas: Yeah, I mean, for a long time, how much of the internet ran on unencrypted HTTP, but it was being served off of an encrypted disk? Great. What have we solved?Corey: Oh, absolutely. It's wild to me. Even now, I still we feel like there should be a reasonable way to handle—to [unintelligible 00:31:17] basically encryption between two points that doesn't depend on the third-party CA's with expiring certs and the rest. Drives me up a wall every time because it's always the worst possible time. It causes the strangest issues and there is something deeply and profoundly wrong with the fact that the failure mode from the user perspective between, “Your connection is being intercepted by a third party,” and, “Holy shit. This certificate expired two hours ago.” Like, those are very different use cases, but the scary warnings have trained people to treat them the same way.Nickolas: Yep. Yep, exactly the same. Ugh.Corey: I really want to thank you for being so generous with your time. If people want to learn more, where's the best place for them to find you?Nickolas: Yeah, so the best place to find out more about Sym is our website, symops.com, SYMOPS dot com. And I should mention that Sym is completely free for teams of up to ten people. If any of you out there listening check it out, please reach out. We'd love to hear about your experiences, help any way we can. And if you want to get in touch with me directly, the best place to do that for now, while it lasts is still Twitter. I'm on there as @nmeans.Corey: And we will, of course, include a link to that in the [show notes 00:32:27]. Thank you so much for agreeing to talk to me about all this stuff. I really appreciate it.Nickolas: Yeah. Thanks so much for having me on, Corey. It's been a lot of fun.Corey: Nick Means, VP of Engineering at Sym. I'm Cloud Economist Corey Quinn, and this has been a promoted guest episode, brought to us by our friends at Sym. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry bitter comment that will get posted in six weeks, after you track down your elusive VP to click the approve button.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Josh Doody, Owner of Fearless Salary Negotiation, joins Corey on Screaming in the Cloud to discuss how to successfully negotiate your salary, and why it's important to do so even in times of economic uncertainty. Corey and Josh chat about some of the hidden reasons why salary negotiation is critical to job seekers, and what goes into determining salary bands behind the scenes. Josh also reveals why he feels there's some stagnancy in the big tech job market, and why it's critical for job seekers to have a balanced view of the value that they provide to employers when negotiating salary. Josh also describes some of the unexpected ways salary negotiations can come up throughout the interview process, and how to best handle the discomfort of negotiation. About JoshJosh is a salary negotiation coach who works with senior software engineers and engineering managers to negotiate job offers with big tech companies. He also wrote Fearless Salary Negotiation: A Step-by-Step Guide to Getting Paid What You're Worth, and recently launched Salary Negotiation Mastery to help folks who aren't able to work with him 1-on-1.Links Referenced: Company website: https://fearlesssalarynegotiation.com Twitter: https://twitter.com/joshdoody LinkedIn: https://www.linkedin.com/in/joshdoody/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Developers are responsible for more than ever these days. Not just the code they write, but also the containers and cloud infrastructure their apps run on. And probably the billing on top of that - which is neither here nor there. And a big part of that responsibility is app security — from code to cloud.That's where Snyk comes in. Snyk is a frictionless security platform that meets teams where they are, automating application security controls across their existing tools, workflows, and the AWS application stack — including seamless integrations with AWS CodePipeline, Amazon EKS, Amazon Inspector and several others.Deploy on AWS. Secure with Snyk. Learn more at snyk.co/scream. That's S-N-Y-K-dot-C-O/scream. And my thanks to them for sponsoring this ridiculous nonsense!Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I have a returning guest today who hasn't been on for a couple of years, at least. Josh Doody is the owner of fearlesssalarynegotiation.com and focuses on a problem that's near and dear to my heart from my previous life as an employee, salary negotiation, specifically emphasizing software engineers, if I have that right. Josh, thanks for joining me.Josh: Yeah. You have it exactly right. It's great to be here, and good to talk to you again, Corey.Corey: I used to be practiced at doing salary negotiations, which is a very roundabout way of saying I got fired a lot, so I got lots of practice at doing it. And I found that it was a very strange experience that was completely orthogonal to anything else that I did in the course of my day-to-day. Now, of course, I you know, negotiate AWS bills for a living among many other things, and do a lot of sales work, and yeah okay, now it's a lot more germane. But back in my engineering life, it was the one time I got to really negotiate that wasn't, you know, haggling with some vendor somewhere when I'm trying to buy a burrito, was salary negotiation, and I felt utterly unprepared for it.Josh: Yeah, I think most people feel that way and you summarized pretty well why that is. And that's, you know, let's say you have a really robust career, you're around for decades, you know, you're working in lots of companies, you might have, I don't know, let's say ten, a dozen job offers that you negotiate, you know, give or take. And that's not very many reps for doing something that's as consequential as, you know, negotiating your actual pay. Which, depending on how senior you are, could be literally negotiating, like, you know, multiple cars' worth of value per year that you're going to [laugh] that you're going to earn. But you don't get the reps, so most people just kind of—I think they kind of don't even think about it until they have to think about it until it's directly in front of them. And then they just kind of power through it, get it over with or even totally ignore it and just get back to the thing that they're doing in their career, which is why you show up to work.Corey: It feels, on some level, like it's one of those areas where people wind up thinking about it long after they really should have. These days, it feels like salary negotiation process, more or less should start when you start debating, huh, maybe I'll change jobs. Like, it feels like it's really that early, not when you have an offer sitting in your inbox that needs a response by the end of the week. Right, wrong, or am I just thinking about this in ridiculous ways?Josh: No, I think you're right. So, you can start thinking about it, you know, you get a job offer in your inbox and you can start thinking how do I negotiate this now, but you know, you're going to be in a less secure position to do a strong negotiation at that moment than you would have had you begun thinking about it when you mentioned, which is, like, you're actually thinking about changing jobs, or, you know, maybe you just got a cold call from a recruiter and they're at a company that you're kind of interested in working with. So, maybe I will talk to this recruiter instead of just blocking them or whatever. And so, the whole process can begin at that moment when they say, “Hey, you know, we have this opportunity that we think you might be interested in. What do you think?”And then, you know, early on, they'll even kind of officially start the negotiation, at least in my mind, where they'll say, “But before we really go too far on this, like, what are you hoping to make here? You know, what are your salary expectations if you come work here?” And you're kind of off to the races at that moment. But even if they don't say that out loud, that's something that you should be thinking about from the beginning, which is, you know, maybe most broadly, how do I position myself to get the best possible version of the job offer that they're willing to give and to leave myself the most latitude to improve that job offer to be the maximum that they can afford to pay me or the maximum that their budget allows or however you want to frame that. So, short answer, yeah, I think you're right that most people think about it as sort of an afterthought, either after they've already started a job and they go, “Huh, I wonder if that guy over there is making more than I am?” Or, you know, “Shoot. I think I moved too fast there. Maybe I should have done something a little bit better.” When they could think about it way earlier in the process than that.Corey: Since I was last on the job market, there have been some changes, at least here in California, that have had a somewhat significant impact, to my understanding. First, job salary changes need to be posted in job ads, which I think is great—and that's occurring in a number of different states—and also it is now against the law, in California—or at least against public policy—to ask what someone's current compensation is and your salary history and dive into that. Now, that's all well and good, but I also have been asked a number of questions that are not exactly… green, when it comes to being in the middle of an interview. And, “You're not legally allowed to ask me that question,” is that a heck of a pushback.Josh: Yeah, I think that I've had a couple conversations about this recently, but also over the past few years, especially on the—you know, you mentioned the two prongs of that idea: what's your current salary, what are you making now? And, you know, what is the salary that you expect to make? And so, kind of one by one, states are outlawing potential employers' ability to ask about what you're currently making. And then I've also heard some agitation lately that there might be some federal legislation that's coming down that might just kind of take that off the table. As you mentioned, recruiters, companies, organizations, however you want to model them are very clever, and so there are always ways, you know, even if they're indirect questions, you know, you don't ask them what they're currently making, but you ask them something that gives you some insight into what they currently might be thinking.Also, if you're in the big tech world—which you mentioned you negotiate AWS contracts—in the big tech world, they don't necessarily have to ask you what you're currently making if they know that you're an L4 software engineer at Google. They can probably approximate it pretty well. And of course, they know that because you're going to have to tell them, you know, with a resume or when you're interviewing, that's kind of how you get in the door. So, that's an interesting thing. But I still say, avoid that. Try to avoid giving him as much information as possible.And I think the most important thing with the current salary idea is, you just don't want to say it out loud. You want to make sure that they can't quite grab onto that because you make it too easy for them when they know what your current salary is to just do sort of a cost-plus version of offering you a job, which is, “Well, you're making this much now. We'll just add 10% to that,” and that's your new job offer, when you know, that's not how you level up quickly and in big ways.And then you mentioned the salary expectations. I do think it's great that a lot of job offers now will have a salary range in them. That's a question that I see a lot is, like, how do I know that I'm not going to waste 25 hours of my personal time and maybe a trip across the country for a job, where when they finally make the offer, it's just laughably low? And the answer is, you know, hopefully, they have something that you can grab onto in the actual job description that says, here's what the range looks like. But even then, you'll notice if you look carefully—I saw one yesterday, and I don't remember where I saw it, but it was like, “Yeah, our range of salary is, you know, 120k up to 290k, depending on geographic region.”And it's like… I mean, technically, that's a salary range, but they don't tell you what the regions are, how they map, and all that stuff. So, you're not getting a lot of information there; you're just getting sort of an approximate number. But it's still helpful to know that information. And it's also helpful to not disclose that information. If you have a number in mind that you're hoping for, it's not in your best interest to share that with the company.So, I think at least what you can do is look at the job description. If they have some kind of a range, take a look at it, see if it feels like, okay, this is something I can work with or if it's just, you know, there's no way that that would ever work for me and you can just pass on it and save yourself some time.Corey: For me, one of the things that always frustrated me was that at the start of looking into a job, there's always the big question that they ask that has been the socially acceptable paths at screwing you over, and the knowing how to answer that is important. But I still bungled it a number of times whenever I was out of practice, which is quite simply, “Okay, what is it that you expect to make in your next role? What are your compensation requirements?” And it feels like answering that at the beginning of the process just completely sets your course for how the rest of that process is going to go.Josh: It does and it's something that's very subtle and clever because most people will not perceive that to be a negotiation tactic when it is. And also you mentioned earlier in the context of, like, asking you what your current salary is that it can be perceived as sort of a gatekeeping question. Like you mentioned, you know, you're in the middle of an interview and somebody pops a question at you like, “What is your [laugh] what's your current salary?” And you're looking at an interviewer and you're thinking, “If I don't give him this information, then I'm saying no to an interviewer, and how's that going to go over?”This is the same kind of thing. When, you know, at any point in the process, they might ask you what your salary expectations are, it could be on the first screening call, it could be right before, they like to hold this till right before they make an offer where you go through the whole interview process and then right before they're going to extend an offer, they say, “Hey, you know, I'm going to go to the hiring committee and make a recommendation that we hire. But before I do that, you know, what are you hoping to make if we actually do extend an offer and I go talk to the comp team?” And it can feel like, well, gosh, I better tell them the answer to that question because they literally just said, basically, like, “I have an offer for you, but first, I need this information from you.” And it can feel really kind of daunting to say, “No, I'm not going to give you that.”So, the question is, you know, should you give them that and how? You shouldn't, as I mentioned earlier. Giving them salary expectations, I'll give kind of a brief summary of why it's not a good idea. I think a good way to reframe that question, you know, what are you what are you hoping to make if you join our team, is, you know, “Hey, you know, we have a giant company here. We've got tens of thousands of employees. We've got thousands of engineers that are at your level and doing your kind of work. We have salary surveys that we run once a quarter, or once a month, that are super expensive. We know what everybody else in the industry is doing. We know what the value of this role is to our company. We know how many other people are applying for this job. We know how many open seats we have. You don't know any of that stuff, but even though you don't know any of that stuff, why don't you take a wild guess what we would pay you to do this job at this company at this moment?”Corey: And then of course, we're going to use it against you later, when you wind up having what you view as a negotiation, like, “Ah, but you said at the beginning of the process that this would be sufficient.”Josh: Yeah. So, that's the problem, right? As you take that wild guess and you're going to do one of two things. It's basically 0% that you're going to hit the nail on the head in terms of you guessed the actual maximum compensation that they would pay you to do the job, it's very unlikely.Corey: You're either going to guess too high and then basically get yourself disqualified—Josh: Yeah.Corey: —you're going to get too low and leave money on the table, or you're going to get it exactly right, but you'll never know whether you got it exactly right or whether you guessed low.Josh: Right. Even if you do guess exactly right, you won't know that you did. And so, of course, if you guess low, like you said, you leave money on the table. And the really pernicious thing is, you could guess low and still feel great about the result and never know it or not find out until the next time you get a job, which is to say, you know, you say a number that's well below the bottom of the minimum that they could pay you and so you say, I don't know, to use round numbers, you say $100,000. And they go, “Great, how about 120?”And you say, “Wow, they must really like me. They're going to pay, I just said 100 and they said 120. That's amazing.” And really what's going on is they're looking at, you know, their internal pay structure and they're like, we can't pay less than 120, like, the pay structure starts at 120. So, we'll pay 120, which is the literal bottom that you could make.You feel like you got a huge win of a 20% bump, but the reality is, you're probably not anywhere near the middle of that pay range and you're way behind the eight ball already. And of course, you could overshoot. And the worst-case scenario is you overshoot so far that you basically disqualify yourself from the process early. So, it's like, if it's on that first screening call, and you say—Corey: And they view you as being fundamentally unserious, where it's a, okay, the compensation for this role is 100 to 130, for example—to use made-up numbers—and you come in asking for 340. It's… okay like, there's no point in even doing a counter and having a negotiation at this point. We are so far apart, that it doesn't work out that way.Josh: Right, which on the surface, seems like oh, well, I just saved a bunch of time. But in reality, what you may have done is sort of like knocked yourself out of the entire hiring funnel for them, when what could have happened is perhaps you could have as you interviewed, you could have aligned better with a more senior role that would have had a higher pay range that you would have been a better fit for, you could have changed their budget based on the way that you present in your interviews and what they perceive from you. And who knows, maybe you actually do get an offer that looks like 340 because they say, “Oh, wow, we had you leveled as a, you know, an L6 and really should be, like, at an L7. So, how about this, you know, this senior or principal or lead role over here that we've been trying to fill for six months, we now realize you might be a good fit for that role. Why don't you go talk to that hiring manager, and if we have to, we'll just put you into that hiring stream?” Instead of, you set a giant number and we got to kick you out because there's no room for you here.Corey: This is all well and good and we're talking about effectively cash comp and salaries, but so many companies these days seem to tie a fair part of their compensation to the equity portion of it. And because remember, everything's up and to the right. Always. The end. Until one day, it's very much not.And now we're taking a look and seeing that, for example, Amazon stock has largely been in the toilet for a couple of years. It's what, 50% off of what it was at the peak.Josh: Yeah.Corey: So it's, on some level, when you're negotiating comp, it feels like you're being asked to predict the future of how well the company does. And at these multibillion-dollar company scales, are you really going to be in a position personally to meaningfully impact the stock price? Like, well, not positively anyway. And it just feels like it's a bit of a shell game where if you can't spot the sucker, it's probably you. Because I wanted to be an engineer, not a stockbroker.Josh: Yeah, I mean, first of all, you're right, that no individual engineer is really going to be impacting the bottom line of Google.Corey: Unless I take the site down.Josh: Right. Well, I was just [laugh]—man, you beat me to the punch on that one. Yeah. So, there is a possibility that one engineer could have a dramatic impact, but not the kind that you would hope if [laugh] you're also tied to their stock price, right? So, there's a couple of ways that I think about this.One of them, you mentioned the Amazon stock going down. So, one thing that's really interesting about that is really what Amazon is doing is they're targeting a total annual compensation number with their stock. And so, they start with their current known stock value—I don't know if they're doing this now, but for many years, they were just kind of building in a year-over-year growth number of 10 to 15%. So, we're going to give you this much total comp and we're targeting 300k total comp per year. And if you kind of map it out based on the base salary and the equity that vests and the signup bonuses they give you in years one and two, then it looks like a pretty flat, like, 300k a year when you build in that stock growth.So, the magic question that I started talking to—and had a couple of internal recruiter friends, like, last year, mid-year last year when things were looking pretty bad, and the question that I don't think that they had an answer to at the time and now they have answered is, well, what do we do when the stock doesn't grow 10 to 15% and actually kind of collapses, like, takes a huge nosedive? And the answer is that Amazon is still targeting a total comp of 300k a year. And they go back and they say, “Well, here's some more RSUs at the current value to kind of makeup for that. Here's your new vesting schedule on these.” They essentially are giving refreshers, and here's the new vesting schedule.And so, at least in Amazon's case, they did kind of try to right the ship. But the reason is that something you alluded to, you're not really getting equity in the company because you impact the company; you're getting equity in the company because it's another way for them to kind of generate, quote-unquote, “Cash flow” of some kind or comp, that isn't, you know, dollars coming off the books. So, this is something I think that's kind of a TBD is, Amazon has now answered this, which is we're going to give him—because otherwise, they're going to have a mass exodus, right, like, if you thought you're going to make 300k a year and you're actually going to make 180k a year, that's a huge dropoff, and you're probably going to be looking elsewhere. So, they say, “Well, here's some more RSUs.”The question is, you know, what will other companies do? All of this is, you know, we're talking about public companies here. So, there's a big difference between, like, Amazon stock, Google stock, whatever—or GSUs, whatever you want to call them—and then private, pre-IPO equity, and all these different things. I see those as much more in the category of what you described, which is, you know, if you're getting stock options on, like, an early stage, you know, like, an early stage startup, right, they're raising, like, their first or second or third round, you are going to have maybe kind of a large impact on the trajectory of the company, but on the price of that you have almost no agency whatsoever because of all the options that they have for dilution and all that other stuff that can go on and whether you even have shares that are going to be liquid at some point and all that stuff. So, I see that as much more like, you've just got to look at the company, the cash that they're paying you, how you feel about that, how you feel about the mission of the company, and understand that you've got, you know, you've got some lotto tickets in that company and who knows, maybe it goes to the moon and you get to go along for the ride, but much less certain than, you know, like I said, like, an Amazon-type situation where they actually will give you even more RSUs if the stock tanks over the course of the year.Corey: What are you seeing these days in terms of the macroeconomic conditions as a result? Like, some wit on Twitter said that the correction in the market has identified the grim reality that there are more engineers making $600,000 a year than there are engineering problems that need $600,000 engineers to fix them. So, there's a certain, are people being overcompensated? Is there a correction in the market? Is that changing the world of salary negotiation and peoples' job mobility?Josh: I think—working backwards—yes, job mobility is affected right now. I mean, I've seen you know, even in my own business, there are just fewer people reaching out and saying, “Hey, I have an offer at a big tech company.” Which is, you know, all over the news, layoffs. First, it was hiring freezes, right? This is late last year, October last year-ish, Q3, Q4, last year. They kind of said, “Oh, we're going to hire—we're going to slow down for a little bit on this hiring.”And then it was layoffs. And so, the last several months have been layoff after a layoff, you know, 5% here, 10% there at lots of different companies. Paradoxically, a lot of those companies are still up into the right, if you're looking at their stock price, lately. And I think a lot of that is back to the first thing that you said, which is, you know, do we have more engineers that are kind of sitting around looking for problems to solve than there are problems to solve? And I think the answer was probably, yes.Certainly, the pandemic, interest rates where they were, and all these other kind of macro-economic things, which I won't opine on too much because I'm not super-educated on them, but I understand them well enough to understand that basically, it was a better investment for a big company to hire an engineer, than necessarily to try to find somewhere to invest that money because interest rates were so low, so it's hard to find a nice quote-unquote, “Risk-free” return on the investment, so they said, “Why not? We'll just hire some engineers and maybe we'll get a bigger ROI there. We'll try a bunch of different projects, we'll put a bunch of people and maybe we'll go to the moon.”Corey: A lot of speculative or strategic hiring—Josh: Yeah.Corey: —and then okay, then you have—something that companies do when they have extra money is they greenlit additional projects. And when things get tight, they wind up effectively removing some of those projects from the table. And what I think people misunderstand in many cases is that compensation of employees is always more expensive than the infrastructure they work on, with very rare exceptions. So, the AWS bill is always secondary to payroll expenses, and fixing AWS cost takes time, effort, and engineering work, whereas laying people off requires a couple of difficult conversations—that companies increasingly seem to be bungling—and that's the end.Josh: Yeah. I think you're right about that. I mean, payroll, it's an old saw in businesses is that payroll is the biggest expense, right? Like, it's very expensive to hire people. But it could be the kind of thing, like you said, “We'll just fire up a bunch of these projects. We've been thinking about them anyway. We can't really invest this money anywhere else for a good return, so we'll take some shots here.” Right?But then interest rates go up and oh, there are places that I can get a nice return on this investment of cash, so maybe, you know, some of these projects that aren't going so well, we're going to shut them down. We're going to lean up a little bit. We're going to increase our margins, reduce our payroll costs, and just kind of ride this economic turmoil out and see how it goes. And who knows, maybe they'll fire some of those projects up later. But yes, it's much easier to say we're laying off 10% of our workforce tomorrow than it is to make a lot of other changes, especially on the expenses side.That's one of the few expenses I think that a company has direct control over and can simply reduce if they choose to. And that's kind of where we are right now, I think. And so, you mentioned economic mobility or job mobility. It's definitely way down. And I think the reason is that, you know, I mean, if I'd been through layoffs at companies that I worked at before, right?It's a really uncomfortable feeling, where the person that was sitting next to you in the office next to you gets laid off and you're sitting there wondering, “Am I going to be next?” And the last thing that you're going to do is start kind of poking your head up and looking for jobs and making it known that you're shopping, or even go ask for a raise or something because you're just trying to keep your head down and maybe the scythe will pass over me [laugh], right? Maybe they're going to miss me in this next round of layoffs if I just keep my mouth shut and I keep typing away here on my keyboard. So, I think a lot of that is going on where people are, if they're still employed, they're happy to be there and they're just going to kind of hunker down. And then if they're not employed, there's not a lot of them, you know, especially if you're coming from big tech, you would want to go most of the time to another big tech company.Like, that's why you're there, a lot of people aspire to work for big tech, they want to be in that ecosystem. But if all the big tech companies are laying people off or freezing hiring, there's nowhere to go. And so, there's nowhere to move if they want to. They don't want to make it known that they're looking to move because they don't want to draw attention to themselves if they're still employed. And if they're unemployed, the options for them to go somewhere are slim, but they probably have a severance package that they're kind of going to milk for a little bit and see if things kind of warm up again and they can go find somewhere to move to. So, everything feels, in the big tech level, there's a lot of inertia right now. People are just kind of sitting back, and there's a lot of friction, and they're just kind of hanging back to see what happens.Corey: And also, at least from my somewhat naive perspective, it feels like when people do get offers and they have made the decision to move on, there's an increasing sense of they should be thankful for what they get and not rock the boat by asking for more. But I vehemently disagree, to be very clear on this. I think that negotiate for the best package you can get. Do it in good faith and be responsible about it, but money that is life-changing to you is a rounding error at best for a lot of these companies. You will always be more invested in this than the counterparty that you're negotiating against. But it just really throws me and on some level, makes me sad watching people take less than they could be getting.Josh: Yeah. I mean, I think that's just the nature of people who are spooked when the economy is doing weird stuff. And it's an understandable reaction to it, but I agree with you. Just yesterday—you know, I'm in a bunch of [laugh] a bunch of different developer Slacks. I don't know which one this was, but I was in a developer Slack—and somebody was saying exactly that.They're like, “Yeah, I got this offer, it seems pretty good. I don't know if I should bother negotiating it, you know? Like, I, I—shouldn't I just be, you know, pretty satisfied with this thing that I got?” And I wrote a long response, which was, the short version of it was basically, “No.” And the reason is, think about all the costs that the company has incurred just to get to the point where they made you an offer?It was expensive for them. Believe me, a lot of money has been spent. They've gotten all the way to the finish line with you. I mean, the number is at least in the thousands of dollars; it's probably in the tens of thousands of dollars, especially if they flew out for an onside or something. If you went through an interview loop, just do the math on, well, I talked to six people for about an hour apiece. That's six hours right there of really expensive time probably at, like [laugh], you know, senior manager and above pay rates.So, they put a lot of money into trying to fill this role. They want to fill the role, especially in this environment. If you're that deep in the process, they've got a role that they probably feel is pretty crucial to be filled. So, you've got a lot of reasons that you should be optimistic about the value that you're bringing to that role and I think it's a mistake to not see what the maximum value is that you can get in return for the work that you're going to provide for them. So, I do think that being scared is not the right response there, again because they've made a significant investment to get to the point of making an offer.And remember their fallback, right, if you negotiate with them and they don't want to give you any more, I have never seen—and I underline the word ‘never—I've never seen that a big tech company, somebody negotiates, and the big tech company says, “Nevermind. Get out of here.” Job offer went away. I've never seen it.Corey: I was about to ask that because I've heard about it at startups. And back in years when I was on Twitter a lot more than I am now, I periodically have people messaging me saying that this happened to them. What should they do? Do I want to put the company on blast and the rest? It's something I learned relatively early on in that process was before I go off half-cocked—which I'm thrilled to do—can I get a screenshot of that email exchange back and forth?Because it hasn't happened often, but once or twice, what I have clearly seen is that the company makes an offer in good faith and the person comes back with what they believe is the professional way to negotiate for more money and it is such a screaming red flag that is basically fists-of-ham-powered here that companies are like, “Oh, thank God. We just learned this giant red flag. We can get out of this super easy by rescinding the offer because of the negotiation, rather than asking them who they think they're speaking to like that.” And that is the way of getting out of it in those cases. I don't think that's particularly common, and as you say, I don't suspect that happens at big tech companies.Josh: I mean, it's not a good look, right? There was a period last year where a big tech company… [laugh] I don't know if this is privileged information or not, but they were actually resending offers, and it's because they had gotten out over their skis. They were hiring way ahead of where they should have been, and then of course, everything turned and they had to start reducing headcount. So, they did, and then they started actually res—Corey: I can think of at least three companies off the top of my head that would qualify for that story. A lot of it came, but no one made an announcement that we're rescinding offers, but it doesn't take much on Twitter when you start seeing wow, 15 people all popped up at the same time claiming that. I wonder if they're telling—Josh: Weird.Corey: —the truth, given they've never—Josh: It's a pattern.Corey: —interacted with each other?Josh: Yes.Corey: Yeah…Josh: So, without putting them on blast, obviously, the reason I'm not saying their names is I would be putting them on bla—it's not a good look, right? Nobody wants to know that they're in the interview process for a company who is known for rescinding offers. And so, you know it wasn't a decision they took lightly. And so, to your point, companies are not just going to willy-nilly start pulling back offers because that's really terrible PR. I mean, it's just not a good idea.So, it's either what you said, which is—and this is something, like when I say, “I've never seen it; underline the word never,” right, what I mean is I work with people one-on-one for a living; that's what I do. None of my clients have ever had a job offer rescinded from big tech company. That's not to say it hasn't happened for reasons like you mentioned.Corey: Yeah, I have to imagine that the emails you help them craft to respond to these things don't start off with, “Now, listen here, asshole…”Josh: Right.Corey: Like, I sort of get the sense that that's not quite the negotiating tone that you take, most days.Josh: [laugh]. No. There's no, like, you know, “I've CC'd my lawyer on this email… and blah, blah—” you know, that's not how I negotiate; it's not a good way to negotiate if you want to get good results and build rapport with people. So, in general, if you follow what I would call, like, kind of good negotiating practices—which is self-serving because I would say that I've created a lot of them for salary negotiations, right—and if you're following the best practices there, everybody's understanding that we're having a professional, business conversation among, you know, [unintelligible 00:26:52] professionals. We're trying to find the best result, that's good for everybody and we're going to get there.And so, as long as you're not—you know, you mentioned, you know [laugh], I say, you know, pounding your fist on the desk and making ultimatums and stuff, like, that's not how I negotiate; you can hear it in the way I talk. You're going to be fine. They're not going to be rescinding offers and therefore, you have pretty much carte blanche to, in good faith, negotiate with them to see if there's more room to negotiate. And how aggressive you're being and what you're asking for, these are all things that are dependent on the situation, right? There's some cases where asking for another half a million a year would be completely absurd; in some cases where it's totally appropriate [laugh] and it just depends on what your situation is.Corey: For some roles, if you just accept the offer as given, you will lose status in their eyes, on some level. For example, one of the challenges we've had with contract negotiation has been when we hire folks to work on negotiations. It's one of those, like, “Okay, do we want somebody who accepts the first offer or do we want someone who really fights us tooth and nail over every aspect of it?” And it's, on some level, it's an extension to the interviewing process there.Josh: Yeah.Corey: I don't know what the right answer is on that I mostly shrug and make that my business partner's problem.Josh: I think it's a good metric to see, especially in your business, like, you want to know not only, like, can they negotiate contracts and all this stuff, but you want to know, like, how savvy are they in terms of business? And I think, in general, a person who just accepts the first offer they get in business, I will not say that they are not savvy because I don't know that, but it's not a signal of savviness, I think, to just outright accept the first thing that comes your way in business, in general.Corey: Oh, when I wind up interviewing people in person and telling them about offers and whatnot, in years past, it was always a, would you like me to sign it right now? It's… to be honest, I'm actually starting to reconsider having given it to you at all because only someone who is deranged is going to sign a contract they haven't read, and we don't try to hire for that.Josh: Right. Yeah, I mean, that's just not—especially when your job is negotiating—you want to know that this person is running a number of filters when they're considering, you know, what is probably a kind of a life-altering decision for them, right? And so, one of those filters is, “Are the terms of this contract good for me? Is there anything dangerous in here?” And one of the filters is, “Am I being appropriately compensated for the value that I'm going to bring?” That's the big one that I focus on, right?And there's a number of those filters and I think—you know, when I'm coaching someone, the first thing that we always say when a job offer comes in is, “Hey, thanks for the offer. I appreciate it. If you wouldn't mind, I'd like”—Corey: Yeah, acknowledge receipt.Josh: Yeah, yeah, “Thank you. I got the offer. Thank you for that.” And also acknowledge it and be thankful. Like, you know, “Hey, I appreciate it.” Like, “We have now made a significant step forward in this whole process that we're going through. I appreciate what you've done to get us here. I appreciate the fact that you're giving me an offer. That demonstrates a lot of trust and all these things. And if you don't mind, I'd like to take a day or two to think it over.” And then the last thing is, “Would you mind sending me a bullet-point summary in email of the numbers that you said, so I make sure I don't mess them up?”Because you're trying to avoid the very unlikely chance that they said numbers and you heard different numbers and then you start negotiating based on the different numbers and everything just kind of go sideways. So, that's the first three things: “Thank you for the offer.” “Can I have a day or two?” “Would you send me a bullet-point summary?” It doesn't have to be formal; just bullet points is fine.Corey: Would always irked me—and I you tend to see this a lot more with early career folks, but there's also this is a common failure mode as well among people who have been in one job for a while where they have gotten completely rusty at doing the interview dance. And they tend to view jobs as being this benevolent gift bestowed upon them by the employer and they become falling over themselves, just thanking them for the opportunity and the rest. And no, no, no, no, no. A job is a mutual exchange of value. You are solving a problem that the company has, and in turn, they are bringing you in and giving you a not inconsiderable amount of money—presumably—to wind up solving that problem for them, you both come out better than you were independently. That is what a job is. Confusing the power dynamic for something else feels, to me at least, like it's the wrong way to view things.Josh: Yeah. I've always not liked even the meta sort of way that we talk about jobs as, like, jobs created, jobs destroyed, somebody gave me a job. I don't know when that term—I would be curious actually, to kind of know the etymology of that term, but like, when we started describing jobs is the thing that was given or taken or—and instead, what it is, is it's a verbal contract or written contract. It's like, “Hey, I'm going to do work for you because I bring value. You're going to pay me because I'm creating value and because it's valuable to you. And we're going to figure out, you know, what's the meet-in-the-middle number, basically, that makes us both feel good about that business transaction.”You as a company can't do what you're doing without people like me. And I as a person have found a good place to flex that particular muscle at your company. That's great for both of us. Let's figure out how, you know, we can both be happy with it. So, it's definitely not that, you know, nobody's really doing anybody favors there. You're both entering into a mutual exchange of value for business reasons.And of course, your business reasons are different than theirs, but that's what they are. So yeah, I like the way that you frame that and you think about it. And I do think it can be a little harmful for people to have that perspective, especially like if they're in a position where they're thinking, “Oh, I'm so thankful that this company is willing to give me this job.” You know, “They're gifting me with this job and they're creating this job for me.” That's actually not what's happening.Corey: Something that I want to talk about, just because I've gone through this process myself as an employee, who interviewed a lot, negotiated a lot, and got hired a lot. Then I started this place and I've been on the other side of the table. And it turns out that it's not that hard to be a human being when you're the hiring manager and making these decisions. And understand the fact that yeah, you may be hiring five people this month, but these people aren't accepting five job offers a month—you hope—and going through that entire process themselves. And extending grace is just not that hard.Like, one thing that we've done since day one here has always been to put our salary compensation for the job in the job posting so we don't waste anyone's time. Where, like, “Well, what do you want to make?” It's like, if someone walks in to buy a car, the salesperson doesn't say, “Well, how much do you want to pay for it?” It doesn't work that way. It's, “This is the thing we're offering. This is the compensation we can build here. We don't do equity, so there's no funny money stories.”And yeah, I know you'd like to make three times more. So, would we, but without growth, that doesn't become sustainable. So, let's talk about how to get there. And being a responsible, decent human being is not that hard in the hiring space, but no one tells those stories because it's more fun, and outrage goes around the internet three times while the truth is still putting its boots on, where the idea of these horrible companies with people who don't know what they're doing just completely kicking themselves.Josh: Yeah, you know, it's funny, I thought, two things kind of flashed in my mind while you were talking. And the first one was, you know, I was a hiring manager for a while. And a lot of the sort of philosophy that I built around, like, asking for raises and promotions, right? Like, I have a process for that that's different than negotiating job offers, but the way that I developed it was as a hiring manager, my employees would say, you know—in their one-on-one or something—like, “Hey, you know, I feel like, you know, here's what happened when I started the company. For these reasons, I feel like I'm like, way behind where I should be in pay. Can you help?”And so, the way that I kind of approached that was, yes, I want to help them, but I cannot really do that on my own. I need a lot of information that I don't have for them. So, what information do I need from them to have them help me help them to get them a raise or get them or promotion, right? And so, I started thinking about it from the manager side of, like, essentially, kind of like a compassionate approach to, like, I need you to give me information and I will do what I can for you. And that was like, my whole philosophy with that, which is, I think I agree with you, but I need you to kind of prove to me that you should be paid more. Not because I don't believe you, but because I can't get you more money if I can't make that case, and I'm not able to make that case on my own, right?And so, I think that there is room for hiring managers to be compassionate in terms of like you said, just putting numbers in a job description, just so the person knows, like, yeah, this seems like it's probably approximately for me. Or you know, like I said, as a hiring manager saying, “Hey listen, I need you to bring me these three things. If you bring me those three things, that'd be the information that I need to go to finance or to HR or whoever and see if we can get you a raise or get you a promotion.” And if we can't, then I'll figure out, like, what are the next steps for you to get there to do that thing. And I think that in general, that's just removing friction from, like, forward-moving business processes and that's a good way to go.I think for you, right, you're saving yourself time, by putting those numbers in the job description, you're saving your applicants time by putting the numbers in the job description, and you're also kind of setting the terms for, like, the conversation that you're going to have, in addition to the abilities that they're bringing, the skills that they're going to bring, the things they're going to do for your company, you're also saving time on what the pay is going to be, what the compensation is going to look like approximately for that role, so that you can say, “Are we having a conversation whose parameters are known to us and that we agree upon to start with? Yes? Okay, great. Let's keep talking.” Otherwise, no, and maybe they should go somewhere else or maybe you need to rework your job listings because nobody is [laugh] applying for that job, right?But it's all data. It's a feedback loop. And it can be done compassionately. It doesn't have to be this, kind of, aggressive, you know, Shark Tank-style, like, I'm going to beat you over the head with this thing and get my result that I want, regardless of how you feel about it or, you know, how it makes you feel as a person.Corey: One last thing that I want to comment on this is that I've done this a fair bit, but if I wound up finding myself on the job market, I would absolutely reach out to you for coaching on the salary piece of it, just because you are a dispassionate third-party who is very aware of what the current state of the market is, you have a bunch of different offerings these days that range from a bunch of free articles on your website all the way up to individualized personalized coaching. I have bullied friends of mine into becoming your clients with a, “If he doesn't justify his fee, I will pay it instead of you.”Josh: [laugh]. Thank you for that, by the way.Corey: Of course. And I've never had to do it because you know what you're doing and the results absolutely speak for themselves. But my question is, what are you doing these days that's between the everything free on a website if you read it and, individualized one-on-one coaching? Are there now points in between those two extremes?Josh: Yeah. I think you actually summarized the whole spectrum pretty well. I mean, I've made—since I started my business seven-and-a-half years ago, one of the primary things that I did to start was, I'm going to create as much free content as I can and make it publicly available, just so that people can find it. Because there's no way that I can talk to tens of thousands, hundreds of thousands of people one-on-one. And so, that's there on fearlesssalarynegotiation.com.The other end is my one-on-one coaching which I developed because, frankly, people were reaching out and saying, “Hey, will you coach me through this?” And I said, “Sure.” And I developed that business. And then in between is, I created a program… three or four months ago, I launched it. It's called Salary Negotiation Mastery, but it is essentially me sitting down late last year with an instructional designer and asking the question, how can I teach the methodology I use in my one-on-one coaching to people who can't afford to hire me or just aren't inclined to hire a consultant to help them do something? And how can I teach that to them in a way that they can execute it on their own to get a good result, or possibly, you know, they're just at an income bracket right now where it doesn't make sense for them to hire me?And so, that's kind of the middle ground there is it's a coaching program, but it's wrapped in a do-it-yourself thing, where you have, you know, worksheets and workbooks and things that you can use to do it yourself using exactly the methodology, even the templates and things that I use with my clients. And the only thing, of course, that you're missing is my brain, but I've put as much of that as I can into the program as possible. So, that's the spectrum is: free articles, Salary Negotiation Mastery in the middle, and then the top tier offering that I have is, like you said, one-on-one bespoke coaching, where I work with somebody one-on-one. And I don't do a lot of that, just because it requires a lot of time and I like to give a ton of focus to everybody that I work with.Corey: Which makes sense because it also feels like it's a very time-sensitive issue as well. Like with AWS bill, great people want it fixed now, but then procurement can slow things down. But that's okay; there's another bill coming next month. Job offers, speaking as a hiring manager, if you accept the job, terrific, that's great. If you don't. Then okay, that's unfortunate, but it happens. But either way, let us know so we can either continue speaking to other people or begin planning for you to show up. So, it feels like there's very much a strong sense of urgency around the entirety of what you do.Josh: Yeah, especially for the coaching. And the whole offering for my coaching offering is really designed to make sure that I have enough bandwidth available for someone to call me. I mean, literally, as we're in this recording right now, I could have gotten an application in the email that would say, “Hey, I have a job offer in hand from Google. It's for this much money, it's for this level, can you help?”Corey: “And they're on the other line. Please respond immediately.”Josh: Yes. And their recruiter is pressuring me for an answer. They want to get back to the hiring manager. And so, I need to be able to respond quickly, get back to that person, have an intro call, get to know them, see if I can help in their situation, kickoff, you know, this afternoon or tomorrow morning, get a counteroffer over in the next 24 to 36 hours, that kind of thing. And so, in order to do that, I've got to build an offering that allows me to have enough bandwidth and, kind of, agency over my schedule so that I can just sort of jump in immediately into the middle of a process that's ongoing and help the person get the best result possible.So, I enjoy that to be honest with you. I like, kind of, being called on in emergency situations like that. It's really good. But of course, I had to structure the offering so that it facilitates it so that I'm not, you know, already booked on the phone eight hours a day and unable to even look at my email until tomorrow or something because it just wouldn't work.Corey: Yeah. There's something to be said for being able to take a vacation.Josh: Yes. Which takes some planning, but can be done [laugh]. And it means I just have to turn off the application sometimes [laugh].Corey: Glad to see things are still going well for you. You started your business a few months before I started mine and it's great to see that we're both still failing to go out of business every month.Josh: [laugh]. That's how I see it, too. I'm still here. Now, what [laugh]? That's every month on the first when I do the books.Corey: [laugh]. I hear you. I really want to thank you for taking the time to speak with me. If people want to learn more—and if they're changing jobs, they absolutely should—where should they go to find out?Josh: fearlesssalarynegotiation.com is the first place to go. I'm also on Twitter. I don't tweet a lot kind of actively, I probably should do better on that, but I'm at @joshdoody on Twitter and I'm very responsive on there. So, you could ping me on there or, you know, connect with me on LinkedIn if you wanted to; I'm also joshdoody there. But fearlesssalarynegotiation.com is the best place to go, especially if you're kind of in a time crunch. Everything is just right there for you to jump in and kind of grab, you know, the free resource that you might need or apply to work with me as a coaching client.Corey: Oh, the template emails are glorious.Josh: Yes. Those are one of my favorite things on the site. They don't look like other emails that people write, and something I take a lot of pride in is communicating well and creating good email templates that help a lot of people.Corey: Oh, in TextExpander, for a decade now, I've had a fill-in-the-blanks templated resignation letter, which it turns out, most people don't have. But I don't need it much these days, but it is useful to wind up giving to people from time to time. Like, “So, how do I tell my boss to take this job and shove it?” It's like—Josh: Well—Corey: —life is long and the industry is small. Go vent to your friends over beers. But there's very little upside and huge potential downside, so write the formal thing. Here you go. And it turns out that it's sort of cathartic, just filling that out. And it's like, oh, that's what this [unintelligible 00:42:05]. And it often helps people step back from the ledge sometimes. Or pushes them right off, depending.Josh: I think that's a useful service.Corey: But yeah, the [unintelligible 00:42:11] template emails are way better than mine.Josh: [laugh]. Well thanks, I appreciate it. It means a lot to me.Corey: [laugh]. Josh Doody the owner of fearlesssalarynegotiation.com. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment and be sure to include your salary expectations.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Eswar Bala, Director of Amazon EKS at AWS, joins Corey on Screaming in the Cloud to discuss how and why AWS built a Kubernetes solution, and what customers are looking for out of Amazon EKS. Eswar reveals the concerns he sees from customers about the cost of Kubernetes, as well as the reasons customers adopt EKS over ECS. Eswar gives his reasoning on why he feels Kubernetes is here to stay and not just hype, as well as how AWS is working to reduce the complexity of Kubernetes. Corey and Eswar also explore the competitive landscape of Amazon EKS, and the new product offering from Amazon called Karpenter.About EswarEswar Bala is a Director of Engineering at Amazon and is responsible for Engineering, Operations, and Product strategy for Amazon Elastic Kubernetes Service (EKS). Eswar leads the Amazon EKS and EKS Anywhere teams that build, operate, and contribute to the services customers and partners use to deploy and operate Kubernetes and Kubernetes applications securely and at scale. With a 20+ year career in software , spanning multimedia, networking and container domains, he has built greenfield teams and launched new products multiple times.Links Referenced: Amazon EKS: https://aws.amazon.com/eks/ kubernetesthemuchharderway.com: https://kubernetesthemuchharderway.com kubernetestheeasyway.com: https://kubernetestheeasyway.com EKS documentation: https://docs.aws.amazon.com/eks/ EKS newsletter: https://eks.news/ EKS GitHub: https://github.com/aws/eks-distro TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It's easy to **BEEP** up on AWS. Especially when you're managing your cloud environment on your own!Mission Cloud un **BEEP**s your apps and servers. Whatever you need in AWS, we can do it. Head to missioncloud.com for the AWS expertise you need. Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. Today's promoted guest episode is brought to us by our friends at Amazon. Now, Amazon is many things: they sell underpants, they sell books, they sell books about underpants, and underpants featuring pictures of books, but they also have a minor cloud computing problem. In fact, some people would call them a cloud computing company with a gift shop that's attached. Now, the problem with wanting to work at a cloud company is that their interviews are super challenging to pass.If you want to work there, but can't pass the technical interview for a long time, the way to solve that has been, “Ah, we're going to run Kubernetes so we get to LARP as if we worked at a cloud company but don't.” Eswar Bala is the Director of Engineering for Amazon EKS and is going to basically suffer my slings and arrows about one of the most complicated, and I would say overwrought, best practices that we're seeing industry-wide. Eswar, thank you for agreeing to subject yourself to this nonsense.Eswar: Hey, Corey, thanks for having me here.Corey: [laugh]. So, I'm a little bit unfair to Kubernetes because I wanted to make fun of it and ignore it. But then I started seeing it in every company that I deal with in one form or another. So yes, I can still sit here and shake my fist at the tide, but it's turned into, “Old Man Yells at Cloud,” which I'm thrilled to embrace, but everyone's using it. So, EKS has recently crossed, I believe, the five-year mark since it was initially launched. What is EKS other than Amazon's own flavor of Kubernetes?Eswar: You know, the best way I can define EKS is, EKS is just Kubernetes. Not Amazon's version of Kubernetes. It's just Kubernetes that we get from the community and offer it to customers to make it easier for them to consume. So, EKS. I've been with EKS from the very beginning when we thought about offering a managed Kubernetes service in 2017.And at that point, the goal was to bring Kubernetes to enterprise customers. So, we have many customers telling us that they want us to make their life easier by offering a managed version of Kubernetes that they've actually beginning to [erupt 00:02:42] at that time period, right? So, my goal was to figure it out, what does that service look like and which customer base should be targeting service towards.Corey: Kelsey Hightower has a fantastic learning tool out there in a GitHub repo called, “Kubernetes the Hard Way,” where he talks you through building the entire thing, start to finish. I wound up forking it and doing that on top of AWS, and you can find that at kubernetesthemuchharderway.com. And that was fun.And I went through the process and my response at the end was, “Why on earth would anyone ever do this more than once?” And we got that sorted out, but now it's—customers aren't really running these things from scratch. It's like the Linux from Scratch project. Great learning tool; probably don't run this in production in the same way that you might otherwise because there are better ways to solve for the problems that you will have to solve yourself when you're building these things from scratch. So, as I look across the ecosystem, it feels like EKS stands in the place of the heavy, undifferentiated lifting of running the Kubernetes control plane so customers functionally don't have to. Is that an effective summation of this?Eswar: That is precisely right. And I'm glad you mentioned, “Kubernetes the Hard Way,” I'm a big fan of that when it came out. And if anyone who did that tutorial, and also your tutorial, “Kubernetes the Harder Way,” would walk away thinking, “Why would I pick this technology when it's super complicated to setup?” But then you see that customers love Kubernetes and you see that reflected in the adoption, even in 2016, 2017 timeframes.And the reason is, it made life easier for application developers in terms of offering web services that they wanted to offer to their customer base. And because of all the features that Kubernetes brought on, application lifecycle management, service discoveries, and then it evolved to support various application architectures, right, in terms of stateless services, stateful applications, and even daemon sets, right, like for running your logging and metrics agents. And these are powerful features, at the end of the day, and that's what drove Kubernetes. And because it's super hard to get going to begin with and then to operate, the day-two operator experience is super complicated.Corey: And the day one experience is super hard and the day two experience of, “Okay, now I'm running it and something isn't working the way it used to. Where do I start,” has been just tremendously overwrought. And frankly, more than a little intimidating.Eswar: Exactly. Right? And that exactly was our opportunity when we started in 2017. And when we started, there was question on, okay, should we really build a service when you have an existing service like ECS in place? And by the way, like, I did work in ECS before I started working in EKS from the beginning.So, the answer then was, it was about giving what customers want. And their space for many container orchestration systems, right, ECS was the AWS service at that point in time. And our thinking was, how do we give customers what they wanted? They wanted a Kubernetes solution. Let's go build that. But we built it in a way that we remove the undifferentiated heavy lifting of managing Kubernetes.Corey: One of the weird things that I find is that everyone's using Kubernetes, but I don't see it in the way that I contextualize the AWS universe, which of course, is on the bill. That's right. If you don't charge for something in AWS Lambda, and preferably a fair bit, I don't tend to know it exists. Like, “What's an IAM and what might that possibly do?” Always have reassuring thing to hear from someone who's often called an expert in this space. But you know, if it doesn't cost money, why do I pay attention to it?The control plane is what EKS charges for, unless you're running a bunch of Fargate-managed pods and containers to wind up handling those things. So, it mostly just shows up as an addenda to the actual big, meaty portions of the belt. It just looks like a bunch of EC2 instances with some really weird behavior patterns, particularly with regard to auto-scaling and crosstalk between all of those various nodes. So, it's a little bit of a murder mystery, figuring out, “So, what's going on in this environment? Do you folks use containers at all?” And the entire Kubernetes shop is looking at me like, “Are you simple?”No, it's just I tend to disregard the lies that customers say, mostly to themselves because everyone has this idea of what's going on in their environment, but the bill speaks. It's always been a little bit of an investigation to get to the bottom of anything that involves Kubernetes at significant points of scale.Eswar: Yeah, you're right. Like if you look at EKS, right, like, we started with managing the control plane to begin with. And managing the control plane is a drop in the bucket when you actually look at the costs in terms of operating a Kubernetes cluster or running a Kubernetes cluster. When you look at how our customers use and where they spend most of their cost, it's about where their applications run; it's actually the Kubernetes data plane and the amount of compute and memory that the applications end of using end up driving 90% of the cost. And beyond that is the storage, beyond that as a networking costs, right, and then after that is the actual control plane costs. So, the problem right now is figuring out, how do we optimize our costs for the application to run on?Corey: On some level, it requires a little bit of understanding of what's going on under the hood. There have been a number of cost optimization efforts that have been made in the Kubernetes space, but they tend to focus around stuff that I find relatively, well, I call it banal because it basically is. You're looking at the idea of, okay, what size instances should you be running, and how well can you fill them and make sure that all the resources per node wind up being taken advantage of? But that's also something that, I guess from my perspective, isn't really the interesting architectural point of view. Whether or not you're running a bunch of small instances or a few big ones or some combination of the two, that doesn't really move the needle on any architectural shift, whereas ingesting a petabyte a month of data and passing 50 petabytes back and forth between availability zones, that's where it starts to get really interesting as far as tracking that stuff down.But what I don't see is a whole lot of energy or effort being put into that. And I mean, industry-wide, to be clear. I'm not attempting to call out Amazon specifically on this. That's [laugh] not the direction I'm taking this in. For once. I know, I'm still me. But it seems to be just an industry-wide issue, where zone affinity for Kubernetes has been a very low priority item, even on project roadmaps on the Kubernetes project.Eswar: Yeah, the Kubernetes does provide ability for customers to restrict their workloads within as particular [unintelligible 00:09:20], right? Like, there is constraints that you can place on your pod specs that end up driving applications towards a particular AZ if they want, right? You're right, it's still left to the customers to configure. Just because there's a configuration available doesn't mean the customers use it. If it's not defaulted, most of the time, it's not picked up.That's where it's important for service providers—like EKS—to offer ability to not only provide the visibility by means of reporting that it's available using tools like [Cue Cards 00:09:50] and Amazon Billing Explorer but also provide insights and recommendations on what customers can do. I agree that there's a gap today. For example in EKS, in terms of that. Like, we're slowly closing that gap and it's something that we're actively exploring. How do we provide insights across all the resources customers end up using from within a cluster? That includes not just compute and memory, but also storage and networking, right? And that's where we are actually moving towards at this point.Corey: That's part of the weird problem I've found is that, on some level, you get to play almost data center archaeologists when you start exploring what's going on in these environments. I found one of the only reliable ways to get answers to some of this stuff has been oral tradition of, “Okay, this Kubernetes cluster just starts hurling massive data quantities at 3 a.m. every day. What's causing that?” And it leads to, “Oh, no no, have you talked to the data science team,” like, “Oh, you have a data science team. A common AWS billing mistake.” And exploring down that particular path sometimes pays dividends. But there's no holistic way to solve that globally. Today. I'm optimistic about tomorrow, though.Eswar: Correct. And that's where we are spending our efforts right now. For example, we recently launched our partnership with Cue Cards, and Cue Cards is now available as an add-on from the Marketplace that you can easily install and provision on Kubernetes EKS clusters, for example. And that is a start. And Cue Cards is amazing in terms of features, in terms of insight it offers, right, it looking into computer, the memory, and the optimizations and insights it provides you.And we are also working with the AWS Cost and Usage Reporting team to provide a native AWS solution for the cost reporting and the insights aspect as well in EKS. And it's something that we are going to be working really closely to solve the networking gaps in the near future.Corey: What are you seeing as far as customer concerns go, with regard to cost and Kubernetes? I see some things, but let's be very clear here, I have a certain subset of the market that I spend an inordinate amount of time speaking to and I always worry that what I'm seeing is not holistically what's going on in the broader market. What are you seeing customers concerned about?Eswar: Well, let's start from the fundamentals here, right? Customers really want to get to market faster, whatever services and applications that they want to offer. And they want to have it cheaper to operate. And if they're adopting EKS, they want it cheaper to operate in Kubernetes in the cloud. They also want a high performance, they also want scalability, and they want security and isolation.There's so many parameters that they have to deal with before they put their service on the market and continue to operate. And there's a fundamental tension here, right? Like they want cost efficiency, but they also want to be available in the market quicker and they want performance and availability. Developers have uptime, SLOs, and SLAs is to consider and they want the maximum possible resources that they want. And on the other side, you've got financial leaders and the business leaders who want to look at the spending and worry about, like, okay, are we allocating our capital wisely? And are we allocating where it makes sense? And are we doing it in a manner that there's very little wastage and aligned with our customer use, for example? And this is where the actual problems arise from [unintelligible 00:13:00].Corey: I want to be very clear that for a long time, one of the most expensive parts about running Kubernetes has not been the infrastructure itself. It's been the people to run this responsibly, where it's the day two, day three experience where for an awful lot of companies like, oh, we're moving to Kubernetes because I don't know we read it in an in-flight magazine or something and all the cool kids are doing it, which honestly during the pandemic is why suddenly everyone started making better IT choices because they're execs were not being exposed to airport ads. I digress. The point, though, is that as customers are figuring this stuff out and playing around with it, it's not sustainable that every company that wants to run Kubernetes can afford a crack SRE team that is individually incredibly expensive and collectively staggeringly so. That it seems to be the real cost is the complexity tied to it.And EKS has been great in that it abstracts an awful lot of the control plane complexity away. But I still can't shake the feeling that running Kubernetes is mind-bogglingly complicated. Please argue with me and tell me I'm wrong.Eswar: No, you're right. It's still complicated. And it's a journey towards reducing the complexity. When we launched EKS, we launched only with managing the control plane to begin with. And that's where we started, but customers had the complexity of managing the worker nodes.And then we evolved to manage the Kubernetes worker nodes in terms two products: we've got Managed Node Groups and Fargate. And then customers moved on to installing more agents in their clusters before they actually installed their business applications, things like Cluster Autoscaler, things like Metric Server, critical components that they have come to rely on, but doesn't drive their business logic directly. They are supporting aspects of driving core business logic.And that's how we evolved into managing the add-ons to make life easier for our customers. And it's a journey where we continue to reduce the complexity of making it easier for customers to adopt Kubernetes. And once you cross that chasm—and we are still trying to cross it—once you cross it, you have the problem of, okay so, adopting Kubernetes is easy. Now, we have to operate it, right, which means that we need to provide better reporting tools, not just for costs, but also for operations. Like, how easy it is for customers to get to the application level metrics and how easy it is for customers to troubleshoot issues, how easy for customers to actually upgrade to newer versions of Kubernetes. All of these challenges come out beyond day one, right? And those are initiatives that we have in flight to make it easier for customers [unintelligible 00:15:39].Corey: So, one of the things I see when I start going deep into the Kubernetes ecosystem is, well, Kubernetes will go ahead and run the containers for me, but now I need to know what's going on in various areas around it. One of the big booms in the observability space, in many cases, has come from the fact that you now need to diagnose something in a container you can't log into and incidentally stopped existing 20 minutes for you got the alert about the issue, so you'd better hope your telemetry is up to snuff. Now, yes, that does act as a bit of a complexity burden, but on the other side of it, we don't have to worry about things like failed hard drives taking systems down anymore. That has successfully been abstracted away by Kubernetes, or you know, your cloud provider, but that's neither here nor there these days. What are you seeing as far as, effectively, the sidecar pattern, for example of, “Oh, you have too many containers and need to manage them? Have you considered running more containers?” Sounds like something a container salesman might say.Eswar: So, running containers demands that you have really solid observability tooling, things that you're able to troubleshoot—successfully—debug without the need to log into the containers itself. In fact, that's an anti-pattern, right? You really don't want a container to have the ability to SSH into a particular container, for example. And to be successful at it demands that you publish your metrics and you publish your logs. All of these are things that a developer needs to worry about today in order to adopt containers, for example.And it's on the service providers to actually make it easier for the developers not to worry about these. And all of these are available automatically when you adopt a Kubernetes service. For example, in EKS, we are working with our managed Prometheus service teams inside Amazon, right—and also CloudWatch teams—to easily enable metrics and logging for customers without having to do a lot of heavy lifting.Corey: Let's talk a little bit about the competitive landscape here. One of my biggest competitors in optimizing AWS bills is Microsoft Excel, specifically, people are going to go ahead and run it themselves because, “Eh, hiring someone who's really good at this, that sounds expensive. We can screw it up for half the cost.” Which is great. It seems to me that one of your biggest competitors is people running their own control plane, on some level.I don't tend to accept the narrative that, “Oh, EKS is expensive that winds up being what 35 bucks or 70 bucks or whatever it is per control plane per cluster on a monthly basis.” Okay, yes, that's expensive if you're trying to stay completely within a free tier perhaps, but if you're running anything that's even slightly revenue-generating or a for-profit company, you will spend far more than that just on people's time. I have no problems—for once—with the EKS pricing model, start to finish. Good work on that. You've successfully nailed it. But are you seeing significant pushback from the industry of, “Nope, we're going to run our own Kubernetes management system instead because we enjoy pain, corporately speaking.”Eswar: Actually, we are in a good spot there, right? Like, at this point, customers who choose to run Kubernetes on AWS by themselves and not adopt EKS just fall into one main category, so—or two main categories: number one, they have existing technical stack built on running Kubernetes on themselves and they'd rather maintain that and not moving to EKS. Or they demand certain custom configurations of the Kubernetes control plane that EKS doesn't support. And those are the only two reasons why we see customers not moving into EKS and prefer to run their own Kubernetes on AWS clusters.[midroll 00:19:46]Corey: It really does seem, on some level, like there's going to be a… I don't want to say reckoning because that makes it sound vaguely ominous and that's not the direction that I intend for things to go in, but there has to be some form of collapsing of the complexity that is inherent to all of this because the entire industry has always done that. An analogy that I fall back on because I've seen this enough times to have the scars to show for it is that in the '90s, running a web server took about a week of spare time and an in-depth knowledge of GCC compiler flags. And then it evolved to ah, I could just unzip a tarball of precompiled stuff, and then RPM or Deb became a thing. And then Yum, or something else, or I guess apt over in the Debian land to wind up wrapping around that. And then you had things like Puppet where it was it was ensure installed. And now it's Docker Run.And today, it's a checkbox in the S3 console that proceeds to yell at you because you're making a website public. But that's neither here nor there. Things don't get harder with time. But I've been surprised by how I haven't yet seen that sort of geometric complexity collapsing of around Kubernetes to make it easier to work with. Is that coming or are we going to have to wait for the next cycle of things?Eswar: Let me think. I actually don't have a good answer to that, Corey.Corey: That's good, at least because if you did, I'd worried that I was just missing something obvious. That's kind of the entire reason I ask. Like, “Oh, good. I get to talk to smart people and see what they're picking up on that I'm absolutely missing.” I was hoping you had an answer, but I guess it's cold comfort that you don't have one off the top of your head. But man, is it confusing.Eswar: Yeah. So, there are some discussions in the community out there, right? Like, it's Kubernetes the right layer to do interact? And there are some tooling that's built on top of Kubernetes, for example, Knative that tries to provide a serverless layer on top of Kubernetes, for example. There are also attempts at abstracting Kubernetes completely and providing tooling that just completely removes any sort of Kubernetes API out of the picture and maybe a specific CI/CD-based solution that takes it from the source and deploys the service without even showing you that there's Kubernetes underneath, right?All of these are evolutions that are being tested out there in the community. Time will tell whether these end up sticking. But what's clear here is the gravity around Kubernetes. All sorts of tooling that gets built on top of Kubernetes, all the operators, all sorts of open-source initiatives that are built to run on Kubernetes. For example, Spark, for example, Cassandra, so many of these big, large-scale, open-source solutions are now built to run really well on Kubernetes. And that is the gravity that's pushing Kubernetes at this point.Corey: I'm curious to get your take on one other, I would consider interestingly competitive spaces. Now, because I have a domain problem, if you go to kubernetestheeasyway.com, you'll wind up on the ECS marketing page. That's right, the worst competition in the world: the people who work down the hall from you.If someone's considering using ECS, Elastic Container Service versus EKS, Elastic Kubernetes Service, what is the deciding factor when a customer's making that determination? And to be clear, I'm not convinced there's a right or wrong answer. But I am curious to get your take, given that you have a vested interest, but also presumably don't want to talk complete smack about your colleagues. But feel free to surprise me.Eswar: Hey, I love ECS, by the way. Like I said, I started my life in the AWS in ECS. So look, ECS is a hugely successful container orchestration service. I know we talk a lot about Kubernetes, I know there's a lot of discussions around Kubernetes, but I wouldn't make it a point that, like, ECS is a hugely successful service. Now, what determines how customers go to?If customers are… if the customers tech stack is entirely on AWS, right, they use a lot of AWS services and they want an easy way to get started in the container world that has really tight integration with other AWS services without them having to configure a lot, ECS is the way, right? And customers have actually seen terrific success adopting ECS for that particular use case. Whereas EKS customers, they start with, “Okay, I want an open-source solution. I really love Kubernetes. I lo—or, I have a tooling that I really like in the open-source land that really works well with Kubernetes. I'm going to go that way.” And those kind of customers end up picking EKS.Corey: I feel like, on some level, Kubernetes has become the most the default API across a wide variety of environments. AWS obviously, but on-prem other providers. It seems like even the traditional VPS companies out there that offer just rent-a-server in the cloud somewhere are all also offering, “Oh, and we have a Kubernetes service as well.” I wound up backing a Kickstarter project that runs a Kubernetes cluster with a shared backplane across a variety of Raspberries Pi, for example. And it seems to be almost everywhere you look.Do you think that there's some validity to that approach of effectively whatever it is that we're going to wind up running in the future, it's going to be done on top of Kubernetes or do you think that that's mostly hype-driven these days?Eswar: It's definitely not hype. Like we see the proof in the kind of adoption we see. It's becoming the de facto container orchestration API. And with all the tooling, open-source tooling that's continuing to build on top of Kubernetes, CNCF tooling ecosystem that's actually spawned to actually support Kubernetes at option, all of this is solid proof that Kubernetes is here to stay and is a really strong, powerful API for customers to adopt.Corey: So, four years ago, I had a prediction on Twitter, and I said, “In five years, nobody will care about Kubernetes.” And it was in February, I believe, and every year, I wind up updating an incrementing a link to it, like, “Four years to go,” “Three years to go,” and I believe it expires next year. And I have to say, I didn't really expect when I made that prediction for it to outlive Twitter, but yet, here we are, which is neither here nor there. But I'm curious to get your take on this. But before I wind up just letting you savage the naive interpretation of that, my impression has been that it will not be that Kubernetes has gone away. That is ridiculous. It is clearly in enough places that even if they decided to rip it out now, it would take them ten years, but rather than it's going to slip below the surface level of awareness.Once upon a time, there was a whole bunch of energy and drama and debate around the Linux virtual memory management subsystem. And today, there's, like, a dozen people on the planet who really have to care about that, but for the rest of us, it doesn't matter anymore. We are so far past having to care about that having any meaningful impact in our day-to-day work that it's just, it's the part of the iceberg that's below the waterline. I think that's where Kubernetes is heading. Do you agree or disagree? And what do you think about the timeline?Eswar: I agree with you; that's a perfect analogy. It's going to go the way of Linux, right? It's here to stay; it just going to get abstracted out if any of the abstraction efforts are going to stick around. And that's where we're testing the waters there. There are many, many open-source initiatives there trying to abstract Kubernetes. All of these are yet to gain ground, but there's some reasonable efforts being made.And if they are successful, they just end up being a layer on top of Kubernetes. Many of the customers, many of the developers, don't have to worry about Kubernetes at that point, but a certain subset of us in the tech world will need to do a deal with Kubernetes, and most likely teams like mine that end up managing and operating their Kubernetes clusters.Corey: So, one last question I have for you is that if there's one thing that AWS loves, it's misspelling things. And you have an open-source offering called Karpenter spelled with a K that is an extending of that tradition. What does Karpenter do and why would someone use it?Eswar: Thank you for that. Karpenter is one of my favorite launches in the last one year.Corey: Presumably because you're terrible at the spelling bee back when you were a kid. But please tell me more.Eswar: [laugh]. So Karpenter, is an open-source flexible and high performance cluster auto-scaling solution. So basically, when your cluster needs more capacity to support your workloads, Karpenter automatically scales the capacity as needed. For people that know the Kubernetes space well, there's an existing component called Cluster Autoscaler that fills this space today. And it's our take on okay, so what if we could reimagine the capacity management solution available in Kubernetes? And can we do something better? Especially for cases where we expect terrific performance at scale to enable cost efficiency and optimization use cases for our customers, and most importantly, provide a way for customers not to pre-plan a lot of capacity to begin with.Corey: This is something we see a lot, in the sense of very bursty workloads where, okay, you're going to steady state load. Cool. Buy a bunch of savings plans, get things set up the way you want them, and call it a day. But when it's bursty, there are challenges with it. Folks love using Spot, but in the event of a sudden capacity shortfall, the question is, is can we spin up capacity to backfill it within those two minutes that we have a warning on that on? And if the answer is no, then it becomes a bit of a non-starter.Customers have had to build an awful lot of those things around EC2 instances that handle a lot of that logic for them in ways that are tuned specifically for their use cases. I'm encouraged to see there's a Kubernetes story around this that starts to remove some of that challenge from the customer side.Eswar: Yeah. So, the burstiness is where complexity comes [here 00:29:42], right? Like many customers for steady state, they know what their capacity requirements are, they set up the capacity, they can also reason out what is the effective capacity needed for good utilization for economical reasons and they can actually pre plan that and set it up. But once burstiness comes in, which inevitably does it at [unintelligible 00:30:05] applications, customers worry about, “Okay, am I going to get the capacity that I need in time that I need to be able to service my customers? And am I confident at it?”If I'm not confident, I'm going to actually allocate capacity beforehand, assuming that I'm going to actually get the burst that I needed. Which means, you're paying for resources that you're not using at the moment. And the burstiness might happen and then you're on the hook to actually reduce the capacity for it once the peak subsides at the end of the [day 00:30:36]. And this is a challenging situation. And this is one of the use cases that we targeted Karpenter towards.Corey: I find that the idea that you're open-sourcing this is fascinating because of two reasons. One, it does show a willingness to engage with the community that… again, it's difficult. When you're a big company, people love to wind up taking issue with almost anything that you do. But for another, it also puts it out in the open, on some level, where, especially when you're talking about cost optimization and decisions that affect cost, it's all out in public. So, people can look at this and think, “Wait a minute, it's not—what is this line of code that means if it's toward the end of the month, crank it up because we might need to hit our numbers.” Like, there's nothing like that in there. At least I'm assuming. I'm trusting that other people have read this code because honestly, that seems like a job for people who are better at that than I am. But that does tend to breed a certain element of trust.Eswar: Right. It's one of the first things that we thought about when we said okay, so we have some ideas here to actually improve the capacity management solution for Kubernetes. Okay, should we do it out in the open? And the answer was a resounding yes, right? I think there's a good story here that actually enables not just AWS to offer these ideas out there, right, and we want to bring it to all sorts of Kubernetes customers.And one of the first things we did is to architecturally figure out all the core business logic of Karpenter, which is, okay, how to schedule better, how quickly to scale, what is the best instance types to pick for this workload. All of that business logic was abstracted out from the actual cloud provider implementation. And the cloud provider implementation is super simple. It's just creating instances, deleting instances, and describing instances. And it's something that we bake from the get-go so it's easier for other cloud providers to come in and to add their support to it. And we as a community actually can take these ideas forward in a much faster way than just AWS doing it.Corey: I really want to thank you for taking the time to speak with me today about all these things. If people want to learn more, where's the best place for them to find you?Eswar: The best place to learn about EKS, right, as EKS evolves, is using our documentation, we have an EKS newsletter that you can go subscribe, and you can also find us on GitHub where we share our product roadmap. So, it's a great places to learn about how EKS is evolving and also sharing your feedback.Corey: Which is always great to hear, as opposed to, you know, in the AWS Console, where we live, waiting for you to stumble upon us, which, yeah. No it's good does have a lot of different places for people to engage with you. And we'll put links to that, of course, in the [show notes 00:33:17]. Thank you so much for being so generous with your time. I appreciate it.Eswar: Corey, really appreciate you having me.Corey: Eswar Bala, Director of Engineering for Amazon EKS. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice telling me why, when it comes to tracking Kubernetes costs, Microsoft Excel is in fact the superior experience.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

An airhacks.fm conversation with Sascha Moellering (@sascha242) about: Schneider CPC, starting programming with C-16, enjoying Finger's Malone, upgrade to C-128, playing Turrican, Manfred Trenz created Turrican and R-Type, publishing a Pommes Game, programming on Amiga 1200, math in game development, implementing a painting application, walking through C pointer and reference hell, from C to Java 1.0 on a Mac 6500 with 200MHz, using Metrowerks JVM, using CodeWarrior, CodeWarrior vs. stormc, Java is a clean language, working on SpiritLink, using Caucho Resin, starting at Accenture, from Accenture to Softlab, building a PaaS solution with JBoss for Allianz, managing hundreds of JVMs with a pizza team, implementing a low latency marketing solution with Vert.x, starting at Zanox, an episode with Arjan Tijms "#184 Piranha: Headless Applets Loaded with Maven", starting at AWS as Account Solution Architect, using quarkus on lambda as a microservice, using POJO asynchronous lambdas, EJB programming restrictions and Lambdas, airhacks discord server, Optimize your Spring Boot application for AWS Fargate, Reactive Microservices Architecture on AWS, Field Notes: Optimize your Java application for Amazon ECS with Quarkus, Field Notes: Optimize your Java application for AWS Lambda with Quarkus, How to deploy your Quarkus application to Amazon EKS, Using GraalVM to Build Minimal Docker Images for Java Applications Sascha Moellering on twitter: @sascha242

In this episode, we discuss running container based workloads anywhere with industry experts: Nathan Walker, Senior Solutions Architect at AWS and Mohan Potheri, Cloud Solutions Architect at Intel Whether your use case(s) require you to have a footprint in:-AWS Cloud-On-Premises-Hybrid Cloud Summary:This talk will kickoff with a brief discussion of Modern Application Architecture to set the stage and then quickly dive into why containers have become an integral part in the Cloud Native Journey. The talk will then focus specifically on Kubernetes based workloads which leverage Amazon EKS within AWS Cloud AND the Amazon EKS Distro for Amazon EKS Anywhere, outside of AWS Cloud. We'll also be joined by a special guest with deep subject matter expertise on Intel backed Amazon EC2 Instances as well as Intel backed On-Prem servers to optimize your Hybrid Cloud Kubernetes Clusters with Amazon EKS and Amazon EKS Anywhere. Link to Blog this podcast discusses: https://community.intel.com/t5/Blogs/Tech-Innovation/Cloud/Leveraging-Amazon-EKS-Anywhere-with-Intel-optimized-instances/post/1411176 *All views shared do NOT reflect current or former employers

On this episode of The Cloud Pod, the team talks about the possible replacement of CEO Sundar Pichai after Alphabet stock went up by just 1.9%, the new support feature of Amazon EKS for Kubernetes, three partner specializations just released by Google, and how clients have responded to the AI Powered Bing and Microsoft Edge. A big thanks to this week's sponsor, Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. This week's highlights

Links: AWS Purity Test  Amazon Detective adds Amazon VPC Flow Logs visualizations for Amazon EKS workloads  AWS Elemental MediaLive adds timecode burn-in  AWS Pricing Calculator now supports optimized pricing estimation for EC2 Dedicated Hosts  Announcing Porting Advisor for Graviton  Now Open — AWS Asia Pacific (Melbourne) Region in Australia  Amazon OpenSearch Serverless is now generally available! AWS Lambda: Resilience under-the-hood  VPC Routing Enhancements and GWLB Deployment Patterns Introducing AWS Lambda runtime management controls 

Amazon EKS Anywhere is a deployment option for Amazon EKS that allows customers to create and operate Kubernetes clusters on customer-managed infrastructure, supported by AWS. With support for bare metal deployments on Amazon EKS Anywhere, customers now have a broader choice of infrastructure for running Kubernetes on-premises. As customers modernize their applications, they want to use Kubernetes consistently between their existing on-premises bare metal infrastructure and the cloud. Running Kubernetes on bare metal infrastructure is complex, and customers spend time, effort and money on infrastructure operations instead of focusing on business innovation. In this podcast, you will learn how Amazon EKS Anywhere on bare metal enables customers to automate all steps - from bare metal hardware provisioning to Kubernetes cluster operations - using a bundled open source toolset built on the foundation of Tinkerbell and Cluster API. Get Started with Bare Metal: https://bit.ly/3XJR5UX Connect with an Amazon EKS Anywhere Specialist today: https://go.aws/408S8zt Learn more about EKS Anywhere: https://go.aws/407ds8w

In the first episode of season 3, Ryan and Bhavin talk to Michael Guarino - the CTO of plural.sh about how plural helps users deploy applications on Kubernetes easily. They discuss the challenges associated with deploying applications consistently across different Kubernetes distributions, and talk about how Plural provides a unified solution that auto-generates Kubernetes manifests, HELM charts, and Terraform files and follows GitOps principles to deploy applications across Amazon EKS, Azure AKS, and Google GKE. News: Kubernetes v1.26: Alpha support for cross-namespace storage data sources Best of 2022: 8 CNCF Projects for Cloud-Native Persistent Storage: https://containerjournal.com/features/8-cncf-projects-for-cloud-native-persistent-storage/ Best distributes file/block for k8s - Reddit 2023 thread - https://www.reddit.com/r/kubernetes/comments/100wdrq/best_distributed_fileblock_storage_for_kubernetes JuiceFS https://juicefs.com/docs/cloud/use_juicefs_in_kubernetes. Chronosphere funding round - https://chronosphere.io/learn/115m-series-c-funding-chronospheres/ Show Links: 1, https://www.plural.sh/ 2. https://app.plural.sh/ 3. https://github.com/pluralsh/plural 4. https://docs.plural.sh/

On The Cloud Pod this week, Amazon announces massive corporate and tech lay offs and S3 Encrypts New Objects By Default, BigQuery multi-statement transactions are now generally available, and Microsoft announces acquisition of Fungible to accelerate datacenter innovation. Thank you to our sponsor, Foghorn Consulting, which provides top notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you're having trouble hiring? Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week. General News: Amazon to lay off 18,000 corporate and tech workers. [1:11] Episode Highlights ⏰ Amazon S3 Encrypts New Objects By Default. [3:09] ⏰ Announcing the GA of BigQuery multi-statement transactions. [13:04] ⏰ Microsoft announces acquisition of Fungible to accelerate datacenter innovation. [17:14] Top Quote

The Cloud Pod recaps all of the positives and negatives of Amazon ReInvent 2022, the annual conference in Las Vegas, bringing together 50,000 cloud computing professionals.  This year's keynote speakers include Adam Selpisky, CEO of Amazon Web Services, Swami Sivasubramanian, Vice President of Data and Machine Learning at AWS and Werner Vogels, Amazon's CTO.  Attendees and web viewers were treated to new features and products, such as AWS Lambda Snapstart for Java Functions, New Quicksight capabilities and quality-of-life improvements to hundreds of services.  Justin, Jonathan, Ryan, Peter and Special guest Joe Daly from the Finops foundation talk about the show and the announcements. Thank you to our sponsor, Foghorn Consulting, which provides top notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you're having trouble hiring? Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week. Episode Highlights ⏰ AWS Pricing Calculator now supports modernization cost estimates for Microsoft workloads. ⏰ AWS Re:Invent 2022 announcements and keynote updates. Top Quote

In this 126th episode of The G2 on 5G, Anshel and Will Cover:1. 5G Techritory Recap2. AWS ReInvent Recap - Vodafone Edge Innovation Lab, JMA 5G XRAN, Amazon EKS & Intel FlexRAN validation3. Bharti Airtel and Meta extend subsea cable to India- what it means for 5G deployments4. Ericsson's November 2022 Mobility Report makes many 5G predictions, including 1 billion 5G connections by year's end and 100m FWA as well, with FWA growing to 300m users by 20285. Verizon executive departs in less than a year - does it signal more trouble?6. Deutsche Telekom claims that its 5G network already covers 94% of the German populationEricsson Nov22 Mobility Report: https://www.ericsson.com/en/reports-and-papers/mobility-report/reports/november-2022

This week we recap the news from AWS re:Invent and discuss application vendors mandating use of specific Kubernetes distros. Plus, some thoughts on dog boarding… Watch the YouTube Live Recording of Episode 389 (https://www.youtube.com/watch?v=h8L0QEIMvOs) Runner-up Titles Everyone gets a Graviton Instance What a Boring re:Invent Part of our brand 17 Days in the Hole Under the Stars, Under the Sea Tighten it up Don't make me pay for security Secure by default That's a great message and I don't believe it Works with Lambda Security, it keeps getting better? Rundown AWS re:Invent What's New at AWS – Cloud Innovation & News - 2022 Archive (https://aws.amazon.com/about-aws/whats-new/2022/?whats-new-content-all.sort-by=item.additionalFields.postDateTime&whats-new-content-all.sort-order=desc&awsf.whats-new-analytics=*all&awsf.whats-new-app-integration=*all&awsf.whats-new-arvr=*all&awsf.whats-new-blockchain=*all&awsf.whats-new-business-applications=*all&awsf.whats-new-cloud-financial-management=*all&awsf.whats-new-compute=*all&awsf.whats-new-containers=*all&awsf.whats-new-customer-enablement=*all&awsf.whats-new-customer%20engagement=*all&awsf.whats-new-database=*all&awsf.whats-new-developer-tools=*all&awsf.whats-new-end-user-computing=*all&awsf.whats-new-mobile=*all&awsf.whats-new-gametech=*all&awsf.whats-new-iot=*all&awsf.whats-new-machine-learning=*all&awsf.whats-new-management-governance=*all&awsf.whats-new-media-services=*all&awsf.whats-new-migration-transfer=*all&awsf.whats-new-networking-content-delivery=*all&awsf.whats-new-quantum-tech=*all&awsf.whats-new-robotics=*all&awsf.whats-new-satellite=*all&awsf.whats-new-security-id-compliance=*all&awsf.whats-new-serverless=*all&awsf.whats-new-storage=*all) Compute Amazon EC2 C7g instances – Compute –Amazon Web Services (https://aws.amazon.com/ec2/instance-types/c7g/?sc_icampaign=aware_ec2-c7gn-instances_reinvent22&sc_ichannel=ha&sc_icontent=awssm-11814_aware_reinvent22&sc_iplace=ribbon&trk=1b39069e-86fc-466c-99c7-4ab2427ddb3a~ha_awssm-11814_aware_reinvent22) Announcing Amazon EC2 M6in, M6idn, R6in, and R6idn network optimized instances (https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-ec2-m6in-m6idn-r6in-r6idn-network-optimized-instances/) Announcing Amazon EC2 Hpc6id instances (https://aws.amazon.com/about-aws/whats-new/2022/11/announcing-amazon-ec2-hpc6id-instances/) AWS Nitro Enclaves now supports Amazon EKS and Kubernetes (https://aws.amazon.com/about-aws/whats-new/2022/11/aws-nitro-enclaves-supports-amazoneks-kubernetes/) Introducing Finch: An Open Source Client for Container Development (https://aws.amazon.com/blogs/opensource/introducing-finch-an-open-source-client-for-container-development/) New – Accelerate Your Lambda Functions with Lambda SnapStart (https://aws.amazon.com/blogs/aws/new-accelerate-your-lambda-functions-with-lambda-snapstart/) Data Announcing Amazon Redshift integration for Apache Spark with Amazon EMR (https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-redshift-integration-apache-spark-amazon-emr/) AWS announces Amazon Redshift integration for Apache Spark (https://aws.amazon.com/about-aws/whats-new/2022/11/aws-announces-amazon-redshift-integration-apache-spark/) AWS announces Amazon Aurora zero-ETL integration with Amazon Redshift (https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-aurora-zero-etl-integration-redshift/) Serverless Open-Source Search Engine – Amazon OpenSearch Serverless (https://aws.amazon.com/opensearch-service/features/serverless/) Introducing AWS Glue 4.0 (https://aws.amazon.com/about-aws/whats-new/2022/11/introducing-aws-glue-4-0/) Security Introducing Amazon Security Lake (Preview) (https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-security-lake-preview/) AWS co-announces release of the Open Cybersecurity Schema Framework (OCSF) (https://aws.amazon.com/blogs/security/aws-co-announces-release-of-the-open-cybersecurity-schema-framework-ocsf-project/) Amazon GuardDuty now protects Amazon Elastic Kubernetes Service clusters (https://aws.amazon.com/about-aws/whats-new/2022/01/amazon-guardduty-elastic-kubernetes-service-clusters/) Solutions AWS CEO: The cloud isn't just about technology (https://www.protocol.com/enterprise/aws-adam-selipsky-cloud) AWS Supply Chain (https://aws.amazon.com/aws-supply-chain/) AWS Clean Room (https://aws.amazon.com/clean-rooms/) Announcing AWS SimSpace Weaver (https://aws.amazon.com/about-aws/whats-new/2022/11/aws-simspace-weaver-available/) Amazon Connect announces Contact Lens agent performance evaluation forms (https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-connect-contact-lens-agent-performance-evaluation-forms/) Introducing Amazon Omics (https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-omics-generally-available/) Corey Quinn on re:Invent (https://twitter.com/QuinnyPig/status/1597664998234345472) Ask SDT — “using a "supported platform" list to drive cross sales.” (https://softwaredefinedtalk.slack.com/archives/C6CDLDCVB/p1669255641385689) (SDT Slack) Relevant to your Interests SigmaOS raises $4 million to build a browser for productivity nerds (https://techcrunch.com/2022/11/16/sigmaos-raises-4-million-to-build-a-browser-for-productivity-nerds/) The Distributed Computing Manifesto (https://www.allthingsdistributed.com/2022/11/amazon-1998-distributed-computing-manifesto.html) Unpacking Musk's "hardcore" marching orders (https://www.axios.com/newsletters/axios-login-3bf3c6e4-d8cd-492c-942d-c7f80719e66b.html?chunk=0&utm_term=emshare#story0) Akeyless secures a cash infusion to help companies manage their passwords, certificates and keys (https://techcrunch.com/2022/11/16/akeyless-secures-a-cash-infusion-to-help-companies-manage-their-passwords-certificates-and-keys/) Vista passes halfway mark to $20bn target for latest flagship (https://www.privateequityinternational.com/vista-passes-halfway-mark-to-20bn-target-for-latest-flagship/) 1Password Will Support Passkeys Starting in Early 2023 (https://www.macrumors.com/2022/11/17/1password-passkeys-support-2023/) Passkeys: the future of authentication in 1Password (https://www.future.1password.com/passkeys/?utm_medium=sign-in-side-panel&utm_source=1password&utm_campaign=passkeys) 10,000 Google Employees Could Be Rated as Low Performers (https://www.theinformation.com/articles/10-000-google-employees-could-be-rated-as-low-performers) Resignations Roil Twitter as Elon Musk Tries Persuading Some Workers to Stay (https://www.nytimes.com/2022/11/17/technology/twitter-elon-musk-ftc.html) Hundreds of employees say no to being part of Elon Musk's ‘extremely hardcore' Twitter (https://www.theverge.com/2022/11/17/23465274/hundreds-of-twitter-employees-resign-from-elon-musk-hardcore-deadline) Security of Passkeys in the Google Password Manager (https://security.googleblog.com/2022/10/SecurityofPasskeysintheGooglePasswordManager.html) With $8.6M in seed funding, Nx wants to take monorepos mainstream (https://techcrunch.com/2022/11/17/with-8-6m-in-seed-funding-nx-wants-to-take-monorepos-mainstream/) Facebook parent Meta winding down some non-core hardware projects (https://www.reuters.com/technology/facebook-parent-meta-winding-down-some-non-core-hardware-projects-2022-11-11/) OpenStack passes 40 million cores in production use (https://www.theregister.com/2022/11/18/openstack_thriving_survey/) A note from CEO Andy Jassy about role eliminations (https://www.aboutamazon.com/news/company-news/a-note-from-ceo-andy-jassy-about-role-eliminations) Twitter is Going Great (https://twitterisgoinggreat.com/) Building Kubernetes Applications with Acorn (https://acorn.io/building-kubernetes-applications-with-acorn/) Platforms at Kubecon 2022 (https://blog.joshgav.com/posts/kubecon-platforms-review) Zoom's looming squeeze (https://www.axios.com/newsletters/axios-login-149ea16b-be11-451a-b4de-5a1e2f8f0ce7.html?chunk=0&utm_term=emshare#story0) Sony's VR headset-console integration could limit sales, but allow depth (https://www.emergingtechbrew.com/stories/2022/11/18/sony-s-vr-headset-console-integration-could-limit-sales-but-allow-depth?utm_campaign=etb&utm_medium=newsletter&utm_source=morning_brew&mid=f642abf4dca6751d0ec109d4cbc6782e) The State of Kubernetes {Open-Source} Security | ARMO (https://www.armosec.io/blog/the-state-of-kubernetes-open-source-security/) Considerations when implementing developer portals in regulated enterprise environments (https://www.redhat.com/en/blog/considerations-when-implementing-developer-portals-regulated-enterprise-environments) Broadcom's proposed $61B VMware acquisition scrutinized by UK regulators (https://techcrunch.com/2022/11/21/broadcoms-proposed-61b-vmware-acquisition-scrutinized-by-uk-regulators/) 2023 may be the year of multicloud Kubernetes (https://www.infoworld.com/article/3679752/2023-may-be-the-year-of-multicloud-kubernetes.html?utm_source=substack&utm_medium=email) Server-side WebAssembly prepares for takeoff in 2023 (https://www.techtarget.com/searchitoperations/news/252527414/Server-side-WebAssembly-prepares-for-takeoff-in-2023?utm_source=substack&utm_medium=email) Zoom shares drop on light forecast as company faces 'heightened deal scrutiny' (https://www.cnbc.com/2022/11/21/zoom-zm-earnings-q3-2023.html?utm_source=newsletter&utm_medium=email&utm_campaign=newsletter_axioslogin&stream=top) What's coming for cloud computing in 2023 (https://www.infoworld.com/article/3680553/whats-coming-for-cloud-computing-in-2023.html) The Rise of Platform Engineering - Software Engineering Daily (https://softwareengineeringdaily.com/2020/02/13/setting-the-stage-for-platform-engineering/) IBM sues Micro Focus, claims it copied mainframe software (https://www.theregister.com/2022/11/22/ibm_sues_micro_focus_for/) How to beat the Kubernetes skills shortage (https://www.infoworld.com/article/3679749/how-to-beat-the-kubernetes-skills-shortage.html) TikTok Couldn't Ensure Accurate Responses To Government Inquiries, A ByteDance Risk Assessment Said (https://www.forbes.com/sites/emilybaker-white/2022/11/28/tiktok-inaccurate-government-inquiries-internal-bytedance-risk-assessment/?sh=7f57dc9723fe) Exclusive: Sam Bankman-Fried says he's down to $100,000 (https://www.axios.com/2022/11/29/sam-bankman-fried-100000-ftx-cftc-regulation?utm_source=newsletter&utm_medium=email&utm_campaign=newsletter_axiosprorata&stream=top) Why Big Tech is not rushing to clone Twitter (https://www.axios.com/newsletters/axios-login-1cea6d1a-1428-448d-b0d3-5da3ae9425ef.html?chunk=0&utm_term=emshare#story0) Amazon Alexa is a “colossal failure,” on pace to lose $10 billion this year (https://arstechnica.com/gadgets/2022/11/amazon-alexa-is-a-colossal-failure-on-pace-to-lose-10-billion-this-year/) I analyzed 290 booths at KubeCon - here are the DevOps trends for 2023 (https://www.uptime.build/post/i-analyzed-290-booths-at-kubecon-here-are-the-devops-trends-for-2023?utm_source=substack&utm_medium=email) Nonsense Billionaires like Elon Musk want to save civilization by having tons of genetically superior kids. Inside the movement to take 'control of human evolution.' (https://www.businessinsider.com/pronatalism-elon-musk-simone-malcolm-collins-underpopulation-breeding-tech-2022-11) Australia: How 'bin chickens' learnt to wash poisonous cane toads (https://www.bbc.com/news/world-australia-63699884) A 12,000 lb. metal sculpture of Elon Musk's head on a goat body riding a rocket parked outside Tesla HQ failed to elicit a response from the billionaire (https://www.businessinsider.com/elon-musk-head-on-goat-body-riding-a-rocket-sculpture-2022-11) The leap second's time will be up in 2035—and tech companies are thrilled (https://www.popsci.com/technology/bipm-abandon-leap-second/) Conferences THAT Conference Texas Speakers and Schedule (https://that.us/events/tx/2023/schedule/). Jan 15th-18th use code SDT for 5% off CloudNativeSecurityCon North America (https://events.linuxfoundation.org/cloudnativesecuritycon-north-america/), Seattle, Feb 1 – 2, 2023 DevOpsDays Birmingham, AL 2023 (https://devopsdays.org/events/2023-birmingham-al/welcome/), April 20 - 21, 2023 Listener Feedback Sudesh shared a list of Tech Companies Hiring (https://airtable.com/shrAPDHg8apj4mnRR/tbl6Kz4KeeCp3HrSM) Send “End of Year” listener questions to questions@softwaredefinedtalk.com (mailto:questions@softwaredefinedtalk.com). SDT news & hype Join us in Slack (http://www.softwaredefinedtalk.com/slack). Get a SDT Sticker! Send your postal address to stickers@softwaredefinedtalk.com (mailto:stickers@softwaredefinedtalk.com) and we will send you free laptop stickers! Follow us on Twitch (https://www.twitch.tv/sdtpodcast), Twitter (https://twitter.com/softwaredeftalk), Instagram (https://www.instagram.com/softwaredefinedtalk/), LinkedIn (https://www.linkedin.com/company/software-defined-talk/) and YouTube (https://www.youtube.com/channel/UCi3OJPV6h9tp-hbsGBLGsDQ/featured). Use the code SDT to get $20 off Coté's book, Digital WTF (https://leanpub.com/digitalwtf/c/sdt), so $5 total. Become a sponsor of Software Defined Talk (https://www.softwaredefinedtalk.com/ads)! Recommendations Brandon: The Complete History & Strategy of Qualcomm (https://www.acquired.fm/episodes/qualcomm) Matt: Kishi Bashi This Must Be The Place (https://www.youtube.com/watch?v=IslMHJFkIME) Carma (https://carma.com.au) car purchase: referral code: REF22-872E Photo Credits Header (https://unsplash.com/photos/K8i-gRJHT_0) CoverArt (https://twitter.com/DevchicaJasmin/status/1597874321510526978)

RE:INVENT NOTICE Jonathan, Ryan and Justin will be live streaming the major keynotes starting Monday Night, followed by Adam's keynote on Tuesday, Swami's keynote on Wednesday and Wrap up our Re:Invent coverage with Werner's keynote on Thursday. Tune into our live stream here on the site or via Twitch/Twitter, etc.  On The Cloud Pod this week, Amazon Time Sync is now available over the internet as a public NTP service, Amazon announces ECS Task Scale-in protection, and Private Marketplace is now in preview. Thank you to our sponsor, Foghorn Consulting, which provides top notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you're having trouble hiring? Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week. Episode Highlights ⏰ Amazon Time Sync is now available over the internet as a public NTP service. ⏰ Amazon announces ECS Task Scale-in protection. ⏰ Private Marketplace is now in preview. Top Quote

On The Cloud Pod this week, Amazon announces Neptune Serverless, Google introduces Google Blockchain Node Engine, and we get some cost management updates from Microsoft. Thank you to our sponsor, Foghorn Consulting, which provides top notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you're having trouble hiring? Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week. General News [1:24]

Links: If you're near Arlington Virgina, come on by Highline this evening at 7PM and let me buy you a drink. Are you confused by AWS's KMS service? Me too. This guide to KMS helped a lot--and you really don't want to be confused by security things. BHIM leaks the details of 7.26 million users and scores themselves an S3 Bucket Negligence Award in the process. Stop doing this! Securely Using External ID for Accessing AWS Accounts Owned by Others - AWS blesses us with a great rundown of how to think about external IDs for accessing AWS accounts.  Use AWS Network Firewall to filter outbound HTTPS traffic from applications hosted on Amazon EKS and collect hostnames provided by SNI- Don't let your sensitive environments connect all willy-nilly (or more formally, all William-Nilliam) to anything they want on the internet.  Last week I mentioned that you might want to enable TouchID to approve sudo requests on macOS. A couple of you pointed out that this setting gets wiped on OS updates, so having a script like this handy to reapply it will likely serve you well.  Cloudfox is a great collection of scripts stuffed into a framework and called a tool that empowers cloud penetration tests. Much like the industry, it biases heavily for AWS; take a look.

On The Cloud Pod this week, the team weighs the merits of bitcoin mining versus hacking. Plus: AWS Trusted Advisor prioritizes Support customers, Google provides impenetrable protection from a major DDoS attack, and Oracle Linux 9 is truly unbreakable. A big thanks to this week's sponsor, Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. This week's highlights

On The Cloud Pod this week, the team chats cloud region wars to establish the true victor. Plus: AWS Storage Day offers a blockhead badge, all the fun of the Microsoft Dev Box, and Google sends people back to sleep with its Cloud Monitoring snooze alert policy. A big thanks to this week's sponsor, Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. This week's highlights

In this week's episodes Bhavin and Ryan dive into the three different operators that MongoDB has for deploying and managing MongoDB with Kubernetes with Developer Advocates Joel Lord & Cedric Clyburn. Come learn about the different ways you can get started using MongoDB with Kubernetes, lessons learned getting started and how to use MongoDB's multi-cloud database service, Atlas. Show Links News Mirantis acquires Amazee.io: https://www.businesswire.com/news/home/20220726005393/en/Mirantis-Acquires-amazee.io-the-Only-ZeroOps-Application-Delivery-Hub-Built-by-Developers-for-Developers Cisco buys Banzai cloud: https://www.fiercetelecom.com/telecom/cisco-buys-kubernetes-start-up-banzai-cloud-to-fuel-cloud-native-connectivity Amazon Detective adds support for Amazon EKS clusters: https://aws.amazon.com/blogs/aws/amazon-detective-supports-kubernetes-workloads-on-amazon-eks-for-security-investigations/ GigaOM report for Cloud Native Kubernetes Storage - https://portworx.com/resource/gigaom-radar-for-cloud-native-kubernetes-data-storage/ Gateway API Beta https://kubernetes.io/blog/2022/07/13/gateway-api-graduates-to-beta/ Thinking about Kubernetes-native way for data protection https://thenewstack.io/identifying-a-kube-native-approach-to-data-protection/ 3 database tools from CNCF July 28 https://containerjournal.com/features/3-cloud-native-database-tools-from-cncf/ Kubernetes Major changes and removals for 1.25 (later in august) https://kubernetes.io/blog/2022/08/04/upcoming-changes-in-kubernetes-1-25/ MongoDB MongoDB Community - https://www.mongodb.com/community/forums/ Mongodb Youtube Channel - https://www.youtube.com/user/mongodb

About ChrisChris Short has been a proponent of open source solutions throughout his over two decades in various IT disciplines, including systems, security, networks, DevOps management, and cloud native advocacy across the public and private sectors. He currently works on the Kubernetes team at Amazon Web Services and is an active Kubernetes contributor and Co-chair of OpenGitOps. Chris is a disabled US Air Force veteran living with his wife and son in Greater Metro Detroit. Chris writes about Cloud Native, DevOps, and other topics at ChrisShort.net. He also runs the Cloud Native, DevOps, GitOps, Open Source, industry news, and culture focused newsletter DevOps'ish.Links Referenced: DevOps'ish: https://devopsish.com/ EKS News: https://eks.news/ Containers from the Couch: https://containersfromthecouch.com opengitops.dev: https://opengitops.dev ChrisShort.net: https://chrisshort.net Twitter: https://twitter.com/ChrisShort TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Coming back to us since episode two—it's always nice to go back and see the where are they now type of approach—I am joined by Senior Developer Advocate at AWS Chris Short. Chris, been a few years. How has it been?Chris: Ha. Corey, we have talked outside of the podcast. But it's been good. For those that have been listening, I think when we recorded I wasn't even—like, when was season two, what year was that? [laugh].Corey: Episode two was first pre-pandemic and the rest. I believe—Chris: Oh. So, yeah. I was at Red Hat, maybe, when I—yeah.Corey: Yeah. You were doing Red Hat stuff, back when you got to work on open-source stuff, as opposed to now, where you're not within 1000 miles of that stuff, right?Chris: Actually well, no. So, to be clear, I'm on the EKS team, the Kubernetes team here at AWS. So, when I joined AWS in October, they were like, “Hey, you do open-source stuff. We like that. Do more.” And I was like, “Oh, wait, do more?” And they were like, “Yes, do more.” “Okay.”So, since joining AWS, I've probably done more open-source work than the three years at Red Hat that I did. So, that's kind of—you know, like, it's an interesting point when I talk to people about it because the first couple months are, like—you know, my friends are like, “So, are you liking it? Are you enjoying it? What's going on?” And—Corey: Do they beat you with reeds? Like, all the questions people have about companies? Because—Chris: Right. Like, I get a lot of random questions about Amazon and AWS that I don't know the answer to.Corey: Oh, when I started telling people, I fixed Amazon bills, I had to quickly pivot that to AWS bills because people started asking me, “Well, can you save me money on underpants?” It's I—Chris: Yeah.Corey: How do you—fine. Get the prime credit card. It docks 5% off the bill, so there you go. But other than that, no, I can't.Chris: No.Corey: It's—Chris: Like, I had to call my bank this morning about a transaction that I didn't recognize, and it was from Amazon. And I was like, that's weird. Why would that—Corey: Money just flows one direction, and that's the wrong direction from my employer.Chris: Yeah. Like, what is going on here? It shouldn't have been on that card kind of thing. And I had to explain to the person on the phone that I do work at Amazon but under the Web Services team. And he was like, “Oh, so you're in IT?”And I'm like, “No.” [laugh]. “It's actually this big company. That—it's a cloud company.” And they're like, “Oh, okay, okay. Yeah. The cloud. Got it.” [laugh]. So, it's interesting talking to people about, “I work at Amazon.” “Oh, my son works at Amazon distribution center,” blah, blah, blah. It's like, cool. “I know about that, but very little. I do this.”Corey: Your son works in Amazon distribution center. Is he a robot? Is normally my next question on that? Yeah. That's neither here nor there.So, you and I started talking a while back. We both write newsletters that go to a somewhat similar audience. You write DevOps'ish. I write Last Week in AWS. And recently, you also have started EKS News because, yeah, the one thing I look at when I'm doing these newsletters every week is, you know what I want to do? That's right. Write more newsletters.Chris: [laugh].Corey: So, you are just a glutton for punishment? And, yeah, welcome to the addiction, I suppose. How's it been going for you?Chris: It's actually been pretty interesting, right? Like, we haven't pushed it very hard. We're now starting to include it in things. Like we did Container Day; we made sure that EKS news was on the landing page for Container Day at KubeCon EU. And you know, it's kind of just grown organically since then.But it was one of those things where it's like, internally—this happened at Red Hat, right—when I started live streaming at Red Hat, the ultimate goal was to do our product management—like, here's what's new in the next version thing—do those live so anybody can see that at any point in time anywhere on Earth, the second it's available. Similar situation to here. This newsletter actually is generated as part of a report my boss puts together to brief our other DAs—or developer advocates—you know, our solutions architects, the whole nine yards about new EKS features. So, I was like, why can't we just flip that into a weekly newsletter, you know? Like, I can pull from the same sources you can.And what's interesting is, he only does the meeting bi-weekly. So, there's some weeks where it's just all me doing it and he ends up just kind of copying and pasting the newsletter into his document, [laugh] and then adds on for the week. But that report meeting for that team is now getting disseminated to essentially anyone that subscribes to eks.news. Just go to the site, there's a subscribe thing right there. And we've gotten 20 issues in and it's gotten rave reviews, right?Corey: I have been a subscriber for a while. I will say that it has less Chris Short personality—Chris: Mm-hm.Corey: —to it than DevOps'ish does, which I have to assume is by design. A lot of The Duckbill Group's marketing these days is no longer in my voice, rather intentionally, because it turns out that being a sarcastic jackass and doing half-billion dollar AWS contracts can not to be the most congruent thing in the world. So okay, we're slowly ameliorating that. It's professional voice versus snarky voice.Chris: Well, and here's the thing, right? Like, I realized this year with DevOps'ish that, like, if I want to take a week off, I have to do, like, what you did when your child was born. You hired folks to like, do the newsletter for you, or I actually don't do the newsletter, right? It's binary: hire someone else to do it, or don't do it. So, the way I structured this newsletter was that any developer advocate on my team could jump in and take over the newsletter so that, you know, if I'm off that week, or whatever may be happening, I, Chris Short, am not the voice. It is now the entire developer advocate team.Corey: I will challenge you on that a bit. Because it's not Chris Short voice, that's for sure, but it's also not official AWS brand voice either.Chris: No.Corey: It is clearly written by a human being who is used to communicating with the audience for whom it is written. And that is no small thing. Normally, when oh, there's a corporate newsletter; that's just a lot of words to say it's bad. This one is good. I want to be very clear on that.Chris: Yeah, I mean, we have just, like, DevOps'ish, we have sections, just like your newsletter, there's certain sections, so any new, what's new announcements, those go in automatically. So, like, that can get delivered to your inbox every Friday. Same thing with new blog posts about anything containers related to EKS, those will be in there, then Containers from the Couch, our streaming platform, essentially, for all things Kubernetes. Those videos go in.And then there's some ecosystem news as well that I collect and put in the newsletter to give people a broader sense of what's going on out there in Kubernetes-land because let's face it, there's upstream and then there's downstream, and sometimes those aren't in sync, and that's normal. That's how Kubernetes kind of works sometimes. If you're running upstream Kubernetes, you are awesome. I appreciate you, but I feel like that would cause more problems and it's worse sometimes.Corey: Thank you for being the trailblazers. The rest of us can learn from your misfortune.Chris: [laugh]. Yeah, exactly. Right? Like, please file your bugs accordingly. [laugh].Corey: EKS is interesting to me because I don't see a lot of it, which is, probably, going to get a whole lot of, “Wait, what?” Moments because wait, don't you deal with very large AWS bills? And I do. But what I mean by that is that EKS, until you're using its Fargate expression, charges for the control plane, which rounds to no money, and the rest is running on EC2 instances running in a company's account. From the billing perspective, there is no difference between, “We're running massive fleets of EKS nodes.” And, “We're managing a whole bunch of EC2 instances by hand.”And that feels like an interesting allegory for how Kubernetes winds up expressing itself to cloud providers. Because from a billing perspective, it just looks like one big single-tenant application that has some really strange behaviors internally. It gets very chatty across AZs when there's no reason to, and whatnot. And it becomes a very interesting study in how to expose aspects of what's going on inside of those containers and inside of the Kubernetes environment to the cloud provider in a way that becomes actionable. There are no good answers for this yet, but it's something I've been seeing a lot of. Like, “Oh, I thought you'd be running Kubernetes. Oh, wait, you are and I just keep forgetting what I'm looking at sometimes.”Chris: So, that's an interesting point. The billing is kind of like, yeah, it's just compute, right? So—Corey: And my insight into AWS and the way I start thinking about it is always from a billing perspective. That's great. It's because that means the more expensive the services, the more I know about it. It's like, “IAM. What is that?” Like, “Oh, I have no idea. It's free. How important could it be?” Professional advice: do not take that philosophy, ever.Chris: [laugh]. No. Ever. No.Corey: Security: it matters. Oh, my God. It's like you're all stars. Your IAM policy should not be. I digress.Chris: Right. Yeah. Anyways, so two points I want to make real quick on that is, one, we've recently released an open-source project called Carpenter, which is really cool in my purview because it looks at your Kubernetes file and says, “Oh, you want this to run on ARM instance.” And you can even go so far as to say, right, here's my limits, and it'll find an instance that fits those limits and add that to your cluster automatically. Run your pod on that compute as long as it needs to run and then if it's done, it'll downsize—eventually, kind of thing—your cluster.So, you can basically just throw a bunch of workloads at it, and it'll auto-detect what kind of compute you will need and then provision it for you, run it, and then be done. So, that is one-way folks are probably starting to save money running EKS is to adopt Carpenter as your autoscaler as opposed to the inbuilt Kubernetes autoscaler. Because this is instance-aware, essentially, so it can say, like, “Oh, your massive ARM application can run here,” because you know, thank you, Graviton. We have those processors in-house. And you know, you can run your ARM64 instances, you can run all the Intel workloads you want, and it'll right size the compute for your workloads.And I'll look at one container or all your containers, however you want to configure it. Secondly, the good folks over at Kubecost have opencost, which is the open-source version of Kubecost, basically. So, they have a service that you can run in your clusters that will help you say, “Hey, maybe this one notes too heavy; maybe this one notes too light,” and you know, give you some insights into Kubernetes spend that are a little bit more granular as far as usage and things like that go. So, those two projects right there, I feel like, will give folks an optimal savings experience when it comes to Kubernetes. But to your point, it's just compute, right? And that's really how we treat it, kind of, here internally is that it's a way to run… compute, Kubernetes, or ECS, or any of those tools.Corey: A fairly expensive one because ignoring entirely for a second the actual raw cost of compute, you also have the other side of it, which is in every environment, unless you are doing something very strange or pre-funding as a one-person startup in your spare time, your payroll costs will it—should—exceed your AWS bill by a fairly healthy amount. And engineering time is always more expensive than services time. So, for example, looking at EKS, I would absolutely recommend people use that rather than rolling their own because—Chris: Rolling their own? Yeah.Corey: —get out of that engineering space where your time is free. I assure you from a business context, it is not. So, there's always that question of what you can do to make things easier for people and do more of the heavy lifting.Chris: Yeah, and to your rather cheeky point that there's 17 ways to run a container on AWS, it is answering that question, right? Like those 17 ways, like, how much of this do you want to run yourself, you could run EKS distro on EC2 instances if you want full control over your environment.Corey: And then run IoT Greengrass core on top within that cluster—Chris: Right.Corey: So, I can run my own Lambda function runtime, so I'm not locked in. Also, DynamoDB local so I'm not locked into AWS. At which point I have gone so far around the bend, no one can help me.Chris: Well—Corey: Pro tip, don't do that. Just don't do that.Chris: But to your point, we have all these options for compute, and specifically containers because there's a lot of people that want to granularly say, “This is where my engineering team gets involved. Everything else you handle.” If I want EKS on Spot Instances only, you can do that. If you want EKS to use Carpenter and say only run ARM workloads, you can do that. If you want to say Fargate and not have anything to manage other than the container file, you can do that.It's how much does your team want to manage? That's the customer obsession part of AWS coming through when it comes to containers is because there's so many different ways to run those workloads, but there's so many different ways to make sure that your team is right-sized, based off the services you're using.Corey: I do want to change gears a bit here because you are mostly known for a couple of things: the DevOps'ish newsletter because that is the oldest and longest thing you've been doing the time that I've known you; EKS, obviously. But when prepping for this show, I discovered you are now co-chair of the OpenGitOps project.Chris: Yes.Corey: So, I have heard of GitOps in the context of, “Oh, it's just basically your CI/CD stuff is triggered by Git events and whatnot.” And I'm sitting here going, “Okay, so from where you're sitting, the two best user interfaces in the world that you have discovered are YAML and Git.” And I just have to start with the question, “Who hurt you?”Chris: [laugh]. Yeah, I share your sentiment when it comes to Git. Not so much with YAML, but I think it's because I'm so used to it. Maybe it's Stockholm Syndrome, maybe the whole YAML thing. I don't know.Corey: Well, it's no XML. We'll put it that way.Chris: Thankfully, yes because if it was, I would have way more, like, just template files laying around to build things. But the—Corey: And rage. Don't forget rage.Chris: And rage, yeah. So, GitOps is a little bit more than just Git in IaC—infrastructure as Code. It's more like Justin Garrison, who's also on my team, he calls it infrastructure software because there's four main principles to GitOps, and if you go to opengitops.dev, you can see them. It's version one.So, we put them on the website, right there on the page. You have to have a declared state and that state has to live somewhere. Now, it's called GitOps because Git is probably the most full-featured thing to put your state in, but you could use an S3 bucket and just version it, for example. And make it private so no one else can get to it.Corey: Or you could use local files: copy-of-copy-of-this-thing-restored-parentheses-use-this-one-dot-final-dot-doc-dot-zip. You know, my preferred naming convention.Chris: Ah, yeah. Wow. Okay. [laugh]. Yeah.Corey: Everything I touch is terrifying.Chris: Yes. Geez, I'm sorry. So first, it's declarative. You declare your state. You store it somewhere. It's versioned and immutable, like I said. And then pulled automatically—don't focus so much on pull—but basically, software agents are applying the desired state from source. So, what does that mean? When it's—you know, the fourth principle is implemented, continuously reconciled. That means those software agents that are checking your desired state are actually putting it back into the desired state if it's out of whack, right? So—Corey: You're talking about agents running it persistently on instances, validating—Chris: Yes.Corey: —a checkpoint on a cron. How is this meaningfully different than a Puppet agent running in years past? Having spent I learned to speak publicly by being a traveling trainer for Puppet; same type of model, and in fact, when I was at Pinterest, we wound up having a fair bit—like, that was their entire model, where they would have—the Puppet's code would live in an S3 bucket that was then copied down, I believe, via Git, and then applied to the instance on a schedule. Like, that sounds like this was sort of a early days GitOps.Chris: Yeah, exactly. Right? Like so it's, I like to think of that as a component of GitOps, right? DevOps, when you talk about DevOps in general, there's a lot of stuff out there. There's a lot of things labeled DevOps that maybe are, or maybe aren't sticking to some of those DevOps core things that make you great.Like the stuff that Nicole Forsgren writes about in books, you know? Accelerate is on my desk for a reason because there's things that good, well-managed DevOps practices do. I see GitOps as an actual implementation of DevOps in an open-source manner because all the tooling for GitOps these days is open-source and it all started as open-source. Now, you can get, like, Flux or Argo—Argo, specifically—there's managed services out there for it, you can have Flux and not maintain it, through an add-on, on EKS for example, and it will reconcile that state for you automatically. And the other thing I like to say about GitOps, specifically, is that it moves at the speed of the Kubernetes Audit Log.If you've ever looked at a Kubernetes audit log, you know it's rather noisy with all these groups and versions and kinds getting thrown out there. So, GitOps will say, “Oh, there's an event for said thing that I'm supposed to be watching. Do I need to change anything? Yes or no? Yes? Okay, go.”And the change gets applied, or, “Hey, there's a new Git thing. Pull it in. A change has happened inGit I need to update it.” You can set it to reconcile on events on time. It's like a cron or it's like an event-driven architecture, but it's combined.Corey: How does it survive the stake through the heart of configuration management? Because before I was doing all this, I wasn't even a T-shaped engineer: you're broad across a bunch of things, but deep in one or two areas, and one of mine was configuration management. I wrote part of SaltStack, once upon a time—Chris: Oh.Corey: —due to a bunch of very strange coincidences all hitting it once, like, I taught people how to use Puppet. But containers ultimately arose and the idea of immutable infrastructure became a thing. And these days when we were doing full-on serverless, well, great, I just wind up deploying a new code bundle to the Lambdas function that I wind up caring about, and that is a immutable version replacement. There is no drift because there is no way to log in and change those things other than through a clear deployment of this as the new version that goes out there. Where does GitOps fit into that imagined pattern?Chris: So, configuration management becomes part of your approval process, right? So, you now are generating an audit log, essentially, of all changes to your system through the approval process that you set up as part of your, how you get things into source and then promote that out to production. That's kind of the beauty of it, right? Like, that's why we suggest using Git because it has functions, like, requests and issues and things like that you can say, “Hey, yes, I approve this,” or, “Hey, no, I don't approve that. We need changes.” So, that's kind of natively happening with Git and, you know, GitLab, GitHub, whatever implementation of Git. There's always, kind of—Corey: Uh, JIF-ub is, I believe, the pronunciation.Chris: JIF-ub? Oh.Corey: Yeah. That's what I'm—Chris: Today, I learned. Okay.Corey: Exactly. And that's one of the things that I do for my lasttweetinaws.com Twitter client that I build—because I needed it, and if other people want to use it, that's great—that is now deployed to 20 different AWS commercial regions, simultaneously. And that is done via—because it turns out that that's a very long to execute for loop if you start down that path—Chris: Well, yeah.Corey: I wound up building out a GitHub Actions matrix—sorry a JIF-ub—actions matrix job that winds up instantiating 20 parallel builds of the CDK deploy that goes out to each region as expected. And because that gets really expensive with native GitHub Actions runners for, like, 36 cents per deploy, and I don't know how to test my own code, so every time I have a typo, that's another quarter in the jar. Cool, but that was annoying for me so I built my own custom runner system that uses Lambda functions as runners running containers pulled from ECR that, oh, it just runs in parallel, less than three minutes. Every time I commit something between I press the push button and it is out and running in the wild across all regions. Which is awesome and also terrifying because, as previously mentioned, I don't know how to test my code.Chris: Yeah. So, you don't know what you're deploying to 20 regions sometime, right?Corey: But it also means I have a pristine, re-composable build environment because I can—Chris: Right.Corey: Just automatically have that go out and the fact that I am making a—either merging a pull request or doing a direct push because I consider main to be my feature branch as whenever something hits that, all the automation kicks off. That was something that I found to be transformative as far as a way of thinking about this because I was very tired of having to tweak my local laptop environment to, “Oh, you didn't assume the proper role and everything failed again and you broke it. Good job.” It wound up being something where I could start developing on more and more disparate platforms. And it finally is what got me away from my old development model of everything I build is on an EC2 instance, and that means that my editor of choice was Vim. I use the VS Code now for these things, and I'm pretty happy with it.Chris: Yeah. So, you know, I'm glad you brought up CDK. CDK gives you a lot of the capabilities to implement GitOps in a way that you could say, like, “Hey, use CDK to declare I need four Amazon EKS clusters with this size, shape, and configuration. Go.” Or even further, connect to these EKS clusters to RDS instances and load balancers and everything else.But you put that state into Git and then you have something that deploys that automatically upon changes. That is infrastructure as code. Now, when you say, “Okay, main is your feature branch,” you know, things happen on main, if this were running in Kubernetes across a fleet of clusters or the globe-wide in 20 regions, something like Flux or Argo would kick in and say, “There's been a change to source, main, and we need to roll this out.” And it'll start applying those changes. Now, what do you get with GitOps that you don't get with your configuration?I mean, can you rollback if you ever have, like, a bad commit that's just awful? I mean, that's really part of the process with GitOps is to make sure that you can, A, roll back to the previous good state, B, roll forward to a known good state, or C, promote that state up through various environments. And then having that all done declaratively, automatically, and immutably, and versioned with an audit log, that I think is the real power of GitOps in the sense that, like, oh, so-and-so approve this change to security policy XYZ on this date at this time. And that to an auditor, you just hand them a log file on, like, “Here's everything we've ever done to our system. Done.” Right?Like, you could get to that state, if you want to, which I think is kind of the idea of DevOps, which says, “Take all these disparate tools and processes and procedures and culture changes”—culture being the hardest part to adopt in DevOps; GitOps kind of forces a culture change where, like, you can't do a CAB with GitOps. Like, those two things don't fly. You don't have a configuration management database unless you absolutely—Corey: Oh, you CAB now but they're all the comments of the pull request.Chris: Right. Exactly. Like, don't push this change out until Thursday after this other thing has happened, kind of thing. Yeah, like, that all happens in GitHub. But it's very democratizing in the sense that people don't have to waste time in an hour-long meeting to get their five minutes in, right?Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: So, would it be overwhelmingly cynical to suggest that GitOps is the means to implement what we've all been pretending to have implemented for the last decade when giving talks at conferences?Chris: Ehh, I wouldn't go that far. I would say that GitOps is an excellent way to implement the things you've been talking about at all these conferences for all these years. But keep in mind, the technology has changed a lot in the, what 11, 12 years of the existence of DevOps, now. I mean, we've gone from, let's try to manage whole servers immutably to, “Oh, now we just need to maintain an orchestration platform and run containers.” That whole compute interface, you go from SSH to a Docker file, that's a big leap, right?Like, you don't have bespoke sysadmins; you have, like, a platform team. You don't have DevOps engineers; they're part of that platform team, or DevOps teams, right? Like, which was kind of antithetical to the whole idea of DevOps to have a DevOps team. You know, everybody's kind of in the same boat now, where we see skill sets kind of changing. And GitOps and Kubernetes-land is, like, a platform team that manages the cluster, and its state, and health and, you know, production essentially.And then you have your developers deploying what they want to deploy in when whatever namespace they've been given access to and whatever rights they have. So, now you have the potential for one set of people—the platform team—to use one set of GitOps tooling, and your applications teams might not like that, and that's fine. They can have their own namespaces with their own tooling in it. Like, Argo, for example, is preferred by a lot of developers because it has a nice UI with green and red dots and they can show people and it looks nice, Flux, it's command line based. And there are some projects out there that kind of take the UI of Argo and try to run Flux underneath that, and those are cool kind of projects, I think, in my mind, but in general, right, I think GitOps gives you the choice that we missed somewhat in DevOps implementations of the past because it was, “Oh, we need to go get cloud.” “Well, you can only use this cloud.” “Oh, we need to go get this thing.” “Well, you can only use this thing in-house.”And you know, there's a lot of restrictions sometimes placed on what you can use in your environment. Well, if your environment is Kubernetes, how do you restrict what you can run, right? Like you can't have an easily configured say, no open-source policy if you're running Kubernetes. [laugh] so it becomes, you know—Corey: Well, that doesn't stop some companies from trying.Chris: Yeah, that's true. But the idea of, like, enabling your developers to deploy at will and then promote their changes as they see fit is really the dream of DevOps, right? Like, same with production and platform teams, right? I want to push my changes out to a larger system that is across the globe. How do I do that? How do I manage that? How do I make sure everything's consistent?GitOps gives you those ways, with Kubernetes native things like customizations, to make consistent environments that are robust and actually going to be reconciled automatically if someone breaks the glass and says, “Oh, I need to run this container immediately.” Well, that's going to create problems because it's deviated from state and it's just that one region, so we'll put it back into state.Corey: It'll be dueling banjos, at some point. You'll try and doing something manually, it gets reverted automatically. I love that pattern. You'll get bored before the computer does, always.Chris: Yeah. And GitOps is very new, right? When you think about the lifetime of GitOps, I think it was coined in, like, 2018. So, it's only four years old, right? When—Corey: I prefer it to ChatOps, at least, as far as—Chris: Well, I mean—Corey: —implementation and expression of the thing.Chris: —ChatOps was a way to do DevOps. I think GitOps—Corey: Well, ChatOps is also a way to wind up giving whoever gets access to your Slack workspace root in production.Chris: Mmm.Corey: But that's neither here nor there.Chris: Mm-hm.Corey: It's yeah, we all like to pretend that's not a giant security issue in our industry, but that's a topic for another time.Chris: Yeah. And that's why, like, GitOps also depends upon you having good security, you know, and good authorization and approval processes. It enforces that upon—Corey: Yeah, who doesn't have one of those?Chris: Yeah. If it's a sole operation kind of deal, like in your setup, your case, I think you kind of got it doing right, right? Like, as far as GitOps goes—Corey: Oh, to be clear, we are 11 people and we do have dueling pull requests and all the rest.Chris: Right, right, right.Corey: But most of the stuff I talk about publicly is not our production stuff, so it really is just me. Just as a point of clarity there. I've n—the 11 people here do not all—the rest of you don't just sit there and clap as I do all the work.Chris: Right.Corey: Most days.Chris: No, I'm sure they don't. I'm almost certain they don't clap… for you. I mean, they would—Corey: No. No, they try and talk me out of it in almost every case.Chris: Yeah, exactly. So, the setup that you, Corey Quinn, have implemented to deploy these 20 regions is kind of very GitOps-y, in the sense that when main changes, it gets updated. Where it's not GitOps-y is what if the endpoint changes? Does it get reconciled? That's the piece you're probably missing is that continuous reconciliation component, where it's constantly checking and saying, “This thing out there is deployed in the way I want it. You know, the way I declared it to be in my source of truth.”Corey: Yeah, when you start having other people getting involved, there can—yeah, that's where regressions enter. And it's like, “Well, I know where things are so why would I change the endpoint?” Yeah, it turns out, not everyone has the state of the entire application in their head. Ideally it should live in—Chris: Yeah. Right. And, you know—Corey: —you know, Git or S3.Chris: —when I—yeah, exactly. When I think about interactions of the past coming out as a new DevOps engineer to work with developers, it's always been, will developers have access to prod or they don't? And if you're in that environment with—you're trying to run a multi-billion dollar operation, and your devs have direct—or one Dev has direct access to prod because prod is in his brain, that's where it's like, well, now wait a minute. Prod doesn't have to be only in your brain. You can put that in the codebase and now we know what is in your brain, right?Like, you can almost do—if you document your code, well, you can have your full lifecycle right there in one place, including documentation, which I think is the best part, too. So, you know, it encourages approval processes and automation over this one person has an entire state of the system in their head; they have to go in and fix it. And what if they're not on call, or in Jamaica, or on a cruise ship somewhere kind of thing? Things get difficult. Like, for example, I just got back from vacation. We were so far off the grid, we had satellite internet. And let me tell you, it was hard to write an email newsletter where I usually open 50 to 100 tabs.Corey: There's a little bit of internet out Californ-ie way.Chris: [laugh].Corey: Yeah it's… it's always weird going from, like, especially after pandemic; I have gigabit symmetric here and going even to re:Invent where I'm trying to upload a bunch of video and whatnot.Chris: Yeah. Oh wow.Corey: And the conference WiFi was doing its thing, and well, Verizon 5G was there but spotty. And well, yeah. Usual stuff.Chris: Yeah. It's amazing to me how connectivity has become so ubiquitous.Corey: To the point where when it's not there anymore, it's what do I do with myself? Same story about people pushing back against remote development of, “Oh, I'm just going to do it all on my laptop because what happens if I'm on a plane?” It's, yeah, the year before the pandemic, I flew 140,000 miles domestically and I was almost never hamstrung by my ability to do work. And my only local computer is an iPad for those things. So, it turns out that is less of a real world concern for most folks.Chris: Yeah I actually ordered the components to upgrade an old Nook that I have here and turn it into my, like, this is my remote code server, that's going to be all attached to GitHub and everything else. That's where I want to be: have Tailscale and just VPN into this box.Corey: Tailscale is transformative.Chris: Yes. Tailscale will change your life. That's just my personal opinion.Corey: Yep.Chris: That's not an AWS opinion or anything. But yeah, when you start thinking about your network as it could be anywhere, that's where Tailscale, like, really shines. So—Corey: Tailscale makes the internet work like we all wanted to believe that it worked.Chris: Yeah. And Wireguard is an excellent open-source project. And Tailscale consumes that and puts an amazingly easy-to-use UI, and troubleshooting tools, and routing, and all kinds of forwarding capabilities, and makes it kind of easy, which is really, really, really kind of awesome. And Tailscale and Kubernetes—Corey: Yeah, ‘network' and ‘easy' don't belong in the same sentence, but in this case, they do.Chris: Yeah. And trust me, the Kubernetes story in Tailscale, there is a lot of there. I understand you might want to not open ports in your VPC, maybe, but if you use Tailscale, that node is just another thing on your network. You can connect to that and see what's going on. Your management cluster is just another thing on the network where you can watch the state.But it's all—you're connected to it continuously through Tailscale. Or, you know, it's a much lighter weight, kind of meshy VPN, I would say, if I had to sum it up in one sentence. That was not on our agenda to talk about at all. Anyways. [laugh]Corey: No, no. I love how many different topics we talk about on these things. We'll have to have you back soon to talk again. I really want to thank you for being so generous with your time. If people want to learn more about what you're up to and how you view these things, where can they find you?Chris: Go to ChrisShort.net. So, Chris Short—I'm six-four so remember, it's Short—dot net, and you will find all the places that I write, you can go to devopsish.com to subscribe to my newsletter, which goes out every week. This year. Next year, there'll be breaks. And then finally, if you want to follow me on Twitter, Chris Short: at @ChrisShort on Twitter. All one word so you see two s's. Like, it's okay, there's two s's there.Corey: Links to all of that will of course be in the show notes. It's easier for people to do the clicky-clicky thing as a general rule.Chris: Clicky things are easier than the wordy things, yes.Corey: Says the Kubernetes guy.Chris: Yeah. Says the Kubernetes guy. Yeah, you like that, huh? Like I said, Argo gives you a UI. [laugh].Corey: Thank you [laugh] so much for your time. I really do appreciate it.Chris: Thank you. This has been fun. If folks have questions, feel free to reach out. Like, I am not one of those people that hides behind a screen all day and doesn't respond. I will respond to you eventually.Corey: I'm right here, Chris. Come on, come on. You're calling me out in front of myself. My God.Chris: Egh. It might take a day or two, but I will respond. I promise.Corey: Thanks again for your time. This has been Chris Short, senior developer advocate at AWS. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice and if it's YouTube, click the thumbs-up button. Whereas if you've hated this podcast, same thing, smash the buttons five-star review and leave an insulting comment that is written in syntactically correct YAML because it's just so easy to do.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Новостной выпуск, говорим о последних релизах с марта по апрель. Новые инстансы в AWS на базе AMD - C6a Amazon RDS Multi-AZ теперь поддерживает две RO реплики. Amazon Aurora Serverless v2 теперь в GA. Уточнение Amazon Aurora Serverless v2 - на текущий момент не поддерживает паузу, работает только с первой версией. AWS Lambda Function URLs - поддержка встроенных HTTPS точек доступа для небольших микросервисов. AWS Lambda теперь поддерживает до 10 ГБ временного хранилища. AWS Fargate увеличение скорости масштабирование в 16 раз. Amazon EKS теперь поддерживает Kubernetes 1.22, все детали этого обновления. Таймкоды: 00:01:00 - Новые инстансы на базе AMD C6a 00:05:39 - Amazon RDS Multi-AZ 00:08:35 - Amazon Aurora Serverless v2 00:12:35 - AWS Lambda Function URLs 00:16:00 - AWS Lambda supports 00:18:50 - AWS Fargate 00:22:10 - Amazon EKS supports k8s 1.22 00:29:00 - Секция новости одной строкой. Телеграм канал который был упомянут во время разговора - https://t.me/aws_notes Если у вас есть вопросы, предложения темы, пишите мне в Linkedin - https://www.linkedin.com/in/vedmich/ или телеграмм https://t.me/VictorVedmich

Cloud Posse holds public "Office Hours" every Wednesday at 11:30am PST to answer questions on all things related to DevOps, Terraform, Kubernetes, CICD. Basically, it's like an interactive "Lunch & Learn" session where we get together for about an hour and talk shop. These are totally free and just an opportunity to ask us (or our community of experts) any questions you may have. You can register here: https://cloudposse.com/office-hoursJoin the conversation: https://slack.cloudposse.com/Find out how we can help your company:https://cloudposse.com/quizhttps://cloudposse.com/accelerate/Learn more about Cloud Posse:https://cloudposse.comhttps://github.com/cloudpossehttps://sweetops.com/https://newsletter.cloudposse.comhttps://podcast.cloudposse.com/[00:00:00] Intro[00:01:17] Atmos Adds Vendoring - pull terraform root modules (or anything) from anywherehttps://github.com/cloudposse/atmos/pull/145[00:07:30] Terraform 1.2 (RC1 just dropped) — adds pre/post conditions, bearer tokenshttps://github.com/hashicorp/terraform/releases/tag/v1.2.0-rc1[00:14:28] Amazon EKS web console adds Kubernetes Resource Viewhttps://aws.amazon.com/blogs/containers/introducing-kubernetes-resource-view-in-amazon-eks-console/[00:18:34] Werf: Consistent delivery toolhttps://werf.io/[00:26:32] Easy-to-follow set of instructions for a strategy that minimizes the cost of NAT gateways in ec2.[00:36:00] How many of you don't commit .terraform.lock.hcl to source control?[00:44:25] Explain to me how crossplane works? [00:53:35] Outro #officehours,#cloudposse,#sweetops,#devops,#sre,#terraform,#kubernetes,#awsSupport the show

Rob is joined by Amazon EKS GM, Chandler Hoisington to discuss the speedy rise of Kubernetes. Chandler explains the power of getting the right tooling for your team and how sometimes the costs of moving fast may be worth the risk. Tune in today!Have a topic you want us to discuss? Someone you want us to interview? Reach out to us on Twitter @circleci!

The Amazon EKS community has added support for Loft Labs' open source project vcluster, which enables you to spin up lightweight, virtual Kubernetes clusters inside the namespaces of an underlying Kubernetes cluster. In this episode of Newsroom, Swapnil Bhartiya sits down with Lukas Gentele, Co-Founder and CEO of Loft Labs, to talk about how the idea of adding support for EKS with vcluster came about and how it is fostering further collaborations and supporting the open source community.

Google Biglake takes the feature of the week with the ability to federate data from multiple data lakes. On The Cloud Pod this week, the team discusses the most expensive way to run a VM (Oracle wins). Plus some exciting developments, an AWS OpenSearch 1.2 update with several new features, and Azure's having a party, so bring your own IP addresses (BYOIP). A big thanks to this week's sponsor, Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. This week's highlights

In the previous Trinites installation (https://trino.io/episodes/24.html), we introduced Kubernetes (k8s) and its concepts and how to use k8s with Trino. After discussing Kubernetes, we did a demo showing how to deploy Trino on k8s.This round, we're going to take the same k8s concepts and dive in a little deeper to help newbies to k8s (KuberNewbies...Kubies?) to but deploy Trino to the cloud (specifically the most common cloud provider, AWS). This takes us from proving Trino is awesome to just you to proving it to your coworkers, boss, and doing so at scale! We will deploy using Amazon's EKS service.- Intro Song: 00:00​- Intro: 00:34- News: 9:08- Concept of the week: ReplicaSets, Deployments, and Services: 27:38- Demo of the month: Deploy Trino k8s to Amazon EKS: 1:21:31- PR of the week: PR 8921: Support TRUNCATE TABLE statement: 1:34:41- Question of the week: How do I run system.sync_partition_metadata with different catalogs?: 1:38:21Show Notes: https://trino.io/episodes/31.htmlShow Page: https://trino.io/broadcast/

In this episode, Ryan and Bhavin interview Xing Yang, Tech Lead at VMware and a co-chair of the CNCF Storage TAG, a co-chair of the Kubernetes Storage SIG, a co-chair of the Data Protection WG in Kubernetes, and a maintainer in Kubernetes CSI. The discussion dives into how Kubernetes SIGs and WGs collaborate, and how people can get started. We also talk about storage features and enhancements introduced in Kubernetes version 1.23 and features and enhancements planned for the Kubernetes 1.24 release. Show links: Xing Yang: https://twitter.com/2000Xyang Storage SIG: https://github.com/kubernetes/community/tree/master/sig-storage Data Protection Working Group: https://github.com/kubernetes/community/tree/master/wg-data-protection Kubernetes Storage SIG Planning Spreadsheet: https://docs.google.com/spreadsheets/d/1t4z5DYKjX2ZDlkTpCnp18icRAQqOE85C1T1r2gqJVck/edit?usp=sharing Amazon GuardDuty support for Amazon EKS clusters - https://aws.amazon.com/about-aws/whats-new/2022/01/amazon-guardduty-elastic-kubernetes-service-clusters/ Diamanti Buys Groudwork and its Kubernetes monitoring capability - https://blocksandfiles.com/2022/01/31/diamanti-buys-groundwork-and-its-k8s-monitoring-capability/ Weaveworks buys Magalix to secure GitOps workflows - https://www.magalix.com/blog/magalix-weaveworks-forging-the-path-of-secure-gitops-workflows MinIO raises series-B funding to become a unicorn - https://www.zdnet.com/article/minio-the-de-facto-open-source-standard-for-multi-cloud-storage-becomes-a-unicorn-after-a-103-million-series-b-round-funding/