Podcasts about aws systems manager

OpenZFS RAID-Z Expansion: A New Era in Storage Flexibility, ZFS Orchestration Tools – Part 1: Snapshots, The Case of UNIX vs. The UNIX System, OpenBGPD 8.8 released, OPNsense 25.1, and more NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) and the BSDNow Patreon (https://www.patreon.com/bsdnow) Headlines OpenZFS RAID-Z Expansion: A New Era in Storage Flexibility (https://freebsdfoundation.org/blog/openzfs-raid-z-expansion-a-new-era-in-storage-flexibility/) ZFS Orchestration Tools – Part 1: Snapshots (https://klarasystems.com/articles/zfs-orchestration-part-1-zfs-snapshots-tools/?utm_source=BSD%20Now&utm_medium=Podcast) News Roundup Manage OpenBSD with AWS Systems Manager (https://rsadowski.de/posts/2025-01-23-manage-openbsd-with-ssm/) TUHS:The Case of UNIX vs. The UNIX System (https://www.tuhs.org/pipermail/tuhs/2025-February/031403.html) OpenBGPD 8.8 released (https://www.undeadly.org/cgi?action=article;sid=20250207192657) OPNsense 25.1 (https://forum.opnsense.org/index.php?topic=45460.msg227323) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) Join us and other BSD Fans in our BSD Now Telegram channel (https://t.me/bsdnow)

AWS Morning Brief for the week of November 25, with Corey Quinn. Links:Enhanced account linking experience across AWS Marketplace and AWS Partner CentralAmazon API Gateway now supports Custom Domain Name for private REST APIsAmazon Aurora Serverless v2 supports scaling to zero capacityAmazon CloudFront now supports Anycast Static IPsAmazon CloudFront now supports additional log formats and destinations for access logsAmazon CloudFront announces VPC originsAmazon CloudWatch launches full visibility into application transactionsAmazon EC2 now provides lineage information for your AMIsAmazon Q Developer in the AWS Management Console now uses the service you're viewing as context for your chatAmazon WorkSpaces introduces support for Rocky LinuxAWS App Studio is now generally availableAWS CloudTrail Lake launches enhanced analytics and cross-account data accessAWS Compute Optimizer now supports rightsizing recommendations for Amazon AuroraAWS Elastic Beanstalk adds support for Node.js 22AWS Lambda supports Amazon S3 as a failed-event destination for asynchronous and stream event sourcesIntroducing an AWS Management Console Visual Update (Preview)The new AWS Systems Manager experience: Simplifying node managementAWS announces Block Public Access for Amazon Virtual Private CloudLoad Balancer Capacity Unit Reservation for Application and Network Load BalancersAnnouncing Idle Recommendations in AWS Compute OptimizerAnnouncing Savings Plans Purchase AnalyzerAWS Lambda turns ten – looking back and looking aheadBoost Engagement with AWS and Amazon AdsBuild fullstack AI apps in minutes with the new Amplify AI KitImportant changes to CloudTrail events for AWS IAM Identity CenterFollow Corey on BlueSky!Follow Last Week In AWS on BlueSky!

Welcome to The Cloud Pod - where the forecast is always cloudy! This week your hosts, Jonathan and Ryan, are talking all about EC2 instances, including changes to AWS Systems Manager and Elastic Disaster Recovery. And speaking of disasters, we're also taking a dive into the ongoing Google DDOS attacks. Plus, we've even thrown a little earthquake warning into the podcast, just for effect.  Titles we almost went with this week: A big thanks to this week's sponsor: Foghorn Consulting provides top-notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you have trouble hiring?  Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week.

Welcome to the newest episode of The Cloud Pod podcast! Justin, Ryan and Matthew are your hosts this week as we discuss all the latest news and announcements in the world of the cloud and AI - including what's new with Google Deepmind, as well as goings on over at the Finops X Conference. Join us!  Titles we almost went with this week:

On this episode of The Cloud Pod, the team discusses Amazon Pi Day, Google's upcoming I/O conference, the agricultural data manager by Microsoft, and the downturn in net profits of Oracle. They also round up cloud migrations by highlighting tools from different cloud service providers that are useful for the process. A big thanks to this week's sponsor, Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. This week's highlights

On this episode of The Cloud Pod, the team discusses the AWS systems manager default enablement option for all EC2 instances in an account, different ideas from leveraging innovators plus subscription using $500 Google credits, the Azure Open Source Day, the new theme for the Oracle OCI Console, and lastly, different ways to migrate to a cloud provider. A big thanks to this week's sponsor, Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. This week's highlights

AWS Morning Brief for the week of February 27, 2023 with Corey Quinn. Links: Amazon OpenSearch Service now lets you schedule service software updates during off-peak hours  AWS App Runner now supports HTTP to HTTPS redirect Announcing the ability to enable AWS Systems Manager by default across all EC2 instances in an account  New: AWS Telco Network Builder – Deploy and Manage Telco Networks Developing portable AWS Lambda functions Using Porting Advisor for Graviton  Query data with DynamoDB Shell – a command line interface for Amazon DynamoDB AWS and Hugging Face collaborate to make generative AI more accessible and cost efficient Branch Insurance improves hiring diversity and accelerates app development using AWS AppSync Gain compliance insights using the open source community for AWS CloudTrail  The true costs of resiliency decisions 

On this episode of The Cloud Pod, the team discusses the upcoming 2023 in-person Google Cloud conference, the accessibility of AWS CloudTrail Lake for non-AWS activity events, the new updates from Azure Chaos studio, and the comparison between Oracle Cloud service and other Cloud providers. They also highlight the application and importance of VPCs in CCOE. A big thanks to this week's sponsor, Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure.  This week's highlights

On this episode of The Cloud Pod, the team sits to talk about AWS's new patching policies, the general availability of Azure OpenAI, and the role of addressing IM or access management challenges in ensuring the seamless transition to the Cloud. A big thanks to this week's sponsor, Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. This week's highlights

RE:INVENT NOTICE Jonathan, Ryan and Justin will be live streaming the major keynotes starting Monday Night, followed by Adam's keynote on Tuesday, Swami's keynote on Wednesday and Wrap up our Re:Invent coverage with Werner's keynote on Thursday. Tune into our live stream here on the site or via Twitch/Twitter, etc.  On The Cloud Pod this week, a new AWS region is open in Spain and NBA and Microsoft team up to transform fan experiences with cloud application modernization. Thank you to our sponsor, Foghorn Consulting, which provides top notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you're having trouble hiring? Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week. General News [0:04]      

About Steve:Steve Rice is Principal Product Manager for AWS AppConfig. He is surprisingly passionate about feature flags and continuous configuration. He lives in the Washington DC area with his wife, 3 kids, and 2 incontinent dogs.Links Referenced:AWS AppConfig: https://go.aws/awsappconfig TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at AWS AppConfig. Engineers love to solve, and occasionally create, problems. But not when it's an on-call fire-drill at 4 in the morning. Software problems should drive innovation and collaboration, NOT stress, and sleeplessness, and threats of violence. That's why so many developers are realizing the value of AWS AppConfig Feature Flags. Feature Flags let developers push code to production, but hide that that feature from customers so that the developers can release their feature when it's ready. This practice allows for safe, fast, and convenient software development. You can seamlessly incorporate AppConfig Feature Flags into your AWS or cloud environment and ship your Features with excitement, not trepidation and fear. To get started, go to snark.cloud/appconfig. That's snark.cloud/appconfig.Corey: Forget everything you know about SSH and try Tailscale. Imagine if you didn't need to manage PKI or rotate SSH keys every time someone leaves. That'd be pretty sweet, wouldn't it? With tail scale, ssh, you can do exactly that. Tail scale gives each server and user device a node key to connect to its VPN, and it uses the same node key to authorize and authenticate.S. Basically you're SSHing the same way you manage access to your app. What's the benefit here? Built in key rotation permissions is code connectivity between any two devices, reduce latency and there's a lot more, but there's a time limit here. You can also ask users to reauthenticate for that extra bit of security. Sounds expensive?Nope, I wish it were. tail scales. Completely free for personal use on up to 20 devices. To learn more, visit snark.cloud/tailscale. Again, that's snark.cloud/tailscaleCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This is a promoted guest episode. What does that mean? Well, it means that some people don't just want me to sit here and throw slings and arrows their way, they would prefer to send me a guest specifically, and they do pay for that privilege, which I appreciate. Paying me is absolutely a behavior I wish to endorse.Today's victim who has decided to contribute to slash sponsor my ongoing ridiculous nonsense is, of all companies, AWS. And today I'm talking to Steve Rice, who's the principal product manager on AWS AppConfig. Steve, thank you for joining me.Steve: Hey, Corey, great to see you. Thanks for having me. Looking forward to a conversation.Corey: As am I. Now, AppConfig does something super interesting, which I'm not aware of any other service or sub-service doing. You are under the umbrella of AWS Systems Manager, but you're not going to market with Systems Manager AppConfig. You're just AWS AppConfig. Why?Steve: So, AppConfig is part of AWS Systems Manager. Systems Manager has, I think, 17 different features associated with it. Some of them have an individual name that is associated with Systems Manager, some of them don't. We just happen to be one that doesn't. AppConfig is a service that's been around for a while internally before it was launched externally a couple years ago, so I'd say that's probably the origin of the name and the service. I can tell you more about the origin of the service if you're curious.Corey: Oh, I absolutely am. But I just want to take a bit of a detour here and point out that I make fun of the sub-service names in Systems Manager an awful lot, like Systems Manager Session Manager and Systems Manager Change Manager. And part of the reason I do that is not just because it's funny, but because almost everything I found so far within the Systems Manager umbrella is pretty awesome. It aligns with how I tend to think about the world in a bunch of different ways. I have yet to see anything lurking within the Systems Manager umbrella that has led to a tee-hee-hee bill surprise level that rivals, you know, the GDP of Guam. So, I'm a big fan of the entire suite of services. But yes, how did AppConfig get its name?Steve: [laugh]. So, AppConfig started about six years ago, now, internally. So, we actually were part of the region services department inside of Amazon, which is in charge of launching new services around the world. We found that a centralized tool for configuration associated with each service launching was really helpful. So, a service might be launching in a new region and have to enable and disable things as it moved along.And so, the tool was sort of built for that, turning on and off things as the region developed and was ready to launch publicly; then the regions launch publicly. It turned out that our internal customers, which are a lot of AWS services and then some Amazon services as well, started to use us beyond launching new regions, and started to use us for feature flagging. Again, turning on and off capabilities, launching things safely. And so, it became massively popular; we were actually a top 30 service internally in terms of usage. And two years ago, we thought we really should launch this externally and let our customers benefit from some of the goodness that we put in there, and some of—those all come from the mistakes we've made internally. And so, it became AppConfig. In terms of the name itself, we specialize in application configuration, so that's kind of a mouthful, so we just changed it to AppConfig.Corey: Earlier this year, there was a vulnerability reported around I believe it was AWS Glue, but please don't quote me on that. And as part of its excellent response that AWS put out, they said that from the time that it was disclosed to them, they had patched the service and rolled it out to every AWS region in which Glue existed in a little under 29 hours, which at scale is absolutely magic fast. That is superhero speed and then some because you generally don't just throw something over the wall, regardless of how small it is when we're talking about something at the scale of AWS. I mean, look at who your customers are; mistakes will show. This also got me thinking that when you have Adam, or previously Andy, on stage giving a keynote announcement and then they mention something on stage, like, “Congratulations. It's now a very complicated service with 14 adjectives in his name because someone's paid by the syllable. Great.”Suddenly, the marketing pages are up, the APIs are working, it's showing up in the console, and it occurs to me only somewhat recently to think about all of the moving parts that go on behind this. That is far faster than even the improved speed of CloudFront distribution updates. There's very clearly something going on there. So, I've got to ask, is that you?Steve: Yes, a lot of that is us. I can't take credit for a hundred percent of what you're talking about, but that's how we are used. We're essentially used as a feature-flagging service. And I can talk generically about feature flagging. Feature flagging allows you to push code out to production, but it's hidden behind a configuration switch: a feature toggle or a feature flag. And that code can be sitting out there, nobody can access it until somebody flips that toggle. Now, the smart way to do it is to flip that toggle on for a small set of users. Maybe it's just internal users, maybe it's 1% of your users. And so, the features available, you can—Corey: It's your best slash worst customers [laugh] in that 1%, in some cases.Steve: Yeah, you want to stress test the system with them and you want to be able to look and see what's going to break before it breaks for everybody. So, you release us to a small cohort, you measure your operations, you measure your application health, you measure your reputational concerns, and then if everything goes well, then you maybe bump it up to 2%, and then 10%, and then 20%. So, feature flags allow you to slowly release features, and you know what you're releasing by the time it's at a hundred percent. It's tempting for teams to want to, like, have everybody access it at the same time; you've been working hard on this feature for a long time. But again, that's kind of an anti-pattern. You want to make sure that on production, it behaves the way you expect it to behave.Corey: I have to ask what is the fundamental difference between feature flags and/or dynamic configuration. Because to my mind, one of them is a means of achieving the other, but I could also see very easily using the terms interchangeably. Given that in some of our conversations, you have corrected me which, first, how dare you? Secondly, okay, there's probably a reason here. What is that point of distinction?Steve: Yeah. Typically for those that are not eat, sleep, and breathing dynamic configuration—which I do—and most people are not obsessed with this kind of thing, feature flags is kind of a shorthand for dynamic configuration. It allows you to turn on and off things without pushing out any new code. So, your application code's running, it's pulling its configuration data, say every five seconds, every ten seconds, something like that, and when that configuration data changes, then that app changes its behavior, again, without a code push or without restarting the app.So, dynamic configuration is maybe a superset of feature flags. Typically, when people think feature flags, they're thinking of, “Oh, I'm going to release a new feature, so it's almost like an on-off switch.” But we see customers using feature flags—and we use this internally—for things like throttling limits. Let's say you want to be able to throttle TPS transactions per second. Or let's say you want to throttle the number of simultaneous background tasks, and say, you know, I just really don't want this creeping above 50; bad things can start to happen.But in a period of stress, you might want to actually bring that number down. Well, you can push out these changes with dynamic configuration—which is, again, any type of configuration, not just an on-off switch—you can push this out and adjust the behavior and see what happens. Again, I'd recommend pushing it out to 1% of your users, and then 10%. But it allows you to have these dials and switches to do that. And, again, generically, that's dynamic configuration. It's not as fun to term as feature flags; feature flags is sort of a good mental picture, so I do use them interchangeably, but if you're really into the whole world of this dynamic configuration, then you probably will care about the difference.Corey: Which makes a fair bit of sense. It's the question of what are you talking about high level versus what are you talking about implementation detail-wise.Steve: Yep. Yep.Corey: And on some level, I used to get… well, we'll call it angsty—because I can't think of a better adjective right now—about how AWS was reluctant to disclose implementation details behind what it did. And in the fullness of time, it's made a lot more sense to me, specifically through a lens of, you want to be able to have the freedom to change how something works under the hood. And if you've made no particular guarantee about the implementation detail, you can do that without potentially worrying about breaking a whole bunch of customer expectations that you've inadvertently set. And that makes an awful lot of sense.The idea of rolling out changes to your infrastructure has evolved over the last decade. Once upon a time you'd have EC2 instances, and great, you want to go ahead and make a change there—or this actually predates EC2 instances. Virtual machines in a data center or heaven forbid, bare metal servers, you're not going to deploy a whole new server because there's a new version of the code out, so you separate out your infrastructure from the code that it runs. And that worked out well. And increasingly, we started to see ways of okay, if we want to change the behavior of the application, we'll just push out new environment variables to that thing and restart the service so it winds up consuming those.And that's great. You've rolled it out throughout your fleet. With containers, which is sort of the next logical step, well, okay, this stuff gets baked in, we'll just restart containers with a new version of code because that takes less than a second each and you're fine. And then Lambda functions, it's okay, we'll just change the deployment option and the next invocation will wind up taking the brand new environment variables passed out to it. How do feature flags feature into those, I guess, three evolving methods of running applications in anger, by which I mean, of course, production?Steve: [laugh]. Good question. And I think you really articulated that well.Corey: Well, thank you. I should hope so. I'm a storyteller. At least I fancy myself one.Steve: [laugh]. Yes, you are. Really what you talked about is the evolution of you know, at the beginning, people were—well, first of all, people probably were embedding their variables deep in their code and then they realized, “Oh, I want to change this,” and now you have to find where in my code that is. And so, it became a pattern. Why don't we separate everything that's a configuration data into its own file? But it'll get compiled at build time and sent out all at once.There was kind of this breakthrough that was, why don't we actually separate out the deployment of this? We can separate the deployment from code from the deployment of configuration data, and have the code be reading that configuration data on a regular interval, as I already said. So now, as the environments have changed—like you said, containers and Lambda—that ability to make tweaks at microsecond intervals is more important and more powerful. So, there certainly is still value in having things like environment variables that get read at startup. We call that static configuration as opposed to dynamic configuration.And that's a very important element in the world of containers that you talked about. Containers are a bit ephemeral, and so they kind of come and go, and you can restart things, or you might spin up new containers that are slightly different config and have them operate in a certain way. And again, Lambda takes that to the next level. I'm really excited where people are going to take feature flags to the next level because already today we have people just fine-tuning to very targeted small subsets, different configuration data, different feature flag data, and allows them to do this like at we've never seen before scale of turning this on, seeing how it reacts, seeing how the application behaves, and then being able to roll that out to all of your audience.Now, you got to be careful, you really don't want to have completely different configurations out there and have 10 different, or you know, 100 different configurations out there. That makes it really tough to debug. So, you want to think of this as I want to roll this out gradually over time, but eventually, you want to have this sort of state where everything is somewhat consistent.Corey: That, on some level, speaks to a level of operational maturity that my current deployment adventures generally don't have. A common reference I make is to my lasttweetinaws.com Twitter threading app. And anyone can visit it, use it however they want.And it uses a Route 53 latency record to figure out, ah, which is the closest region to you because I've deployed it to 20 different regions. Now, if this were a paid service, or I had people using this in large volume and I had to worry about that sort of thing, I would probably approach something that is very close to what you describe. In practice, I pick a devoted region that I deploy something to, and cool, that's sort of my canary where I get things working the way I would expect. And when that works the way I want it to I then just push it to everything else automatically. Given that I've put significant effort into getting deployments down to approximately two minutes to deploy to everything, it feels like that's a reasonable amount of time to push something out.Whereas if I were, I don't know, running a bank, for example, I would probably have an incredibly heavy process around things that make changes to things like payment or whatnot. Because despite the lies, we all like to tell both to ourselves and in public, anything that touches payments does go through waterfall, not agile iterative development because that mistake tends to show up on your customer's credit card bills, and then they're also angry. I think that there's a certain point of maturity you need to be at as either an organization or possibly as a software technology stack before something like feature flags even becomes available to you. Would you agree with that, or is this something everyone should use?Steve: I would agree with that. Definitely, a small team that has communication flowing between the two probably won't get as much value out of a gradual release process because everybody kind of knows what's going on inside of the team. Once your team scales, or maybe your audience scales, that's when it matters more. You really don't want to have something blow up with your users. You really don't want to have people getting paged in the middle of the night because of a change that was made. And so, feature flags do help with that.So typically, the journey we see is people start off in a maybe very small startup. They're releasing features at a very fast pace. They grow and they start to build their own feature flagging solution—again, at companies I've been at previously have done that—and you start using feature flags and you see the power of it. Oh, my gosh, this is great. I can release something when I want without doing a big code push. I can just do a small little change, and if something goes wrong, I can roll it back instantly. That's really handy.And so, the basics of feature flagging might be a homegrown solution that you all have built. If you really lean into that and start to use it more, then you probably want to look at a third-party solution because there's so many features out there that you might want. A lot of them are around safeguards that makes sure that releasing a new feature is safe. You know, again, pushing out a new feature to everybody could be similar to pushing out untested code to production. You don't want to do that, so you need to have, you know, some checks and balances in your release process of your feature flags, and that's what a lot of third parties do.It really depends—to get back to your question about who needs feature flags—it depends on your audience size. You know, if you have enough audience out there to want to do a small rollout to a small set first and then have everybody hit it, that's great. Also, if you just have, you know, one or two developers, then feature flags are probably something that you're just kind of, you're doing yourself, you're pushing out this thing anyway on your own, but you don't need it coordinated across your team.Corey: I think that there's also a bit of—how to frame this—misunderstanding on someone's part about where AppConfig starts and where it stops. When it was first announced, feature flags were one of the things that it did. And that was talked about on stage, I believe in re:Invent, but please don't quote me on that, when it wound up getting announced. And then in the fullness of time, there was another announcement of AppConfig now supports feature flags, which I'm sitting there and I had to go back to my old notes. Like, did I hallucinate this? Which again, would not be the first time I'd imagine such a thing. But no, it was originally how the service was described, but now it's extra feature flags, almost like someone would, I don't know, flip on a feature-flag toggle for the service and now it does a different thing. What changed? What was it that was misunderstood about the service initially versus what it became?Steve: Yeah, I wouldn't say it was a misunderstanding. I think what happened was we launched it, guessing what our customers were going to use it as. We had done plenty of research on that, and as I mentioned before we had—Corey: Please tell me someone used it as a database. Or am I the only nutter that does stuff like that?Steve: We have seen that before. We have seen something like that before.Corey: Excellent. Excellent, excellent. I approve.Steve: And so, we had done our due diligence ahead of time about how we thought people were going to use it. We were right about a lot of it. I mentioned before that we have a lot of usage internally, so you know, that was kind of maybe cheating even for us to be able to sort of see how this is going to evolve. What we did announce, I guess it was last November, was an opinionated version of feature flags. So, we had people using us for feature flags, but they were building their own structure, their own JSON, and there was not a dedicated console experience for feature flags.What we announced last November was an opinionated version that structured the JSON in a way that we think is the right way, and that afforded us the ability to have a smooth console experience. So, if we know what the structure of the JSON is, we can have things like toggles and validations in there that really specifically look at some of the data points. So, that's really what happened. We're just making it easier for our customers to use us for feature flags. We still have some customers that are kind of building their own solution, but we're seeing a lot of them move over to our opinionated version.Corey: This episode is brought to us in part by our friends at Datadog. Datadog's SaaS monitoring and security platform that enables full stack observability for developers, IT operations, security, and business teams in the cloud age. Datadog's platform, along with 500 plus vendor integrations, allows you to correlate metrics, traces, logs, and security signals across your applications, infrastructure, and third party services in a single pane of glass.Combine these with drag and drop dashboards and machine learning based alerts to help teams troubleshoot and collaborate more effectively, prevent downtime, and enhance performance and reliability. Try Datadog in your environment today with a free 14 day trial and get a complimentary T-shirt when you install the agent.To learn more, visit datadoghq/screaminginthecloud to get. That's www.datadoghq/screaminginthecloudCorey: Part of the problem I have when I look at what it is you folks do, and your use cases, and how you structure it is, it's similar in some respects to how folks perceive things like FIS, the fault injection service, or chaos engineering, as is commonly known, which is, “We can't even get the service to stay up on its own for any [unintelligible 00:18:35] period of time. What do you mean, now let's intentionally degrade it and make it work?” There needs to be a certain level of operational stability or operational maturity. When you're still building a service before it's up and running, feature flags seem awfully premature because there's no one depending on it. You can change configuration however your little heart desires. In most cases. I'm sure at certain points of scale of development teams, you have a communications problem internally, but it's not aimed at me trying to get something working at 2 a.m. in the middle of the night.Whereas by the time folks are ready for what you're doing, they clearly have that level of operational maturity established. So, I have to guess on some level, that your typical adopter of AppConfig feature flags isn't in fact, someone who is, “Well, we're ready for feature flags; let's go,” but rather someone who's come up with something else as a stopgap as they've been iterating forward. Usually something homebuilt. And it might very well be you have the exact same biggest competitor that I do in my consulting work, which is of course, Microsoft Excel as people try to build their own thing that works in their own way.Steve: Yeah, so definitely a very common customer of ours is somebody that is using a homegrown solution for turning on and off things. And they really feel like I'm using the heck out of these feature flags. I'm using them on a daily or weekly basis. I would like to have some enhancements to how my feature flags work, but I have limited resources and I'm not sure that my resources should be building enhancements to a feature-flagging service, but instead, I'd rather have them focusing on something, you know, directly for our customers, some of the core features of whatever your company does. And so, that's when people sort of look around externally and say, “Oh, let me see if there's some other third-party service or something built into AWS like AWS AppConfig that can meet those needs.”And so absolutely, the workflows get more sophisticated, the ability to move forward faster becomes more important, and do so in a safe way. I used to work at a cybersecurity company and we would kind of joke that the security budget of the company is relatively low until something bad happens, and then it's, you know, whatever you need to spend on it. It's not quite the same with feature flags, but you do see when somebody has a problem on production, and they want to be able to turn something off right away or make an adjustment right away, then the ability to do that in a measured way becomes incredibly important. And so, that's when, again, you'll see customers starting to feel like they're outgrowing their homegrown solution and moving to something that's a third-party solution.Corey: Honestly, I feel like so many tools exist in this space, where, “Oh, yeah, you should definitely use this tool.” And most people will use that tool. The second time. Because the first time, it's one of those, “How hard could that be out? I can build something like that in a weekend.” Which is sort of the rallying cry of doomed engineers who are bad at scoping.And by the time that they figure out why, they have to backtrack significantly. There's a whole bunch of stuff that I have built that people look at and say, “Wow, that's a really great design. What inspired you to do that?” And the absolute honest answer to all of it is simply, “Yeah, I worked in roles for the first time I did it the way you would think I would do it and it didn't go well.” Experience is what you get when you didn't get what you wanted, and this is one of those areas where it tends to manifest in reasonable ways.Steve: Absolutely, absolutely.Corey: So, give me an example here, if you don't mind, about how feature flags can improve the day-to-day experience of an engineering team or an engineer themselves. Because we've been down this path enough, in some cases, to know the failure modes, but for folks who haven't been there that's trying to shave a little bit off of their journey of, “I'm going to learn from my own mistakes.” Eh, learn from someone else's. What are the benefits that accrue and are felt immediately?Steve: Yeah. So, we kind of have a policy that the very first commit of any new feature ought to be the feature flag. That's that sort of on-off switch that you want to put there so that you can start to deploy your code and not have a long-lived branch in your source code. But you can have your code there, it reads whether that configuration is on or off. You start with it off.And so, it really helps just while developing these things about keeping your branches short. And you can push the mainline, as long as the feature flag is off and the feature is hidden to production, which is great. So, that helps with the mess of doing big code merges. The other part is around the launch of a feature.So, you talked about Andy Jassy being on stage to launch a new feature. Sort of the old way of doing this, Corey, was that you would need to look at your pipelines and see how long it might take for you to push out your code with any sort of code change in it. And let's say that was an hour-and-a-half process and let's say your CEO is on stage at eight o'clock on a Friday. And as much as you like to say it, “Oh, I'm never pushing out code on a Friday,” sometimes you have to. The old way—Corey: Yeah, that week, yes you are, whether you want to or not.Steve: [laugh]. Exactly, exactly. The old way was this idea that I'm going to time my release, and it takes an hour-and-a-half; I'm going to push it out, and I'll do my best, but hopefully, when the CEO raises her arm or his arm up and points to a screen that everything's lit up. Well, let's say you're doing that and something goes wrong and you have to start over again. Well, oh, my goodness, we're 15 minutes behind, can you accelerate things? And then you start to pull away some of these blockers to accelerate your pipeline or you start editing it right in the console of your application, which is generally not a good idea right before a really big launch.So, the new way is, I'm going to have that code already out there on a Wednesday [laugh] before this big thing on a Friday, but it's hidden behind this feature flag, I've already turned it on and off for internals, and it's just waiting there. And so, then when the CEO points to the big screen, you can just flip that one small little configuration change—and that can be almost instantaneous—and people can access it. So, that just reduces the amount of stress, reduces the amount of risk in pushing out your code.Another thing is—we've heard this from customers—customers are increasing the number of deploys that they can do per week by a very large percentage because they're deploying with confidence. They know that I can push out this code and it's off by default, then I can turn it on whenever I feel like it, and then I can turn it off if something goes wrong. So, if you're into CI/CD, you can actually just move a lot faster with a number of pushes to production each week, which again, I think really helps engineers on their day-to-day lives. The final thing I'm going to talk about is that let's say you did push out something, and for whatever reason, that following weekend, something's going wrong. The old way was oop, you're going to get a page, I'm going to have to get on my computer and go and debug things and fix things, and then push out a new code change.And this could be late on a Saturday evening when you're out with friends. If there's a feature flag there that can turn it off and if this feature is not critical to the operation of your product, you can actually just go in and flip that feature flag off until the next morning or maybe even Monday morning. So, in theory, you kind of get your free time back when you are implementing feature flags. So, I think those are the big benefits for engineers in using feature flags.Corey: And the best way to figure out whether someone is speaking from a position of experience or is simply a raving zealot when they're in a position where they are incentivized to advocate for a particular way of doing things or a particular product, as—let's be clear—you are in that position, is to ask a form of the following question. Let's turn it around for a second. In what scenarios would you absolutely not want to use feature flags? What problems arise? When do you take a look at a situation and say, “Oh, yeah, feature flags will make things worse, instead of better. Don't do it.”Steve: I'm not sure I wouldn't necessarily don't do it—maybe I am that zealot—but you got to do it carefully.Corey: [laugh].Steve: You really got to do things carefully because as I said before, flipping on a feature flag for everybody is similar to pushing out untested code to production. So, you want to do that in a measured way. So, you need to make sure that you do a couple of things. One, there should be some way to measure what the system behavior is for a small set of users with that feature flag flipped to on first. And it could be some canaries that you're using for that.You can also—there's other mechanisms you can do that to: set up cohorts and beta testers and those kinds of things. But I would say the gradual rollout and the targeted rollout of a feature flag is critical. You know, again, it sounds easy, “I'll just turn it on later,” but you ideally don't want to do that. The second thing you want to do is, if you can, is there some sort of validation that the feature flag is what you expect? So, I was talking about on-off feature flags; there are things, as when I was talking about dynamic configuration, that are things like throttling limits, that you actually want to make sure that you put in some other safeguards that say, “I never want my TPS to go above 1200 and never want to set it below 800,” for whatever reason, for example. Well, you want to have some sort of validation of that data before the feature flag gets pushed out. Inside Amazon, we actually have the policy that every single flag needs to have some sort of validation around it so that we don't accidentally fat-finger something out before it goes out there. And we have fat-fingered things.Corey: Typing the wrong thing into a command structure into a tool? “Who would ever do something like that?” He says, remembering times he's taken production down himself, exactly that way.Steve: Exactly, exactly, yeah. And we've done it at Amazon and AWS, for sure. And so yeah, if you have some sort of structure or process to validate that—because oftentimes, what you're doing is you're trying to remediate something in production. Stress levels are high, it is especially easy to fat-finger there. So, that check-and-balance of a validation is important.And then ideally, you have something to automatically roll back whatever change that you made, very quickly. So AppConfig, for example, hooks up to CloudWatch alarms. If an alarm goes off, we're actually going to roll back instantly whatever that feature flag was to its previous state so that you don't even need to really worry about validating against your CloudWatch. It'll just automatically do that against whatever alarms you have.Corey: One of the interesting parts about working at Amazon and seeing things in Amazonian scale is that one in a million events happen thousands of times every second for you folks. What lessons have you learned by deploying feature flags at that kind of scale? Because one of my problems and challenges with deploying feature flags myself is that in some cases, we're talking about three to five users a day for some of these things. That's not really enough usage to get insights into various cohort analyses or A/B tests.Steve: Yeah. As I mentioned before, we build these things as features into our product. So, I just talked about the CloudWatch alarms. That wasn't there originally. Originally, you know, if something went wrong, you would observe a CloudWatch alarm and then you decide what to do, and one of those things might be that I'm going to roll back my configuration.So, a lot of the mistakes that we made that caused alarms to go off necessitated us building some automatic mechanisms. And you know, a human being can only react so fast, but an automated system there is going to be able to roll things back very, very quickly. So, that came from some specific mistakes that we had made inside of AWS. The validation that I was talking about as well. We have a couple of ways of validating things.You might want to do a syntactic validation, which really you're validating—as I was saying—the range between 100 and 1000, but you also might want to have sort of a functional validation, or we call it a semantic validation so that you can make sure that, for example, if you're switching to a new database, that you're going to flip over to your new database, you can have a validation there that says, “This database is ready, I can write to this table, it's truly ready for me to switch.” Instead of just updating some config data, you're actually going to be validating that the new target is ready for you. So, those are a couple of things that we've learned from some of the mistakes we made. And again, not saying we aren't making mistakes still, but we always look at these things inside of AWS and figure out how we can benefit from them and how our customers, more importantly, can benefit from these mistakes.Corey: I would say that I agree. I think that you have threaded the needle of not talking smack about your own product, while also presenting it as not the global panacea that everyone should roll out, willy-nilly. That's a good balance to strike. And frankly, I'd also say it's probably a good point to park the episode. If people want to learn more about AppConfig, how you view these challenges, or even potentially want to get started using it themselves, what should they do?Steve: We have an informational page at go.aws/awsappconfig. That will tell you the high-level overview. You can search for our documentation and we have a lot of blog posts to help you get started there.Corey: And links to that will, of course, go into the [show notes 00:31:21]. Thank you so much for suffering my slings, arrows, and other assorted nonsense on this. I really appreciate your taking the time.Steve: Corey thank you for the time. It's always a pleasure to talk to you. Really appreciate your insights.Corey: You're too kind. Steve Rice, principal product manager for AWS AppConfig. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment. But before you do, just try clearing your cookies and downloading the episode again. You might be in the 3% cohort for an A/B test, and you [want to 00:32:01] listen to the good one instead.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About MattMatt is a Sr. Architect in Belfast, an AWS DevTools Hero, Serverless Architect, Author and conference speaker. He is focused on creating the right environment for empowered teams to rapidly deliver business value in a well-architected, sustainable and serverless-first way.You can usually find him sharing reusable, well architected, serverless patterns over at cdkpatterns.com or behind the scenes bringing CDK Day to life.Links Referenced: Previous guest appearance: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/slinging-cdk-knowledge-with-matt-coulter/ The CDK Book: https://thecdkbook.com/ Twitter: https://twitter.com/NIDeveloper TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the best parts about, well I guess being me, is that I can hold opinions that are… well, I'm going to be polite and call them incendiary, and that's great because I usually like to back them in data. But what happens when things change? What happens when I learn new things?Well, do I hold on to that original opinion with two hands at a death grip or do I admit that I was wrong in my initial opinion about something? Let's find out. My guest today returns from earlier this year. Matt Coulter is a senior architect since he has been promoted at Liberty Mutual. Welcome back, and thanks for joining me.Matt: Yeah, thanks for inviting me back, especially to talk about this topic.Corey: Well, we spoke about it a fair bit at the beginning of the year. And if you're listening to this, and you haven't heard that show, it's not that necessary to go into; mostly it was me spouting uninformed opinions about the CDK—the Cloud Development Kit, for those who are unfamiliar—I think of it more or less as what if you could just structure your cloud resources using a programming language you claim to already know, but in practice, copy and paste from Stack Overflow like the rest of us? Matt, you probably have a better description of what the CDK is in practice.Matt: Yeah, so we like to say it's imperative code written in a declarative way, or declarative code written in an imperative way. Either way, it lets you write code that produces CloudFormation. So, it doesn't really matter what you write in your script; the point is, at the end of the day, you still have the CloudFormation template that comes out of it. So, the whole piece of it is that it's a developer experience, developer speed play, that if you're from a background that you're more used to writing a programming language than a YAML, you might actually enjoy using the CDK over writing straight CloudFormation or SAM.Corey: When I first kicked the tires on the CDK, my first initial obstacle—which I've struggled with in this industry for a bit—is that I'm just good enough of a programmer to get myself in trouble. Whenever I wind up having a problem that StackOverflow doesn't immediately shine a light on, my default solution is to resort to my weapon of choice, which is brute force. That sometimes works out, sometimes doesn't. And as I went through the CDK, a couple of times in service to a project that I'll explain shortly, I made a bunch of missteps with it. The first and most obvious one is that AWS claims publicly that it has support in a bunch of languages: .NET, Python, there's obviously TypeScript, there's Go support for it—I believe that went generally available—and I'm sure I'm missing one or two, I think? Aren't I?Matt: Yeah, it's: TypeScript, JavaScript, Python Java.Net, and Go. I think those are the currently supported languages.Corey: Java. That's the one that I keep forgetting. It's the block printing to the script that is basically Java cursive. The problem I run into, and this is true of most things in my experience, when a company says that we have deployed an SDK for all of the following languages, there is very clearly a first-class citizen language and then the rest that more or less drift along behind with varying degrees of fidelity. In my experience, when I tried it for the first time in Python, it was not a great experience for me.When I learned just enough JavaScript, and by extension TypeScript, to be dangerous, it worked a lot better. Or at least I could blame all the problems I ran into on my complete novice status when it comes to JavaScript and TypeScript at the time. Is that directionally aligned with what you've experienced, given that you work in a large company that uses this, and presumably, once you have more than, I don't know, two developers, you start to take on aspects of a polyglot shop no matter where you are, on some level?Matt: Yeah. So personally, I jump between Java, Python, and TypeScript whenever I'm writing projects. So, when it comes to the CDK, you'd assume I'd be using all three. I typically stick to TypeScript and that's just because personally, I've had the best experience using it. For anybody who doesn't know the way CDK works for all the languages, it's not that they have written a custom, like, SDK for each of these languages; it's a case of it uses a Node process underneath them and the language actually interacts with—it's like the compiled JavaScript version is basically what they all interact with.So, it means there are some limitations on what you can do in that language. I can't remember the full list, but it just means that it is native in all those languages, but there are certain features that you might be like, “Ah,” whereas, in TypeScript, you can just use all of TypeScript. And my first inclination was actually, I was using the Python one and I was having issues with some compiler errors and things that are just caused by that process. And it's something that talking in the cdk.dev Slack community—there is actually a very active—Corey: Which is wonderful, I will point out.Matt: [laugh]. Thank you. There is actually, like, an awesome Python community in there, but if you ask them, they would all ask for improvements to the language. So, personally if someone's new, I always recommend they start with TypeScript and then branch out as they learn the CDK so they can understand is this a me problem, or is this a problem caused by the implementation?Corey: From my perspective, I didn't do anything approaching that level of deep dive. I took a shortcut that I find has served me reasonably well in the course of my career, when I'm trying to do something in Python, and you pull up a tutorial—which I'm a big fan of reading experience reports, and blog posts, and here's how to get started—and they all have the same problem, which is step one, “Run npm install.” And that's “Hmm, you know, I don't recall that being a standard part of the Python tooling.” It's clearly designed and interpreted and contextualized through a lens of JavaScript. Let's remove that translation layer, let's remove any weird issues I'm going to have in that transpilation process, and just talk in the language it written in. Will this solve my problems? Oh, absolutely not, but it will remove a subset of them that I am certain to go blundering into like a small lost child trying to cross an eight-lane freeway.Matt: Yeah. I've heard a lot of people say the same thing. Because the CDK CLI is a Node process, you need it no matter what language you use. So, if they were distributing some kind of universal binary that just integrated with the languages, it would definitely solve a lot of people's issues with trying to combine languages at deploy time.Corey: One of the challenges that I've had as I go through the process of iterating on the project—but I guess I should probably describe it for those who have not been following along with my misadventures; I write blog posts about it from time to time because I need a toy problem to kick around sometimes because my consulting work is all advisory and I don't want to be a talking head-I have a Twitter client called lasttweetinaws.com. It's free; go and use it. It does all kinds of interesting things for authoring Twitter threads.And I wanted to deploy that to a bunch of different AWS regions, as it turns out, 20 or so at the moment. And that led to a lot of interesting projects and having to learn how to think about these things differently because no one sensible deploys an application simultaneously to what amounts to every AWS region, without canary testing, and having a phased rollout in the rest. But I'm reckless, and honestly, as said earlier, a bad programmer. So, that works out. And trying to find ways to make this all work and fit together led iteratively towards me discovering that the CDK was really kind of awesome for a lot of this.That said, there were definitely some fairly gnarly things I learned as I went through it, due in no small part to help I received from generous randos in the cdk.dev Slack team. And it's gotten to a point where it's working, and as an added bonus, I even mostly understand what he's doing, which is just kind of wild to me.Matt: It's one of those interesting things where because it's a programming language, you can use it out of the box the way it's designed to be used where you can just write your simple logic which generates your CloudFormation, or you can do whatever crazy logic you want to do on top of that to make your app work the way you want it to work. And providing you're not in a company like Liberty, where I'm going to do a code review, if no one's stopping you, you can do your crazy experiments. And if you understand that, it's good. But I do think something like the multi-region deploy, I mean, with CDK, if you'd have a construct, it takes in a variable that you can just say what the region is, so you can actually just write a for loop and pass it in, which does make things a lot easier than, I don't know, try to do it with a YAML, which you can pass in parameters, but you're going to get a lot more complicated a lot quicker.Corey: The approach that I took philosophically was I wrote everything in a region-agnostic way. And it would be instantiated and be told what region to run it in as an environment variable that CDK deploy was called. And then I just deploy 20 simultaneous stacks through GitHub Actions, which invoke custom runners that runs inside of a Lambda function. And that's just a relatively basic YAML file, thanks to the magic of GitHub Actions matrix jobs. So, it fires off 20 simultaneous processes and on every commit to the main branch, and then after about two-and-a-half minutes, it has been deployed globally everywhere and I get notified on anything that fails, which is always fun and exciting to learn those things.That has been, overall, just a really useful experiment and an experience because you're right, you could theoretically run this as a single CDK deploy and then wind up having an iterate through a list of regions. The challenge I have there is that unless I start getting into really convoluted asynchronous concurrency stuff, it feels like it'll just take forever. At two-and-a-half minutes a region times 20 regions, that's the better part of an hour on every deploy and no one's got that kind of patience. So, I wound up just parallelizing it a bit further up the stack. That said, I bet they are relatively straightforward ways, given the async is a big part of JavaScript, to do this simultaneously.Matt: One of the pieces of feedback I've seen about CDK is if you have multiple stacks in the same project, it'll deploy them one at a time. And that's just because it tries to understand the dependencies between the stacks and then it works out which one should go first. But a lot of people have said, “Well, I don't want that. If I have 20 stacks, I want all 20 to go at once the way you're saying.” And I have seen that people have been writing plugins to enable concurrent deploys with CDK out of the box. So, it may be something that it's not an out-of-the-box feature, but it might be something that you can pull in a community plug-in to actually make work.Corey: Most of my problems with it at this point are really problems with CloudFormation. CloudFormation does not support well, if at all, secure string parameters from the AWS Systems Manager parameter store, which is my default go-to for secret storage, and Secrets Manager is supported, but that also cost 40 cents a month per secret. And not for nothing, I don't really want to have all five secrets deployed to Secrets Manager in every region this thing is in. I don't really want to pay $20 a month for this basically free application, just to hold some secrets. So, I wound up talking to some folks in the Slack channel and what we came up with was, I have a centralized S3 bucket that has a JSON object that lives in there.It's only accessible from the deployment role, and it grabs that at deploy time and stuffs it into environment variables when it pushes these things out. That's the only stateful part of all of this. And it felt like that is, on some level, a pattern that a lot of people would benefit from if it had better native support. But the counterargument that if you're only deploying to one or two regions, then Secrets Manager is the right answer for a lot of this and it's not that big of a deal.Matt: Yeah. And it's another one of those things, if you're deploying in Liberty, we'll say, “Well, your secret is unencrypted at runtime, so you probably need a KMS key involved in that,” which as you know, the costs of KMS, it depends on if it's a personal solution or if it's something for, like, a Fortune 100 company. And if it's personal solution, I mean, what you're saying sounds great that it's IAM restricted in S3, and then that way only at deploy time can be read; it actually could be a custom construct that someone can build and publish out there to the construct library—or the construct hub, I should say.Corey: To be clear, the reason I'm okay with this, from a security perspective is one, this is in a dedicated AWS account. This is the only thing that lives in that account. And two, the only API credentials we're talking about are the application-specific credentials for this Twitter client when it winds up talking to the Twitter API. Basically, if you get access to these and are able to steal them and deploy somewhere else, you get no access to customer data, you get—or user data because this is not charge for anything—you get no access to things that have been sent out; all you get to do is submit tweets to Twitter and it'll have the string ‘Last Tweet in AWS' as your client, rather than whatever normal client you would use. It's not exactly what we'd call a high-value target because all the sensitive to a user data lives in local storage in their browser. It is fully stateless.Matt: Yeah, so this is what I mean. Like, it's the difference in what you're using your app for. Perfect case of, you can just go into the Twitter app and just withdraw those credentials and do it again if something happens, whereas as I say, if you're building it for Liberty, that it will not pass a lot of our Well-Architected reviews, just for that reason.Corey: If I were going to go and deploy this at a more, I guess, locked down environment, I would be tempted to find alternative approaches such as having it stored encrypted at rest via KMS in S3 is one option. So, is having global DynamoDB tables that wind up grabbing those things, even grabbing it at runtime if necessary. There are ways to make that credential more secure at rest. It's just, I look at this from a real-world perspective of what is the actual attack surface on this, and I have a really hard time just identifying anything that is going to be meaningful with regard to an exploit. If you're listening to this and have a lot of thoughts on that matter, please reach out I'm willing to learn and change my opinion on things.Matt: One thing I will say about the Dynamo approach you mentioned, I'm not sure everybody knows this, but inside the same Dynamo table, you can scope down a row. You can be, like, “This row and this field in this row can only be accessed from this one Lambda function.” So, there's a lot of really awesome security features inside DynamoDB that I don't think most people take advantage of, but they open up a lot of options for simplicity.Corey: Is that tied to the very recent announcement about Lambda getting SourceArn as a condition key? In other words, you can say, “This specific Lambda function,” as opposed to, “A Lambda in this account?” Like that was a relatively recent Advent that I haven't fully explored the nuances of.Matt: Yeah, like, that has opened a lot of doors. I mean, the Dynamo being able to be locked out in your row has been around for a while, but the new Lambda from SourceArn is awesome because, yeah, as you say, you can literally say this thing, as opposed to, you have to start going into tags, or you have to start going into something else to find it.Corey: So, I want to talk about something you just alluded to, which is the Well-Architected Framework. And initially, when it launched, it was a whole framework, and AWS made a lot of noise about it on keynote stages, as they are want to do. And then later, they created a quote-unquote, “Well-Architected Tool,” which let's be very direct, it's the checkbox survey form, at least the last time I looked at it. And they now have the six pillars of the Well-Architected Framework where they talk about things like security, cost, sustainability is the new pillar, I don't know, absorbency, or whatever the remainders are. I can't think of them off the top of my head. How does that map to your experience with the CDK?Matt: Yeah, so out of the box, the CDK from day one was designed to have sensible defaults. And that's why a lot of the things you deploy have opinions. I talked to a couple of the Heroes and they were like, “I wish it had less opinions.” But that's why whenever you deploy something, it's got a bunch of configuration already in there. For me, in the CDK, whenever I use constructs, or stacks, or deploying anything in the CDK, I always build it in a well-architected way.And that's such a loaded sentence whenever you say the word ‘well-architected,' that people go, “What do you mean?” And that's where I go through the six pillars. And in Liberty, we have a process, it used to be called SCORP because it was five pillars, but not SCORPS [laugh] because they added sustainability. But that's where for every stack, we'll go through it and we'll be like, “Okay, let's have the discussion.” And we will use the tool that you mentioned, I mean, the tool, as you say, it's a bunch of tick boxes with a text box, but the idea is we'll get in a room and as we build the starter patterns or these pieces of infrastructure that people are going to reuse, we'll run the well-architected review against the framework before anybody gets to generate it.And then we can say, out of the box, if you generate this thing, these are the pros and cons against the Well-Architected Framework of what you're getting. Because we can't make it a hundred percent bulletproof for your use case because we don't know it, but we can tell you out of the box, what it does. And then that way, you can keep building so they start off with something that is well documented how well architected it is, and then you can start having—it makes it a lot easier to have those conversations as they go forward. Because you just have to talk about the delta as they start adding their own code. Then you can and you go, “Okay, you've added these 20 lines. Let's talk about what they do.” And that's why I always think you can do a strong connection between infrastructure-as-code and well architected.Corey: As I look through the actual six pillars of the Well-Architected Framework: sustainability, cost optimization, performance, efficiency, reliability, security, and operational excellence, as I think through the nature of what this shitpost thread Twitter client is, I am reasonably confident across all of those pillars. I mean, first off, when it comes to the cost optimization pillar, please, don't come to my house and tell me how that works. Yeah, obnoxiously the security pillar is sort of the thing that winds up causing a problem for this because this is an account deployed by Control Tower. And when I was getting this all set up, my monthly cost for this thing was something like a dollar in charges and then another sixteen dollars for the AWS config rule evaluations on all of the deploys, which is… it just feels like a tax on going about your business, but fine, whatever. Cost and sustainability, from my perspective, also tend to be hand-in-glove when it comes to this stuff.When no one is using the client, it is not taking up any compute resources, it has no carbon footprint of which to speak, by my understanding, it's very hard to optimize this down further from a sustainability perspective without barging my way into the middle of an AWS negotiation with one of its power companies.Matt: So, for everyone listening, watch as we do a live well-architected review because—Corey: Oh yeah, I expect—Matt: —this is what they are. [laugh].Corey: You joke; we should do this on Twitter one of these days. I think would be a fantastic conversation. Or Twitch, or whatever the kids are using these days. Yeah.Matt: Yeah.Corey: And again, if so much of it, too, is thinking about the context. Security, you work for one of the world's largest insurance companies. I shitpost for a living. The relative access and consequences of screwing up the security on this are nowhere near equivalent. And I think that's something that often gets lost, per the perfect be the enemy of the good.Matt: Yeah that's why, unfortunately, the Well-Architected Tool is quite loose. So, that's why they have the Well-Architected Framework, which is, there's a white paper that just covers anything which is quite big, and then they wrote specific lenses for, like, serverless or other use cases that are shorter. And then when you do a well-architected review, it's like loose on, sort of like, how are you applying the principles of well-architected. And the conversation that we just had about security, so you would write that down in the box and be, like, “Okay, so I understand if anybody gets this credential, it means they can post this Last Tweet in AWS, and that's okay.”Corey: The client, not the Twitter account, to be clear.Matt: Yeah. So, that's okay. That's what you just mark down in the well-architected review. And then if we go to day one on the future, you can compare it and we can go, “Oh. Okay, so last time, you said this,” and you can go, “Well, actually, I decided to—” or you just keep it as a note.Corey: “We pivoted. We're a bank now.” Yeah.Matt: [laugh]. So, that's where—we do more than tweets now. We decided to do microtransactions through cryptocurrency over Twitter. I don't know but if you—Corey: And that ends this conversation. No no. [laugh].Matt: [laugh]. But yeah, so if something changes, that's what the well-architected reviews for. It's about facilitating the conversation between the architect and the engineer. That's all it is.Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: And the lens is also helpful in that this is a serverless application. So, we're going to view it through that lens, which is great because the original version of the Well-Architected Tool is, “Oh, you built this thing entirely in Lambda? Have you bought some reserved instances for it?” And it's, yeah, why do I feel like I have to explain to AWS how their own systems work? This makes it a lot more streamlined and talks about this, though, it still does struggle with the concept of—in my case—a stateless app. That is still something that I think is not the common path. Imagine that: my code is also non-traditional. Who knew?Matt: Who knew? The one thing that's good about it, if anybody doesn't know, they just updated the serverless lens about, I don't know, a week or two ago. So, they added in a bunch of more use cases. So, if you've read it six months ago, or even three months ago, go back and reread it because they spent a good year updating it.Corey: Thank you for telling me that. That will of course wind up in next week's issue of Last Week in AWS. You can go back and look at the archives and figure out what week record of this then. Good work. One thing that I have learned as well as of yesterday, as it turns out, before we wound up having this recording—obviously because yesterday generally tends to come before today, that is a universal truism—is it I had to do a bit of refactoring.Because what I learned when I was in New York live-tweeting the AWS Summit, is that the Route 53 latency record works based upon where your DNS server is. Yeah, that makes sense. I use Tailscale and wind up using my Pi-hole, which lives back in my house in San Francisco. Yeah, I was always getting us-west-1 from across the country. Cool.For those weird edge cases like me—because this is not the common case—how do I force a local region? Ah, I'll give it its own individual region prepend as a subdomain. Getting that to work with both the global lasttweetinaws.com domain as well as the subdomain on API Gateway through the CDK was not obvious on how to do it.Randall Hunt over at Caylent was awfully generous and came up with a proof-of-concept in about three minutes because he's Randall, and that was extraordinarily helpful. But a challenge I ran into was that the CDK deploy would fail because the way that CloudFormation was rendered in the way it was trying to do stuff, “Oh, that already has that domain affiliated in a different way.” I had to do a CDK destroy then a CDK deploy for each one. Now, not the end of the world, but it got me thinking, everything that I see around the CDK more or less distills down to either greenfield or a day one experience. That's great, but throw it all away and start over is often not what you get to do.And even though Amazon says it's always day one, those of us in, you know, real companies don't get to just treat everything as brand new and throw away everything older than 18 months. What is the day two experience looking like for you? Because you clearly have a legacy business. By legacy, I of course, use it in the condescending engineering term that means it makes actual money, rather than just telling really good stories to venture capitalists for 20 years.Matt: Yeah. We still have mainframes running that make a lot of money. So, I don't mock legacy at all.Corey: “What's that piece of crap do?” “Well, about $4 billion a year in revenue. Perhaps show some respect.” It's a common refrain.Matt: Yeah, exactly. So yeah, anyone listening, don't mock legacy because as Corey says, it is running the business. But for us when it comes to day two, it's something that I'm actually really passionate about this in general because it is really easy. Like I did it with CDK patterns, it's really easy to come out and be like, “Okay, we're going to create a bunch of starter patterns, or quickstarts”—or whatever flavor that you came up with—“And then you're going to deploy this thing, and we're going to have you in production and 30 seconds.” But even day one later that day—not even necessarily day two—it depends on who it was that deployed it and how long they've been using AWS.So, you hear these stories of people who deployed something to experiment, and they either forget to delete, it cost them a lot of money or they tried to change it and it breaks because they didn't understand what was in it. And this is where the community starts to diverge in their opinions on what AWS CDK should be. There's a lot of people who think that at the minute CDK, even if you create an abstraction in a construct, even if I create a construct and put it in the construct library that you get to use, it still unravels and deploys as part of your deploy. So, everything that's associated with it, you don't own and you technically need to understand that at some point because it might, in theory, break. Whereas there's a lot of people who think, “Okay, the CDK needs to go server side and an abstraction needs to stay an abstraction in the cloud. And then that way, if somebody is looking at a 20-line CDK construct or stack, then it stays 20 lines. It never unravels to something crazy underneath.”I mean, that's one pro tip thing. It'd be awesome if that could work. I'm not sure how the support for that would work from a—if you've got something running on the cloud, I'm pretty sure AWS [laugh] aren't going to jump on a call to support some construct that I deployed, so I'm not sure how that will work in the open-source sense. But what we're doing at Liberty is the other way. So, I mean, we famously have things like the software accelerator that lets you pick a pattern or create your pipelines and you're deployed, but now what we're doing is we're building a lot of telemetry and automated information around what you deployed so that way—and it's all based on Well-Architected, common theme. So, that way, what you can do is you can go into [crosstalk 00:26:07]—Corey: It's partially [unintelligible 00:26:07], and partially at a glance, figure out okay, are there some things that can be easily remediated as we basically shift that whole thing left?Matt: Yeah, so if you deploy something, and it should be good the second you deploy it, but then you start making changes. Because you're Corey, you just start adding some stuff and you deploy it. And if it's really bad, it won't deploy. Like, that's the Liberty setup. There's a bunch of rules that all go, “Okay, that's really bad. That'll cause damage to customers.”But there's a large gap between bad and good that people don't really understand the difference that can cost a lot of money or can cause a lot of grief for developers because they go down the wrong path. So, that's why what we're now building is, after you deploy, there's a dashboard that'll just come up and be like, “Hey, we've noticed that your Lambda function has too little memory. It's going to be slow. You're going to have bad cold starts.” Or you know, things like that.The knowledge that I have had the gain through hard fighting over the past couple of years putting it into automation, and that way, combined with the well-architected reviews, you actually get me sitting in a call going, “Okay, let's talk about what you're building,” that hopefully guides people the right way. But I still think there's so much more we can do for day two because even if you deploy the best solution today, six months from now, AWS are releasing ten new services that make it easier to do what you just did. So, someone also needs to build something that shows you the delta to get to the best. And that would involve AWS or somebody thinking cohesively, like, these are how we use our products. And I don't think there's a market for it as a third-party company, unfortunately, but I do think that's where we need to get to, that at day two somebody can give—the way we're trying to do for Liberty—advice, automated that says, “I see what you're doing, but it would be better if you did this instead.”Corey: Yeah, I definitely want to spend more time thinking about these things and analyzing how we wind up addressing them and how we think about them going forward. I learned a lot of these lessons over a decade ago. I was fairly deep into using Puppet, and came to the fair and balanced conclusion that Puppet was a steaming piece of crap. So, the solution was that I was one of the very early developers behind SaltStack, which was going to do everything right. And it was and it was awesome and it was glorious, right up until I saw an environment deployed by someone else who was not as familiar with the tool as I was, at which point I realized hell is other people's use cases.And the way that they contextualize these things, you craft a finely balanced torque wrench, it's a thing of beauty, and people complain about the crappy hammer. “You're holding it wrong. No, don't do it that way.” So, I have an awful lot of sympathy for people building platform-level tooling like this, where it works super well for the use case that they're in, but not necessarily… they're not necessarily aligned in other ways. It's a very hard nut to crack.Matt: Yeah. And like, even as you mentioned earlier, if you take one piece of AWS, for example, API Gateway—and I love the API Gateway team; if you're listening, don't hate on me—but there's, like, 47,000 different ways you can deploy an API Gateway. And the CDK has to cover all of those, it would be a lot easier if there was less ways that you could deploy the thing and then you can start crafting user experiences on a platform. But whenever you start thinking that every AWS component is kind of the same, like think of the amount of ways you're can deploy a Lambda function now, or think of the, like, containers. I'll not even go into [laugh] the different ways to run containers.If you're building a platform, either you support it all and then it sort of gets quite generic-y, or you're going to do, like, what serverless cloud are doing though, like Jeremy Daly is building this unique experience that's like, “Okay, the code is going to build the infrastructure, so just build a website, and we'll do it all behind it.” And I think they're really interesting because they're sort of opposites, in that one doesn't want to support everything, but should theoretically, for their slice of customers, be awesome, and then the other ones, like, “Well, let's see what you're going to do. Let's have a go at it and I should hopefully support it.”Corey: I think that there's so much that can be done on this. But before we wind up calling it an episode, I had one further question that I wanted to explore around the recent results of the community CDK survey that I believe is a quarterly event. And I read the analysis on this, and I talked about it briefly in the newsletter, but it talks about adoption and a few other aspects of it. And one of the big things it looks at is the number of people who are contributing to the CDK in an open-source context. Am I just thinking about this the wrong way when I think that, well, this is a tool that helps me build out cloud infrastructure; me having to contribute code to this thing at all is something of a bug, whereas yeah, I want this thing to work out super well—Docker is open-source, but you'll never see me contributing things to Docker ever, as a pull request, because it does, as it says on the tin; I don't have any problems that I'm aware of that, ooh, it should do this instead. I mean, I have opinions on that, but those aren't pull requests; those are complete, you know, shifts in product strategy, which it turns out is not quite done on GitHub.Matt: So, it's funny I, a while ago, was talking to a lad who was the person who came up with the idea for the CDK. And CDK is pretty much the open-source project for AWS if you look at what they have. And the thought behind it, it's meant to evolve into what people want and need. So yes, there is a product manager in AWS, and there's a team fully dedicated to building it, but the ultimate aspiration was always it should be bigger than AWS and it should be community-driven. Now personally, I'm not sure—like you just said it—what the incentive is, given that right now CDK only works with CloudFormation, which means that you are directly helping with an AWS tool, but it does give me hope for, like, their CDK for Terraform, and their CDK for Kubernetes, and there's other flavors based on the same technology as AWS CDK that potentially could have a thriving open-source community because they work across all the clouds. So, it might make more sense for people to jump in there.Corey: Yeah, I don't necessarily think that there's a strong value proposition as it stands today for the idea of the CDK becoming something that works across other cloud providers. I know it technically has the capability, but if I think that Python isn't quite a first-class experience, I don't even want to imagine what other providers are going to look like from that particular context.Matt: Yeah, and that's from what I understand, I haven't personally jumped into the CDK for Terraform and we didn't talk about it here, but in CDK, you get your different levels of construct. And is, like, a CloudFormation-level construct, so everything that's in there directly maps to a property in CloudFormation, and then L2 is AWS's opinion on safe defaults, and then L3 is when someone like me comes along and turns it into something that you may find useful. So, it's a pattern. As far as I know, CDK for Terraform is still on L1. They haven't got the rich collection—Corey: And L4 is just hiring you as a consultant—Matt: [laugh].Corey: —to come in fix my nonsense for me?Matt: [laugh]. That's it. L4 could be Pulumi recently announced that you can use AWS CDK constructs inside it. But I think it's one of those things where the constructs, if they can move across these different tools the way AWS CDK constructs now work inside Pulumi, and there's a beta version that works inside CDK for Terraform, then it may or may not make sense for people to contribute to this stuff because we're not building at a higher level. It's just the vision is hard for most people to get clear in their head because it needs articulated and told as a clear strategy.And then, you know, as you said, it is an AWS product strategy, so I'm not sure what you get back by contributing to the project, other than, like, Thorsten—I should say, so Thorsten who wrote the book with me, he is the number three contributor, I think, to the CDK. And that's just because he is such a big user of it that if he sees something that annoys him, he just comes in and tries to fix it. So, the benefit is, he gets to use the tool. But he is a super user, so I'm not sure, outside of super users, what the use case is.Corey: I really want to thank you for, I want to say spending as much time talking to me about this stuff as you have, but that doesn't really go far enough. Because so much of how I think about this invariably winds up linking back to things that you have done and have been advocating for in that community for such a long time. If it's not you personally, just, like, your fingerprints are all over this thing. So, it's one of those areas where the entire software developer ecosystem is really built on the shoulders of others who have done a lot of work that came before. Often you don't get any visibility of who those people are, so it's interesting whenever I get to talk to someone whose work I have directly built upon that I get to say thank you. So, thank you for this. I really do appreciate how much more straightforward a lot of this is than my previous approach of clicking in the console and then lying about it to provision infrastructure.Matt: Oh, no worries. Thank you for the thank you. I mean, at the end of the day, all of this stuff is just—it helps me as much as it helps everybody else, and we're all trying to do make everything quicker for ourselves, at the end of the day.Corey: If people want to learn more about what you're up to, where's the best place to find you these days? They can always take a job at Liberty; I hear good things about it.Matt: Yeah, we're always looking for people at Liberty, so come look up our careers. But Twitter is always the best place. So, I'm @NIDeveloper on Twitter. You should find me pretty quickly, or just type Matt Coulter into Google, you'll get me.Corey: I like it. It's always good when it's like, “Oh, I'm the top Google result for my own name.” On some level, that becomes an interesting thing. Some folks into it super well, John Smith has some challenges, but you know, most people are somewhere in the middle of that.Matt: I didn't used to be number one, but there's a guy called the Kangaroo Kid in Australia, who is, like, a stunt driver, who was number one, and [laugh] I always thought it was funny if people googled and got him and thought it was me. So, it's not anymore.Corey: Thank you again for, I guess, all that you do. And of course, taking the time to suffer my slings and arrows as I continue to revise my opinion of the CDK upward.Matt: No worries. Thank you for having me.Corey: Matt Coulter, senior architect at Liberty Mutual. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and leave an angry comment as well that will not actually work because it has to be transpiled through a JavaScript engine first.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About AndrewI create free cloud certification courses and somehow still make money.Links: ExamPro Training, Inc.: https://www.exampro.co/ PolyWork: https://www.polywork.com/andrewbrown LinkedIn: https://www.linkedin.com/in/andrew-wc-brown Twitter: https://twitter.com/andrewbrown TranscriptAndrew: Hello, and welcome to Screaming in the Cloud with your host, Chief cloud economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: This episode is sponsored in part by our friends at Rising Cloud, which I hadn't heard of before, but they're doing something vaguely interesting here. They are using AI, which is usually where my eyes glaze over and I lose attention, but they're using it to help developers be more efficient by reducing repetitive tasks. So, the idea being that you can run stateless things without having to worry about scaling, placement, et cetera, and the rest. They claim significant cost savings, and they're able to wind up taking what you're running as it is in AWS with no changes, and run it inside of their data centers that span multiple regions. I'm somewhat skeptical, but their customers seem to really like them, so that's one of those areas where I really have a hard time being too snarky about it because when you solve a customer's problem and they get out there in public and say, “We're solving a problem,” it's very hard to snark about that. Multus Medical, Construx.ai and Stax have seen significant results by using them. And it's worth exploring. So, if you're looking for a smarter, faster, cheaper alternative to EC2, Lambda, or batch, consider checking them out. Visit risingcloud.com/benefits. That's risingcloud.com/benefits, and be sure to tell them that I said you because watching people wince when you mention my name is one of the guilty pleasures of listening to this podcast.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is… well, he's challenging to describe. He's the co-founder and cloud instructor at ExamPro Training, Inc. but everyone knows him better as Andrew Brown because he does so many different things in the AWS ecosystem that it's sometimes challenging—at least for me—to wind up keeping track of them all. Andrew, thanks for joining.Andrew: Hey, thanks for having me on the show, Corey.Corey: How do I even begin describing you? You're an AWS Community Hero and have been for almost two years, I believe; you've done a whole bunch of work as far as training videos; you're, I think, responsible for #100daysofcloud; you recently started showing up on my TikTok feed because I'm pretending that I am 20 years younger than I am and hanging out on TikTok with the kids, and now I feel extremely old. And obviously, you're popping up an awful lot of places.Andrew: Oh, yeah. A few other places like PolyWork, which is an alternative to LinkedIn, so that's a space that I'm starting to build up on there as well. Active in Discord, Slack channels. I'm just kind of everywhere. There's some kind of internet obsession here. My wife gets really mad and says, “Hey, maybe tone down the social media.” But I really enjoy it. So.Corey: You're one of those folks where I have this challenge of I wind up having a bunch of different AWS community Slacks and cloud community, Slacks and Discords and the past, and we DM on Twitter sometimes. And I'm constantly trying to figure out where was that conversational thread that I had with you? And tracking it down is an increasingly large search problem. I really wish that—forget the unified messaging platform. I want a unified search platform for all the different messaging channels that I'm using to talk to people.Andrew: Yeah, it's very hard to keep up with all the channels for myself there. But somehow I do seem to manage it, but just with a bit less sleep than most others.Corey: Oh, yeah. It's like trying to figure out, like, “All right, he said something really useful. What was that? Was that a Twitter DM? Was it on that Slack channel? Was it that Discord? No, it was on that brick that he threw through my window with a note tied to it. There we go.”That's always the baseline stuff of figuring out where things are. So, as I mentioned in the beginning, you are the co-founder and cloud instructor at ExamPro, which is interesting because unlike most of the community stuff that you do and are known for, you don't generally talk about that an awful lot. What's the deal there?Andrew: Yeah, I think a lot of people give me a hard time because they say, Andrew, you should really be promoting yourself more and trying to make more sales, but that's not why I'm out here doing what I'm doing. Of course, I do have a for-profit business called ExamPro, where we create cloud certification study courses for things like AWS, Azure, GCP, Terraform, Kubernetes, but you know, that money just goes to fuel what I really want to do, is just to do community activities to help people change their lives. And I just decided to do that via cloud because that's my domain expertise. At least that's what I say because I've learned up on in the last four or five years. I'm hoping that there's some kind of impact I can make doing that.Corey: I take a somewhat similar approach. I mean, at The Duckbill Group, we fixed the horrifying AWS bill, but I've always found that's not generally a problem that people tend to advertise having. On Twitter, like, “Oh, man, my AWS bill is killing me this month. I've got to do something about it,” and you check where they work, and it's like a Fortune 50. It's, yeah, that moves markets and no one talks about that.So, my approach was always, be out there, be present in the community, talk about this stuff, and the people who genuinely have billing problems will eventually find their way to me. That was always my approach because turning everything I do into a sales pitch doesn't work. It just erodes confidence, it reminds people of the used mattress salesman, and I just don't want to be that person in that community. My approach has always been if I can help someone with a 15-minute call or whatnot, yeah, let's jump on a phone call. I'm not interested in nickel-and-diming folks.Andrew: Yeah. I think that if you're out there doing a lot of hard work, and a lot of it, it becomes undeniable the value you're putting out there, and then people just will want to give you money, right? And for me, I just feel really bad about taking anybody's money, and so even when there's some kind of benefit—like my courses, I could charge for access for them, but I always feel I have to give something in terms of taking somebody's money, but I would never ask anyone to give me their money. So, it's bizarre. [laugh] so.Corey: I had a whole bunch of people a year or so after I started asking, like, “I really find your content helpful. Can I buy you a cup of coffee or something?” And it's, I don't know how to charge people a dollar figure that doesn't have a comma in it because it's easy for me to ask a company for money; that is the currency of effort, work, et cetera, that companies are accustomed to. People view money very differently, and if I ask you personally for money versus your company for money, it's a very different flow. So, my solution to it was to build the annual charity t-shirt drive, where it's, great, spend 35 bucks or whatever on a snarky t-shirt once a year for ten days and all proceeds go to benefit a nonprofit that is, sort of, assuaged that.But one of my business philosophies has always been, “Work for free before you work for cheap.” And dealing with individuals and whatnot, I do not charge them for things. It's, “Oh, can you—I need some advice in my career. Can I pay you to give me some advice?” “No, but you can jump on a Zoom call with me.” Please, the reason I exist at all is because people who didn't have any reason to did me favors, once upon a time, and I feel obligated to pay that forward.Andrew: And I appreciate, you know, there are people out there that you know, do need to charge for their time. Like—Corey: Oh. Oh, yes.Andrew: —I won't judge anybody that wants to. But you know, for me, it's just I can't do it because of the way I was raised. Like, my grandfather was very involved in the community. Like, he was recognized by the city for all of his volunteer work, and doing volunteer work was, like, mandatory for me as a kid. Like, every weekend, and so for me, it's just like, I can't imagine trying to take people's money.Which is not a great thing, but it turns out that the community is very supportive, and they will come beat you down with a stick, to give you money to make sure you keep doing what you're doing. But you know, I could be making lots of money, but it's just not my priority, so I've avoided any kind of funding so like, you know, I don't become a money-driven company, and I will see how long that lasts, but hopefully, a lot longer.Corey: I wish you well. And again, you're right; no shade to anyone who winds up charging for their time to individuals. I get it. I just always had challenges with it, so I decided not to do it. The only time I find myself begrudging people who do that are someone who picked something up six months ago and decided, oh, I'm going to build some video course on how to do this thing. The end. And charge a bunch of money for it and put myself out as an expert in that space.And you look at what the content they're putting out is, and one, it's inaccurate, which just drives me up a wall, and two, there's a lack of awareness that teaching is its own skill. In some areas, I know how to teach certain things, and in other areas, I'm a complete disaster at it. Public speaking is a great example. A lot of what I do on the public speaking stage is something that comes to me somewhat naturally. So, can you teach me to be a good public speaker? Not really, it's like, well, you gave that talk and it was bad. Could you try giving it only make it good? Like, that is not a helpful coaching statement, so I stay out of that mess.Andrew: Yeah, I mean, it's really challenging to know, if you feel like you're authority enough to put something out there. And there's been a few courses where I didn't feel like I was the most knowledgeable, but I produced those courses, and they had done extremely well. But as I was going through the course, I was just like, “Yeah, I don't know how any this stuff works, but this is my best guess translating from here.” And so you know, at least for my content, people have seen me as, like, the lens of AWS on top of other platforms, right? So, I might not know—I'm not an expert in Azure, but I've made a lot of Azure content, and I just translate that over and I talk about the frustrations around, like, using scale sets compared to AWS auto-scaling groups, and that seems to really help people get through the motions of it.I know if I pass, at least they'll pass, but by no means do I ever feel like an expert. Like, right now I'm doing, like, Kubernetes. Like, I have no idea how I'm doing it, but I have, like, help with three other people. And so I'll just be honest about it and say, “Hey, yeah, I'm learning this as well, but at least I know I passed, so you know, you can pass, too.” Whatever that's worth.Corey: Oh, yeah. Back when I was starting out, I felt like a bit of a fraud because I didn't know everything about the AWS billing system and how it worked and all the different things people can do with it, and things they can ask. And now, five years later, when the industry basically acknowledges I'm an expert, I feel like a fraud because I couldn't possibly understand everything about the AWS billing system and how it works. It's one of those things where the more you learn, the more you realize that there is yet to learn. I'm better equipped these days to find the answers to the things I need to know, but I'm still learning things every day. If I ever get to a point of complete and total understanding of a given topic, I'm wrong. You can always go deeper.Andrew: Yeah, I mean, by no means am I even an expert in AWS, though people seem to think that I am just because I have a lot of confidence in there and I produce a lot of content. But that's a lot different from making a course than implementing stuff. And I do implement stuff, but you know, it's just at the scale that I'm doing that. So, just food for thought for people there.Corey: Oh, yeah. Whatever, I implement something. It's great. In my previous engineering life, I would work on large-scale systems, so I know how a thing that works in your test environment is going to blow up in a production scale environment. And I bring those lessons, written on my bones the painful way, through outages, to the way that I build things now.But the stuff that I'm building is mostly to keep my head in the game, as opposed to solving an explicit business need. Could I theoretically build a podcast transcription system on top of Transcribe or something like that for these episodes? Yeah. But I've been paying a person to do this for many years to do it themselves; they know the terms of art, they know how this stuff works, and they're building a glossary as they go, and understanding the nuances of what I say and how I say it. And that is the better business outcome; that's the answer. And if it's production facing, I probably shouldn't be tinkering with it too much, just based upon where the—I don't want to be the bottleneck for the business functioning.Andrew: I've been spending so much time doing the same thing over and over again, but for different cloud providers, and the more I do, the less I want to go deep on these things because I just feel like I'm dumping all this information I'm going to forget, and that I have those broad strokes, and when I need to go deep dive, I have that confidence. So, I'd really prefer people were to build up confidence in saying, “Yes, I think I can do this.” As opposed to being like, “Oh, I have proof that I know every single feature in AWS Systems Manager.” Just because, like, our platform, ExamPro, like, I built it with my co-founder, and it's a quite a system. And so I'm going well, that's all I need to know.And I talk to other CTOs, and there's only so much you need to know. And so I don't know if there's, like, a shift between—or difference between, like, application development where, let's say you're doing React and using Vercel and stuff like that, where you have to have super deep knowledge for that technical stack, whereas cloud is so broad or diverse that maybe just having confidence and hypothesizing the work that you can do and seeing what the outcome is a bit different, right? Not having to prove one hundred percent that you know it inside and out on day one, but have the confidence.Corey: And there's a lot of validity to that and a lot of value to it. It's the magic word I always found in interviewing, on both sides of the interview table, has always been someone who's unsure about something start with, “I'm not sure, but if I had to guess,” and then say whatever it is you were going to say. Because if you get it right, wow, you're really good at figuring this out, and your understanding is pretty decent. If you're wrong, well, you've shown them how you think but you've also called them out because you're allowed to be wrong; you're not allowed to be authoritatively wrong. Because once that happens, I can't trust anything you say.Andrew: Yeah. In terms of, like, how do cloud certifications help you for your career path? I mean, I find that they're really well structured, and they give you a goal to work towards. So, like, passing that exam is your motivation to make sure that you complete it. Do employers care? It depends. I would say mostly no. I mean, for me, like, when I'm hiring, I actually do care about certifications because we make certification courses but—Corey: In your case, you're a very specific expression of this that is not typical.Andrew: Yeah. And there are some, like, cases where, like, if you work for a larger cloud consultancy, you're expected to have a professional certification so that customers feel secure in your ability to execute. But it's not like they were trying to hire you with that requirement, right? And so I hope that people realize that and that they look at showing that practical skills, by building up cloud projects. And so that's usually a strong pairing I'll have, which is like, “Great. Get the certifications to help you just have a structured journey, and then do a Cloud project to prove that you can do what you say you can do.”Corey: One area where I've seen certifications act as an interesting proxy for knowledge is when you have a company that has 5000 folks who work in IT in varying ways, and, “All right. We're doing a big old cloud migration.” The certification program, in many respects, seems to act as a bit of a proxy for gauging where people are on upskilling, how much they have to learn, where they are in that journey. And at that scale, it begins to make some sense to me. Where do you stand on that?Andrew: Yeah. I mean, it's hard because it really depends on how those paths are built. So, when you look at the AWS certification roadmap, they have the Certified Cloud Practitioner, they have three associates, two professionals, and a bunch of specialties. And I think that you might think, “Well, oh, solutions architect must be very popular.” But I think that's because AWS decided to make the most popular, the most generic one called that, and so you might think that's what's most popular.But what they probably should have done is renamed that Solution Architect to be a Cloud Engineer because very few people become Solutions Architect. Like that's more… if there's Junior Solutions Architect, I don't know where they are, but Solutions Architect is more of, like, a senior role where you have strong communications, pre-sales, obviously, the role is going to vary based on what companies decide a Solution Architect is—Corey: Oh, absolutely take a solutions architect, give him a crash course in finance, and we call them a cloud economist.Andrew: Sure. You just add modifiers there, and they're something else. And so I really think that they should have named that one as the cloud engineer, and they should have extracted it out as its own tier. So, you'd have the Fundamental, the Certified Cloud Practitioner, then the Cloud Engineer, and then you could say, “Look, now you could do developer or the sysops.” And so you're creating this path where you have a better trajectory to see where people really want to go.But the problem is, a lot of people come in and they just do the solutions architect, and then they don't even touch the other two because they say, well, I got an associate, so I'll move on the next one. So, I think there's some structuring there that comes into play. You look at Azure, they've really, really caught up to AWS, and may I might even say surpass them in terms of the quality and the way they market them and how they construct their certifications. There's things I don't like about them, but they have, like, all these fundamental certifications. Like, you have Azure Fundamentals, Data Fundamentals, AI Fundamentals, there's a Security Fundamentals.And to me, that's a lot more valuable than going over to an associate. And so I did all those, and you know, I still think, like, should I go translate those over for AWS because you have to wait for a specialty before you pick up security. And they say, like, it's intertwined with all the certifications, but, really isn't. Like—and I feel like that would be a lot better for AWS. But that's just my personal opinion. So.Corey: My experience with AWS certifications has been somewhat minimal. I got the Cloud Practitioner a few years ago, under the working theory of I wanted to get into the certified lounge at some of the events because sometimes I needed to charge things and grab a cup of coffee. I viewed it as a lounge pass with a really strange entrance questionnaire. And in my case, yeah, I passed it relatively easily; if not, I would have some questions about how much I actually know about these things. As I recall, I got one question wrong because I was honest, instead of going by the book answer for, “How long does it take to restore an RDS database from a snapshot?”I've had some edge cases there that give the wrong answer, except that's what happened. And then I wound up having that expire and lapse. And okay, now I'll do it—it was in beta at the time, but I got the sysops associate cert to go with it. And that had a whole bunch of trivia thrown into it, like, “Which of these is the proper syntax for this thing?” And that's the kind of question that's always bothered me because when I'm trying to figure things like that out, I have entire internet at my fingertips. Understanding the exact syntax, or command-line option, or flag that needs to do a thing is a five-second Google search away in most cases. But measuring for people's ability to memorize and retain that has always struck me as a relatively poor proxy for knowledge.Andrew: It's hard across the board. Like Azure, AWS, GCP, they all have different approaches—like, Terraform, all of them, they're all different. And you know, when you go to interview process, you have to kind of extract where the value is. And I would think that the majority of the industry, you know, don't have best practices when hiring, there's, like, a superficial—AWS is like, “Oh, if you do well, in STAR program format, you must speak a communicator.” Like, well, I'm dyslexic, so that stuff is not easy for me, and I will never do well in that.So like, a lot of companies hinge on those kinds of components. And I mean, I'm sure it doesn't matter; if you have a certain scale, you're going to have attrition. There's no perfect system. But when you look at these certifications, and you say, “Well, how much do they match up with the job?” Well, they don't, right? It's just Jeopardy.But you know, I still think there's value for yourself in terms of being able to internalize it. I still think that does prove that you have done something. But taking the AWS certification is not the same as taking Andrew Brown's course. So, like, my certified cloud practitioner was built after I did GCP, Oracle Cloud, Azure Fundamentals, a bunch of other Azure fundamental certifications, cloud-native stuff, and then I brought it over because was missing, right? So like, if you went through my course, and that I had a qualifier, then I could attest to say, like, you are of this skill level, right?But it really depends on what that testament is and whether somebody even cares about what my opinion of, like, your skillset is. But I can't imagine like, when you have a security incident, there's going to be a pop-up that shows you multiple-choice answer to remediate the security incident. Now, we might get there at some point, right, with all the cloud automation, but we're not there yet.Corey: It's been sort of thing we've been chasing and never quite get there. I wish. I hope I live to see it truly I do. My belief is also that the value of a certification changes depending upon what career stage someone is at. Regardless of what level you are at, a hiring manager or a company is looking for more or less a piece of paper that attests that they're to solve the problem that they are hiring to solve.And entry-level, that is often a degree or a certification or something like that in the space that shows you have at least the baseline fundamentals slash know how to learn things. After a few years, I feel like that starts to shift into okay, you've worked in various places solving similar problems on your resume that the type that we have—because the most valuable thing you can hear when you ask someone, “How would we solve this problem?” Is, “Well, the last time I solved it, here's what we learned.” Great. That's experience. There's no compression algorithm for experience? Yes, there is: Hiring people with experience.Then, at some level, you wind up at the very far side of people who are late-career in many cases where the piece of paper that shows that they know what they're doing is have you tried googling their name and looking at the Wikipedia article that spits out, how they built fundamental parts of a system like that. I think that certifications are one of those things that bias for early-career folks. And of course, partners when there are other business reasons to get it. But as people grow in seniority, I feel like the need for those begins to fall off. Do you agree? Disagree? You're much closer to this industry in that aspect of it than I am.Andrew: The more senior you are, and if you have big names under your resume there, no one's going to care if you have certification, right? When I was looking to switch careers—I used to have a consultancy, and I was just tired of building another failed startup for somebody that was willing to pay me. And I'm like—I was not very nice about it. I was like, “Your startup's not going to work out. You really shouldn't be building this.” And they still give me the money and it would fail, and I'd move on to the next one. It was very frustrating.So, closed up shop on that. And I said, “Okay, I got to reenter the market.” I don't have a computer science degree, I don't have big names on my resume, and Toronto is a very competitive market. And so I was feeling friction because people were not valuing my projects. I had, like, full-stack projects, I would show them.And they said, “No, no. Just do these, like, CompSci algorithms and stuff like that.” And so I went, “Okay, well, I really don't want to be doing that. I don't want to spend all my time learning algorithms just so I can get a job to prove that I already have the knowledge I have.” And so I saw a big opportunity in cloud, and I thought certifications would be the proof to say, “I can do these things.”And when I actually ended up going for the interviews, I didn't even have certifications and I was getting those opportunities because the certifications helped me prove it, but nobody cared about the certifications, even then, and that was, like, 2017. But not to say, like, they didn't help me, but it wasn't the fact that people went, “Oh, you have a certification. We'll get you this job.”Corey: Yeah. When I'm talking to consulting clients, I've never once been asked, “Well, do you have the certifications?” Or, “Are you an AWS partner?” In my case, no, neither of those things. The reason that we know what we're doing is because we've done this before. It's the expertise approach.I question whether that would still be true if we were saying, “Oh, yeah, and we're going to drop a dozen engineers on who are going to build things out of your environment.” “Well, are they certified?” is a logical question to ask when you're bringing in an external service provider? Or is this just a bunch of people you found somewhere on Upwork or whatnot, and you're throwing them at it with no quality control? Like, what is the baseline level experience? That's a fair question. People are putting big levels of trust when they bring people in.Andrew: I mean, I could see that as a factor of some clients caring, just because like, when I used to work in startups, I knew customers where it's like their second startup, and they're flush with a lot of money, and they're deciding who they want to partner with, and they're literally looking at what level of SSL certificate they purchased, right? Like now, obviously, they're all free and they're very easy to get to get; there was one point where you had different tiers—as if you would know—and they would look and they would say—Corey: Extended validation certs attend your browser bar green. Remember those?Andrew: Right. Yeah, yeah, yeah. It was just like that, and they're like, “We should partner with them because they were able to afford that and we know, like…” whatever, whatever, right? So, you know, there is that kind of thought process for people at an executive level. I'm not saying it's widespread, but I've seen it.When you talk to people that are in cloud consultancy, like solutions architects, they always tell me they're driven to go get those professional certifications [unintelligible 00:22:19] their customers matter. I don't know if the customers care or not, but they seem to think so. So, I don't know if it's just more driven by those people because it's an expectation because everyone else has it, or it's like a package of things, like, you know, like the green bar in the certifications, SOC 2 compliance, things like that, that kind of wrap it up and say, “Okay, as a package, this looks really good.” So, more of an expectation, but not necessarily matters, it's just superficial; I'm not sure.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: You've been building out certifications for multiple cloud providers, so I'm curious to get your take on something that Forrest Brazeal, who's now head of content over at Google Cloud, has been talking about lately, the idea that as an engineer is advised to learn more than one cloud provider; even if you have one as a primary, learning how another one works makes you a better engineer. Now, setting aside entirely the idea that well, yeah, if I worked at Google, I probably be saying something fairly similar.Andrew: Yeah.Corey: Do you think there's validity to the idea that most people should be broad across multiple providers, or do you think specialization on one is the right path?Andrew: Sure. Just to contextualize for our listeners, Google Cloud is highly, highly promoting multi-cloud workloads, and one of their flagship products is—well, they say it's a flagship product—is Anthos. And they put a lot of money—I don't know that was subsidized, but they put a lot of money in it because they really want to push multi-cloud, right? And so when we say Forrest works in Google Cloud, it should be no surprise that he's promoting it.But I don't work for Google, and I can tell you, like, learning multi-cloud is, like, way more valuable than just staying in one vertical. It just opened my eyes. When I went from AWS to Azure, it was just like, “Oh, I'm missing out on so much in the industry.” And it really just made me such a more well-rounded person. And I went over to Google Cloud, and it was just like… because you're learning the same thing in different variations, and then you're also poly-filling for things that you will never touch.Or like, I shouldn't say you never touch, but you would never touch if you just stayed in that vertical when you're learning. So, in the industry, Azure Active Directory is, like, widespread, but if you just stayed in your little AWS box, you're not going to notice it on that learning path, right? And so a lot of times, I tell people, “Go get your CLF-C01 and then go get your AZ-900 or AZ-104.” Again, I don't care if people go and sit the exams. I want them to go learn the content because it is a large eye-opener.A lot of people are against multi-cloud from a learning perspective because say, it's too much to learn all at the same time. But a lot of people I don't think have actually gone across the cloud, right? So, they're sitting from their chair, only staying in one vertical saying, “Well, you can't learn them all at the same time.” And I'm going, “I see a way that you could teach them all at the same time.” And I might be the first person that will do it.Corey: And the principles do convey as well. It's, “Oh, well I know how SNS works on AWS, so I would never be able to understand how Google Pub/Sub works.” Those are functionally identical; I don't know that is actually true. It's just different to interface points and different guarantees, but fine. You at least understand the part that it plays.I've built things out on Google Cloud somewhat recently, and for me, every time I do, it's a refreshing eye-opener to oh, this is what developer experience in the cloud could be. And for a lot of customers, it is. But staying too far within the bounds of one ecosystem does lend itself to a loss of perspective, if you're not careful. I agree with that.Andrew: Yeah. Well, I mean, just the paint more of a picture of differences, like, Google Cloud has a lot about digital transformation. They just updated their—I'm not happy that they changed it, but I'm fine that they did that, but they updated their Google Digital Cloud Leader Exam Guide this month, and it like is one hundred percent all about digital transformation. So, they love talking about digital transformation, and those kind of concepts there. They are really good at defining migration strategies, like, at a high level.Over to Azure, they have their own cloud adoption framework, and it's so detailed, in terms of, like, execution, where you go over to AWS and they have, like, the worst cloud adoption framework. It's just the laziest thing I've ever seen produced in my life compared to out of all the providers in that space. I didn't know about zero-trust model until I start using Azure because Azure has Active Directory, and you can do risk-based policy procedures over there. So, you know, like, if you don't go over to these places, you're not going to get covered other places, so you're just going to be missing information till you get the job and, you know, that job has that information requiring you to know it.Corey: I would say that for someone early career—and I don't know where this falls on the list of career advice ranging from, “That is genius,” to, “Okay, Boomer,” but I would argue that figuring out what companies in your geographic area, or the companies that you have connections with what they're using for a cloud provider, I would bias for learning one enough to get hired there and from there, letting what you learn next be dictated by the environment you find yourself in. Because especially larger companies, there's always something that lives in a different provider. My default worst practice is multi-cloud. And I don't say that because multi-cloud doesn't exist, and I'm not saying it because it's a bad idea, but this idea of one workload—to me—that runs across multiple providers is generally a challenge. What I see a lot more, done intelligently, is, “Okay, we're going to use this provider for some things, this other provider for other things, and this third provider for yet more things.” And every company does that.If not, there's something very strange going on. Even Amazon uses—if not Office 365, at least exchange to run their email systems instead of Amazon WorkMail because—Andrew: Yeah.Corey: Let's be serious. That tells me a lot. But I don't generally find myself in a scenario where I want to build this application that is anything more than Hello World, where I want it to run seamlessly and flawlessly across two different cloud providers. That's an awful lot of work that I struggle to identify significant value for most workloads.Andrew: I don't want to think about securing, like, multiple workloads, and that's I think a lot of friction for a lot of companies are ingress-egress costs, which I'm sure you might have some knowledge on there about the ingress-egress costs across providers.Corey: Oh, a little bit, yeah.Andrew: A little bit, probably.Corey: Oh, throwing data between clouds is always expensive.Andrew: Sure. So, I mean, like, I call multi-cloud using multiple providers, but not in tandem. Cross-cloud is when you want to use something like Anthos or Azure Arc or something like that where you extend your data plane or control pla—whatever the plane is, whatever plane across all the providers. But you know, in practice, I don't think many people are doing cross-cloud; they're doing multi-cloud, like, “I use AWS to run my primary workloads, and then I use Microsoft Office Suite, and so we happen to use Azure Active Directory, or, you know, run particular VM machines, like Windows machines for our accounting.” You know?So, it's a mixed bag, but I do think that using more than one thing is becoming more popular just because you want to use the best in breed no matter where you are. So like, I love BigQuery. BigQuery is amazing. So, like, I ingest a lot of our data from, you know, third-party services right into that. I could be doing that in Redshift, which is expensive; I could be doing that in Azure Synapse, which is also expensive. I mean, there's a serverless thing. I don't really get serverless. So, I think that, you know, people are doing multi-cloud.Corey: Yeah. I would agree. I tend to do things like that myself, and whenever I see it generally makes sense. This is my general guidance. When I talk to individuals who say, “Well, we're running multi-cloud like this.” And my response is, “Great. You're probably right.”Because I'm talking in the general sense, someone building something out on day one where they don't know, like, “Everyone's saying multi-cloud. Should I do that?” No, I don't believe you should. Now, if your company has done that intentionally, rather than by accident, there's almost certainly a reason and context that I do not have. “Well, we have to run our SaaS application in multiple cloud providers because that's where our customers are.” “Yeah, you should probably do that.” But your marketing, your billing systems, your back-end reconciliation stuff generally does not live across all of those providers. It lives in one. That's the sort of thing I'm talking about. I think we're in violent agreement here.Andrew: Oh, sure, yeah. I mean, Kubernetes obviously is becoming very popular because people believe that they'll have a lot more mobility, Whereas when you use all the different managed—and I'm still learning Kubernetes myself from the next certification I have coming out, like, study course—but, you know, like, those managed services have all different kind of kinks that are completely different. And so, you know, it's not going to be a smooth process. And you're still leveraging, like, for key things like your database, you're not going to be running that in Kubernetes Cluster. You're going to be using a managed service.And so, those have their own kind of expectations in terms of configuration. So, I don't know, it's tricky to say what to do, but I think that, you know, if you have a need for it, and you don't have a security concern—like, usually it's security or cost, right, for multi-cloud.Corey: For me, at least, the lock-in has always been twofold that people don't talk about. More—less lock-in than buy-in. One is the security model where IAM is super fraught and challenging and tricky, and trying to map a security model to multiple providers is super hard. Then on top of that, you also have the buy-in story of a bunch of engineers who are very good at one cloud provider, and that skill set is not in less demand now than it was a year ago. So okay, you're going to start over and learn a new cloud provider is often something that a lot of engineers won't want to countenance.If your team is dead set against it, there's going to be some friction there and there's going to be a challenge. I mean, for me at least, to say that someone knows a cloud provider is not the naive approach of, “Oh yeah, they know how it works across the board.” They know how it breaks. For me, one of the most valuable reasons to run something on AWS is I know what a failure mode looks like, I know how it degrades, I know how to find out what's going on when I see that degradation. That to me is a very hard barrier to overcome. Alternately, it's entirely possible that I'm just old.Andrew: Oh, I think we're starting to see some wins all over the place in terms of being able to learn one thing and bring it other places, like OpenTelemetry, which I believe is a cloud-native Kubernetes… CNCF. I can't remember what it stands for. It's like Linux Foundation, but for cloud-native. And so OpenTelemetry is just a standardized way of handling your logs, metrics, and traces, right? And so maybe CloudWatch will be the 1.0 of observability in AWS, and then maybe OpenTelemetry will become more of the standard, right, and so maybe we might see more managed services like Prometheus and Grafa—well, obviously, AWS has a managed Prometheus, but other things like that. So, maybe some of those things will melt away. But yeah, it's hard to say what approach to take.Corey: Yeah, I'm wondering, on some level, whether what the things we're talking about today, how well that's going to map forward. Because the industry is constantly changing. The guidance I would give about should you be in cloud five years ago would have been a nuanced, “Mmm, depends. Maybe for yes, maybe for no. Here's the story.” It's a lot less hedge-y and a lot less edge case-y these days when I answer that question. So, I wonder in five years from now when we look back at this podcast episode, how well this discussion about what the future looks like, and certifications, and multi-cloud, how well that's going to reflect?Andrew: Well, when we look at, like, Kubernetes or Web3, we're just seeing kind of like the standardized boilerplate way of doing a bunch of things, right, all over the place. This distributed way of, like, having this generic API across the board. And how well that will take, I have no idea, but we do see a large split between, like, serverless and cloud-natives. So, it's like, what direction? Or we'll just have both? Probably just have both, right?Corey: [Like that 00:33:08]. I hope so. It's been a wild industry ride, and I'm really curious to see what changes as we wind up continuing to grow. But we'll see. That's the nice thing about this is, worst case, if oh, turns out that we were wrong on this whole cloud thing, and everyone starts exodusing back to data centers, well, okay. That's the nice thing about being a small company. It doesn't take either of us that long to address the reality we see in the industry.Andrew: Well, that or these cloud service providers are just going to get better at offering those services within carrier hotels, or data centers, or on your on-premise under your desk, right? So… I don't know, we'll see. It's hard to say what the future will be, but I do believe that cloud is sticking around in one form or another. And it basically is, like, an essential skill or table stakes for anybody that's in the industry. I mean, of course, not everywhere, but like, mostly, I would say. So.Corey: Andrew, I want to thank you for taking the time to speak with me today. If people want to learn more about your opinions, how you view these things, et cetera. Where can they find you?Andrew: You know, I think the best place to find me right now is Twitter. So, if you go to twitter.com/andrewbrown—all lowercase, no spaces, no underscores, no hyphens—you'll find me there. I'm so surprised I was able to get that handle. It's like the only place where I have my handle.Corey: And we will of course put links to that in the [show notes 00:34:25]. Thanks so much for taking the time to speak with me today. I really appreciate it.Andrew: Well, thanks for having me on the show.Corey: Andrew Brown, co-founder and cloud instructor at ExamPro Training and so much more. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment telling me that I do not understand certifications at all because you're an accountant, and certifications matter more in that industry.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

On The Cloud Pod this week, Jonathan reveals his love for “Twilight.” Plus GCP kicks off Google Cloud Next and announces Google Distributed Cloud, and Azure admits to a major DDoS attack.  A big thanks to this week's sponsors: Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. JumpCloud, which offers a complete platform for identity, access, and device management — no matter where your users and devices are located.  This week's highlights

Setting a new record for delay in editing, you can finally listen to Arjen, JM, and Guy discuss the news from April 2021. This was recorded nearly two months before it was released. News Finally in Sydney Amazon Transcribe Custom Language Models now support Australian English, British English, Hindi and US Spanish Multi-Attach for Provisioned IOPS io2 Now Available in Thirteen Additional AWS Regions AWS Transit Gateway Connect is now available in additional AWS Regions AWS CloudShell is now available in the Asia Pacific (Mumbai), Asia Pacific (Sydney), and Europe (Frankfurt) regions Serverless API Gateway Amazon API Gateway custom domain names now support multi-level base path mappings Lambda AWS Lambda@Edge changes duration billing granularity from 50ms down to 1ms Amazon CloudWatch Lambda Insights Now Supports AWS Lambda Container Images (General Availability) Amazon RDS for PostgreSQL Integrates with AWS Lambda AWS Lambda@Edge now supports Node 14.x Step Functions AWS Step Functions adds new data flow simulator for modelling input and output processing EventBridge Amazon EventBridge introduces support for cross-Region event bus targets AWS Chatbot now expands coverage of AWS Services monitored through Amazon EventBridge Amplify Data management is now generally available in the AWS Amplify Admin UI Amplify iOS now available via Swift Package Manager (SPM) AWS Amplify now orchestrates multiple Amazon DynamoDB GSI updates in a single deployment Containers eksctl now supports creating node groups using resource specifications and dry run mode AWS Secrets Manager Delivers Provider for Kubernetes Secrets Store CSI Driver EC2 & VPC Amazon EC2 Auto Scaling introduces Warm Pools to accelerate scale out while saving money Amazon VPC Flow Logs announces out-of-the-box integration with Amazon Athena MacSec Encryption for some Direct Connect (apologies, linking to this prevents the podcast from getting published :shrug:) New AWS Storage Gateway management console simplifies gateway creation and management AWS Batch now supports EFS volumes at the job level AWS Backup now supports cost allocation tags for Amazon EFS Backups Internet Group Management Protocol (IGMP) Multicast on AWS Transit Gateway is now available in major AWS regions worldwide Amazon EC2 enables replacing root volumes for quick restoration and troubleshooting Announcing availability of Red Hat Enterprise Linux with High availability for Amazon EC2 AWS Nitro Enclaves now supports Windows operating system Dev & Ops Dev Amazon CodeGuru Reviewer Updates: New Predictable Pricing Model Up To 90% Lower and Python Support Moves to GA | AWS News Blog Now available credential profile support for AWS SSO and Assume Role with MFA in the AWS Toolkit for Visual Studio AWS CodeDeploy improves support for EC2 deployments with Auto Scaling Groups AWS SAM CLI now supports AWS CDK applications - public preview Better together: AWS SAM and AWS CDK | AWS Compute Blog Proton AWS Proton allows adding and removing instances from an existing service AWS Proton introduces customer-managed environments AWS Proton adds an API to cancel deployments CloudFormation You can now deploy CloudFormation Stacks concurrently across multiple AWS regions using AWS CloudFormation StackSets AWS CloudFormation Command Line Interface (CFN-CLI) now supports TypeScript AWS CloudFormation Modules now Provides YAML and Delimiter Support Now reference latest AWS Systems Manager parameter values in AWS CloudFormation templates without specifying parameter versions You can now use macros and transforms in CloudFormation templates to create AWS CloudFormation StackSets Control Tower AWS Control Tower introduces changes to preventive S3 guardrails and updates to S3 bucket encryption protocols AWS Control Tower now provides configurable naming during Landing Zone setup Systems Manager AWS Systems Manager Run Command now displays more logs and enables log download from the console AWS Systems Manager Parameter Store now supports easier public parameter discoverability Customers can now use ServiceNow to track operational items related to AWS resources AWS Systems Manager Parameter Store now supports removal of parameter labels AWS Systems Manager now supports Amazon Elastic Container Service clusters AWS Systems Manager OpsCenter and Explorer now integrate with AWS Security Hub for diagnosis and remediation of security findings Security Firewalls How to Get Started with Amazon Route 53 Resolver DNS Firewall for Amazon VPC | AWS News Blog Reduce Unwanted Traffic on Your Website with New AWS WAF Bot Control | AWS News Blog AWS Firewall Manager now supports centralized management of Amazon Route 53 Resolver DNS Firewall AWS Firewall Manager now supports centralized deployment of the new AWS WAF Bot Control across your organization AWS WAF now supports Labels to improve rule customization and reporting Identity Review last accessed information to identify unused EC2, IAM, and Lambda permissions and tighten access for your IAM roles AWS Identity and Access Management now makes it easier to relate a user's IAM role activity to their corporate identity Other AWS Config launches the ability to track and visualize compliance change history of conformance packs AWS Security Hub Automated Response & Remediation Solution adds support for AWS Foundational Security Best Practices standard You now can use AWS CloudTrail to log Amazon DynamoDB Streams data-plane API activity Data Storage & Processing Glue Detect outliers and use dedicated transforms to handle outliers in AWS Glue DataBrew AWS Glue DataBrew now supports time-based, pattern-based and customizable parameters to create dynamic datasets AWS announces preview of AWS Glue custom blueprints AWS Glue now supports cross-account reads from Amazon Kinesis Data Streams AWS Glue now supports missing value imputation based on machine learning AWS announces data sink capability for the Glue connectors AWS Glue DataBrew announces native console integration with Amazon AppFlow to connect to data from SaaS (Software as a Service) applications and AWS services (in Preview) Redshift AQUA (Advanced Query Accelerator) – A Speed Boost for Your Amazon Redshift Queries | AWS News Blog Announcing cross-VPC support for Amazon Redshift powered by AWS PrivateLink Announcing general availability of Amazon Redshift native console integration with partners Announcing general availability of Amazon Redshift native JSON and semi-structured data support EMR Amazon EMR Release 5.33 now supports 10 new instance types Amazon EMR Studio is now generally available Athena Announcing general availability of Amazon Athena ML powered by Amazon SageMaker User Defined Functions (UDF) are now generally available for Amazon Athena RDS Amazon RDS for SQL Server now supports Extended Events Amazon RDS on VMware networking now simplified and more secure Other Amazon FSx and AWS Backup announce support for copying file system backups across AWS Regions and AWS accounts AWS Batch increases job scheduling and EC2 instance scaling performance Amazon Elasticsearch Service now supports integration with Microsoft Power BI AWS Ground Station now supports data delivery to Amazon S3 Amazon ElastiCache now supports publishing Redis logs to Amazon CloudWatch Logs and Kinesis Data Firehose AI & ML SageMaker Decrease Your Machine Learning Costs with Instance Price Reductions and Savings Plans for Amazon SageMaker | AWS News Blog New options to trigger Amazon SageMaker Pipeline executions ( EventBridge) Other Detect abnormal equipment behavior with Amazon Lookout for Equipment — now generally available Amazon Fraud Detector now supports Batch Fraud Predictions Get estimated run time for forecast creation jobs while using Amazon Forecast Amazon Kendra launches dynamic relevance tuning Other Cool Stuff WorkSpaces Amazon WorkSpaces webcam support now Generally Available Amazon WorkSpaces now supports smart cards with the WorkSpaces macOS client application IVS Amazon Interactive Video Service adds new Cloudwatch Metrics Amazon Interactive Video Service adds support for recording live streams to Amazon S3 Connect Amazon Connect launches audio device settings for the custom Contact Control Panel (CCP) Amazon Connect allows contact center managers to configure agent settings in a custom Contact Control Panel (CCP) Other AWS RoboMaker now supports the ability to configure tools for simulation jobs Amazon AppStream 2.0 adds support for fully managed image updates Amazon Managed Service for Grafana now supports Grafana Enterprise upgrade, Grafana version 7.5, Open Distro for Elasticsearch integration, and AWS Billing reports AWS Cloud9 now supports Amazon Linux 2 environments CloudWatch Metric Streams – Send AWS Metrics to Partners and to Your Apps in Real Time | AWS News Blog Announcing open source robotics projects for AWS DeepRacer Announcing Moving Graphs for CloudWatch Dashboards Amazon Nimble Studio – Build a Creative Studio in the Cloud | AWS News Blog AWS Snow Family now enables you to order, track, and manage long-term pricing Snow jobs The Nanos AWS Console Mobile Application adds support for Asia Pacific (Osaka) region (Arjen) Amazon Connect reduces telephony rates in Cyprus, Belgium, and Portugal (Guy) AWS Cloud9 now supports Amazon Linux 2 environments (Jean-Manuel) Sponsors Gold Sponsor Innablr Silver Sponsors AC3 CMD Solutions DoIT International

About MartinMartin Mao is the co-founder and CEO of Chronosphere. He was previously at Uber, where he led the development and SRE teams that created and operated M3. Prior to that, he was a technical lead on the EC2 team at AWS and has also worked for Microsoft and Google. He and his family are based in our Seattle hub and he enjoys playing soccer and eating meat pies in his spare time.Links: Chronosphere: https://chronosphere.io/ Email: contact@chronosphere.io TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the Cloud: low effort, high visibility and detection. To learn more, visit lacework.com.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I've often talked about observability, or as I tend to think of it when people aren't listening, hipster monitoring. Today, we have a promoted episode from a company called Chronosphere, and I'm joined today by Martin Mao, their CEO and co-founder. Martin, thank you for coming on the show and suffering my slings and arrows.Martin: Thanks for having me on the show, Corey, and looking forward to our conversation today.Corey: So, before we dive into what you're doing now, I'm always a big sucker for origin stories. Historically, you worked at Microsoft and Google, but then you really sort of entered my sphere of things that I find myself having to care about when I'm lying awake at night and the power goes out by working on the EC2 team over at AWS. Tell me a little bit about that. You've hit the big three cloud providers at this point. What was that like?Martin: Yeah, it was an amazing experience, I was a technical lead on one of the EC2 teams, and I think when an opportunity like that comes up on such a core foundational project for the cloud, you take it. So, it was an amazing opportunity to be a part of leading that team at a fairly early stage of AWS and also helping them create a brand new service from scratch, which was AWS Systems Manager, which was targeted at fleet-wide management of EC2 instances, so—Corey: I'm a tremendous fan of Systems Manager, but I'm still looking for the person who named Systems Manager Session Manager because, at this point, I'm about to put a bounty out on them. Wonderful service; terrible name.Martin: That was not me. So, yes. But yeah, no, it was a great experience, for sure, and I think just seeing how AWS operated from the inside was an amazing learning experience for me. And being able to create foundational pieces for the cloud was also an amazing experience. So, only good things to say about my time at AWS.Corey: And then after that, you left and you went to Uber where you led development and SRE teams that created and operated something called M3. Alternately, I'm misreading your bio, and you bought an M3 from BMW and went to drive for Uber. Which is it?Martin: I wish it was the second one, but unfortunately, it is the first one. So yes, I did leave AWS and joined Uber in 2015 to lead a core part of their monitoring and eventually larger observability team. And that team did go on to build open-source projects such as M3—which perhaps we should have thought about the name and the conflict with the car when we named it at the time—and other projects such as Jaeger for distributed tracing as well, and a logging backend system, too. So, yeah, definitely spent many years there building out their observability stack.Corey: We're going to tie a theme together here. You were at Microsoft, you were at Google, you were at AWS, you were at Uber, and you look at all of this and decide, “All right. My entire career has been spent in large companies doing massive globally scaled things. I'm going to go build a small startup.” What made you decide that, all right, this is something I'm going to pursue?Martin: So, definitely never part of the plan. As you mentioned, a lot of big tech companies, and I think I always got a lot of joy building large distributed systems, handling lots of load, and solving problems at a really grand scale. And I think the reason for doing a startup was really the situation that we were in. So, at Uber as I mentioned, myself and my co-founder led the core part of the observability team there, and we were lucky to happen to solve the problem, not just for Uber but for the broader community, especially the community adopting cloud-native architecture. And it just so happened that we were solving the problem of Uber in 2015, but the rest of the industry has similar problems today.So, it was almost the perfect opportunity to solve this now for a broader range of companies out there. And we already had a lot of the core technology built-in open-source as well. So, it was more of an opportunity rather than a long-term plan or anything of that sort, Corey.Corey: So, before we dive into the intricacies of what you've built, I always like to ask people this question because it turns out that the only thing that everyone agrees on is that everyone else is wrong. What is the dividing line, if any, between monitoring and observability?Martin: That's a great question, and I don't know if there's an easy answer.Corey: I mean, my cynical approach is that, “Well, if you call it monitoring, you don't get to bring in SRE-style salaries. Call it observability and no one knows what the hell we're talking about, so sure, it's a blank check at that point.” It's cynical, and probably not entirely correct. So, I'm curious to get your take on it.Martin: Yeah, for sure. So, you know, there's definitely a lot of overlap there, and there's not really two separate things. In my mind at least, monitoring, which has been around for a very long time, has always been around notification and having visibility into your systems. And then as the system's got more complex over time, being able to understand that and not just have visibility into it but understand it a little bit more required, perhaps, additional new data types to go and solve those problems. And that's how, in my mind, monitoring sort of morphed into observability. So, perhaps one is a subset of the other, and they're not competing concepts there. But at least that's my opinion. I'm sure there are plenty out there that would, perhaps, disagree with that.Corey: On some level, it almost hits to the adage of, past a certain point of scale with distributed systems, it's never a question of is the app up or down, it's more a question of how down is it? At least that's how it was explained to me at one point, and it was someone who was incredibly convincing, so I smiled and nodded and never really thought to question it any deeper than that. But I look back at the large-scale environments I've been in, and yeah, things are always on fire, on some level, and ideally, there are ways to handle and mitigate that. Past a certain point, the approach of small-scale systems stops working at large scale. I mean, I see that over in the costing world where people will put tools up on GitHub of, “Hey, I ran this script, and it works super well on my 10 instances.”And then you try and run the thing on 10,000 instances, and the thing melts into the floor, hits rate limits left and right because people don't think in terms of those scales. So, it seems like you're sort of going from the opposite end. Well, this is how we know things work at large scale; let's go ahead and build that out as an initially smaller team. Because I'm going to assume, not knowing much about Chronosphere yet, that it's the sort of thing that will help a company before they get to the hyperscaler stage.Martin: A hundred percent, and you're spot on there, Corey. And it's not even just a company going from small-stage, small-scale simple systems to more complicated ones, actually, if you think about this shift in the cloud right now, it's really going from cloud to cloud-native. So, going from VMs to container on the infrastructure tier, and going from monoliths to microservices. So, it's not even the growth of the company, necessarily, or the growth of the load that the system has to handle, but this shift to containers and microservices heavily accelerates the growth of the amount of data that gets produced, and that is causing a lot of these problems.Corey: So, Uber was famous for disrupting, effectively, the taxi market. What made you folks decide, “I know. We're going to reinvent observability slash monitoring while we're at it, too.” What was it about existing approaches that fell down and, I guess, necessitated you folks to build your own?Martin: Yeah, great question, Corey. And actually, it goes to the first part; we were disrupting the taxi industry, and I think the ability for Uber to iterate extremely fast and respond as a business to changing market conditions was key to that disruption. So, monitoring and observability was a key part of that because you can imagine it was providing all of the real-time visibility to not only what was happening in our infrastructure and applications, but the business as well. So, it really came out of a necessity more than anything else. We found that in order to be more competitive, we had to adopt what is probably today known as cloud-native architecture, adopt running on containers and microservices so that we can move faster, and along with that, we found that all of the existing monitoring tools we were using, weren't really built for this type of environment. And it was that that was the forcing function for us to create our own technologies that were really purpose-built for this modern type of environment that gave us the visibility we needed to, to be competitive as a company and a business.Corey: So, talk to me a little bit more about what observability is. I hear people talking about it in terms of having three pillars; I hear people talking about it, to be frank, in a bunch of ways so that they're trying to, I guess, appropriate the term to cover what they already are doing or selling because changing vocabulary is easier than changing an entire product philosophy. What is it?Martin: Yeah, we actually had a very similar view on observability, and originally we thought that it is a combination of metrics, logs, and traces, and that's a very common view. You have the three pillars, it's almost like three checkboxes; you tick them off, and you have, quote-unquote, “Observability.” And that's actually how we looked at the problem at Uber, and we built solutions for each one of those and we checked all three boxes. What we've come to realize since then is perhaps that was not the best way to look at it because we had all three, but what we realized is that actually just having all three doesn't really help you with the ultimate goal of what you want from this platform, and having more of each of the types of data didn't really help us with that, either. So, taking a step back from there and when we really looked at it, the lesson that we learned in our view on observability is really more from an end-user perspective, rather than a data type or data input perspective.And really, from an end-user perspective, if you think about why you want to use your monitoring tool or your observability tool, you really want to be notified of issues and remediate them as quickly as possible. And to do that, it really just comes down to answering three questions. “Can I get notified when something is wrong? Yes or no? Do I even know something is wrong?”The second question is, “Can I triage it quickly to know what the impact is? Do I know if it's impacting all of my customers or just a subset of them, and how bad is the issue? Can I go back to sleep if I'm being paged at two o'clock in the morning?”And the third one is, “Can I figure out the underlying root cause to the problem and go and actually fix it?” So, this is how we think about the problem now, is from the end-user perspective. And it's not that you don't need metrics, logs, or distributed traces to solve the problem, but we are now orienting our solution around solving the problem for the end-user, as opposed to just orienting our solution around the three data types, per se.Corey: I'm going to self-admit to a fun billing experience I had once with a different monitoring vendor whom I will not name because it turns out, you can tell stories, you can name names, but doing both gets you in trouble. It was a more traditional approach in a simpler time, and they wound up sending me a message saying, “Oh, we're hitting rate limits on CloudWatch. Go ahead and open a ticket asking for them to raise it.” And in a rare display of foresight, AWS respond to my ticket with a, “We can do this, but understand at this level of concurrency, it will cost something like $90,000 a month on increased charges, with that frequency, for that many metrics.” And that was roughly twice what our AWS bill was in those days, and, “Oh.” So, I'm curious as to how you can offer predictable pricing when you can have things that emit so much data so quickly. I believe you when you say you can do it; I'm just trying to understand the philosophy of how that works.Martin: As I said earlier, we started to approach this by trying to solve it in a very engineering fashion where we just wanted to create more efficient backend technology so that it would be cheaper for the increased amount of data. What we realized over time is that no matter how much cheaper we make it, the amount of data being produced, especially from monitoring and observability, kept increasing, and not even in a linear fashion but in an exponential fashion. And because of that, it really switched the problem not to how efficiently can we store this, it really changed our focus of the problem to how our users using this data, and do they even understand the data that's being produced? So, in addition to the couple of properties I mentioned earlier, around cost accounting and rate-limiting—those are definitely required—the other things we try to make available for our end-users is introspection tools such that they understand the type of data that's being produced. It's actually very easy in the monitoring and observability world to write a single line of code that actually produces a lot of data, and most developers don't understand that that single line of code produces so much data.So, our approach to this is to provide a tool so that developers can introspect and understand what is produced on the backend side, not what is being inputted from their code, and then not only have an understanding of that but also dynamic ways to deal with it. So that again, when they hit the rate limit, they don't just have to monitor it less, they understand that, “Oh, I inserted this particular label and now I have 20 times the amount of data that I needed before. Do I really need that particular label in there> and if not, perhaps dropping it dynamically on the server-side is a much better way of dealing with that problem than having to roll back your code and change your metric instrumentation.” So, for us, the way to deal with it is not to just make the backend even more efficient, but really to have end-users understand the data that they're producing, and make decisions on which parts of it is really useful and which parts of it do they, perhaps not want or perhaps want to retain for shorter periods of time, for example, and then allow them to actually implement those changes on that data on the backend. And that is really how the end-users control the bills and the cost themselves.Corey: So, there are a number of different companies in the observability space that have different approaches to what they solve for. In some cases, to be very honest, it seems like, well, I have 15 different observability and monitoring tools. Which ones do you replace? And the answer is, “Oh, we're number 16.” And it's easy to be cynical and down on that entire approach, but then you start digging into it and they're actually right.I didn't expect that to be the case. What was your perspective that made you look around the, let's be honest, fairly crowded landscape of observability companys' tools that gave insight into the health status and well being of various applications in different ways, and say, “You know, no one's quite gotten this right, yet. I have a better idea.”Martin: Yeah, you're completely correct, and perhaps the previous environments that everybody was operating in, there were a lot of different tools for different purposes. A company would purchase an infrastructure monitoring tool, or perhaps even a network monitoring tool, and then they would have, perhaps, an APM solution for the applications, and then perhaps BI tools for the business. So, there was always historically a collection of different tools to go and solve this problem. And I think, again, what has really happened recently with this shift to cloud-native recently is that the need for a lot of this data to be in a single tool has become more important than ever. So, you think about your microservices running on a single container today, if a single container dies in isolation without knowing, perhaps, which microservice was running on it doesn't mean very much, and just having that visibility is not going to be enough, just like if you don't know which business use case that microservice was serving, that's not going to be very useful for you, either.So, with cloud-native architecture, there is more of a need to have all of this data and visibility in a single tool, which hasn't historically happened. And also, none of the existing tools today—so if you think about both the existing APM solutions out there and the existing hosted solutions that exist in the world today, none of them were really built for a cloud-native environment because you can think about even the timing that these companies were created at, you know, back in early 2010s, Kubernetes and containers weren't really a thing. So, a lot of these tools weren't really built for the modern architecture that we see most companies shifting towards. So, the opportunity was really to build something for where we think the industry and everyone's technology stack was going to be as opposed to where the technology stack has been in the past before. And that was really the opportunity there, and it just so happened that we had built a lot of these solutions for a similar type environment for Uber many years before. So, leveraging a lot of our lessons learned there put us in a good spot to build a new solution that we believe is fairly different from everything else that exists today in the market, and it's going to be a good fit for companies moving forward.Corey: So, on your website, one of the things that you, I assume, put up there just to pick a fight—because if there's one thing these people love, it's fighting—is a use case is outgrowing Prometheus. The entire story behind Prometheus is, “Oh, it scales forever. It's what the hyperscalers would use. This came out of the way that Google does things.” And everyone talks about Google as if it's this mythical Valhalla place where everything is amazing and nothing ever goes wrong. I've seen the conference talks. And that's great. What does outgrowing Prometheus look like?Martin: Yeah, that's a great question, Corey. So, if you look at Prometheus—and it is the graduated and the recommended monitoring tool for cloud-native environments—if you look at it and the way it scales, actually, it's a single binary solution, which is great because it's really easy to get started. You deploy a single instance, and you have ingestion, storage, and visibility, and dashboarding, and alerting, all packaged together into one solution, and that's definitely great. And it can scale by itself to a certain point and is definitely the recommended starting point, but as you really start to grow your business, increase your cluster sizes, increase the number of applications you have, actually isn't a great fit for horizontal scale. So, by default, there isn't really a high availability and horizontal scale built into Prometheus by default, and that's why other projects in the CNCF, such as Cortex and Thanos were created to solve some of these problems.So, we looked at the problem in a similar fashion, and when we created M3, the open-source metrics platform that came out of Uber, it was also approaching it from this different perspective where we built it to be horizontally scalable, and highly reliable from the beginning, but yet, we don't really want it to be a, let's say, competing project with Prometheus. So, it is actually something that works in tandem with Prometheus, in the sense that it can ingest Prometheus metrics and you can issue Prometheus query language queries against it, and it will fulfill those. But it is really built for a more scalable environment. And I would say that once a company starts to grow and they run into some of these pain points and these pain points are surrounding how reliable a Prometheus instance is, how you can scale it up beyond just giving it more resources on the VM that it runs on, vertical scale runs out at a certain point. Those are some of the pain points that a lot of companies do run into and need to solve eventually. And there are various solutions out there, both in open-source and in the commercial world, that are designed to solve those pain points. M3 being one of the open-source ones and, of course, Chronosphere being one of the commercial ones.Corey: This episode is sponsored in part by Salesforce. Salesforce invites you to “Salesforce and AWS: Whats Ahead for Architects, Admins and Developers” on June 24th at 10AM, Pacific Time. Its a virtual event where you'll get a first look at the latest innovations of the Salesforce and AWS partnership, and have an opportunity to have your questions answered. Plus you'll get to enjoy an exclusive performance from Grammy Award winning artist The Roots! I think they're talking about a band, not people with super user access to a system. Registration is free at salesforce.com/whatsahead.Corey: Now, you've also gone ahead and more or less dangled raw meat in front of a tiger in some respects here because one of the things that you wind up saying on your site of why people would go with Chronosphere is, “Ah, this doesn't allow for bill spike overages as far as what the Chronosphere bill is.” And that's awesome. I love predictable pricing. It's sort of the antithesis of cloud bills. But there is the counterargument, too, which is with many approaches to monitoring, I don't actually care what my monitoring vendor is going to charge me because they wind up costing me five times more, just in terms of CloudWatch charges. How does your billing work? And how do you avoid causing problems for me on the AWS side, or other cloud provider? I mean, again, GCP and Azure are not immune from this.Martin: So, if you look at the built-in solutions by the cloud providers, a lot of those metrics and monitoring you get from those like CloudWatch or Stackdriver, a lot of it you get included for free with your AWS bill already. It's only if you want additional data and additional retention, do you choose to pay more there. So, I think a lot of companies do use those solutions for the default set of monitoring that they want, especially for the AWS services, but generally, a lot of companies have custom monitoring requirements outside of that in the application tier, or even more detailed monitoring in the infrastructure that is required, especially if you think about Kubernetes.Corey: Oh, yeah. And then I see people using CloudWatch as basically a monitoring, or metric, or log router, which at its price point, don't do that. [laugh]. It doesn't end well for anyone involved.Martin: A hundred percent. So, our solution and our approach is a little bit different. So, it doesn't actually go through CloudWatch or any of these other inbuilt cloud-hosted solutions as a router because, to your point, there's a lot of cost there as well. It actually goes and collects the data from the infrastructure tier or the applications. And what we have found is that not only does the bill for monitoring climb exponentially—and not just as you grow; especially as you shift towards cloud-native architecture—our very first take of solving that problem is to make the backend a lot more efficient than before so it just is cheaper overall.And we approached it that way at Uber, and we had great results there. So, when we created an—originally before M3, 8% of Uber's infrastructure bill was spent on monitoring all the infrastructure and the application. And by the time we were done with M3, the cost was a little over 1%. So, the very first solution was just make it more efficient. And that worked for a while, but what we saw is that over time, this grew again.And there wasn't any more efficiency, we could crank out of the backend storage system. There's only so much optimization you can do to the compression algorithms in the backend and how much you can get there. So, what we realized the problem shifted towards was not, can we store this data more efficiently because we're already reaching limitations there, and what we noticed was more towards getting the users of this data—so individual developers themselves—to start to understand what data is being produced, how they're using it, whether it's even useful, and then taking control from that perspective. And this is not a problem isolated to the SRE team or the observability team anymore; if you think about modern DevOps practices, every developer needs to take control of monitoring their own applications. So, this responsibility is really in the hands of the developers.And the way we approached this from a Chronosphere perspective is really in four steps. The first one is that we have cost accounting so that every developer, and every team, and the central observability team know how much data is being produced. Because it's actually a hard thing to measure, especially in the monitoring world. It's—Corey: Oh, yeah. Even AWS bills get this wrong. Like if you're sending data between one availability zone to another in the same region, it charges a penny to leave an AZ and a penny to enter an AZ in that scenario. And the way that they reflect this on the bill is they double it. So, if you're sending one gigabyte across AZ link in a month, you'll see two gigabytes on the bill and that's how it's reflected. And that is just a glimpse of the monstrosity that is the AWS billing system. But yeah, exposing that to folks so they can understand how much data their application is spitting off? Forget it. That never happens.Martin: Right. Right. And it's not even exposing it to the company as a whole, it's to each use case, to each developer so they know how much data they are producing themselves. They know how much of the bill is being consumed. And then the second step in that is to put up bumper lanes to that so that once you hit the limit, you don't just get a surprise bill at the end of the month.When each developer hits that limit, they rate-limit themselves and they only impact their own data; there is no impact to the other developers or to the other teams, or to the rest of the company. So, we found that those two were necessary initial steps, and then there were additional steps beyond that, to help deal with this problem.Corey: So, in order for this to work within a multi-day lag, in some cases, it's a near certainty that you're looking at what is happening and the expense that is being incurred in real-time, not waiting for it to pass its way through the AWS billing system and then do some tag attribution back.Martin: A hundred percent. It's in real-time for the stream of data. And as I mentioned earlier, for the monitoring data we are collecting, it goes straight from the customer environment to our backend so we're not waiting for it to be routed through the cloud providers because, rightly so, there is a multi-day or multi-hour delay there. So, as the data is coming straight to our backend, we are actively in real-time measuring that and cost accounting it to each individual team. And in real-time, if the usage goes above what is allocated, will actually limit that particular team or that particular developer, and prevent them by default from using more. And with that mechanism, you can imagine that's how the bill is controlled and controlled in real-time.Corey: So, help me understand, on some level; is your architecture then agent-based? Is it a library that gets included in the application code itself? All of the above and more? Something else entirely? Or is this just such a ridiculous question that you can't believe that no one has ever asked it before?Martin: No, it's a great question, Corey, and would love to give some more insight there. So, it is an agent that runs in the customer environment because it does need to be something there that goes and collects all the data we're interested in to send it to the backend. This agent is unlike a lot of APM agents out there where it does, sort of, introspection, things like that. We really believe in the power of the open-source community, and in particular, open-source standards like the Prometheus format for metrics. So, what this agent does is it actually goes and discovers Prometheus endpoints exposed by the infrastructure and applications, and scrapes those endpoints to collect the monitoring data to send to the backend.And that is the only piece of software that runs in our customer environments. And then from that point on, all of the data is in our backend, and that's where we go and process it and get visibility into the end-users as well as store it and make it available for alerting and dashboarding purposes as well.Corey: So, when did you found Chronosphere? I know that you folks recently raised a Series B—congratulations on that, by the way; that generally means, at least if I understand the VC world correctly, that you've established product-market fit and now we're talking about let's scale this thing. My experience in startup land was, “Oh, we've raised a Series B, that means it's probably time to bring in the first DevOps hire.” And that was invariably me, and I wound up screaming and freaking out for three months, and then things were better. So, that was my exposure to Series B.But it seems like, given what you do, you probably had a few SRE folks kicking around, even on the product team because everything you're saying so far absolutely resonates with the experiences someone who has run these large-scale things in production. No big surprise there. Is that where you are? I mean, how long have you been around?Martin: Yeah, so we've been around for a couple of years thus far—so still a relatively new company, for sure. A lot of the core team were the team that both built the underlying technology and also ran it in production the many years at Uber, and that team is now here at Chronosphere. So, you can imagine from the very beginning, we had DevOps and SREs running this hosted platform for us. And it's the folks that actually built the technology and ran it for years running it again, outside of Uber now. And then to your first question, yes, we did establish fairly early on, and I think that is also because we could leverage a lot of the technology that we had built at Uber, and it sort of gave us a boost to have a product ready for the market much faster.And what we're seeing in the industry right now is the adoption of cloud-native is so fast that it's sort of accelerating a need of a new monitoring solution that historical solutions, perhaps, cannot handle a lot of the use cases there. It's a new architecture, it's a new technology stack, and we have the solution purpose-built for that particular stack. So, we are seeing fairly fast acceleration and adoption of our product right now.Corey: One problem that an awful lot of monitoring slash observability companies have gotten into in the last few years—at least it feels this way, and maybe I'm wildly incorrect—is that it seems that the target market is the Ubers of the world, the hyperscalers where once you're at that scale, then you need a tool like this, but if you're just building a standard three-tier web app, oh, you're nowhere near that level of scale. And the problem with go-to-market in those stories inherently seems that by the time you are a hyperscalers, you have already built a somewhat significant observability apparatus, otherwise you would not have survived or stayed up long enough to become a hyperscalers. How do you find that the on-ramp looks? I mean, your website does talk about, “When you outgrow Prometheus.” Is there a certain point of scale that customers should be at before they start looking at things like Chronosphere?Martin: I think if you think about the companies that are born in the cloud today and how quickly they are running and they are iterating their technology stack, monitoring is so critical to that. It's the real-time visibility of these changes that are going out multiple times a day is critical to the success and growth of a lot of new companies. And because of how critical that piece is, we're finding that you don't have to be a giant hyperscalers like Uber to need technology like this. And as you rightly pointed out, you need technology like this as you scale up. And what we're finding is that while a lot of large tech companies can invest a lot of resources into hiring these teams and building out custom software themselves, generally, it's not a great investment on their behalf because those are not companies that are selling monitoring technology as their core business.So generally, what we find is that it is better for companies to perhaps outsource or purchase, or at least use open-source solutions to solve some of these problems rather than custom-build in-house. And we're finding that earlier and earlier on in a company's lifecycle, they're needing technology like this.Corey: Part of the problem I always ran into was—again, I come from the old world of grumpy Unix sysadmins—for me, using Nagios was my approach to monitoring. And that's great when you have a persistent stateful, single node or a couple of single nodes. And then you outgrow it because well, now everything's ephemeral and by the time you realize that there's an outage or an issue with a container, the container hasn't existed for 20 minutes. And you better have good telemetry into what's going on and how your application behaves, especially at scale because at that point, edge cases, one-in-a-million events happen multiple times a second, depending upon scale, and that's a different way of thinking. I've been somewhat fortunate in that, in my experience at least, I've not usually had to go through those transformative leaps.I've worked with Prometheus, I've worked with Nagios, but never in the same shop. That's the joy of being a consultant. You go into one environment, you see what they're doing and you take notes on what works and what doesn't, you move on to the next one. And it's clear that there's a definite defined benefit to approaching observability in a more modern way. But I despair the idea of trying to go from one to the other. And maybe that just speaks to a lack of vision for me.Martin: No, I don't think that's the case at all, Corey. I think we are seeing a lot of companies do this transition. I don't think a lot of companies go and ditch everything that they've done. And things that they put years of investment into, there's definitely a gradual migration process here. And what we're seeing is that a lot of the newer projects, newer environments, newer efforts that have been kicked off are being monitored and observed using modern technology like Prometheus.And then there's also a lot of legacy systems which are still going to be around and legacy processes which are still going to be around for a very long time. It's actually something we had to deal with that at Uber as well; we were actually using Nagios and a StatsD Graphite stack for a very long time before switching over to a more modern tag-like system like Prometheus. So—Corey: Oh, modern Nagios. What was it, uh… that's right, Icinga. That's what it was.Martin: Yes, yes. It was actually the system that we were using Uber. And I think for us, it's not just about ditching all of that investment; it's really about supporting this migration as well. And this is why both in the open-source technology M3, we actually support both the more legacy data types, like StatsD and the Graphite query language, as well as the more modern types like Prometheus and PromQL. And having support for both allows for a migration and a transition.And not even a complete transition; I'm sure there will always be StatsD, Graphite data in a lot of these companies because they're just legacy applications that nobody owns or touches anymore, and they're just going to be lying around for a long time. So, it's actually something that we proactively get ahead of and ensure that we can support both use cases even though we see a lot of companies and trending towards the modern technology solutions, for sure.Corey: The last point I want to raise has always been a personal, I guess, area of focus for me. I allude to it, sometimes; I've done a Twitter thread or two on it, but on your website, you say something that completely resonates with my entire philosophy, and to be blunt is why in many cases, I'm down on an awful lot of vendor tooling across a wide variety of disciplines. On the open-source page on your site, near the bottom, you say, and I quote, “We want our end-users to build transferable skills that are not vendor or product-specific.” And I don't think I've ever seen a vendor come out and say something like that. Where did that come from?Martin: Yeah. If you look at the core of the company, it is built on top of open-source technology. So, it is a very open core company here at Chronosphere, and we really believe in the power of the open-source community and in particular, perhaps not even individual projects, but industry standards and open standards. So, this is why we don't have a proprietary protocol, or proprietary agent, or proprietary query language in our product because we truly believe in allowing our end-users to build these transferable skills and industry-standard skills. And right now that is using Prometheus as the client library for monitoring and PromQL as the query language.And I think it's not just a transferable skill that you can bring with you across multiple companies, it is also the power of that broader community. So, you can imagine now that there is a lot more sharing of, “Hey, I am monitoring, for example, MongoDB. How should I best do that?” Those skills can be shared because the common language that they're all speaking, the queries that everybody is sharing with each other, the dashboards everybody is sharing with each other, are all, sort of, open-source standards now. And we really believe in the power that and we really do everything we can to promote that. And that is why in our product, there isn't any proprietary query language, or definitions of dashboarding, or [learning 00:35:39] or anything like that. So yeah, it is definitely just a core tenant of the company, I would say.Corey: It's really something that I think is admirable, I've known too many people who wind up, I guess, stuck in various environments where the thing that they work on is an internal application to the company, and nothing else like it exists anywhere else, so if they ever want to change jobs, they effectively have a black hole on their resume for a number of years. This speaks directly to the opposite. It seems like it's not built on a lock-in story; it's built around actually solving problems. And I'm a little ashamed to say how refreshing that is [laugh] just based upon what that says about our industry.Martin: Yeah, Corey. And I think what we're seeing is actually the power of these open-source standards, let's say. Prometheus is actually having effects on the broader industry, which I think is great for everybody. So, while a company like Chronosphere is supporting these from day one, you see how pervasive the Prometheus protocol and the query language are that actually all of these probably more traditional vendors providing proprietary protocols and proprietary query languages all actually have to have Prometheus—or not ‘have to have,' but we're seeing that more and more of them are having Prometheus compatibility as well. And I think that just speaks to the power of the industry, and it really benefits all of the end-users and the industry as a whole, as opposed to the vendors, which we are really happy to be supporters of.Corey: Thank you so much for taking the time to speak with me today. If people want to learn more about what you're up to, how you're thinking about these things, where can they find you? And I'm going to go out on a limb and assume you're also hiring.Martin: We're definitely hiring right now. And you can find us on our website at chronosphere.io or feel free to shoot me an email directly. My email is martin@chronosphere.io. Definitely massively hiring right now, and also, if you do have problems trying to monitor your cloud-native environment, please come check out our website and our product.Corey: And we will, of course, include links to that in the [show notes 00:37:41]. Thank you so much for taking the time to speak with me today. I really appreciate it.Martin: Thanks a lot for having me, Corey. I really enjoyed this.Corey: Martin Mao, CEO and co-founder of Chronosphere. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment speculating about how long it took to convince Martin not to name the company ‘Observability Manager Chronosphere Manager.'Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

This week on The Cloud Pod, Justin is away so the rest of the team has taken the opportunity to throw him under the bus.     A big thanks to this week's sponsors: Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. This week's highlights The Pentagon has had enough of the kids fighting so no one gets the toy. Amazon has given developers the happy ending they've always wanted. Google is playing with fire and hopes no one gets burnt. JEDI: Play Nice Pentagon officials are considering pulling the plug on the star-crossed JEDI cloud-computing project. Reminds us of when we were kids and our parents took toys away when we couldn't play nice together. Amazon Web Services: We've Made All the Money AWS announces a price reduction for Amazon Managed Service for Prometheus. That's an awful lot of samples.   Amazon Virtual Private Cloud (VPC) announces pricing change for VPC Peering. Just get rid of the ridiculous data transfer fees!    AWS Organizations launches a new console experience. We're excited to try this out! AWS announces IAM Access Control for Apache Kafka on Amazon MSK. This is great.  AWS Systems Manager now includes Incident Manager to resolve IT incidents faster. This might initially fall short of some of the other offerings on the market.  AWS Local Zones are now open in Boston, Miami and Houston. They're continuing on the Oracle model of racks in random garages.  Amazon now lets you create Microsoft SQL Server Instances of Amazon RDS on AWS Outposts. A big hooray for people using Outposts.  Google Cloud Platform: Smells A Bit Google announces Agent Assist for Chat is now in Preview. Hopefully this is better than predictive text, which is often highly inappropriate. Google releases a handy new Google Cloud, AWS and Azure product map. This press release has an Oracle smell about it.  Browse and query Google Cloud Spanner databases from Visual Studio Code. We can see this being welcomed by developers.   Azure: So Pretty Azure releases a new logo. We think it kind of looks like a Google icon.  Multiple new features for Azure VPN Gateway are now generally available. Really great features!  Enabling Azure Site Recovery while creating Azure Virtual Machines is now generally available. Something about this feels clunky. The next installment of the low code development series is now available. Spoiler alert: it's not that riveting.    TCP Lightning Round Ryan blatantly stole Justin's jokes but still takes this week's point, leaving scores at Justin (7), Ryan (4), Jonathan (7).  Other headlines mentioned: Amazon QuickSight Launches Threshold Alerts Amazon DevOps Guru now generally available with additional capabilities Amazon Pinpoint Announces Journey Pause and Resume Azure Backup: Operational backup for Azure Blobs is now generally available Append blob support in Azure Data Lake Storage is now generally available Amazon SageMaker Automatic Model Tuning now supports up to 10x faster tuning and enables exploring up to 20X more models Amazon CloudWatch Synthetics supports cron expression for scheduling Amazon CloudFront announces price cuts in India and Asia Pacific regions Amazon Elasticsearch Service now offers AWS Graviton2 (M6g, C6g, R6g, and R6gd) instances3 Amazon Athena drivers now support Azure AD and PingFederate authentication Migration Evaluator announces a faster way to project AWS cloud costs with Quick Insights Amazon EKS managed node groups adds support for Kubernetes node taints  Things Coming Up Announcing Google Cloud 2021 Summits [frequently updated] Save the date: AWS Containers events in May AWS Regional Summits — May 10–19 Microsoft Build — May 19–21 (Digital) Google Financial Services Summit — May 27th  Harness Unscripted Conference — June 16–17 Google Cloud Next — Not announced yet (one site says Moscone is reserved June 28–30) Google Cloud Next 2021 — October 12–14, 2021 AWS re:Invent — November 29–December 3 — Las Vegas Oracle Open World (no details yet) 

Good surprise? Your friends jumping out with cake on your birthday. Bad surprise? Your application is down. Tune in to listen to Nicki Stone, Sr. Software Engineer chat with Dave Cliffe, Sr. Product Manager and Oren Nachman, Sr. Development Manager about AWS Systems Manager's newest capability: Incident Manager. Incident Manager helps you prepare for incidents with automated response plans that bring the right people and information together, so you can get back to that surprise birthday party. Learn more: https://aws.amazon.com/systems-manager/features/#Incident_Manager Read the blog: https://aws.amazon.com/blogs/aws/resolve-it-incidents-faster-with-incident-manager-a-new-capability-of-aws-systems-manager/

Un épisode sur deux du podcast est consacré à une brève revue des principales nouveautés AWS. Cette semaine, nous parlons de Amazon FinSpace, un service pour faciliter la vie des analystes financiers, d'un nouveau volet ajouté à AWS Systems Manager, de la possibilité de déployer des minis fonctions sur les edges de Amazon CloudFront, d'un nouveau SDK en version Alpha pour les développeurs Rust et d'un cours en ligne et gratuit sur Amazon DynamoDB.

Un épisode sur deux du podcast est consacré à une brève revue des principales nouveautés AWS. Cette semaine, nous parlons de Amazon FinSpace, un service pour faciliter la vie des analystes financiers, d'un nouveau volet ajouté à AWS Systems Manager, de la possibilité de déployer des minis fonctions sur les edges de Amazon CloudFront, d'un nouveau SDK en version Alpha pour les développeurs Rust et d'un cours en ligne et gratuit sur Amazon DynamoDB.

About BradAuthor and Senior Executive Editor, Bloomberg TechnologyBrad Stone is the author of four books, including Amazon Unbound: Jeff Bezos and the Invention of a Global Empire,published by Simon & Schuster in May 2021. It traces the transformation of Amazon into one of the largest and most feared companies of the world and the accompanying emergence of Jeff Bezos as the richest man alive. Brad is also the author of The Everything Store: Jeff Bezos and the Age of Amazon, which chronicled the foundational early years of the company and its founder. The book, a New York Times and Wall Street Journal bestseller, was translated into more than 35 languages and won the 2013 Financial Times/Goldman Sachs Business Book of the Year Award. In 2017, he also published The Upstarts: Uber, Airbnb, and the Battle for the New Silicon Valley.Brad is Senior Executive Editor for Global Technology at Bloomberg Newswhere he oversees a team of 65 reporters and editors that covers high-tech companies, startups, cyber security and internet trends around the world. Over the last ten years, as a writer for Bloomberg Businessweek, he’s authored over two dozen cover stories on companies such as Apple, Google, Amazon, Softbank, Twitter, Facebook and the Chinese internet juggernauts Didi, Tencent and Baidu. He’s a regular contributor to Bloomberg’s technology newsletter Fully Charged, and to the daily Bloomberg TV news program, Bloomberg Technology. He was previously a San Francisco-based correspondent for The New York Times and Newsweek. A graduate of Columbia University, he is originallyfrom Cleveland, Ohio and lives in the San Francisco Bay Area with his wifeand three daughtersLinks: The Everything Store: https://www.amazon.com/Everything-Store-Jeff-Bezos-Amazon/dp/0316219282/ Amazon Unbound: https://www.amazon.com/Amazon-Unbound-Invention-Global-Empire/dp/1982132612/ Andy Jassy book review: https://www.amazon.com/gp/customer-reviews/R1Q4CQQV1ALSN0/ref=cm_cr_getr_d_rvw_ttl?ie=UTF8&ASIN=B00FJFJOLC TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It’s an awesome approach. I’ve used something similar for years. Check them out. But wait, there’s more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It’s awesome. If you don’t do something like this, you’re likely to find out that you’ve gotten breached, the hard way. Take a look at this. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That’s canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I’m a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored in part by VMware. Because let’s face it, the past year hasn’t been kind to our AWS bills, or honestly any cloud bills. The pandemic had a bunch of impacts: it forced us to move workloads to the cloud sooner than we would have otherwise, we saw strange patterns such as user traffic drops off but infrastructure spend doesn’t. What do you do about it? Well, the CloudLIVE 2021 virtual conference is your chance to connect with people wrestling with the same type of thing, be they practitioners, vendors in the space, leaders of thought—ahem, ahem—and get some behind the scenes look into various ways different companies are handling this. Hosted by CloudHealth by VMware on May 20, the CloudLIVE 2021 conference will be 100% virtual and 100% free to attend, so you really have no excuses for missing out on this opportunity to deal with people who care about cloud bills. Visit cloudlive.com/coreyto learn more and save your virtual seat today. That’s cloud L-I-V-E slash Corey. C-O-R-E-Y. Drop the E, we’re all in trouble. My thanks to VMware for sponsoring this ridiculous episode.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. Sometimes people tell me that I should write a book about Amazon. And that sounds awful. But to be sure, today, my guest is Brad Stone, someone who has written not one, but two books about Amazon, one of which coming out on May 11th, or as most of you will know while listening to this, today. Brad, thanks for joining me.Brad: Corey, thanks for having me.Corey: So, what on earth would inspire you to not just write a book about one of what is in many ways an incredibly secretive company, but then to go back and do it again?Brad: Yeah. I’m a glutton for punishment. And Corey, my hair right now is completely white way before it should be, and I think that Amazon might be responsible for some of that. So, as you contemplate your own project, consider that this company—will you already know: it can age you. They are sometimes resistant to scrutiny.So, to answer your question, I set out to write The Everything Store back in 2011, and this was a much smaller company. It was a cute little tiny internet company of about $100 billion in market value. And poor, impoverished Jeff Bezos maybe had, I’d be guessing maybe $50 billion.So anyway, it was a much different time. And that was a great experience. The company was kind of flowering as the book came out. And to my surprise, it was embraced not by Bezos or the management team, who maybe we’ll talk about didn’t love it, but by Amazon employees, and customers, and competitors, and prospective employees. And I was really proud of it that this had become a kind of definitive account of the early years of the company.And then a funny thing happened. The little cute little internet company became a juggernaut, a $1.5 trillion market cap. Bezos is the wealthiest guy in the world now with a $200 billion fortune, and Alexa, and the rise of AWS, and the Go store, and incursions into India and Mexico and other countries, I mean, so much had changed, and my definitive history felt a little out of date. And so back in 2017—also a different world, Bezos is a happily married man; he’s the CEO of Amazon, Amazon’s headquarters are in Seattle only—I set out to research and write Amazon Unbound. And as I was writing the story, yeah, just, like, the ground kept shifting under my feet.Corey: Not a lot changes in the big sphere. I mean, one of the things that Bezos said is, “Oh, what’s going to be different in 10 years? I think the better question is, ‘what’s going to not be different in 10 years?’” but watching the company shift, watching it grow, just from the outside has been a real wild ride, I’ve got to say. And I restrict myself primarily to the AWS parts because well, there’s too much to cover if you go far beyond that, and two, it’s a very different place with very different challenges around it.I viewed The Everything Store when it came out and I read that, almost like it was a biography of Jeff Bezos himself. And in some respects, Amazon Unbound feels like it hews in that direction as well, but it also goes beyond that. How do you approach separating out the story of Amazon from the story of Jeff Bezos?Brad: Yeah, you’re putting your finger on almost the core challenge, and the adjoining challenge, which is how do you create a narrative, a linear story? Often readers want a chronological story out of a miasma of overlapping events, and initiatives, and challenges. Amazon’s really decentralized; everything is happening at once. Bezos is close to some things, he was very close to Alexa. He is really distant from other things.Andy Jassy for years had a lot of independence to run AWS. So, how do you tell that story, and then keep Bezos in the center? I mean, Andy Jassy and Jeff Wilke and everyone, I mean, those are great business people. Not necessarily dynamic personalities as, Corey, you know well, but people want to read about Jeff Bezos. He is a larger-than-life figure.He’s a pioneer. He’s an innovator. He’s controversial. And so the challenge all along is to, kind of, keep him in the center. And so that’s just, like, a writing challenge. It’s a narrative challenge.And the lucky thing is that Amazon does tend to orbit around Jeff Bezos’s brain. And so in all the storytelling, even the AWS bits of the book, which we can talk about, as an author, you can always bring Bezos back just by following the facts. You’ll eventually get, in the evolution of any story, to an S Team meeting, or to an acquisition discussion where Jeff had an impact, said something insightful, walked out of a meeting, raise the bar, had impossibly high standards. So, the last thing I’ll say is, because Amazon’s so decentralized, when you write these books you have to talk to a lot of people. And then you get all the pieces of the puzzle, and you start to assemble them, and the challenge as a writer is to, kind of, keep Bezos, your main character in the lens at all times, never let him drift too far out.Corey: One of the things that I learned from it was just the way that Bezos apparently talks to his senior executives, as far as, “I will invest in this project, more than you might think I would.” I guess I’ve never really heard of a budget meeting talking about, “I”—in the first person—“Will invest.” Like, that is what happens, but for some reason the business books never put it quite that starkly or frame it quite that way. But in hindsight, it made a lot of things of my own understanding of Amazon fall into place. That makes sense.Brad: He’s got a lot of levers, ways in which he’ll back a new initiative or express his support. And one of them is simply how he spends his time. So, with Alexa in the early years, he would meet once or twice a week with that team. But another lever is just the amount of investment. And oftentimes teams will come to him—the India team is a great example—they’ll come to the S Team with a budget, and they’ll list out their priorities and their goals for the coming year, and he’ll say, “You know, you’re thinking about this all wrong. Don’t constrain yourself. Tell us what the goals are, tell us what the opportunity is, then we’ll figure out how much it costs.”And his mindset is like you can kind of break up opportunity into two categories: one are the land grabs, the big immediate opportunities where he will go all out, and India was a great example of that, I think the failed fire phone was another example, Prime Video, he doesn’t cap the investment, he wants to win. And then there are the more greenfield opportunities that he thinks he can go slower on and groceries for a long time was in that category. And there the budgets might be more constrained. The other example is the much older businesses, just like the retail business. That’s 20 years old—I have a chapter about that—and the advertising business, and he recognized that the retail business wasn’t profitable and it was depending on advertising as a crutch, and he blew it up because he thinks that those older divisions shouldn’t require investment; they should be able to stand on their own.Corey: One quote you had as well, that just really resonated with me, as far as basically my entire ethos of how I make fun of Amazon is—and I’m going to read the excerpt here. My apologies. You have to listen to your own words being read back toward you—Brad: [laugh].Corey: These were typically Amazonian names: geeky, obscure, and endlessly debated inside AWS since—according to an early AWS exec—Bezos had once mused, “You know, the name is about 3% of what matters, but sometimes 3% is the difference between winning and losing.” And I just want to call that out because I don’t think I’ve ever seen an AWS exec ever admit that names might be even 3% worth of important. Looking at how terrible some of their service names are, I would say that 3% might be an aspirational target for their worldview.Brad: [laugh]. Let me throw this back at you, Corey. Have you ever figured out why certain AWS services are Amazon and why others are AWS?Corey: I did. I got to sit down—in the before times—with then the VP of Global Marketing, Ariel Kelman—who’s now Oracle’s Chief Marketing Officer—and Jeff Barr. And the direction that they took that in was that if you could use an AWS service without getting into the AWS weeds of a bunch of other services, then it was called Amazon whatever. Amazon S3, for example, as a primitive service doesn’t need a bunch of other AWS services hooked into it, so that gets the Amazon moniker. Whereas if you’re dealing with a service that requires the integration of a whole bunch of AWS in the weeds stuff—Brad: Mmm, right.Corey: —then it’s AWS. For example, AWS Systems Manager is useless without a whole bunch of other Amazon services. And they say they don’t get it perfectly right all the time, but that is the direction that it’s gone in. And for better or worse, I still have to look a lot of them up myself because I don’t care nearly as much as their branding people do.Brad: Right. Well, I’ll tell you in the chapter about AWS, that quote comes up when the team is contemplating the names of the databases. And they do go into long debates, and I remember talking to Charlie Bell about the search for Redshift, and they go back and forth on it, and the funny thing about that one was, of course, Oracle interpreted it as a competitive slight. Its corporate color, I guess, being red, which he intended it more as a physics term. But yeah, when they were launching Aurora and Redshift, they contemplated those names quite a bit. And I don’t know if it’s 3%. I don’t know if it does matter, but certainly, those services have become really important to a lot of businesses.Corey: Oh, yeah. And once you name something, it’s really hard to rename it. And AWS does view for—better or worse—APIs as a promise, so when you build something and presented a certain way, they’re never going to turn it off. Our grandkids are going to have to deal with some of these decisions once they get into computers. That’s a problem.And I understand the ethos behind it, but again, it’s easy to make fun of names; it’s an accessible thing because let’s be very real here, a lot of what AWS does is incredibly inaccessible to people who don’t live in this space. But naming is something that everyone can get behind making fun of.Brad: Absolutely. Yep. And [laugh] it’s perhaps why they spend a lot of time on it because they know that this is going to be the shingle that they hang out to the world. I don’t know that they’re anticipating your ridicule, but it’s obviously key to the marketing process for them.Corey: Some of the more aware ones do. But that’s a different topic for a different time. One question I have for you that I wrestle with myself is I’ve been spending the last four years or so basically studying AWS all the time. And there’s a lot of things they get right; there’s a lot of things that they get wrong. But for better or worse, it’s very difficult not to come away from an in-depth study with an appreciation for an awful lot of the things that they do. At least for me.I’m not saying that I fall in love with the company and will excuse them their wrongs; I absolutely do not do that. But it is hard, bordering on impossible for me, to not come away with a deep respect for a lot of the things that they do and clearly believe. How do you feel about that? Looking at Amazon, do you come away with this with, “Ooh. Remind me to never to become a Prime member and get rid of everything with an Amazon logo in my house,” versus the you’re about to wind up wondering if they can hire you for some esoteric role? Where do you fall on that spectrum?Brad: I think I’m probably with you. I come away with an admiration. And look, I mean, let me say upfront, I am a Prime member. I have a Alexas in my home, probably more than my wife and kids are comfortable with. We watch Prime Video, we have Prime Video.We order from Amazon all the time, we ordered from Whole Foods. I’m an Amazon customer, and so part of my appreciation comes from, like all other customers, the fact that Amazon uniquely restores time to our lives rather than extracts it. I wouldn’t say that about the social networks, right? You know, those can be time-wasters. Amazon’s a great efficiency machine.But in terms of my journalism, you know, now two books and this big in-depth study in Amazon Unbound, and you have to admire what they have built. I mean, a historic American institution that has not only changed our economic reality, in ways good and bad but over the last year and a half, in the pandemic was among the few institutions that functioned properly and served as a kind of lifeline. And there is a critique in Amazon Unbound and we can talk about it, but it’s hard to come away—I think you said it well—it’s hard to come away after studying this company and studying the top executives, and how Jeff Bezos, thinks and how he has conceived products without real admiration for what they have built over the last 25 years.Corey: Well, let’s get into your critique of Amazon. What do you think is, from what you’ve seen with all of the years of research you put into this company, what’s the worst thing about them?Brad: Well, that’s a good way to put it, Corey. [laugh]. Let me—Corey: [laugh]. It’s like, talk about a target-rich opportunity. Like, “Oh, wow. It’s like my children. I can’t stand any of them. How in the world could I pick just one?” But give it a shot.Brad: Right. Well, let me start this way, which is I often will listen to their critiques from Amazon critics—and I’m sure you might feel this way as well—and just think, like, “Do they get it?” They’ll argue that Amazon exercised its size and might to buy the companies that led to Alexa. As I write in the Alexa chapter, that’s not true at all. They bought a couple of small companies, and those executives were all horrified at what Amazon was trying to do, and then they made it work.Or the critics will say, “Fifty percent or more of internet users start their product searches on Amazon. Amazon has lock-in.” That’s not true either. Lock-in on the internet is only as strong as a browser window that remains open. And you could always go find a competitor or search on a search engine.So, I find at least some of the public criticism to be a little specious. And often, these are people that complained about Walmart for ten years. And now Amazon’s the big, bad boogeyman.Corey: Oh, I still know people who refuse to do business with Walmart but buy a bunch of stuff from Amazon, and I’m looking at these things going, any complaints you have about Walmart are very difficult to avoid mapping to Amazon.Brad: Here’s maybe the distillation of the critique that’s an Amazon Unbound. We make fun of Facebook for, “Move fast and break things.” And they broke things, including, potentially, our democracy. When you look at the creation of the Amazon Marketplace, Jeff wanted a leader who can answer the question, “How would you bring a million sellers into the Amazon Marketplace?” And what that tells you is he wanted to create a system, a self-service system, where you could funnel sellers the world over into the system and sell immediately.And that happened, and a lot of those sellers, there was no friction, and many of them came from the Wild West of Chinese eCommerce. And you had—inevitably because there were no guardrails—you had fraud and counterfeit, and all sorts of lawsuits and damage. Amazon moved fast and broke things. And then subsequently tried to clean it up. And if you look at the emergence of the Amazon supply chain and the logistics division, the vans that now crawl our streets, or the semi-trailers on our highways, or the planes.Amazon moved fast there, too. And the first innings of that game were all about hiring contractors, not employees, getting them on the road with a minimum of guidance. And people died. There were accidents. You know, there weren’t just drivers flinging packages into our front yards, or going to the bathroom on somebody’s porch.That happened, but there were also accidents and costs. And so I think some of the critique is that Amazon, despite its profession that it focuses only on customers, is also very competitor-aware and competitor-driven, and they move fast, often to kind of get ahead of competitors, and they build the systems and they’re often self-service systems, and they avoid employment where it’s possible, and the result have been costs to society, the cost of moving quickly. And then on the back-end when there are lawsuits, Amazon attempts to either evade responsibility or settle cases, and then hide those from the public. And I think that is at the heart of what I show in a couple of ways in Amazon Unbound. And it’s not just Amazon; it’s very typical right now of corporate America and particularly tech companies.And part of it is the state of the laws and regulations that allow the companies to get away with it, and really restrict the rights of plaintiffs, of people who are wronged from extracting significant penalties from these companies and really changing their behavior.Corey: Which makes perfect sense. I have the luxury of not having to think about that by having a mental division and hopefully one day a real division between AWS and Amazon’s retail arm. For me at least, the thing I always had an issue with was their treatment of staff in many respects. It is well known that in the FAANG constellation of tech companies, Facebook, Amazon, Apple, Netflix, and Google, apparently, it’s an acronym and it’s cutesy. People in tech think they’re funny.But the problem is that Amazon’s compensation is significantly below that. One thing I loved in your book was that you do a breakdown of how those base salaries work, how most of it is stock-based and with a back-loaded vesting and the rest, and looking through the somewhat lengthy excerpt—but I will not read your own words to you this time—it more or less completely confirms what I said in my exposé of this, which means if we’re wrong, we’re both wrong. And we’ve—and people have been very convincing and very unified across the board. We’re clearly not wrong. It’s nice to at least get external confirmation of some of the things that I stumble over.Brad: But I think this is all part of the same thing. What I described as the move fast and break things mentality, often in a race with competition, and your issues about the quality, the tenor of work, and the compensation schemes, I think maybe and this might have been a more elegant answer to your question, we can wrap it all up under the mantle of empathy. And I think it probably starts with the founder and soon-to-be-former CEO. And look, I mean, an epic business figure, a builder, an inventor, but when you lay out the hierarchy of qualities, and attributes, and strengths, maybe empathy with the plight of others wasn’t near the top. And when it comes to the treatment of the workforce, and the white-collar employees, and the compensation schemes, and how they’re very specifically designed to make people uncomfortable, to keep them running fast, to churn them out if they don’t cut it, and the same thing in the workforce, and then the big-scale systems and marketplace and logistics—look, maybe empathy is a drag, and not having it can be a business accelerant, and I think that’s what we’re talking about, right?That some of these systems seem a little inhumane, and maybe to their credit, when Amazon recognizes that—or when Jeff has recognized it00, he’s course-corrected a little bit. But I think it’s all part of that same bundle. And maybe perversely, it’s one of the reasons why Amazon has succeeded so much.Corey: I think that it’s hard to argue against the idea of culture flowing from the top. And every anecdote I’ve ever heard about Jeff Bezos, never having met the man myself, is always filtered through someone else; in many cases, you. But there are a lot of anecdotes from folks inside Amazon, folks outside Amazon, et cetera, and I think that no one could make a serious argument that he is not fearsomely intelligent, penetratingly insightful, and profoundly gifted in a whole bunch of different ways. People like to say, “Well, he started Amazon with several $100,000 and loan from his parents, so he’s not really in any ways a self-made anything.” Well, no one is self-made. Let’s be very clear on that.But getting a few $100,000 to invest in a business, especially these days, is not that high of a stumbling block for an awful lot of folks similarly situated. He has had outsized success based upon where he started and where he wound up ending now. But not a single story that I’ve ever heard about him makes me think, yeah, that’s the kind of guy I want to be friends with. That’s the kind of guy I want to invite to a backyard barbecue and hang out with, and trade stories about our respective kids, and just basically have a social conversation with. Even a business conversation doesn’t feel like it would be particularly warm or compelling.It would be educational, don’t get me wrong, but he doesn’t strike me as someone who really understands empathy in any meaningful sense. I’m sure he has those aspects to him. I’m sure he has a warm, wonderful relationship with his kids, presumably because they still speak to him, but none of that ever leaks through into his public or corporate persona.Brad: Mmm, partially agree, partially disagree. I mean, certainly maybe the warmth you’re right on, but this is someone who’s incredibly charismatic, who is incredibly smart, who thinks really deeply about the future, and has intense personal opinions about current events. And getting a beer with him—which I have not done—with sound fantastic. Kicking back at the fireplace at his ranch in Texas, [laugh] to me, I’m sure it’s tremendously entertaining to talk to him. But when it comes to folks like us, Corey, I have a feeling it’s not going to happen, whether you want to or not.He’s also incredibly guarded around the jackals of the media, so perhaps it doesn’t make a difference one way or another. But, yeah, you’re right. I mean, he’s all business at work. And it is interesting that the turnover in the executive ranks, even among the veterans right now, is pretty high. And I don’t know, I mean, I think Amazon goes through people in a way, maybe a little less on the AWS side. You would know that better than me. But—Corey: Yes and no. There’s been some turnover there that you can also pretty easily write down to internal political drama—for lack of a better term—palace intrigue. For lack of a better term. When, for example, Adam Selipsky is going to be the new CEO of AWS as Andy Jesse ascends to be the CEO of all Amazon—the everything CEO as it were. And that has absolutely got to have rubbed some people in unpleasant ways.Let’s be realistic here about what this shows: he quit AWS to go be the CEO of Tableau, and now he’s coming back to run AWS. Clearly, the way to get ahead there is to quit. And that might not be the message they’re intending to send, but that’s something that people can look at and take away, that leaving a company doesn’t mean you can’t boomerang and go back there at a higher level in the future.Brad: Right.Corey: And that might be what people are waking up to because it used to be a culture of once you’re out, you’re out. Clearly not the case anymore. They were passed over for a promotion they wanted, “Well, okay, I’m going to go talk to another company. Oh, my God, they’re paying people in yachts.” And it becomes, at some level, time for something new.I don’t begrudge people who decide to stay; I don’t begrudge people who decide to leave, but one of my big thrusts for a long time has been understand the trade-offs of either one of those decisions and what the other side looks like so you go into it with your eyes open. And I feel like, on some level, a lot of folks there didn’t necessarily feel that they could have their eyes open in the way that they can now.Brad: Mm-hm. Interesting. Yeah. Selipsky coming back, I never thought about that, sends a strong message. And Amazon wants builders, and operators, and entrepreneurial thinking at the top and in the S Team. And the fact that Andy had a experienced leadership team at AWS and then went outside it for the CEO could be interpreted as pretty demotivating for that team. Now, they’ve all worked with Adam before, and I’ve met him and he seems like a great guy so maybe there are no hard feelings, but—Corey: I never have. He left a few months before I started this place. So, it—I get the sense that he knew I was coming and said, “Well, better get out of here. This isn’t going to go well at all.”Brad: [laugh]. I actually went to interview him for this book, and I sat in his office at Tableau thinking, “Okay, here’s a former AWS guy,” and I got to tell you, he was really on script and didn’t say anything bad, and I thought, “Okay, well, that wasn’t the best use of my time.” He was great to meet, and it was an interesting conversation, but the goss he did not deliver. And so when I saw that he got this job, I thought, well, he’s smart. He smartly didn’t burn any bridges, at least with me.Corey: This episode is sponsored in part by our friends at ChaosSearch., you could run Elasticsearch or Elastic Cloud—or OpenSearch as they’re calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you’ve come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you’re using Elasticsearch, consider not running Elasticsearch. They’re also available now in the AWS marketplace if you’d prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like HubSpot, Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: No. And it’s pretty clear that you don’t get to rise to those levels without being incredibly disciplined with respect to message. I don’t pity Andy Jesse’s new job wherein a key portion of the job description is going to be testifying before Congress. Without going into details, I’ve been in situations where I’ve gotten to ask him questions before in a real-time Q&A environment, and my real question hidden behind the question was, “How long can I knock him off of his prepared talking points?” Because I—Brad: Good luck. [laugh].Corey: Yeah. I got the answer: about two and a half seconds, which honestly was a little bit longer than I thought I would get. But yeah, incredibly disciplined and incredibly insightful, penetrating answers, but they always go right back to talking points. And that’s what you have to do at that level. I’ve heard stories—it may have been from your book—that Andy and Adam were both still friendly after Adam’s departure, they would still hang out socially and clearly, relationships are still being maintained, if oh, by the way, you’re going to be my successor. It’s kind of neat. I’m curious to see how this plays out once that transition goes into effect.Brad: Yeah, it’ll be interesting. And then also, Andy’s grand homecoming to the other parts of the business. He started in the retail organization. He was Jeff’s shadow. He ran the marketing department at very early Amazon.He’s been in all those meetings over the years, but he’s also been very focused on AWS. So, I would imagine there’s a learning curve as he gets back into the details of the other 75% of Amazon.Corey: It turns out that part of the business has likely changed in the last 15 years, just a smidgen when every person you knew over there is now 10,000 people. There was an anecdote in your book that early on in those days, Andy Jesse was almost let go as part of a layoff or a restructuring, and Jeff Bezos personally saved his job. How solid is that?Brad: Oh, that is solid. An S Team member told me that, who was Andy’s boss at the time. And the story was, in the late 90s—I hope I remember this right—there was a purge of the marketing department. Jeff always thought that marketing—in the early days marketing was purely satisfying customers, so why do we need all these people? And there was a purge of the marketing department back when Amazon was trying to right-size the ship and get profitable and survive the dotcom bust.And Jeff intervened in the layoffs and said, “Not Andy. He’s one of the most—yeah, highest ceiling folks we have.” And he made him his first full-time shadow. Oh, and that comes right from an S Team member. I won’t say the name because I can’t remember if that was on or off the record.But yeah, it was super interesting. You know what? I’ve always wondered how good of a identifier of talent and character is Bezos. And he has some weaknesses there. I mean, obviously, in his personal life, he certainly didn’t identify Lauren Sánchez’s brother as the threat that he became.You know, I tell the story in the book of the horrific story of the CEO of Amazon Mexico, who Jeff interviewed, and they hired and then later ended up what appears to be hiring an assassin to kill his wife. I tell the story in the book. It’s a horrible story. So, not to lay that at the feet of Jeff Bezos, of course, but he often I think, moves quickly. And I actually have a quote from a friend of his in the book saying, “It’s better to not be kind of paranoid, and the”—sort of—I can’t remember what the quote is.It’s to trust people rather than be paranoid about everyone. And if you trust someone wrongly, then you of course-correct. With Andy, though, he somehow had an intuitive sense that this guy was very high potential, and that’s pretty impressive.Corey: You’re never going to bet a thousand. There’s always going to be people that slip through the cracks. But learning who these people are and getting different angles on them is always interesting. Every once in a while—and maybe I’m completely wrong on this, but never having spent time one on one with Andy Jassy, I have to rely on other folks and different anecdotes, most of them, I can’t disclose the source of, but every time that I wind up hearing about these stories, and maybe I’m projecting here, but there are aspects of him where it seems like there is a genuinely nice person in there who is worried, on some level, that people are going to find out that he’s a nice person.Brad: [laugh]. I think he is. He’s extraordinarily nice. He seems like a regular guy, and what’s sort of impressive is that obviously he’s extraordinarily wealthy now, and unlike, let’s say Bezos, who’s obviously much more wealthy, but who, who really has leaned into that lifestyle, my sense is Andy does not. He’s still—I don’t know if he’s on the corporate jet yet, but at least until recently he wasn’t, and he presents humbly. I don’t know if he’s still getting as close from wherever, [unintelligible 00:32:50] or Nordstroms.Corey: He might be, but it is clear that he’s having them tailored because fit is something—I spent a lot of time in better years focusing on sartorial attention, and wherever he’s sourcing them from aside, they fit well.Brad: Okay, well, they didn’t always. Right?Corey: No. He’s, he’s… there’s been a lot of changes over the past decade. He is either discovered a hidden wellspring of being one of the best naturally talented speakers on the planet, or he’s gone through some coaching to improve in those areas. Not that he was bad at the start, but now he’s compelling.Brad: Okay. Well, now we’re talking about his clothes and his speaking style. But—Corey: Let’s be very honest here. If he were a woman, we would have been talking about that as the beginning topic of this. It’s on some level—Brad: Or we wouldn’t have because we’d know it’s improper these days.Corey: We would like to hope. But I am absolutely willing to turn it back around.Brad: [laugh]. Anyway.Corey: So, I’m curious, going back a little bit to criticisms here, Amazon has been criticized roundly by regulators and Congress and the rest—folks on both sides of the aisle—for a variety of things. What do you see is being the fair criticisms versus the unfair criticisms?Brad: Well, I mean, I think we covered some of the unfair ones. But there’s one criticism that Amazon uses AWS to subsidize other parts of the business. I don’t know how you feel about that, but until recently at least, my reading of the balance sheet was that the enormous profits of AWS were primarily going to buy more AWS. They were investing in capital assets and building more data centers.Corey: Via a series of capital leases because cash flow is king in how they drive those things there. Oh, yeah.Brad: Right. Yeah. You know, and I illustrate in the book how when it did become apparent that retail was leaning on advertising, Jeff didn’t accept that. He wanted retail to stand on its own, and it led to some layoffs and fiercer negotiations with brands, higher fees for sellers. Advertising is the free cash flow that goes in Prime movies, and TV shows, and Alexa, and stuff we probably don’t know about.So, this idea that Amazon is sort of improperly funneling money between the divisions to undercut competitors on price, I think we could put that in the unfair bucket. In the fair bucket, those are the things that we can all look at and just go, “Okay, that feels a little wrong.” So, for an example, the private brand strategy. Now, of course, every supermarket and drugstore is going to line their shelves with store brands. But when you go to an Amazon search results page these days, and they are pockmarked with Amazon brands, and Whole Foods brands, and then sponsored listings, the pay-to-play highest bidder wins.And then we now know that, at least for a couple of years, Amazon managers, private label managers were kind of peeking at the third-party data to figure out what was selling and what they should introduce is a private Amazon brand. It just feels a little creepy that Amazon as the everything store is so different than your normal Costco or your drugstore. The shelves are endless; Amazon has the data, access to the data, and the way that they’re parlaying their valuable real estate and the data at their disposal to figure out what to launch, it just feels a little wrong. And it’s a small part of their business, but I think it’s one where they’re vulnerable. The other thing is, in the book, I tried to figure out how can I take the gauge of third-party sellers?There’s so many disgruntled voices, but do they really speak for everyone? And so instead of going to the enemies, I went to every third-party seller that had been mentioned in Jeff Bezos’s shareholder letters over the past decade. And these were the allies. These were the success stories that Bezos was touting in his sacrosanct investor letter, and almost to a one, they had all become disgruntled. And so the way in which the rules of the marketplace change, the way that the fees go up, and the difficulty that sellers often have in getting a person or a guiding hand at Amazon to help them with those changes, that kind of feels wrong.And I think that maybe that’s not a source of regulation, but it could be a source of disruptive competition. If somebody can figure out how to create a marketplace that caters to sellers a little better with lower fees, then they could do to Amazon with Amazon years ago did to eBay. And considering that Marketplace is now a preponderance of sales more than even retail on amazon.com, that can end up hurting the company.Corey: Yeah, at some point, you need to continue growing things, and you’ve run out of genuinely helpful ways, and in turn in start to have to modify customer behavior in order to continue doing things, or expand into brand new markets. We saw the AWS bleeding over into Alexa as an example of that. And I think there’s a lot of interesting things still to come in spaces like that. It’s interesting watching how the Alexa ecosystem has evolved. There’s still some very basic usability bugs that drive me nuts, but at the same token, it’s not something that I think we’re going to see radically changing the world the next five years. It feels like a hobby, but also a lucrative one, and keeps people continuing to feed into the Amazon ecosystem. Do you see that playing out differently?Brad: Wait, with Alexa?Brad: Absolutely.Brad: Yeah. I agree with you. I mean, it feels like there was more promise in the early years, and that maybe they’ve hit a little bit of a wall in terms of the AI and the natural language understanding. It feels like the ecosystem that they tried to build, the app store-like ecosystem of third-party skills makers, that hasn’t crystallized in the way we hoped—in the way they hoped. And then some of these new devices like the glasses or the wristband that have Alexa feel, just, strange, right?Like, I’m not putting Alexa on my face. And those haven’t done as well. And so yeah, I think they pioneered a category: Alexa plays music and answers basic queries really well, and yet it hasn’t quite been conversational in the way that I think Jeff Bezos had hoped in the early days. I don’t know if it’s a profitable business now. I mean, they make a lot of money on the hardware, but the team is huge.I think it was, like, 10,000 people the last I checked. And the R&D costs are quite large. And they’re continuing to try to improve the AI, so I think Jeff Bezos talks about the seeds, and then the main businesses, and I don’t think Alexa has graduated yet. I think there’s still a little bit of a question mark.Corey: It’s one of those things that we remain to see. One last thing that I wanted to highlight and thank you for, was that when you wrote the original book, The Everything Store, Andy Jassy wrote a one-star review. It went into some depth about all the things that, from his perspective, you got wrong, were unfair about, et cetera.And that can be played off as a lot of different things, but you can almost set that aside for a minute and look at it as the really only time in recent memory that Andy Jassy has sat down and written something, clearly himself, and then posted it publicly. He writes a lot—Amazon has a writing culture—but they don’t sign their six-pagers. It’s very difficult to figure out where one person starts and one person stops. This shows that he is a gifted writer in many respects, and I don’t think we have another writing sample from him to compare it to.Brad: So, Corey, you’re saying I should be honored by his one-star review of The Everything Store?Corey: Oh, absolutely.Brad: [laugh].Corey: He, he just ignores me. You actually got a response.Brad: I got a response. Well.Corey: And we’ll put a link to that review in the [show notes 00:40:10] because of course we will.Brad: Yes, thank you. Do you—remember, other Amazon executives also left one-star reviews. And Jeff’s wife, and now ex-wife Mackenzie left a one-star review. And it was a part of a, I think a little bit of a reflexive reaction and campaign that Jeff himself orchestrated at my—this was understanding now, in retrospect. After the book came out, he didn’t like it.He didn’t like aspects of his family life that were represented in the book, and he asked members of the S Team to leave bad reviews. And not all of them did, and Andy did. So, you wonder why he’s CEO now. No, I’m kidding about that. But you know what?It ended up, kind of perversely, even though that was uncomfortable in the moment, ended up being good for the first book. And I’ve seen Andy subsequently, and no hard feelings. I don’t quite remember what his review said. Didn’t it, strangely, like, quote a movie or something like that?Corey: I recall that it did. It went in a bunch of different directions, and at the end—it ended with, “Well, maybe someday he’ll write the actual story. And I’m not trying to bait anyone into doing it, but this book isn’t it.” Well, in the absence of factual corrections, that’s what we go with. That is also a very Amazonian thing. They don’t tell their own story, but they’re super quick to correct the record—Brad: Yeah.Corey: —after someone says a thing.Brad: But I don’t recall him making many specific claims of anything I got wrong. But why don’t we hope that there’s a sequel review for Amazon Unbound? I will look forward to that from Andy.Corey: I absolutely hope so. It’s one of those things that we just really, I guess, hope goes in a positive direction. Now, I will say I don’t try to do any reviews that are all positive. And that’s true. There’s one thing that you wrote that I vehemently disagree with.Brad: Okay, let’s hear it.Corey: Former Distinguished Engineer and VP at AWS, Tim Bray, who resigned on conscientious objector grounds, more or less, has been a guest on the show, and I have to say, you did him dirty. You described him—Brad: How did I—what did I do? Mm-hm.Corey: Oh, I quote, “Bray, a fedora-wearing software developer”—which is true, but still is evocative in an unpleasant way—“And one of the creators of the influential web programming language, XML”—which is true, but talk about bringing up someone’s demons to haunt them. Oh, my starts.Brad: [laugh]. But wait. How is the fedora-wearing pejorative?Corey: Oh, it has a whole implication series of, and entire subculture of the gamer types, people who are misogynist, et cetera. It winds up being an unfair characterization—Brad: But he does wear a fedora.Corey: He does. And he can pull it off. He has also mentioned that he is well into retirement age, and it was a different era when he wore one. But that’s not something that people often will associate with him. It’s—Brad: I’m so naive. You’re referring to things that I do not understand what the implication was that I made. But—Corey: Oh, spend more time with the children of Reddit. You’ll catch on quickly.Brad: [laugh]. I try, I try not to do that. But thank you, Corey.Corey: Of course. So, thank you so much for taking the time to go through what you’ve written. I’m looking forward to seeing the reaction once the book is published widely. Where can people buy it? There’s an easy answer, of course, of Amazon itself, but is there somewhere you would prefer them to shop?Brad: Well, everyone can make their own decisions. I flattered if anyone decides to pick up the book. But of course, there is always their independent bookstore. On sale now.Corey: Excellent. And we will, of course, throw a link to the book in the [show notes 00:43:31]. Brad, thank you so much for taking the time to speak with me. I really appreciate it.Brad: Corey, it’s been a pleasure. Thank you.Corey: Brad Stone, author of Amazon Unbound: Jeff Bezos and the Invention of a Global Empire, on sale now wherever fine books are sold—and crappy ones, too. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice and then a multi-paragraph, very long screed telling me exactly what I got wrong.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

最新情報を "ながら" でキャッチアップ！ ラジオ感覚放送 「毎日AWS」 おはようございます、水曜日担当の福島です。 今日は 4/6 に出たアップデートをピックアップしてご紹介 感想は Twitter にて「#サバワ」をつけて投稿してください！ ■ トークスクリプト https://blog.serverworks.co.jp/everyday-aws-175 ■ UPDATE PICKUP  AWS Systems ManagerのRun Commandログの利便性が向上  Amazon WorkSpacesでWebカメラの利用が一般提供開始   Amazon WorkSpacesは、macOS用のWorkSpacesクライアントでスマートカードをサポート   Amazon SageMaker PipelinesがAmazon EventBridgeと統合   Amazon MQが大阪リージョンで利用可能に   Amazon Macieが大阪リージョンで利用可能に ■ サーバーワークスSNS Twitter / Facebook ■ サーバーワークスブログ サーバーワークスエンジニアブログ

Der offizielle deutschsprachige Podcast rund um Amazon Web Services (AWS), für Neugierige, Cloud-Einsteiger und AWS-Experten, produziert von Dennis Traub, Developer Advocate bei AWS. Bei Fragen, Anregungen und Feedback wendet euch gerne direkt an Dennis auf Twitter (@dtraub) oder per Mail an traubd@amazon.com. In dieser Episode spricht Dennis über die Sicherheitsrisiken des traditionellen SSH-Zugriffs auf Instanzen in der Cloud und stellt den AWS Systems Manager Session Manager als sichere und einfache Alternative vor. Links zum Thema: - AWS Systems Manager: Gain operational insights and take action on AWS resources - https://aws.amazon.com/systems-manager/ - User Guide: AWS Systems Manager Session Manager - https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html Für mehr Infos, Tipps und Tricks rund um AWS und die Cloud folgt Dennis auf: - Twitter - https://twitter.com/dtraub  - Twitch - https://www.twitch.tv/dennis_at_work  - YouTube - https://www.youtube.com/dennistraub

In another slightly delayed episode Arjen, JM, and Guy talk about all the many things that were announced in October. But before that, they will first discuss exactly how badly Lex understands "a fair shake of the sauce bottle".   Talk to us in our Slack or on Twitter!   The News   Finally in Sydney Amazon Connect supports Amazon Lex bots using the Australian English dialect Amazon EC2 G4dn Bare Metal Instances with NVIDIA T4 Tensor Core GPUs, now available in 15 additional regions AWS IoT SiteWise is now available in Asia Pacific (Singapore) and Asia Pacific (Sydney) AWS regions Amazon Relational Database Service (RDS) Snapshot Export to S3 available in additional regions Serverless Introducing AWS Lambda Extensions – In preview | AWS Compute Blog Announcing Amazon CloudWatch Lambda Insights (preview) New – Use AWS PrivateLink to Access AWS Lambda Over Private AWS Network | AWS News Blog Amazon EventBridge announces support for Dead Letter Queues AWS Step Functions now supports Amazon Athena service integration Amazon API Gateway now supports disabling the default REST API endpoint Containers Amazon EKS now supports Kubernetes version 1.18 Amazon EKS now supports the Los Angeles AWS Local Zones Amazon EKS now supports configurable Kubernetes service IP address range Amazon ECS extensions for AWS Cloud Development Kit now available as a Developer Preview AWS Elastic Beanstalk Adds Support for Running Multi-Container Applications on AL2 based Docker Platform Fluent Bit supports Amazon S3 as a destination to route container logs AWS App Mesh supports cross account sharing of ACM Private Certificate Authority Introducing the AWS Load Balancer Controller AWS Copilot CLI launches v0.5 to let users deploy scheduled jobs and more EC2 & VPC AWS Nitro Enclaves – Isolated EC2 Environments to Process Confidential Data | AWS News Blog Announcing SSL/TLS certificates for Amazon EC2 instances with AWS Certificate Manager (ACM) for Nitro Enclaves New – Application Load Balancer Support for End-to-End HTTP/2 and gRPC | AWS News Blog AWS Compute Optimizer enhances EC2 instance type recommendations with Amazon EBS metrics AWS Cloud Map simplifies service discovery with optional parameters AWS Global Accelerator launches port overrides AWS IoT SiteWise launches support for VPC private links AWS Site-to-Site VPN now supports health notifications Dev & Ops AWS CloudFormation now supports increased limits on five service quotas AWS CloudFormation Guard – an open-source CLI for infrastructure compliance – is now generally available AWS CloudFormation Drift Detection now supports CloudFormation Registry resource types Amazon CloudWatch Synthetics now supports prebuilt canary monitoring dashboard Amazon CloudWatch Synthetics launches Recorder to generate user flow scripts for canaries AWS and Grafana Labs launch AWS X-Ray data source plugin Now author AWS Systems Manager Automation runbooks using Visual Studio Code AWS Systems Manager now supports free-text search of runbooks AWS Systems Manager now allows filtering automation executions by applications or environments Now use AWS Systems Manager to view vulnerability identifiers for missing patches on your Linux instances Port forwarding sessions created using Session Manager now support multiple simultaneous connections Now customize your Session Manager shell environment with configurable shell profiles AWS End of Support Migration Program for Windows Server now available as a self-serve solution for customers EC2 Image Builder now supports AMI distribution across AWS accounts Announcing general availability of waiters in the AWS SDK for Java 2.x Porting Assistant for .NET is now open source Amazon Corretto 8u272, 11.0.9, 15.0.1 quarterly updates are now available Security AWS Config adds 15 new sample conformance pack templates and introduces simplified setup experience for conformance packs AWS IAM Access Analyzer now supports archive rules for existing findings AWS AppSync adds support for AWS WAF AWS Shield now provides global and per-account event summaries to all AWS customers Amazon CloudWatch Logs now supports two subscription filters per log group Amazon S3 Object Ownership is available to enable bucket owners to automatically assume ownership of objects uploaded to their buckets Protect Your AWS Compute Optimizer Recommendation Data with customer master keys (CMKs) Stored in AWS Key Management Service Manage access to AWS centrally for Ping Identity users with AWS Single Sign-On Amazon Elasticsearch Service adds native SAML Authentication for Kibana Amazon Inspector has expanded operating system support for Red Hat Enterprise Linux (RHEL) 8, Ubuntu 20.04 LTS, Debian 10, and Windows Server 2019 Data Storage & Processing New – Amazon RDS on Graviton2 Processors | AWS News Blog Amazon ElastiCache now supports M6g and R6g Graviton2-based instances Easily restore an Amazon RDS for MySQL database from your MySQL 8.0 backup Amazon RDS for PostgreSQL supports concurrent major version upgrades of read replicas Amazon Aurora enables dynamic resizing for database storage space AWS Lake Formation now supports Active Directory and SAML providers for Amazon Athena AWS Lake Formation now supports cross account database sharing Now generally available – design and visualize Amazon Keyspaces data models more easily by using NoSQL Workbench You now can manage access to Amazon Keyspaces by using temporary security credentials for the Python, Go, and Node.js Cassandra drivers Amazon ElastiCache on Outposts is now available Amazon EMR now supports placing your EMR master nodes in distinct racks to reduce risk of simultaneous failure Amazon EMR integration with AWS Lake Formation is now generally available Amazon EMR now provides up to 35% lower cost and up to 15% improved performance for Spark workloads on Graviton2-based instances AWS Glue Streaming ETL jobs support schema detection and evolution AWS Glue supports reading from self-managed Apache Kafka AWS Glue crawlers now support Amazon DocumentDB (with MongoDB compatibility) and MongoDB collections Amazon Kinesis Data Analytics now supports Force Stop and a new Autoscaling status Kinesis Client Library now enables multi-stream processing Announcing cross-database queries for Amazon Redshift (preview) Amazon Redshift announces support for Lambda UDFs and enables tokenization New Amazon Neptune engine release now enforces a minimum version of TLS 1.2 and SSL client connections AWS Database Migration Service now supports Amazon DocumentDB (with MongoDB compatibility) as a source AI & ML Amazon SageMaker Autopilot now Creates Machine Learning Models up to 40% Faster with up to 200% Higher Accuracy Now launch Amazon SageMaker Studio in your Amazon Virtual Private Cloud (VPC) Amazon SageMaker Price Reductions – Up to 18% for ml.P3 and ml.P2 instances Amazon SageMaker Studio Notebooks now support custom images Amazon Rekognition adds support for six new content moderation categories Amazon Rekognition now detects Personal Protective Equipment (PPE) such as face covers, head covers, and hand covers on persons in images Amazon Transcribe announces support for AWS PrivateLink for Batch APIs Amazon Kendra now supports custom data sources Amazon Kendra adds Confluence Server connector Amazon Textract announces improvements to reduce average API processing times by up to 20% Other Cool Stuff AWS DeepRacer announces new Community Races updates Amazon WorkSpaces introduces sharing images across accounts AWS Batch now supports Custom Logging Configurations, Swap Space, and Shared Memory Amazon Connect supports Amazon Lex bots using the British English dialect Amazon Connect chat now provides automation and personalization capabilities with whisper flows CloudWatch Application Insights offers new, improved user interface CloudWatch Application Insights adds EBS volume and API Gateway metrics Announcing AWS Budgets price reduction Announcing AWS Budgets Actions Resource Access Manager Support is now available on AWS Outposts Announcing Amazon CloudFront Origin Shield Announcing AWS Distro for OpenTelemetry in Preview Introducing Amazon SNS FIFO – First-In-First-Out Pub/Sub Messaging | AWS News Blog Amazon SNS now supports selecting the origination number when sending SMS messages Amazon SES now offers list and subscription management capabilities Nano candidates Amazon WorkDocs now supports Dark Mode on iOS Amazon Corretto 8u272, 11.0.9, 15.0.1 quarterly updates are now available AWS OpsWorks for Configuration Management now supports new version of Chef Automate Sponsors   Gold Sponsor Innablr   Silver Sponsors AC3 CMD Solutions DoIT International  

最新情報を "ながら" でキャッチアップ！ ラジオ感覚放送 「毎日AWS！」 おはようございます、サーバーワークスの加藤です。 今日は 11/5 に出たアップデート11件をご紹介。 感想は Twitter にて「#サバワ」をつけて投稿してください！ ■ UPDATE ラインナップ AWS Fargate がコンテナの構成とメトリクスに関する複数の改善を発表 Amazon Elasticsearch Service がカスタムドメインに対応 Savings Plans アラートが利用可能に Amazon Kendra が Confluence Cloud に対応 Amazon EMR が AWS Service Quotas に対応 AWS Client VPN が Client Connect Handler 機能をサポート Amazon CloudWatchアラームが AWS Systems Manager OpsCenter アクションをサポート Amazon Redshift の JDBC および Python ドライバーがオープンソースに AWS Systems Manager 高速セットアップがターゲットとしてリソースグループをサポート AWS Lambda がイベントソースとして Amazon MQ をサポート 組み込み C 言語用 AWS IoT SDK の新しいバージョンが登場 ■ サーバーワークスSNS Twitter / Facebook ■ サーバーワークスブログ サーバーワークスエンジニアブログ

最新情報を "ながら" でキャッチアップ！ ラジオ感覚放送 「毎日AWS！」 おはようございます、サーバーワークスの加藤です。 今日は 10/21 に出たアップデート9件をご紹介。 感想は Twitter にて「#サバワ」をつけて投稿してください！ ■ UPDATE ラインナップ AWS Systems Manager のセッションマネージャー機能でシェル環境をカスタマイズできるように Amplify プロジェクトから既存の Cognito ユーザープール・IDプールを利用できるように Amazon EC2 Hibernation 機能がI3, M5ad, R5ad に対応 AWS Global Accelerator がポートオーバーライドをサポート Amazon Connect が新しいメトリクス、エージェント接続時間に対応 CloudWatch Application Insights が EBS ボリュームと API Gateway のメトリクスを追加 Amazon MSK が Apache Kafka バージョン 2.6.0 をサポート AWS Distro for OpenTelemetry をプレビューとして発表 Amazon QLDB Go Driver が一般利用可能に ■ サーバーワークスSNS Twitter / Facebook ■ サーバーワークスブログ サーバーワークスエンジニアブログ

最新情報を "ながら" でキャッチアップ！ ラジオ感覚放送 「毎日AWS！」 おはようございます、サーバーワークスの加藤です。 今日は 10/16 に出たアップデート7件をご紹介。 感想は Twitter にて「#サバワ」をつけて投稿してください！ ■ UPDATE ラインナップ Amazon Rekognition が個人用防護具を検出できるように AWS Systems Manager がランブックの全文検索に対応 AWS Systems Manager パッチマネージャが Amazon Linux のすべてのパッチのカタログを提供 Resource Access Manager が AWS Outposts をサポート Amazon EMR が Graviton2ベースの M6g インスタンスタイプに対応 Amazon QuickSight がダッシュボード上にフィルターコントローラーを設置できるように AWS Launch Wizard が AWS Backint Agent を用いた SAP HANA のバックアップをサポート ■ サーバーワークスSNS Twitter / Facebook ■ サーバーワークスブログ サーバーワークスエンジニアブログ

A bit later than planned, but Arjen, Jean-Manuel, and Guy are back to talk about the AWS news from September 2020. This episode contains Arjen talking about what's wrong with the SSO APIs, Jean-Manuel showing off his Quantum computing knowledge, and Guy giving a sauce bottle a fair shake? The News   Finally in ANZ Amazon Lex launches support for Australian English Urban Dictionary: Fair shake of the sauce bottle Amazon RDS M6g and R6g instances powered by AWS Graviton2 processors are now available in Asia Pacific regions Amazon RDS M6g and R6g instance types, powered by AWS Graviton2 processors: In preview and now supported on more database versions Amazon CloudFront launches in two new countries - Mexico and New Zealand Serverless AWS Step Functions increases payload size to 256KB API Gateway HTTP APIs now supports Lambda and IAM authorization options AWS Step Functions adds support for AWS X-Ray AWS Lambda adds console support for visualizing AWS Step Functions workflows Amazon API Gateway now supports mutual TLS authentication mTLS auth with AWS API Gateway | by Koustubha Kale | Contino Engineering Mutual TLS auth with AWS API Gateway Part 2 - check certificate revocation | by Koustubha Kale | Contino Engineering Amazon EventBridge Schema Registry announces support for JSON Schema Containers Announcing the General Availability of Bottlerocket, a new open source Linux-based operating system purpose-built to run containers EKS Now Supports Creation and Management of Fargate Profiles Using AWS CloudFormation Amazon EKS now supports assigning EC2 security groups to Kubernetes pods Amazon CloudWatch now monitors Prometheus metrics from Container environments AWS and Docker extend collaboration to launch new features in Docker Desktop Docker Open Sources Compose for Amazon ECS and Microsoft ACI - Docker Blog Amazon ECS is now available in the Los Angeles AWS Local Zones EC2 & VPC New EC2 T4g Instances – Burstable Performance Powered by AWS Graviton2 – Try Them for Free | AWS News Blog Amazon EC2/Spot Fleet now support modifying instance types and weights on the run Announcing AWS PrivateLink support for Amazon Textract Amazon CodeGuru Profiler now supports AWS PrivateLink Amazon Lightsail now offers new OS blueprints Application Load Balancers now support AWS Outposts AWS Elastic Beanstalk now supports sharing of an Application Load Balancer among Elastic Beanstalk environments Amazon CloudWatch Agent is now Open Source and included with Amazon Linux 2 Dev & Ops AWS Systems Manager now supports all current versions of Ubuntu AWS X-Ray launches Auto-Instrumentation Agent for Java AWS X-Ray launches anomaly detection-based actionable insights in preview Amazon CloudWatch Synthetics strengthens end-to-end canary run debugging with X-Ray traces Systems Manager now supports on-demand patching with just two clicks Amazon CloudWatch Synthetics now supports enhanced monitoring for Broken Link and GUI Workflow Blueprints Amazon CloudFront announces support for Brotli compression AWS Systems Manager Explorer now supports grouping and customization of operational data sources Announcing event logging and self-upgrade capabilities in SSM Agent, with new version 3.0 Announcing the General Availability of Amazon Corretto 15 Security AWS Single Sign-On adds account assignment APIs and AWS CloudFormation support to automate multi-account access management Fixing AWS SSO's CloudFormation | ig.nore.me GitHub - ArjenSchwarz/awstools: A little application to help with more complex AWS functions cloudformation-macros/SSOFixer at master · ArjenSchwarz/cloudformation-macros · GitHub Now available AWS SSO credential profile support in the AWS Toolkit for JetBrains IDEs Amazon CloudFront announces real-time logs Amazon CloudFront announces support for TLSv1.3 for viewer connections Amazon CloudWatch Dashboards now supports sharing AWS Backup Will Automatically Copy Tags from Nested EBS Volumes to EC2 Recovery Points Enforce encryption for Amazon Elastic File System resources using AWS IAM Amazon Detective introduces IAM Role Session Analysis Data Lifecycle Manager now supports multiple schedules within in a single lifecycle policy AWS Backup supports application-consistent backups of Microsoft workloads on EC2 Introducing AWS Cost Anomaly Detection (Preview) Storage & Databases Announcing Data API for Amazon Redshift Amazon Redshift now supports 100K tables in a single cluster Amazon RDS for SQL Server Now Supports Native Backup/Restore on DB Instances with Read Replicas Amazon Aurora Increases Maximum Storage Size to 128TB Amazon Elasticsearch Service now offers T3 Instances Amazon ElastiCache is now available in the AWS Local Zones in Los Angeles (LA) Now it's even easier to connect JetBrains IDEs to Amazon RDS or Redshift Databases Amazon EFS integrates with AWS Systems Manager to simplify management of Amazon EFS clients AI & ML Amazon Textract supports customer S3 buckets Other Cool Stuff AWS announces a 86%+ price reduction for AWS IoT Events Amazon WorkSpaces introduces Microsoft Office Professional bundle for Bring Your Own Windows License WorkSpaces Amazon WorkSpaces introduces support for cross-Region redirection Amazon Connect launches contact flow management APIs Amazon Connect launches APIs that list prompts within your instance Amazon Connect launches API to configure routing profiles programmatically AWS CloudFormation now supports StackSets Resource Type in the CloudFormation Registry Introducing AWS Perspective Announcing new AWS Wavelength Zones in Atlanta, New York City, and Washington DC Queuing purchases of Savings Plans Amazon Braket now offers D-Wave's Advantage quantum system for quantum annealing The D-Wave 2000Q (PDF) The Nano Candidates Amazon Elasticsearch Service now offers T3 Instances (Jean-Manuel) AWS Fargate increases default resource count service quotas (Guy) AWS IQ now provides short URLs for expert profiles (Arjen)   Sponsors   Gold Sponsor Innablr   Silver Sponsors AC3 CMD Solutions DoIT International  

最新情報を "ながら" でキャッチアップ！ ラジオ感覚放送 「毎日AWS！」 おはようございます、サーバーワークスの加藤です。 今日は 9/29 に出たアップデート6件をご紹介。 感想は Twitter にて「#サバワ」をつけて投稿してください！ ■ UPDATE ラインナップ Amazon EFS が AWS Systems Manager と統合 - EFS クライアントの管理を簡素化 Amazon CloudFront がメキシコとニュージーランドでサービス提供開始 AWS Marketplace が Discovery API をリリース Amazon EventBridge スキーマレジストリが JSON スキーマをサポート Amazon Braket が D-Wave 製の Advantage 量子システムを提供開始 Amazon Textract が S3 バケットへの結果出力に対応 ■ サーバーワークスSNS Twitter / Facebook ■ サーバーワークスブログ サーバーワークスエンジニアブログ

最新情報を "ながら" でキャッチアップ！ ラジオ感覚放送 「毎日AWS！」 おはようございます、サーバーワークスの加藤です。 今日は 8/12 に出たアップデート13件をご紹介。 感想は Twitter にて「#サバワ」をつけて投稿してください！ ■ UPDATE ラインナップ AWS Systems Manager Distributor で有名な3rdパーティ製品のエージェントを管理できるように Amazon S3 Access Points が Copy API をサポート AWS Storage Gateway ハードウェアアプライアンスを16のリージョンに拡大 Amazon Forecast がたたみ込みニューラルネットワークを使用 - 予測モデルを最大2倍早く、最大30%高い精度でトレーニング Amazon NeptuneがNeptune Workbenchを用いたグラフの可視化を発表 AWS Glue が Glue workflows の停止と再開をサポート AWS Lambda が Amazon Linux 2環境でGo及びカスタムランタイムをサポート AWS Lambda がAmazon Corretto を利用したJava 8ランタイムをサポート Amazon Cognitoユーザープールがトークンの有効期限のカスタマイズをサポート Amazon ElastiCache がリソースベースの権限ポリシーをサポート AWS IoT Device Defender が監査結果抑制機能を開始 Amazon FSx for Lustre が高性能HDDベースの新しい共有ストレージを発表 AWS Solutions Library に AWS Security Hub を用いた自動セキュリティ対応ソリューションが追加 ■ サーバーワークスSNS Twitter / Facebook ■ サーバーワークスブログ サーバーワークスエンジニアブログ

最新情報を "ながら" でキャッチアップ！ ラジオ感覚放送 「毎日AWS！」 おはようございます、サーバーワークスの加藤です。 今日は 7/16 に出た2件のアップデートをご紹介。 感想は YouTube のコメント欄または Twitter にて「#サバワ」をつけて投稿してください！ ■ UPDATE ラインナップ APN パートナー向けに4つの新しいコースを発表 AWS Systems Manager メンテナンスウィンドウのスケジュールでオフセットの追加をサポート ■ サーバーワークスSNS Twitter / Facebook ■ サーバーワークスブログ サーバーワークスエンジニアブログ

最近現場でレガシーだったインフラのリニューアルをガチでやってるのでその話 始まり:検証環境が不足してて困っていたため、自分が環境構築することに。 しかしインフラに詳しい人がすでに退職していたので誰もインフラさわれない状態になっていた。。 jenkinsおじさんグッバイ before: stg・本番共にビルド・デプロイを1台のjenkinsでやっていた after: jenkinsをやめてcodebuild + codedeployのビルド・デプロイに変更し、サーバーいらずに。 監視をZabbixとPrometheus+GrafanaからCloudWatchに全面移行 上記をほぼ自動化 terraform 特に監視設定をterraformでやったのは初めてだったのでナレッジはたまった ansible インフラに関わってる時にAWSの最近のサービスについてもいくつか知れたし活用した AWS Chatbot SNSとAmazon Chime/Slackの仲介人。SNSをトリガーとして簡単にslackに通知できる CloudWatch SyntheticsでHTTP監視をする AWS Systems Manager セッションマネージャーでSSH・SCPできるようになりました 新規プロダクト立ち上げを今主導していて、Kotlin+SpringBootを採用した話

AWS management and governance services can help your organization become and remain agile while enabling you to maintain control over costs, compliance, and security. Join us to hear AWS service leaders discuss their vision and the latest launches from the AWS management and governance teams, including innovations you can leverage now from Amazon CloudWatch, AWS Config, AWS Organizations, AWS Service Catalog, AWS Control Tower, AWS Systems Manager, and much more. We are joined onstage by current AWS customers who discuss how they use management and governance services today.

In this session, we walk through what you need to do to be prepared to respond to security incidents in your AWS environments. We start off with planning best practices, move through the configurations that will help deliver protective and detective controls, then finally show you how you can improve your response capability. Learn how AWS Organizations, AWS Identity and Access Management (IAM), Amazon GuardDuty, AWS Security Hub, AWS Lambda, AWS WAF, AWS Systems Manager, and AWS Key Management Service (AWS KMS) can help take you from protect and detect to respond and recover.

You can use an expanding set of services to automate many common management tasks in your AWS environment: patching, configuration updates, software stack deployments, and more. In this session, we explore how you can use AWS management tools for automation, including through the use of self-service runbooks. We discuss the many options available, including AWS CloudFormation, AWS Service Catalog, and AWS Systems Manager.

Justin is back from vacation and gets the podcast back on track. Justin, Peter and Jonathan talk about their guest spot on roaring elephants and Justin’s AWS lambda fireside chat video. Elasticsearch sues AWS over trademark infringement, AWS gets its IQ raised, Oracle gets fedramp certified cloud regions and Google enhances their github app for cloud build.  Plus the world famous lightning round. Sponsors: Foghorn Consulting – fogops.io/thecloudpod Follow Up Topics Roaring Elephant https://roaringelephant.org/2019/10/08/episode-161-the-cloudpod-weather-report-part-1/ AWS  https://www.youtube.com/watch?v=8Aq2DIMRIIg&t=1s  General News/Topics Oracle Launches FedRAMP-Authorized Government Cloud Regions Oracle will add 2,000 jobs and 20 data centers in cloud infrastructure push AWS faces Elasticsearch lawsuit for trademark infringement Ansible holds the pole position for automation, but is it too good and too small? AWS Now use AWS Systems Manager to execute complex Ansible playbooks AWS DataSync News – S3 Storage Class Support and Much More AWS IQ – Get Help from AWS Certified Third Party Experts on Demand EC2 High Memory Update – New 18 TB and 24 TB Insta

AWS introduces new kernel panic API trigger, Azure storage gets complicated, and Google’s big query gets a terraform module. Sponsors: Foghorn Consulting – fogops.io/thecloudpod Follow Up Cloudflare files for IPO, revealing revenue of $129M in first half of 2019 Topics General News Alibaba blows past earnings estimates cloud business hits 4.5b run rate Digital Ocean launches new managed MySQL and Redis Database Services AWS New – Trigger a Kernel Panic to Diagnose Unresponsive EC2 Instances Amazon Prime Day 2019 – Powered by AWS AWS App Mesh now supports routing based on HTTP headers and specifying route priorities Easily enable AWS Systems Manager capabilities with Quick Setup Amazon ECS Now Supports Per-Container Swap Space Parameters 081319 Amazon Letter to Sen Wyden RE Consumer Data.pdf Original letter: https://www.wyden.senate.gov/imo/media/doc/080519%20Letter%20to%20Amazon%20re%20Capital%20One%20Hack.pdf 

Simon and Nicki run through some interesting new AWS capabilities for customers as well as a look at the upcoming re:MARS conference (https://remars.amazon.com/). 0:29 - Databases 1:20 - Analytics 1:52 - Compute 3:22 - IoT 4:05 - Customer Engagement 5:07 - Networking 5:34 - Developer Tools 7:46 - Application Integration 8:20 - Game Tech 8:42 - Media Services 9:24 - Management and Governance 12:41 - re:MARS Topic || Databases Amazon DynamoDB adds support for switching encryption keys to encrypt your data at rest | https://aws.amazon.com/about-aws/whats-new/2019/02/amazon-dynamodb-adds-support-for-switching-encryption-keys-to-encrypt-your-data-at-rest/ Amazon ElastiCache for Redis adds support for Redis 5.0.3 and the ability to change Redis command names | https://aws.amazon.com/about-aws/whats-new/2019/02/amazon-elasticache-for-redis-adds-support-for-redis-503-and-the-ability-to-change-redis-command-names/ Performance Insights is Generally Available on Amazon RDS for SQL Server | https://aws.amazon.com/about-aws/whats-new/2019/03/performance-insights-is-generally-available-for-sql-server/ Topic || Analytics Amazon QuickSight Supports Row Level Security Enabled Email Reports, New Analytical Capabilities and More | https://aws.amazon.com/about-aws/whats-new/2019/03/amazon-quickSight-supports-row-level-security-enabled-email-reports-new-analytical-capabilities-and-more/ Topic || Compute AWS Step Functions Adds Tag-Based Permissions | https://aws.amazon.com/about-aws/whats-new/2019/03/aws-step-functions-adds-tag-based-permissions/ AWS ParallelCluster support for Amazon FSx Lustre | https://aws.amazon.com/about-aws/whats-new/2019/03/aws-parallelcluster-support-for-amazon-fsx-lustre/ Announcing the Preupgrade Assistant to Migrate to Amazon Linux 2 From Amazon Linux AMI | https://aws.amazon.com/about-aws/whats-new/2019/03/announcing_the_amazon_linux_2_preupgrade_assistant/ Topic || IoT AWS IoT Greengrass Introduces New Networking Configurations and Group Permission Settings | https://aws.amazon.com/about-aws/whats-new/2019/03/aws-iot-greengrass-introduces-new-networking-configurations-group-permission-settings/ Topic || Customer Engagement Amazon Connect Simplifies Adding AWS Lambda Functions to Contact Flows | https://aws.amazon.com/about-aws/whats-new/2019/02/amazon-connect-simplifies-adding-aws-lambda-functions-to-contact-flows/ Introducing new AWS Digital Customer Experience Competency Partner Solutions | https://aws.amazon.com/about-aws/whats-new/2019/03/introducing-new-aws-digital-customer-experience-competency/ Topic || Networking Announcing the new AWS Direct Connect Console | https://aws.amazon.com/about-aws/whats-new/2019/03/announcing-the-new-aws-direct-connect-console/ Topic || Developer Tools Amazon Corretto 11 is Now Available as a Release Candidate | https://aws.amazon.com/about-aws/whats-new/2019/03/amazon-corretto-11-is-now-available-as-a-release-candidate/ AWS Amplify Console Adds Support for Instant CDN Cache Invalidation and Delta Deployments | https://aws.amazon.com/about-aws/whats-new/2019/03/aws-amplify-console-adds-support-for-instant-cdn-cache-invalidation-and-delta-deployments/ AWS CodeCommit Supports VPC Endpoints | https://aws.amazon.com/about-aws/whats-new/2019/03/aws-codecommit-supports-vpc-endpoints/ Automate Releases to the AWS Serverless Application Repository using AWS CodePipeline | https://aws.amazon.com/about-aws/whats-new/2019/03/automate-releases-to-the-aws-serverless-application-repository-using-aws-codepipeline/ Topic || Application Integration New Amazon SNS Console Now Available | https://aws.amazon.com/about-aws/whats-new/2019/03/new-amazon-sns-console-now-available/ Topic || Game Tech Identity and Access Management (IAM) Roles Now Available for Amazon GameLift | https://aws.amazon.com/about-aws/whats-new/2019/03/identity-and-access-management--iam--roles-now-available-for-ama/ Topic || Media Services AWS Elemental MediaLive Adds Support for Encrypted HLS and VPC Inputs | https://aws.amazon.com/about-aws/whats-new/2019/03/aws-elemental-medialive-adds-supports-for-encrypted-hls-and-vpc-inputs/ AWS Elemental MediaLive Now Supports Pausing Channel Delivery on a Schedule | https://aws.amazon.com/about-aws/whats-new/2019/03/aws-elemental-medialive-now-supports-pausing-channel-delivery-on-a-schedule/ AWS Elemental MediaLive Simplifies Sending Live Streams to AWS Elemental MediaPackage | https://aws.amazon.com/about-aws/whats-new/2019/03/aws-elemental-medialive-simplifies-sending-live-streams-to-aws-elemental-mediapackage/ Topic || Management and Governance AWS Systems Manager now supports on-premises instance management for large hybrid environments | https://aws.amazon.com/about-aws/whats-new/2019/03/AWS_Systems_Manager_on-premises_instance_management_for_large_hybrid_environments/ AWS CloudFormation Coverage Updates for AWS RAM, AWS Robomaker, Amazon ApiGateway, and more | https://aws.amazon.com/about-aws/whats-new/2019/02/aws-cloudformation-coverage-updates-for-aws-ram--aws-robomaker--/ whats-new/2019/02/amazon-elasticache-for-redis-adds-support-for-redis-503-and-the-ability-to-change-redis-command-names/ AWS License Manager adds new capabilities to track on premises usage, number of instances, and vCPUs based on Optimize CPU settings | https://aws.amazon.com/about-aws/whats-new/2019/02/NewLicenseManagervCPU/ AWS License Manager enhances support for tracking instances on premises | https://aws.amazon.com/about-aws/whats-new/2019/03/LicenseManagerOnPremises/

This week Matt Adorjan (@mda590) joins us to talk about AWS’s open distro for ElasticSearch, Breaking up big tech, and F5 acquiring Nginx. Plus the lightning round and Cool Tools with Jonathan. Thanks to our sponsors! Foghorn Consulting: fogops.io/thecloudpod Audible: audibletrial.com/thecloudpod Show Topics How does your cloud storage grow? With a scalable plan and a price drop Azure Premium Blob storage now in public preview Simply Enterprise Threat Detection and Protection with Google Cloud Security Services Elizabeth Warren bold plan to break up big tech Azure Devops Server 2019 now available AWS Announces Open Distro for Elastic Search Adrian Cockcroft publishes blog on keeping open source open Elastic.Co Response to open distros open source F5 Acquires NGINX Lightning Round AWS Performance Insights is now GA for SQL Server https://aws.amazon.com/about-aws/whats-new/2019/03/performance-insights-is-generally-available-for-sql-server/ AWS SSM on-Premise now handles large hybrid environments (Previously less than 1000 instances) https://aws.amazon.com/about-aws/whats-new/2019/03/AWS_Systems_Manager_on-premises_instance_management_for_large_hybrid_environments/ AWS Coretto 11 is now available as a r

Learn how Symantec uses AWS to provide complete, integrated security solutions that monitor and protect companies and governments from hackers. Hear about lessons learned from how Symantec scaled up its infrastructure to analyze billions of logs every day to detect the world's most sophisticated cyber attacks, and you'll see how Symantec integrates with native AWS services, like Amazon GuardDuty, AWS Lambda, and AWS Systems Manager, into its own security solutions to provide even better security in the cloud. This session is brought to you by AWS partner, Symantec Corporation.

In this session, Verizon shares how it uses AWS Systems Manager for inventory, compliance, and patch management solutions. Learn about the challenges that large enterprises face when they attempt to retrofit legacy solutions for cloud environments, and discover best practices for using AWS Systems Manager for minimal access policies, custom Amazon Machine Images, tagging policies, encryption, and more.