Podcast appearances and mentions of pete cheslock

Let's take another look at the topic of Platform Engineering, but perhaps with a different perspective. Pete Cheslock turns his attention to Platform Engineering, and if it's really anything new, or just what DevOps has always meant?

Let's take another look at the topic of Platform Engineering, but perhaps with a different perspective. Pete Cheslock turns his attention to Platform Engineering, and if it's really anything new, or just what DevOps has always meant?

About PetePete is currently the Head of Growth And Community for AppMap, the open source dynamic runtime code analyzer. Pete also works with early stage startups, helping them navigate the complex world of early stage new product development.Pete also fully acknowledges his profile pic is slightly out of date, but has been too lazy to update it to reflect current hair growth trends.Links:AppMap: https://appmap.io/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: If you asked me to rank which cloud provider has the best developer experience, I'd be hard-pressed to choose a platform that isn't Google Cloud. Their developer experience is unparalleled and, in the early stages of building something great, that translates directly into velocity. Try it yourself with the Google for Startups Cloud Program over at cloud.google.com/startup. It'll give you up to $100k a year for each of the first two years in Google Cloud credits for companies that range from bootstrapped all the way on up to Series A. Go build something, and then tell me about it. My thanks to Google Cloud for sponsoring this ridiculous podcast.Corey: Cloud native just means you've got more components or microservices than anyone (even a mythical 10x engineer) can keep track of. With OpsLevel, you can build a catalog in minutes and forget needing that mythical 10x engineer. Now, you'll have a 10x service catalog to accompany your 10x service count. Visit OpsLevel.com to learn how easy it is to build and manage your service catalog. Connect to your git provider and you're off to the races with service import, repo ownership, tech docs, and more. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn and this is probably my favorite recurring episode slash tradition, every year. I drag Pete Cheslock on who talks with me about his experience at re:Invent. Last year, Pete, you didn't go. The year before, none of us went because it was all virtual, but it feels like we're finally getting back into the swing of things. How are you, Pete?Pete: I am doing great. It is always a pleasure. It was amazing to see other humans in person at a industry event. As weird as it sounds to say that, you know, it was great to be in Vegas [laugh], it was mostly great, just because there were other humans there too that I wanted to see.Corey: Because this is going to confuse folks who haven't been following our various adventures, these days, you are the Head of Growth and Community at AppMap. But you and I have been talking for years and you did a stint working at The Duckbill Group here with us as a cloud economist. Ah, I miss those days. It was fun working with you and being able to bother you every day as opposed to just on special occasions like this.Pete: Yeah, I know. I got to slide into your Slack DMs in addition, and then when I didn't get a response, I would slide into your Twitter DMs. It worked out perfectly. So yeah, it's been a wild ride. I mean, I took an interlude from my startup journey by continually working at tech startups.And yeah, I got to join onboard the Duckbill and have, you know, a really wonderful time cutting bills and diving into all of the amazing parts of people's Amazon usage. But I am also equally broken in my brain, and continually said to myself, “Maybe I'll do another startup.” [laugh].Corey: Right. And it turns out that we're not a startup. Everyone likes to think we are. It's like, oh, okay—like Amazon, for example, has us historically in their startup division as far as how they—the buckets as they put different accounts into. And if you look at us through that lens, it's yeah, we're a specific kind of startups, specifically a failing startup—or failed—because to us growth is maybe we'll hire one or two people next year, as opposed to, “Oh, yeah, we're going to TEDx this place.” No, yeah, we're building a lifestyle business very much by design.Pete: I'd be very curious how many account managers actually Duckbill has kind of churned through because usually, you get to keep your account manager if you're growing at a pretty incredible clip. And it's kind of a bellwether for, like, how fast are we—are we growing so fast that we have kept our account manager for multiple years?Corey: Your timing is apt. We're a six-year-old company and I just met our fourth account manager last week.Pete: [laugh].Corey: No it's, honestly, what happens with AWS account managers is the only way you get to keep them is if your spend trajectory on AWS matches their career trajectory inside of AWS. Because if you outpace them, they'll give you to someone that they view as being more senior, whereas if they outpace you, they're going to stop dealing with the small accounts and move on to the bigger ones. Honestly, at this point, I've mostly stopped dealing with my account managers. I had one that was just spectacular. It was sad to see him get promoted; good for him.But I get tired of trying to explain the lunacy that is me to someone on the AWS side every year. It just doesn't make sense because my accounts are super weird and when they try and suggest the usual things that work for 99.995% of AWS customers and things they care about, it falls to custard when it comes to me specifically. And that's not on them; it's because I'm weird and broken.Pete: I'm remembering now one of the best account managers that I ever worked with at a startup, years and years ago. She was with us for a couple of years, pretty solidly. And then, you know, because careers are long and jobs are short, when I was at The Duckbill Group again, doing work, turns out she was the account manager on this other thing, you know? Which, like, looking at the company she was account manager for was like 500x [laugh] my previous company, so I was like, “Oh, yeah. You're clearly moving up in the world because my company did not 500x.” So, sometimes you got to chase the ones who are.Corey: So, let's talk about re:Invent. This felt like the first re:Invent post-pandemic. And let's be clear, I wound up getting Covid by the end, so I don't recommend that to everyone. But let's be clear, this was not a re:Invent were anyone officially accepted that Covid existed. I was one of the only people wearing masks to most of the events I was at. Great load of good that did me.But it was big. It was the usual sixty-some-odd-thousand people that had been in previous years, as opposed to the hard cap of 30 or so that they had last year so it felt smaller and more accessible. Nope. Right back to bizarre numbers of people. But fewer sponsors than most years, so it felt like their budget was severely constrained. And they were trying to have not as many sponsors, but still an overwhelming crush of attendees. It felt odd, but definitely very large scale.Pete: Yeah, I can echo that a hundred percent. I'm sure we've talked about this in previous ones, but I've had the pleasure—well, I don't know, some might call it not a pleasure, but it's been a pleasure to watch re:Invent grow over so many years. I went to the first re:Invent. A company I was at actually sponsored it. And remembering back to that first re:Invent, it was kind of quaint by comparison.There were 4000 people at the first re:Invent, which again, it's a big conference, especially when a lot of the conferences that I think I was really attending at the time were like, you know, 600, 1000, maybe tops. To go to a 4000-person event in Vegas especially, it's again, in the same Expo Hall it's been since that first one, it still felt big. But every person stayed in the Venetian. Pretty much everyone was in the same hotel, all of the attendees that year. All the talks were there.There was, you know, a lot [laugh]—I mean, a lot less of everything that was there. And so, watching it grow over time, not only as a sponsor because I've actually been—kind of worked re:Invent as a, like, a booth person for many of these years for multiple different sponsors and had to coordinate that aspect of it, but then also a couple of times just being more, like, attendee, right, just someone who could go and kind of consume the content. This year was more on the side of being more of an attendee where I got to just kind of experience the Expo Hall. You know, I actually spent a lot of time in the Expo Hall because a big part of why I was there was—Corey: To get t-shirts.Pete: Yeah, we'll get to—I was running low on not only t-shirts but socks. My socks were really worse for wear the last few years. I had to, like, re-up that, right [laugh]?Corey: Yeah, you look around. It's like, “Well, none of you people have, like, logoed pants? What's the deal here? Like, I have to actually buy those myself. I don't—I'm not here to spend money.”Pete: Yeah, I know. So. And so yeah, this year, it felt—it was like Covid wasn't a thing. It wasn't in anyone's mind. Just walking around—Vegas in general, obviously, it's kind of in its own little bubble, but, you know, I've been to other events this year that were much more controlled and had a lot more cautious attendees and this was definitely not like that at all. It felt very much, like the last one I was at. The last one I was at was 2019 and it was a big huge event with probably 50,000-plus people. And this one felt like to me at least, attendee-wise, it definitely felt bigger than that one in a lot of ways.Corey: I think that when all is said and done, it was a good event, but it wasn't necessarily what a lot of folks were expecting. What was your take on the content and how the week played out?Pete: Yeah, so I do, in many ways, kind of miss [laugh] the event of yore that was a little bit more of a targeted, focused event. And I understand that it will never be that kind of event anymore. Maybe they start splitting it off to be, you know, there's—just felt much more like a builder event in previous years. The content in the keynotes, you know, the big keynotes and things like that would be far more, these big, iterative improvements to the cloud. That's something that always felt kind of amazing to see. I mean, for years and years, it was like, “Who's ready for another re:Invent price drop?” Right? It was all about, like, what's the next big price drop going to be?Corey: Was it though because I never was approaching with an eye toward, “Oh, great. What are they going to cut prices on now?” That feels like the least interesting things that ever came out of re:Invent, at least for me. It's, what are they doing architecturally that lets me save money, yes. Or at least do something interesting architecturally, great. I didn't see Lambda when it first came out, for example, as a cost opportunity, although, of course, it became one. I saw it as this is a neat capability that I'm looking forward to exploring.Pete: Yeah, and I think that's what was really cool about some of those early ones is these, like, big things would get released. Like, Lambda was a big thing that got released. There was just these larger types of services coming out. And I think it's one of your quotes, right? Like, there's a service for everyone, but every service isn't for everyone.Corey: Yeah.Pete: And I feel like, you know, again, years ago, looking back, it felt like more of the services were more geared towards the operational, the early adopters of Amazon, a lot of those services was for those people. And it makes sense. They got to spread out further, they've got to have kind of a wider reach to grow into all of these different areas. And so, when they come out with things that, yeah, to me, I'm like, “This is ridiculous feature. Who would ever use this?” Like, there's probably a dozen other people at different companies that are obscenely excited because they're at some enterprise that has been ignored for years and now finally they're getting the exact tooling that they need, right?Corey: That made sense for a long time. I think that now, the idea that we're going to go and see an Andy Jassy-era style feature drop of, “Here's five new databases and a whole new compute platform and 17,000 more ways to run containers,” is not necessarily what is good for the platform, certainly not good for customers. I think that we're seeing an era of consolidation where, okay, you have all these services to do a thing. How do I pick which one to use? How do I get onto a golden path that I can also deviate from without having to rebuild everything? That's where customers seem to be. And it feels like AWS has been relatively slow to acknowledge or embrace that to me.Pete: Yeah, a lot of the services, you know, are services they're probably building for just their own internal purposes, as well. You know, I know, they are for a while very motivated to get off anything Oracle-related, so they started building these services that would help migrate, you know, away from Oracle because they were trying to do it themselves. But also, it's like, there's still—I mean, I talk with friends of mine who have worked at Amazon for many years and I'm always fascinated by how excited they are still to be there because they're operating at a scale that just doesn't exist anywhere else, right? It's like, they're off on their lone island that go into work somewhere else is almost going backwards because you've already solved problems at this lower level of scale. That's obviously not what you want to be doing anymore.And at the scale that they're at for some of these services, even like the core services, the small improvements they're making, they seem so simple and basic, like a tiny EBS improvement, you're like, “Ugh, that's so boring.” But at their level of scale for, like, something like an EBS, like one of those top five services, the impact of that tiny little change is probably even so amazingly impactful. Like it's just so huge [laugh], you know, inside that scope of the business that is just—that's what—if you really start pulling the thread, you're like, “Wow, actually, that is a massive improvement.” It just doesn't feel that way because it's just oh, it's just this tiny little thing [laugh]. It's like, just almost—it's too simple. It's too simple to be complex, except at massive scale.Corey: Exactly. The big problem I ran into is, I should have noticed it last year, but it was Adam Selipsky's first re:Invent and I didn't want to draw too many conclusions on it, but now we have enough dots to make a line—specifically two—where he is not going to do the Andy Jassy thing of getting on stage and reading off of a giant 200 item list of new feature and service announcements, which in AWS parlance, are invariably the same thing, and they wind up rolling all of that out. And me planning my content schedule for re:Quinnvent around that was a mistake. I had to cancel a last-minute rebuttal to his keynote because there was so little there of any substance that all would have been a half-hour-long personal attack and I try not to do that.Pete: Mmm. Yeah, the online discussion, I feel like, around the keynote was really, like, lackluster. It was yeah, like you said, very devoid of… not value; it's not really the right word, but just substance and heft to it. And maybe, look, we were just blessed with many, many years of these dense, really, really dense, full keynotes that were yeah, just massive feature drops, where here's a thing and here's a thing, and it was almost that, like, Apple-esque style kind of keynote where it was like, we're just going to bombard you with so many amazing things that kind of is in a cohesive storyline. I think that's the thing that they were always very good about in the past was having a cohesive story to tell about all of these crazy features.All of these features that they were just coming out with at this incredible velocity, they could weave the story around it. And you felt like, yeah, keynote was whatever hour, two hours long, but it would go by—it always felt like it would go by quickly because they were just they had down kind of really tight messaging and kept your attention the whole way through because you were kind of like, “Well, what's next? There's always—there's more. There's got to be more.” And there would be, right? There would be that payoff.Corey: I'm glad that they recognized that what got them here won't get them there, but I do wish that they had done a better job of signaling that to us in more effective ways. Does that make any sense?Pete: Yeah, that's an interesting… it's kind of an interesting thought exercise. I mean, you kind of mentioned before earlier, before we started recording, the CMO job is still available, it's still open [laugh] at AWS. So, if this was a good way to attract a top-tier CMO, I'd almost feel like if you were that person to come in and be like, “Hey, this did not work. Here are the following reasons and here's what you need to do to improve it.” Like, you might have a pretty solid shot of landing that role [laugh].Corey: Yeah, I'm not trying to make people feel intentionally bad over it. This stuff is very hard, particularly at scale. The problem I had with his keynote was not in fact that he was a bad speaker, far from it. He was good a year ago, he's clearly put work and energy into becoming better over the past year. From a technical analysis of how is Adam Selipsky as a public speaker, straight A's as far as I'm concerned, and I spent a lot of time focusing on this stuff myself as a professional speaker myself. I have no problems with how he wound up delivering any of the content. My problem was with the content itself. It feels like he was let down by the content team.Pete: Yeah, it definitely felt not as dense or as rich as we had come to expect in previous years. I don't think it was that the content didn't exist. It's not like they didn't build just as much, if not way, way more than they have in previous years. It just seemed to just not be part of the talk.I don't know. I always kind of wonder, too, is this just an audience thing? Which is, like, maybe I'm just not the audience for his talk, right? Was there someone else in that Expo Hall, someone else watching the stream, that was just kept on the edge of their seat hearing these stories? I don't know. I'm really kind of curious. Like, you know, are we only representing this one slice of the pie, basically?Corey: I think part of the problem is that re:Invent has grown so big, that it doesn't know what it wants to be anymore. Is it a sales event? By the size of the Expo Hall, yeah, it kind of is. Is it a partner expo where they talk about how they're going to milk various companies? Possibly. There's certainly one of those going on.There was an analyst summit that I attended for a number of days during re:Invent this year. They have a whole separate thing for press. The community has always been a thriving component of re:Invent, at least for me, and seeing those folks is always terrific. Is it supposed to be where they dump a whole bunch of features and roadmap information? Is it time for them to wind up having executive meetings with their customers? It tries to be all of those things and as a result, at this scale it feels like it is failing to be any of them, at least in an effective, cohesive way.Pete: Yeah, and you really nailed each of the personas, like, of a re:Invent attendee. I've talked to many people who are considering going to re:Invent, and they're, “I don't really know if I want to go, but I really want to go to some sessions, and I really want to do this.” And I always have to kind of push back and say, “Look, if you're only going there to attend talks,” like, just don't bother, right? As everyone knows, the talks are all recorded, you can watch them later. I did have conversations with some engineer, principal engineer level software folks that were there and the prevailing consensus from chatting with those folks, kind of anecdotally, is that, like, they had actually a lot of struggles even getting into some of these sessions, which for anyone who has been to re:Invent in the last, I don't know, four or five years, like, it's still a challenge, right?There's—you got to register for a lot of these talks way far in advance, there'll be a standby list, there'll be a standby line. It's a lot of a lot. And so, there's not usually a ton of value there. And so, I always try to say, like, “If you're going to re:Invent your, kind of, main purpose to go would be more for networking,” or just you're going because of the human interaction that you hope to get out of it, right, the high bandwidth conversations that are really hard to do in other areas. And I think you've nailed a bunch of those, right? Like, an analyst briefing is really efficient if you can get all the analysts in a room versus doing one-off analyst meetings.Meeting with big enterprises and hearing their thoughts and feelings and needs and requirements, you can get a lot of those conversations. And especially, too, if, like, talking to an enterprise and they got a dozen people all spread over the world, well you can get them all in one room, like, that's pretty amazing in this world. And then on the sales side, I feel like granted, I spent most of my time in the Expo Hall, but that was probably the area that I think you said earlier which I really picked up on, which was the balance between sponsors and attendees felt out of whack. Like it felt like there were way more attendees than the sponsors that I would have kind of expect to see.Now, there were a lot of sponsors on that Expo Hall and it took days. I mean, I was on the Expo Hall for days walking around and chatting with different companies and people. But one of the things that I saw that I have never seen before was a number of sponsorship booths, right—and these are booths that are, like, prebuilt, ten by ten-foot size or the smaller ones—that were blanks. They were like, you know, like, in a low-quality car where you have blank buttons that, like, if you paid more you get that feature. Walking around, there was a nonzero number of just straight-up empty booth, blank booths around which, I don't know, like, that felt kind of telling. Like, did they not sell all their sponsorships? Has that ever happened? I don't even know. But this was I felt like the first I've had—Corey: Or did companies buy the sponsorship and then realize that it was so expensive to go on top of it, throwing bad money after good might not have made sense. Because again, when people—Pete: Right.Corey: —brought out these sponsorships, in many cases, it was in the very early days of the growing recession we've seen now. And they may have been more optimistic, their circumstances may certainly have changed. I do know that pricing for re:Invent sponsorships was lower this past year than it had been in previous years. In 2019, for example, they had two Expo Halls, one at the ARIA Quad and the other at the Venetian. They had just one this year, which made less running around on my part, but still.Pete: Yeah, I do remember that, that they had so many sponsors. What I would say about the sponsors that there's two parts of this that were actually interesting. One, you're definitely right. As someone who has sponsored re:Invent before and has had to navigate that world, you are likely going to commit to the sponsorship as early as June, you know, could even be earlier than June depending on how big of a thing that you're doing. But it's early. It's usually in the summertime that you're—if you haven't made a decision by the summertime, like, you could actually not get a booth, right?And this was, I remember, the last one that I had sponsored was maybe 2018, 2019. And, like, you don't want those last few booths. Like, they put you in the back and not a good way. But going there, were a lot of—I did notice a lot of booths that had pretty massive layoffs who still had the booths, you know, and again, large booths, large companies, which again, same thing. I kind of am like, “Wow, like, how many employees did that booth cost you, right?”Because like [laugh], some of these booths are hundreds of thousands of dollars to sponsor. And then the other thing that I actually noticed, too, which I was honestly a little surprised by, with the exception of the Datadog booth; I love my friends at Datadog, they have the most amazingly aggressive booth BDRs who are always just, they'll get you if you're, like, hovering near them. And there's always someone to talk to over there. Like, they staff it really, really well. But there were some other booths that I was actually really interested in talking to some of the people to learn about their technology, that I actually waited to talk to someone. Like, I waited for someone to talk to, and then finally I'm like, “You know what? I'm going to come back.”And then I came back and waited again. So, it's like, how many of these sponsors obviously spent a lot of money to go there, then months later, they start looking at the people that they have to support this, they've already had some layoffs and probably sent a much smaller audience there to actually, like, operate the booth.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: One bright light that I did enjoy and I always enjoy, though I'm not certain how actionable it is in the direct sense, was Peter DeSantis' Monday Night Live keynote. It was great. I mean, the one criticism I had of it—on Twitter at the time, before that thing melted and exploded—was that it was a bit mislabeled because it really should have been what it turned into midway through of surprise computer science lecture with Professor DeSantis. And I was totally there for it. But it was fun just watching some of the deep magic that makes this all work be presented in a way that most of us normally don't get to see.Pete: Well, this was the first year they did not do their Midnight Madness over-the-top kind of thing. And I also I don't recall that I saw them doing one of the other things I feel like is at night is they're, like, giant wing-eating competition. Am I wrong? Did they do that this year and I just missed hearing about it?Corey: They did not. Turns out that competitive Gluttony is not as compelling as it once was. But they also canceled their Midnight Madness event a month or two before re:Invent itself. What was super weird to me was that there was no event—community or otherwise—that sprung forth to seize that mantle. So, you had a whole bunch of people who were used to going for several hours that night to a big event with nothing to do.And at 9 p.m. they started just dumping a whole bunch of service releases in their blog and RSS feeds and the rest, and it just felt very weird and discordant. Like, do they think that we have nothing better to do than sit here and read through this on a Sunday night where we would have otherwise been at a party? Well yeah, in my case, I'm super sad and of course, I had nothing better to do that night. But most people had things going on.Pete: Yeah. Yeah, exactly. I think also, if you—maybe it's a little bit better now but I don't know when you have to buy that many chicken wings in advance, but with supply chains being what they are and the cost of chicken wings, I mean, not that I track the cost of chicken wings, except I absolutely do every time I go to Costco, they're up substantially. So, that was probably a contributing factor to the wing-eating contest: supply chain pain and suffering. But yeah, it's really interesting that just even in what some of the sponsors kind of were doing this year over previous years, I doubt they did this in 2021—but maybe, I don't know—but definitely not in 2019, something that I don't recall to this level was the sponsors essentially booking out entire restaurants near the venue every single day of the conference.And so again, if you were at this event like we were, and you at the end of the thing, were just like, I just want to sit and I've got a handful of friends, I want to sit and, like, have a drink, and just, like, chat and catch up and hear how the day went and everything else, finding a place to actually go to do that was very, very hard to do. And the thing that I noticed was—again, seemed like it was new this year; I don't recall it in 2019 to this level is, there were a lot of the big sponsors that had just booked a whole restaurant, breakfast, lunch, and dinner, like, from open to close, fully booked it, which was honestly, brilliant.Corey: Oh, yeah. If you bring 200, 300 people to an event, you've got to feed him somehow. And, “Hey, can we just rent out your restaurant for the entirety of this week?” Is not out of the question compared to what you'd even spend just reimbursing that sea of people to go and eat somewhere else.Pete: Exactly. The reason—I'm approaching this from, like, a business perspective—if I had a large group of enterprise salespeople and they need a place to book meetings, well, it's super compelling if I'm being courted by one of these salespeople and they're like, “Hey, come and have breakfast. Come and grab a coffee.” You know, and there's a place where you can sit down and quietly enjoy that meal or coffee while having a sale. Like, I'll have that sales conversation and I'm going to be way more motivated to show up to it because you're telling me it's like, this is where we're going to meet.Versus some of my friends were trying to, like, coordinate a lunch or a coffee and it's like, do we want to go to the Starbucks that has 500 people in line or do we want to walk four hotels, you know, down the street to find a bar that has video poker that no one will be sitting at and that we can just sit down and talk, right? It kind of felt like those were your two options.Corey: One thing that MongoDB did is rented out the Sugarcane restaurant. And they did this a couple of years in a row and they wound up effectively making it available to community leaders, heroes, and whatnot, for meetings or just a place to sit down and catch your breath. And I think that was a brilliant approach. You've gone to the trouble of setting this thing up for meetings for your execs and whatnot. Why not broaden it out?You can't necessarily do it for everyone, for obvious reasons, but it was nice to just reach out to folks in your orbit and say, “Yeah, this is something available to you.” I thought that was genius. And I—Pete: Oh yeah.Corey: —wish I thought of doing something like that. Let's be clear. I also wish I had rent-out-Sugarcane-for-a-week budget. But you know—Pete: [laugh].Corey: —we take what we can get.Pete: Yeah. That'll be a slight increase to the Spite Budget to support that move.Corey: Just a skosh, yeah.Pete: Yeah, the MongoDB, they were one that I do remember had done it similarly. I don't know if they had done it, kind of, full-time before, but a friend of mine work there, had invited me over and said, “Hey, like, come by, let's grab a drink. You know, we've got this hotel, you know, this restaurant kind of booked out.” And that was back in 2019. Really enjoyed it.And yeah, I noticed it was like, you know, basically, they had this area available, again, a place to sit down, to open your laptop, to respond to some emails, making it available to community people should have been a no-brainer to, really, all of these other sponsors that may have times of less kind of attendance, right? So obviously, at any of the big meals, maybe that's when you can't make it available for all the people you want to, but there's going to be off hours in between times that making that available and offering that up generates a supreme amount of goodwill, you know, in the community because you know, you're just looking for a place to sit out and drink some water [laugh].Corey: Yeah, that was one challenge that I saw across the board. There were very few places to just sit and work on something. And I'm not talking a lounge everywhere around every corner was needed necessarily or even advisable. No, the problem I've got was that I just wanted to sit down for two or three minutes and just type up an email quickly, or a tweet or something, and nope, you're walking and moving the whole time.Pete: Yeah. Now honestly, this would be a—this was a big missed opportunity for the Amazon event planning folks. There was a lot of unnecessary space usage that I understand why they had it. Here's an area you could play Foursquare, here's an area that had seesaws that you could sit on. Like, just, I don't know, kitschy stuff like that, and it was kind of off to the side or whatever.Those areas honestly, like, we're kind of off to the side, they were a little bit quieter. Would have been a great spot to just, like, load up some chairs and couches and little coffee tables and just having places that people could sit down because what ended up happening—and I'm sure you saw it just like I did—is that any hallway that had somewhere that you could lean your back against had a line of people just sitting there on their laptops because again, a lot of us are at this event, but we're also have jobs that we're working at, too, and at some point during the day, you need to check in, you need to check some stuff out. It felt like a lack of that kind of casual space that you can just relax in. And when you add on top of all the restaurants nearby being essentially fully booked, it really, really leaves you hanging for any sort of area to sit and relax and just check a thing or talk to a person or anything like that.Corey: Yeah, I can't wait to see what lessons get learned from this and how it was a mapping to next year, across the board. Like, I have a laundry list of things that I'm going to do differently at re:Invent next year. I do every year. And sometimes it works out; sometimes it really doesn't. And it's a constant process of improvement.I mean, one of the key breakthroughs for me was when I finally internalized the idea that, yeah, this isn't going to be like most jobs where I get fired in the next six months, where when I'm planning to go to re:Invent this is not the last re:Invent I will be at in my current capacity, doing what I do professionally. And that was no small thing. Where oh, yeah. So, I'm already making plans, not just for next re:Invent, but laying the groundwork for the re:Invent after that.Pete: Yeah, I mean, that's smart way to do it. And especially, too, when you don't consider yourself an analyst, even though you obviously are an analyst. Maybe you do consider yourself an analyst, but you're [laugh] more, you know, you're also the analyst who will go and actually use the product and start being like, “Why does this work the way it does?” But you're kind of a little bit the re:Invent target audience in a lot of ways, right? You're kind of equal parts on the analyst expert and user as well. It's like you kind of touch in a bunch of those areas.But yeah, I mean, I would say the one part that I definitely enjoyed was the nature walk that you did. And just seeing the amount of people that also enjoyed that and came by, it was kind of surreal to watch you in, like, full safari garb, basically meandering through the Expo Hall with this, like, trail of, like, backpacks [laugh] following you around. It was a lot of fun. And, you know, it's like stuff like that, where people are looking for interesting takes on, kind of, the state of something that is its own organism. Like, the Expo Hall is kind of its own thing that is outside of the re:Invent control. It's kind of whatever is made up by the people who are actually sponsoring it.Corey: Yeah, it was neat to see it play out. I'm curious to see how it winds up continuing to evolve in future years. Like right now, the Nature Walk is a blast, but it was basically at the top and I had something like 50 people following me around at one point. And that is too big for the Expo Hall. And I'm not there to cause a problem for AWS. Truly, I'm not. So, I need to find ways to embrace that in ways that don't kill the mojo or the energy but also don't create problems for, you know, the company whose backup I am perched upon, yelling more or less ridiculous things.Pete: [laugh]. I think it was particularly interested in how many people I'd be walking by and every once a while I would see, like, a friend of mine, someone actually working one of the booths and just be like, “What's going on here?” Like, I know one of my friends even said, “Yeah, like, nothing draws a crowd like a crowd.” And you can almost see more people [laugh] just, like, connecting themselves onto this safari train moving their way through. Yeah, it's a sight to see, that's for sure [laugh].Corey: Yeah, I'll miss aspects of this. Again, nothing can ever stay the same, on some level. You've got to wind up continuing to evolve and grow or you wind up more or less just frozen in place[ and nothing great ever happens for you.Pete: Yeah, I mean, again, Expo Hall has gone through these different iterations, and I—you know, when it does come to the event, as I kind of think back, I probably have spent most of my time actually in the Expo Hall, usually just related to the fact that, like, when you're a sponsor, like, you're just—that's where you're at. For better or worse, you're going to be in there. And especially if you're a sponsor, you want to check out what other sponsors are doing because you want to get ideas around things that you might want to try in later years. I mentioned Datadog before because Datadog to this day continues to have the best-designed booth ever, right? Like when it comes to a product that is highly demoable, I've been myself as a sponsor, it has always been a struggle to have a very effective demo setup.And I actually remember, kind of, recommending to a startup that I was at years ago, I'm doing a demo setup that was very, very similar to how Datadog did it because it was brilliant, where you have this, like, octagon around a main area of tables, and having double-sided demo stations. A lot more people are doing this now, but again, as I walked by I was again reminded just how effective that setup is because not only do you have people that just they don't want to talk, they just want to look, and they can kind of safely stand there and look, but you also have enough people staffing the booth for conversations that for, like me who actually might want to ask for questions, I don't have to wait and I can get an answer and be taken care of right away, versus some other booths. This year, one of the areas that I actually really enjoyed—and I don't even know the details of, like, how it all came about—but it looked like some sort of like Builder's Expo. I don't know if you remember walking by there, but there was a whole area of different people who had these little IoT or various powered things. One of them was, like, a marble sorting thing that was set up with a bunch of AWS services. I think there was like the Simple Beer Service V… four or five at this point. I had one of those iterations.It was some sort of mixture between Amazon software services that were powering these, like, physical things that you can interact with. But what was interesting is like, I have no idea, like, how it was set up, and who—I'm assuming it was Amazon specific—but each of these little booths were like chocked up with information about who they were and what they built, which gave it a feel of, like, this was like a last-minute builder event thing. It didn't feel like it was a highly produced thing. It had a much more casual feel to it, which honestly got me more interested to spend time there and check out the different booths.Corey: It was really nice to be able to go and I feel like you got to see all of the booths and whatnot. I know in previous years, it feels like you go looking for specific companies and you never find them. And you thought, “Oh”—Pete: [laugh].Corey: —“They must not have been here.” You find out after the fact, oh no, just you were looking in the wrong direction because there was so much to see.Pete: There were definitely still a couple of those. I had a list of a handful of booths I wanted to stop by, either to say hi to someone I knew who was going to be there or just to chat with them in general, there was a couple that I had to do a couple loops to really track them down. But yeah, I mean, it didn't feel as overly huge as a previous one, or as previous ones. I don't know, maybe it was like the way they designed it, the layout was maybe a little bit more efficient so that you could do loops through, like, an outer loop and an inner loop and actually see everything, or if it just was they just didn't have enough sponsors to truly fill it out and maybe that's why it felt like it was a little bit more approachable.I mean, it was still massive. I mean, it was still completely over the top, and loud and shiny lights and flashing things and millions of people. But it is kind of funny that, like, if you do enough of these, you can start to say, “Oh well, I don't know, it's still felt… a little bit less [laugh] for some reason.”Corey: Yeah, just a smidgen. Yeah. Pete, it is always a pleasure to get your take on re:Invent and see what you saw that I didn't and vice versa. And same time next year, same place?Pete: Yeah. I mean, like I said, one of my favorite parts of re:Invent is, you know, we always try to schedule, like, an end-of-event breakfast when we're both just supremely exhausted. Most of us don't even have a voice by the end. But just being able to, like, catch up and do our quick little recap and then obviously to be able to get on a podcast and talk about it is always a lot of fun. And yeah, thanks again for having me. This is—it's always, it's always a blast.Corey: It really is. Pete Cheslock, Head of Growth and Community at AppMap. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and then put something insulting about me in the next keynote because you probably work on that content team.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About “Matty”Matt Stratton is a Staff Developer Advocate at Pulumi, founder and co-host of the popular Arrested DevOps podcast, and the global chair of the DevOpsDays set of conferences.Matt has over 20 years of experience in IT operations and is a sought-after speaker internationally, presenting at Agile, DevOps, and cloud engineering focused events worldwide. Demonstrating his keen insight into the changing landscape of technology, he recently changed his license plate from DEVOPS to KUBECTL.He lives in Chicago and has three awesome kids, whom he loves just a little bit more than he loves Diet Coke. Matt is the keeper of the Thought Leaderboard for the DevOps Party Games online game show and you can find him on Twitter at @mattstratton.Links Referenced Pulumi: https://www.pulumi.com/ Arrested DevOps: https://www.arresteddevops.com/ 8bits.tv: https://8bits.tv Twitter: https://twitter.com/mattstratton LinkedIn: https://www.linkedin.com/in/mattstratton/ speaking.mattstratton.com: https://speaking.mattstratton.com twitch.tv/Pulumi: https://twitch.tv/Pulumi 8bit.tv: https://8bit.tv duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats V-U-L-T-R.com slash screaming.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Returning today for yet another round on the Screaming in the Cloud podcast is my dear friend, and hopefully yours as well, Matty Stratton. Since the last time we spoke, you've changed jobs, Mattie; you're now a staff developer advocate at Pulumi. I don't believe you were the last time you were on this show, but memory escapes me.Matty: You know, I was just wondering that myself, and I guess we'll have to go back to the archives.Corey: Yes, but that sounds like work, so we're going to roll with it anyway.Matty: Everyone who's listening, go do the homework for us. And, like, just tweet and let us know what my job was last time.Corey: And yell at us if we get it wrong, of course.Matty: Yell at us if we get it right.Corey: In the interest of being, well, I guess a little on the judgey side—because why not I tend to be good at that.Matty: I was hoping to be on the judgey side on this show.Corey: Oh, absolutely. You have a very strange career trajectory, in that—the companies you work for and how that winds up going back and forth. But when we first met, you were at Chef; and Chef, great company. And after that it was PagerDuty; great company.Matty: [laugh].Corey: And then it was IBM Hat, which I—was it Red Hat, was it IBM side?Matty: For me, it was Red Hat.Corey: So, it went from Chef, which is great, and a company that was doing a lot of things on the container side of the world became a thing and mutable infrastructure did sort of change Chef's business model. And then you went to PagerDuty, the wake-you-up-in-the-middle-of-the-night service named after some legacy technologies. And should be very direct in the popular consciousness, IBM views pagers as newfangled technology in some circles, in some areas, so it feels like you were traveling back in time a bit, again and again and again. On the federal side as well which, for excellent reasons, is not usually the absolute bow wave of innovation because you don't usually want your government doing that in some ways. And now you've leapfrogged into Pulumi, which is sort of the bleeding edge of the modern way we think about provisioning cloud infrastructure.It feels like it's a very interesting trajectory. Now, this is speaking as a complete outsider, I'm going to assume that's not how you view basically any characterization of any of those companies I've just named. How do you view it?Matty: You know, I don't know that I necessarily disagree with the way that you've put everything, but there's some nuance and some interesting stuff when it comes to that. So, I'm going to specifically talk about the Red Hat thing; why did I leave PagerDuty? And one of the interesting things is, I actually had an offer from Pulumi at the time that I took the job from Red Hat. So, it actually took me a year to come and work at Pulumi. And the little bit of the short answer is Red Hat backed up a big truck of money. And we all have a price.Corey: Yeah, the dulcet tones of a dump truck full of gold bricks emptying itself into your backyard, it's hard to say no to.Matty: The reason that I want to bring that up is that has nothing to do with specifically Red Hat the company versus other companies. It was the role. It was a sales-oriented role, so if you don't know, sales gets paid a lot of money and there's good reason. One of the reasons—again, if you don't work in sales, you don't necessarily know this—is, the last day of the quarter, you will have your VP of sales talking, he'll be like, “Corey, you are amazing. I love you. Look at this big deal you brought in.” Twenty-four hours later, “What have you done for me lately?”Corey: Mm-hm.Matty: That didn't matter, right? And I remember the CEO of PagerDuty—so Jen Tejada—at one of the sales kickoffs I was at, she said—you know, because salespeople, like, you might know this, like, the top sales reps in the company, they go on trips, they have all this stuff—and Jen said, you know, “I've got engineers here that are like, well, I don't understand.” It's like, “How come the salespeople get to go to Bermuda or do whatever?” And she's like, “Would you like your paycheck to change every quarter based upon specifically what you did and have the stress of what have you done all this stuff? No? Okay, cool. Then you can keep”—you know, there's a trade-off. So, the point of that was—Corey: And as your paycheck gets smaller, you're getting closer and closer to losing your job because a salesperson needs to perform to keep. It's very feast or famine. It's a heck of a role, and I have nothing but respect for people who can do it.Matty: And people can do it well. And I do feel like a lot of people don't understand how sales works, especially in a larger organization, and I think it's really important. So, one of the things that was interesting is we've all—I shouldn't say all, but many of us have worked in jobs that have some form of variable compensation, some kind of annual bonus. So, let's say for example, at x company I'm working at, they're like, “Mattie, your bonus is equal to 10% of your paycheck.” Well, the most it could be, generally speaking, it's like, let's say that your bonus would be, I'm just going to make up a number and say it's a $10,000 bonus.That's the most it could be, and that's if everything is amazing. Maybe I'll get a little more. Now, your commission, your what they call your on-target earnings and sales, they'll tell you a number and they'll say, “Okay, Corey, you're on-target earnings are, say $200,000.” And you're like, “Oh.” But whatever.The thing is, if you're only getting you're on-target earnings, you probably are needing to look for another job. So, you remember, like, we hear it differently, those of us that have done bonuses in a non-sales way. We're like, “But that's not a lot.” You're like, “No, but what they tell you your commission is, it's actually… it better end up being more or else you have trouble.” Anyway, point is—Corey: And in some cases, it could be a significant multiple of that number as well, for top performers.Matty: Absolutely.Corey: The upside is always interesting, and calculating out the nuances of the sales plan is always a challenge, speaking as a business owner. It is a very specific field that has a bunch of nuance to it. Something I learned very early on is that if you manage salespeople as if they were engineers, or manage engineers as if they were salespeople, you are going to have an absolutely terrible time.Matty: I think one of the things that, along those lines, I've have had conversations with people who work in different parts of technology, different parts of the business, who their long-term desire is to be a CEO, and I'm like, you really should go spend some time working in sales because most CEOs—again, this is blunt, but it's true—if you think about it, what is the area of the business that they pay the most attention to? And I don't mean, they don't care about the other stuff, but who is the person on the executive team that the CEO is mostly joined at the hip with, and it's your chief revenue officer, it's your head of sales because you have to understand that, you have to understand pipeline, how that—you have to understand a lot of things as a CEO, but if you don't know how sales works—it doesn't mean know how to sell but know the ideas behind it. I mean, you should know how to sell, but you know what I mean?Corey: Yeah, I think every CEO is selling. It is a sales job, whether that is selling the company to prospective employees, whether it is selling strategic partnerships, whether it's being brought in to help close strategic deals, et cetera, you're always selling in that role.Matty: That's a very good point. I should rephrase that, where I wasn't saying you don't need to know—Corey: CEO who has no idea how to sell [unintelligible 00:07:42] the fundamentals of—like, you put them in a meeting, and they wind up saying the wrong thing and pooching the deal, yeah, they're not CEO for very long.Matty: It's not just knowing how to sell, it's understanding how a sales process works. That's sort of the thing.Corey: I'll take it one step further beyond that, and that is that I believe that every professional is working in sales and is selling something, but not everyone's aware of it“. Well, I'm an engineer, and I don't do any sort of sales work.” Well, I hear about that from folks who are—“I have all these great ideas, but none of them ever get implemented.” Well, you're not doing an effective job of selling the idea. “I keep getting put up for promotion and not getting it,” or, “I'm not doing well in job interviews.” Or, “I'm trying to get a raise and it just isn't working for me.” And every job has elements of sales to it. I'd argue a lot of facets of modern life have sales elements to it.Matty: They do and I think the reason that people get hung out—I agree with you; I could not agree with you more. I have a talk I used to give called “The Five Love Languages of DevOps” but it was really a talk about effecting organizational change, and you have to be a salesperson, right? But I think we have this—and this is a much larger topic because it comes into how people always want to distance themselves from sales—we have this thing in our head that when we think of sales, we think of tricky people. Shysters, right? Someone that's trying to, like, pull a fast one on us, like the used car salesperson thing.And I'm like, that's not most salespeople. Like, salespeople want you to—because when we talk about learning how to sell, it's not learning how to trick somebody. It's actually learning about how to—I mean, here's the biggest thing. You want to know—we talk about DevOps all the time and stuff like that, you know, and empathy. You want to know one of the most important skills of a salesperson is? Freaking empathy.Because you need to be able to understand what your prospect—and that's if you've, you know, there's the book, The Challenger Sale, which like all business books can be summarized in a blog post, right, so you can just go read the blog post about The Challenger Sale; that'll tell you everything you need to know, but a good salesperson that's a challenger-style salesperson knows the customer better than they know themselves and knows there problems they might have that they're not aware of. And it's not because they're smarter; they have a different perspective. So, the same thing is true. So, to Corey's point, we're always selling. And even whether it's figuratively, like, conceptually—but I used to say when I was a Chef I said, the two best sales—most effective salespeople at Chef were Adam Jacob, the founder, and Nathan Harvey, the VP of community.Sales engineers are powerful because a customer will tell things to a sales engineer they won't tell the rep because they think the rep is trying to take advantage of them, which isn't true. Most important conversations that happen are on the walk from the front desk to the conference room. How many conversations would I have with the SRE, or whatever, who was the one who came to get me from reception, and we're just walking to the conference room. I learned so much there than in any other discovery session? You know, and then you use that to be—Corey: And there's not such thing as an easy sale either. And I think that gets overlooked a lot. Like, here at The Duckbill Group, if you bring us in on a consulting engagement to fix your AWS bill, you will turn a profit on that engagement. That has always been true. And we are quite literally selling money.It is effectively one of the easiest possible sales you can make; it is incredibly easy to calculate out what the ROI looks like on any of these things, and it's great, and we still have a full-on enterprise sales force because that is what it takes to wind up getting deals done when you're selling business-to-business. These are not selling t-shirts to the masses. It is a nuanced field, and honestly, when I'm interviewing people, one of the easiest ways for me to discount someone as a potential hire is that they start talking smack about sales because it is clear, first, they lack empathy, and secondly, they don't understand what sales does.Matty: One of the things that I think people who are not connected with it don't understand that again, back to Corey's point about because selling is hard, and selling internally is hard. So, this is the thing. So, you can have a champion inside your prospect who's, like, “I'm all about hiring Duckbill.” But they have to convince other people. So, what are salespeople really good at doing? They're really good at helping you build your business case to be able to get your thing that you want.Corey: How to turn your champion into an effective advocate for the thing that's going to make their job easier because they're not the person that signs off on it.Matty: And they're not the expert. Like, this used to happen when I was at Chef and I would have a customer who was like, “Okay.” They go and buy a bunch of licenses, and they're like, “Well, it didn't get deployed.” And we're like, “Well, how can we help you?” And they're like, “Well, no, it's just internal stuff. We got to convince people or whatever.”And I was like, “So, what you need to do is what you're telling me, what you need to do is sell Chef, right?” “Uh-huh.” There is nobody on this planet better at selling Chef than Chef. So, that's where that comes in because again, that's how everybody wins. So anyway, I went there because I was getting paid like a salesperson.Also, I one thing I wanted to touch on. So, you're right, usually, public sector is not seen as the most cutting edge. One of the things that's interesting at Red Hat, especially on the sales side—and friends of mine who are working on the commercial side may disagree with this, but it's generally not been true—what they call NAPS, so the North America Public Sector, I used to say I was a NAPS specialist, which sounded awesome. Because that was my title, I was NAPS specialist; I specialized in NAPS—is actually—Corey: Your status in the internal messaging system should always be sleeping at that point, why not?Matty: Sleeping. Yeah. But it's sort of known that actually the kind of emergent tech group and sales inside of the public sector, inside Red Hat, is very innovative compared to other ones. So, a lot of stuff was created there. So, it was we were doing something around a transformation office that wasn't being done in the same way anywhere else, so it was very exciting.So, I—also was the opportunity to go and work with people like Andrew Clay Shafer and John Willis and people that were—you know, it was all the people I was going to get to work with. So, that got me excited to be there. And then Covid happened, and I got news for you. Like, my job was to have challenging conversations with people about how they should do work differently. It's pretty easy to tune somebody out on the Zoom, it's a lot harder to tune somebody out when they're challenging you in a room.So, it was very hard to do this job during Covid, so our team really kind of disbanded towards the end of the year. I was really on the fence to join in the first place, and the person who was referring me to come work on the team who wanted to convince me said, you know, “What's holding you back?” And I said, “Well, it's not”—I said, “I really like developer advocacy. I like DevRel. That's not this job.” And he said, “Hey. Come try this for a year, and… if it turns out you didn't like it or wasn't for you, then go back and do DevRel.”And so that's sort of what happened. And I have seen though I am much happier in a smaller organization that's creating—you know, like, I like to feel my impact. I think everybody should spend some time in a large org because if you're going to be working with other people—right, you know what I mean—especially if you're a vendor, if you work on the vendor side like I do and stuff, Corey, you and I've talked before about background and doing developer advocacy, and I always say that, like, I do DevRel on easy mode because it's very easy for me to have empathy for my prospects and community because I did the job for 20 years. It's not impossible to be effective doing this job if you haven't literally done it. It's just that much harder. So, I [crosstalk 00:15:04]—Corey: It's a lot harder. And there's a credibility question and the rest. Yeah.Matty: I do this on easy mode. I can sit there and I can say, “Yes, I feel your pain. I literally did it for 20 years.”Corey: And you're at a point, too, let's be clear here, that you have a gravitas to you. I use you as my default example when I talk about, like, the expression of DevRel in that if you—like back when you were at PagerDuty, which I guess dates the reference a bit, but it was, okay. If you sit down and say you're doing on-call wrong, now I've been around this industry at that point 15 years or so, and I'm pretty sure I'm not. But if you're going to say that you have already got my attention in a constructive way, not in a, “Well, let me just tear this apart.” It's, no, no. I'm about to learn something by whatever it is you're about to say. And it's very hard to have that level of credibility without having done the role.Matty: That's true. Without doing it in that way. I mean, this is [crosstalk 00:15:59]—Corey: In the practitioner way of practicing the thing for which you are advocating. Like, someone telling me that I'm doing on-call wrong, who has never themselves been in a role where they themselves were on call is a little lacking in the authenticity department. It's not impossible and it can't be overcome.Matty: And you have to do it in a different way, right?Corey: Yes.Matty: And this goes back to another thing that I say a lot—my pithy Stratton quote is, “DevRel contains multitudes,” right? So, this is one of the things that we ran into, like, when we're building out our advocacy team at PagerDuty, it was seeing sort of my boss was an amazing dude and everything like that. I love him, but like, we don't scale horizontally. Our team was made up of enough of different kinds of people that, like, the way that I was able to do it because I had a certain experience, you couldn't expect that out of another one of my teammates because they actually had a different way of doing it that was just as effective, but in a different way because they have a different background, they have a different—so that's—Corey: And there's so many ways to do DevRel. Oh, yeah. Like, I'm going to call it my own bias here where when I think about DevRel, I think about it through a lens of the way I approach things, and when I give conference talks, of how I present myself, and the rest. And my approach would absolutely be aligned with what I just described, “So, you're doing AWS billing wrong.” And based upon who I am, and what I do, I can make that claim with some credibility.If I were relatively new to the industry and giving a talk about AWS billing, I would not lead that way because it does not present nearly as well, and it's going to call into question a whole bunch of skepticism. I would instead approach it as, “Here are some interesting facets about AWS billing that you may or may not be aware of.” There are different ways to approach it. Let's also be clear that it's not just conference talks; it can be blog posts, it can be documentation, it can be writing sample code, it can be Twitter, it could be TikTok of all things. There are so many ways to communicate with an audience, and your audience is wherever you happen to find them.Ideally, not in line at the Starbucks harassing the poor person in front who's just trying to order their coffee, but you know, as long as it's all consensual, talk to people who are interested in this stuff, wherever they happen to be.Matty: I think that's a really important statement you said there towards the end, which is meet people where they are, whether that's where you want them to be or not. And this comes up, it's interesting because one of the things—I'm a big believer in repurposing of content, and that's just partially because of effectiveness, but it's like, hey, if I give a talk, I should make that a blog post, I should make it a video, I should do a code example. And it's not so much because then I can hit all my OKRs with my boss.—I mean, that's part of it, right?—but not everybody likes the same kind of content.You know, there are people who really like videos, and there are people who are like, “I don't want to learn from a video at all.” And there's two ways you can approach that. One is you can say, “You're wrong. Videos are better. You should watch all my videos.” And take a guess about how well that's going to work with them getting your information or say, “I'll meet you where you are.”And I learned this even well before doing DevRel when I just thought about internal communication at an organization I was at when I was at Apartments.com and I was like, how do we get information? And you can't just say, like, well, we have this email we send to everybody. Well, everybody doesn't read email, right? So, it could be, maybe some people like RSS feeds, they want to capture it there. And the example I always gave was the most effective way that I ever saw that information was communicated inside our organization was signs in the restroom.Corey: Oh, yeah. That's a well-renowned way of doing it. That I think that Google pioneered this for a while. They had these all these things up about interesting things going on inside the—Matty: Oh—Corey: —company, about the way some systems worked—Matty: —I was at Google office and using the restroom, and I was standing there, and right in front of me with a whole good practice on cross-site scripting vulnerabilities. I guarantee they probably sent that email to everybody, it's probably been in meetings, and the people who saw it, [unintelligible 00:19:53] they saw it in the restroom.Corey: Now, of course, I'm sure they probably sell ads on those sheets, but okay.Matty: Yeah. You know, a little bit of that. When I was at Apartments.com, the floor that I worked on, the main restroom I used was a shared restroom with another office, which meant corporate never put anything up in there, and there was actually a fair amount of stuff that I didn't know about because I ignored it everywhere else and [unintelligible 00:20:14] anyway. So, the point is, back—if you will do work in person, which who is doing that anymore and why bother?—your most effective way to communicate. So, if you can figure out how to do DevRel in signs in a restroom at a conference—ohh, conferences should sell sponsorship of restroom signs.Corey: The jokes write themselves and almost certainly violate the code of conduct of at least four different [unintelligible 00:20:38], but it works. It works.Matty: [laugh]. We'll take those to Twitter.Corey: You've been around the industry for a while. You are one of the cohosts of the Arrested DevOps podcast; you've been instrumental in organizing a number of DevOps Days… or Devs-Ops days, however you want to mis-pluralize that is fine by me; roll with it. Ant—Matty: We argue more about the capitalization than the pluralization.Corey: Very fair. I want to talk to you a little bit of how the DevOps movement slash community slash role has evolved. For a long time now, it's been, “Great. So, where are the DevOps people sitting?” And then when you hear the shouted response of, “It's not a job. It's a culture,” good work. You found them. Now, you can go talk to them and all. What has changed over the past few years in the world of DevOps?Matty: So, I am fond of saying you can't buy DevOps, but I can sell it to you.Corey: Oh, absolutely. You're an exemplary DevOps salesman.Matty: Yeah. So, what happened? When we think back across the decade-plus, you know, back since 2009, one of the things I think that's interesting is, when we look at things like DevSecOps, or the other portmanteaus that are being created. It's a little bit like that meme, right, with the astronaut: “Wait. You mean, it's been DevSecOps all along?” You know, it's, “Yes, always has.”That's the thing. Like, for those who don't know, Andrew Clay Shafer is best known as coining the term. And I love Andrew, but wow, is it the worst name in the world for what we're talking about. Because it makes us all think that it's only about development and operations. And it's always been about cross-functional across all of those things. And if it helps us to give it a different name, great.Corey: It's replacing dysfunction with cross-function.Matty: Yes. There we go. That's DevOps right there. That's the best definition of DevOps I've heard. You heard it here.Corey: That one coins a phrase, in case you wondered.Matty: So, we still use the term CALMS to say what is about: It's about Culture, Automation, Lean, Measurement, and Sharing. That's held up for a reason. For something that was scrawled on a napkin in 2010, there's a reason we still talk that way. It sounds like we talk about culture more than anything else, and it's not because it's more important. It's because it's the one that we have to scream from the rooftops.You don't have to convince engineers to play with automation tools; they're going to do it. That's fine, right? So, they're all equal. Now, that said, what's changed is we have definitely found DevOps to feel a lot more that it's about automation. It's about the technology. We've veered away from the people to your statement about, like, “Oh, it's a culture, not a ti”—well, it's all of these things.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: Well, one thing I do want to call out because the whole point of having you on the show, of course, is to embarrass you with proof-positive, for example, that you are in fact, a good person at heart despite, you know, your dubious friendship with people like me, is we both used to be adamant about the idea of DevOps is not a role, not a job title, and we both stopped, but for different reasons. The reason that I stopped was that I took a job as the director of DevOps at a company because I was trying to solve about five or six different things that were important for me to negotiate for, and job title did not make the cut of impactful changes. You had a far less self-serving reason for no longer picking that particular fight. What was it?Matty: [laugh]. I do want to call out one of my favorite jokes which is not supposed to be gatekeeping, but it's making fun of Corey so it's okay—Corey: Hmm.Matty: —Nathan Harvey said years ago, and it was actually I think, intended as a shot at our friend Pete Cheslock, who also has had the title of director of DevOps, which said, “The only DevOps tool is a person that calls themselves director of DevOps.”Corey: Oh, absolutely. It's super lucrative. I was really insulted by that and cried all the way to the bank.Matty: Uh-huh. Now, I'll tell you there's two reasons that I've changed my tune on—you know, I used to say it's not a tool, title, or team. I still will agree that it's not a tool. The title and team—and the reason for that is twofold, and neither of which are self-serving other than I don't want people to think I'm a jerk. The first reason that deviated me from a little bit was again, to go back to your friend and mine, Pete Cheslock, he gave a talk, I don't remember where it was, but he made the point where he said, “You look at it, the title ‘DevOps engineer' is a 30 to 35% pay bump, so it's like, I don't care what you call yourself. Go get paid.” So, that's that.Corey: Yes.Matty: So, first of all, I was like cool—Corey: J. Paul Reed did a whole talk-pay thing that shined a light on that.Matty: Absolutely. The one that I think is more empathetic and probably was… is maybe a little more important—or equally so—Ian Coldwater has pointed out before, and this really resonated with me, is that when we get on Twitter and are like, “Oh, my God. DevOps engineer is not a real title, blah, blah, blah.” The people that hear that are the people who have that title. They did not give themselves that title. It's very exclusionary, and all that will happen out of that is it doesn't eff—Corey: “I'm going to go quit my job and not be able to make rent this month.” “Why?” “Because Twitter said that my job title was bad.”Matty: Yeah.Corey: All the reasons to quit a job, I promise you job title is not one of them. Unless it is something horrifying, as into the territory of discriminating or belittling. There are always exceptions to every rule, but by and large, “That's a ridiculous job title,” is not the reason to quit a job. Says the self-proclaimed chief cloud economist.Matty: Totally yeah. I mean, like, you know what is very similar? There's a meme about, like, every time people want to make fun of a political figure or something and they'll make fun of them being overweight, or any kind of thing, and the meme is like, the only people who hear that are your friends that have a similar condition, not the actual person you're making fun of, so all you're doing there is hurting people who… so that's a similar thing.Now, I will say—and I think you and I might disagree about this a little bit, so that'll be fine—Corey: I hope so.Matty: So, when I hear—and actually the title doesn't do this, for me; it's actually very specifically a DevOps team. When people say, “We have a DevOps team.” This is not a perfect analogy when I say it's a code smell; I call it an organizational smell. And what I mean by that—it's not as bad as a code smell—what it does is it makes me ask more questions. If it's relevant to me to ask questions. It might be none of my damn business. If you tweet that I'm on the DevOps team, I'm not going to come into your mentions and start questioning your existence, but—Corey: Oh please, I have way better personal attacks than that.Matty: Oh, yeah. But if I'm working with you and we're working on that, or we're having a conversation, and it comes up that you have a team called DevOps Team, I'm going to ask questions because that could be, okay or it could be, [sigh] I want to use the word dangerous lightly; it's not, but like, counter-effective. And the reason for that is if the DevOps team is the one who does all your automation and you haven't really enabled other squads and all you've done is move a silo around, doesn't make you a bad person, but that's not the most effective way you could be. So, it makes me start to ask questions, right? But sometimes DevOps teams are people who lead in the organization, they are empowerment teams, maybe they run dojo, maybe they are subject matter experts that help.As long as there are good bridges still being built, it's not bad, right? So, it just—again, it raises questions. It's not inherently wrong. I am sure that… Pulumi where wo—actually, many of the tools I've worked with have been called DevOps tools; I will still tell you there's no tool that gives you DevOps, right? You can't—Corey: But when other people—like, read as ‘buyers'—refer to you as the ‘DevOps tool company,' well, you can be right or you can make a sale, in some cases.Matty: [laugh]. Yeah, I'm not going to tell you—Corey: On some level, you have to meet people where they are, and this is a part of that. I say that in full sincerity. Same story with the idea of culture. I hear this question all the time, “How do we wind up making all of our engineers aware of AWS billing issues?” And to a point, you should have understanding that when you turn something on it runs forever, bigger things cost more than smaller things, but the knowledge fits on an index card.You shouldn't have every engineer wanting to—or needing to—become deep experts in this space. Having a centralized team that specializes in that, at a sufficient level of org size and maturity, makes an awful lot of sense, and they can float around. But yeah, having the AWS bill team, in some cases is the right answer and others it's the complete wrong answer, and it really does depend. I think the way that we solve this problem, authoritatively, is a way that neither you nor I can argue with it because the only source for authoritative DevOps answers is from the source itself, and that is, of course, Emily Freeman, whose treatise on the subject, DevOps for Dummies, despite the weird title, is absolutely fantastic work that gives insight into all of this. And are you prepared to tell her she's wrong? Because I'm certainly not.Matty: Well, there are plenty of people who will. As we know.Corey: Yes. And we call them shitheads if we're being perfectly honest with you.Matty: Yeah. [laugh].Corey: The internet what a ple—no, Emily is an absolute treasure in the space and I'm continuing to watch her meteoric rise with nothing other than pure admiration. It is just spectacular to see her succeed.Matty: I could not agree more. This is something I struggle with a little bit. I don't think Emily would mind me saying it this way. This is the thing where you don't want to sound condescending, but I always love when I look at people and it's not—it's going to come off a little bit about, like, “I knew them when,” and it's not like I was a Corey Quinn fan before he went pop, but I love to see and remember where we all came from, and it's true of myself and it's true of other people, but that's one of my favorite things is I love to see my friends succeed.Corey, I love to see what you've done. Like, I think back to when we knew each other. I'm not saying you weren't successful, but it's funny, this [unintelligible 00:30:08] sounds a little condescending to be like, oh, I'm so proud of you, but I am. And I'm impressed. It's great to see.And Emily's another example. Like, I remember when I first met Emily, and not like I was any big deal, either, but it's like, everybody comes from somewhere, right? Like Jacquie Grindrod who just recently left Hashi, I remember when she started to get into DevRel and I was talking to her because she's like, “I may be thinking I want to do this thing.” And you look and you see these people. And it's not supposed to be like, “Oh, I remember when you were like the cute little baby DevRel.” It's not like that.And it's like, it's just impressive to see—and not even impressive. It's you like to see people who do good work and have a good heart and want to help people grow and be successful. And I'll tell you something, here—we're going to get real for a second—you can be jealous of them. It's okay. And I'm going to be honest, there are times that—Emily and Corey are both good friends of mine, and there are times that I'm like, “Wow. I'm a little jealous of you. Sometimes I'm a lot jealous of you. Sometimes I'm not at all.” So, I'm telling everybody, it's okay to be jealous. [laugh].Corey: I agree with the sentiment that I changed the word ‘envious' because envy is one of those, like—Matty: Okay.Corey: —“I want that, too,” whereas jealousy is a lot more a shade of, “I want to have it and I don't want them to.” And I don't believe that's the direction you're heading in. [laugh].Matty: No. Thank you. No, you're exactly right. Envy is the better one yeah because it's never—Corey: Now, I recently learned the distinction there by getting very wrong and saying things I didn't intend to imply, which is why I bring it up. Again, let my mistake be something others can learn from. Sometimes the best purpose I can serve in this industry is as a counter-example.Matty: Example. I was going to say, you know, just for everybody, I remember at the beginning, you know, Corey said, “Maybe we'll learn something.” I'm like, I guess that's what we learned [laugh] is the difference between envy and jealousy.Corey: Yeah.Matty: [unintelligible 00:31:50] gotta say, you know, it took us half an hour to get there. But you know.Corey: No no. And I appreciate your friendship throughout the years. Like, you were one of those people that has been something of a guiding star, where it's, sometimes I get it right, sometimes I get it wrong, and you've always been someone who has been very willing to share which side of the divide you think I'm on with anything that I've done. And for lack of a better term, you knew me before I basically bought ink by the barrel. And back when I was just the conference speaker that had to follow one of your ridiculous talks, like, “Oh, God. Those are big shoes to fill. I'd better learn how to give a conference talk.” So, most of what I become is your fault. But I do want to thank you for your guidance over the years on these things.Matty: Can we tell the real story about how I claim ownership of The Duckbill Group?Corey: By all means, take it away.Matty: Oh, okay. So, [[laugh]] I honestly still think that I should have a part ownership in The Duckbill Group because for those of you who don't know, Corey mentioned that I had worked at PagerDuty, and actually that job came down between the two of us and Corey didn't get it. And then went and started his own company and became famous and amazing. So really, it's because of me is what I'm trying to get at. I—Corey: To be fair, they made the right hire. Which one of us do you think makes the better employee, let's be very clear?Matty: [laugh].Corey: And yeah, I am thrilled to deal in you in on ownership of The Duckbill Group because the way we're structured, you cannot have ownership without also assuming liability. So yeah—Matty: [laugh].Corey: I would love to dump legal responsibility for my shenanigans on someone else. Come on in. Yeah, there's always a cutting edge to everything else. But no, you're right. I always wonder what would have happened if that decision had gone differently.And I'm very glad it played out the way that it did. You were the right hire for the company in a way that I never would have been. But I would have given it a good try for a while before they begrudgingly had to fire me or I sensed the axe was coming and left on my own. That is the nature of me as an employee. You have a very different perspective because you're good at things that I'm terrible at.Matty: And vice versa. It was interesting. You just talked about, like, how would things go different? So I—yesterday—just recorded—I don't know when it's going to come out—I was on a podcast called 8 Bits—so it's 8bits.tv—and it's really a show about people's journey through tech.And what was interesting that came out of that conversation was, first of all, how much of how I got to where I am is because of spite. Which you're going to have to go back and listen to the episode to hear the whole story of all the spite. But we did talk about, like, those junction points that happen that seem innocuous. And it's like, I made this one choice that wasn't even necessarily a choice and you follow all the forking logic that gets you to, Corey, you and I are sitting here on a podcast right now. How many decisions that weren't even decisions? There's the alternate universe where this doesn't happen where this doesn't exist, right?Corey: It's weird how this stuff all works. Years before I'd met either one of you, you videotaped my wife's law school musical and burned it to CD. We found that out when you were here over dinner one night.Matty: That was my favorite thing.Corey: It was surreal.Matty: Yeah, I was at dinner with Corey and his wife and we got into a conversation about that she had gone to law school in Chicago. And I was like, “Oh, funny thing. Like, I produced the video of the law school mu”—and she was like, “Wait, what was that?” And I couldn't even remember. I had to, like, dig back into, like, an old blog post. And was that and then yeah, and Bethany, like—Corey: She walks into the other room and comes back with a DVD that you burned, your handwriting on it.Matty: Yeah.Corey: Yeah.Matty: Yeah, pretty much. Yeah. The world is small. Be nice to everybody.Corey: It never hurts. I want to thank you for taking time out of your day to basically tell stories once again. It's always good to talk to you. If people want to learn more about who you are, what you're up to, where's the best place they can find you.Matty: So, really the best place is Twitter. You know, so I'm at @mattstratton on Twitter. If you're not a Twitter person, that's okay. LinkedIn is not great for fi—I don't always remember to post stuff there. If you want to know about upcoming, you know, so if you go to speaking.mattstratton.com, that has all my previous talks, my upcoming talks, and things as hopefully we'll have more and more of that.And yeah, and every week, I stream on twitch.tv/Pulumi on Thursdays. And it's not webinars, it's not slick demos, it's just me screwing around and sometimes having fun people on, and sometimes just proving how little I know about coding. So yeah, good times. Thank you for having me on, again, Corey. It's always fun.Corey: Of course. Links to all that's going on in the [show notes 00:36:20]. And as always, it's a pleasure.Matty: Also, I will say, Corey, I'll give you the link to that 8bit.tv, if you want to put that in the [show notes 00:36:28]—Corey: Oh, of course, we will.Matty: —if people want to go and find that. Because I think it's similar, connected to what we talked about.Corey: Good. I look forward to listening to it myself. Mattie Stratton, staff developer advocate at Pulumi. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a long angry comment detailing that DevOps is in fact a role and here's what it means, and then go ahead and describe a sysadmin.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About PetePete does many startup things at Allma. Links: Last Tweet in AWS: https://lasttweetinaws.com Twitter: https://twitter.com/petecheslock LinkedIn: https://www.linkedin.com/in/petecheslock/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I am joined—as is tradition, for a post re:Invent wrap up, a month or so later, once everything is time to settle—by my friend and yours, Pete Cheslock. Pete, how are you?Pete: Hi, I'm doing fantastic. New year; new me. That's what I'm going with.Corey: That's the problem. I keep hoping for that, but every time I turn around, it's still me. And you know, honestly, I wouldn't wish that on anyone.Pete: Exactly. [laugh]. I wouldn't wish you on me either. But somehow I keep coming back for this.Corey: So, in two-thousand twenty—or twenty-twenty, as the children say—re:Invent was fully virtual. And that felt weird. Then re:Invent 2021 was a hybrid event which, let's be serious here, is not really those things. They had a crappy online thing and then a differently crappy thing in person. But it didn't feel real to me because you weren't there.That is part of the re:Invent tradition. There's a midnight madness thing, there's a keynote where they announce a bunch of nonsense, and then Pete and I go and have brunch on the last day of re:Invent and decompress, and more or less talk smack about everything that crosses our minds. And you weren't there this year. I had to backfill you with Tim Banks. You know, the person that I backfield you with here at The Duckbill Group as a principal cloud economist.Pete: You know, you got a great upgrade in hot takes, I feel like, with Tim.Corey: And other ways, too, but it's rude of me to say that to you directly. So yeah, his hot takes are spectacular. He was going to be doing this with me, except you cannot mess with tradition. You really can't.Pete: Yeah. I'm trying to think how many—is this third year? It's at least three.Corey: Third or fourth.Pete: Yeah, it's at least three. Yeah, it was, I don't want to say I was sad to not be there because, with everything going on, it's still weird out there. But I am always—I'm just that weird person who actually likes re:Invent, but not for I feel like the reasons people think. Again, I'm such an extroverted-type person, that it's so great to have this, like, serendipity to re:Invent. The people that you run into and the conversations that you have, and prior—like in 2019, I think was a great example because that was the last one I had gone to—you know, having so many conversations so quickly because everyone is there, right? It's like this magnet that attracts technologists, and venture capital, and product builders, and all this other stuff. And it's all compressed into, like, you know, that five-day span, I think is the biggest part that makes so great.Corey: The fear in people's eyes when they see me. And it was fun; I had a pair of masks with me. One of them was a standard mask, and no one recognizes anyone because, masks, and the other was a printout of my ridiculous face, which was horrifyingly uncanny, but also made it very easy for people to identify me. And depending upon how social I was feeling, I would wear one or the other, and it worked flawlessly. That was worth doing. They really managed to thread the needle, as well, before Omicron hit, but after the horrors of last year. So, [unintelligible 00:03:00]—Pete: It really—Corey: —if it were going on right now, it would not be going on right now.Pete: Yeah. I talk about really—yeah—really just hitting it timing-wise. Like, not that they could have planned for any of this, but like, as things were kind of not too crazy and before they got all crazy again, it feels like wow, like, you know, they really couldn't have done the event at any other time. And it's like, purely due to luck. I mean, absolute one hundred percent.Corey: That's the amazing power of frugality. Because the reason is then is it's the week after Thanksgiving every year when everything is dirt cheap. And, you know, if there's one thing that I one-point-seve—sorry, their stock's in the toilet—a $1.6 trillion company is very concerned about, it is saving money at every opportunity.Pete: Well, the one thing that was most curious about—so I was at the first re:Invent in-what—2012 I think it was, and there was—it was quaint, right?—there was 4000 people there, I want to say. It was in the thousands of people. Now granted, still a big conference, but it was in the Sands Convention Center. It was in that giant room, the same number of people, were you know, people's booths were like tables, like, eight-by-ten tables, right? [laugh].It had almost a DevOpsDays feel to it. And I was kind of curious if this one had any of those feelings. Like, did it evoke it being more quaint and personable, or was it just as soulless as it probably has been in recent years?Corey: This was fairly soulless because they reduced the footprint of the event. They dropped from two expo halls down to one, they cut the number of venues, but they still had what felt like 20,000 people or something there. It was still crowded, it was still packed. And I've done some diligent follow-ups afterwards, and there have been very few cases of Covid that came out of it. I quarantined for a week in a hotel, so I don't come back and kill my young kids for the wrong reasons.And that went—that was sort of like the worst part of it on some level, where it's like great. Now I could sit alone at a hotel and do some catch-up and all the rest, but all right I'd kind of like to go home. I'm not used to being on the road that much.Pete: Yeah, I think we're all a little bit out of practice. You know, I haven't been on a plane in years. I mean, the travel I've done more recently has been in my car from point A to point B. Like, direct, you know, thing. Actually, a good friend of mine who's not in technology at all had to travel for business, and, you know, he also has young kids who are under five, so he when he got back, he actually hid in a room in their house and quarantine himself in the room. But they—I thought, this is kind of funny—they never told the kids he was home. Because they knew that like—Corey: So, they just thought the house was haunted?Pete: [laugh].Corey: Like, “Don't go in the west wing,” sort of level of nonsense. That is kind of amazing.Pete: Honestly, like, we were hanging out with the family because they're our neighbors. And it was like, “Oh, yeah, like, he's in the guest room right now.” Kids have no idea. [laugh]. I'm like, “Oh, my God.” I'm like, I can't even imagine. Yeah.Corey: So, let's talk a little bit about the releases of re:Invent. And I'm going to lead up with something that may seem uncharitable, but I don't think it necessarily is. There weren't the usual torrent of new releases for ridiculous nonsense in the same way that there have been previously. There was no, this service talks to satellites in space. I mean, sure, there was some IoT stuff to manage fleets of cars, and giant piles of robots, and cool, I don't have those particular problems; I'm trying to run a website over here.So okay, great. There were enhancements to a number of different services that were in many cases appreciated, in other cases, irrelevant. Werner said in his keynote, that it was about focusing on primitives this year. And, “Why do we have so many services? It's because you asked for it… as customers.”Pete: [laugh]. Yeah, you asked for it.Corey: What have you been asking for, Pete? Because I know what I've been asking for and it wasn't that. [laugh].Pete: It's amazing to see a company continually say yes to everything, and somehow, despite their best efforts, be successful at doing it. No other company could do that. Imagine any other software technology business out there that just builds everything the customers ask for. Like from a product management business standpoint, that is, like, rule 101 is, “Listen to your customers, but don't say yes to everything.” Like, you can't do everything.Corey: Most companies can't navigate the transition between offering the same software in the Cloud and on a customer facility. So, it's like, “Ooh, an on-prem version, I don't know, that almost broke the company the last time we tried it.” Whereas you have Amazon whose product strategy is, “Yes,” being able to put together a whole bunch of things. I also will challenge the assertion that it's the primitives that customers want. They don't want to build a data center out of popsicle sticks themselves. They want to get something that solves a problem.And this has been a long-term realization for me. I used to work at Media Temple as a senior systems engineer running WordPress at extremely large scale. My websites now run on WordPress, and I have the good sense to pay WP Engine to handle it for me, instead of doing it myself because it's not the most productive use of my time. I want things higher up the stack. I assure you I pay more to WP Engine than it would cost me to run these things myself from an infrastructure point of view, but not in terms of my time.What I see sometimes as the worst of all worlds is that AWS is trying to charge for that value-added pricing without adding the value that goes along with it because you still got to build a lot of this stuff yourself. It's still a very janky experience, you're reduced to googling random blog posts to figure out how this thing is supposed to work, and the best documentation comes from externally. Whereas with a company that's built around offering solutions like this, great. In the fullness of time, I really suspect that if this doesn't change, their customers are going to just be those people who build solutions out of these things. And let those companies capture the up-the-stack margin. Which I have no problem with. But they do because Amazon is a company that lies awake at night actively worrying that someone, somewhere, who isn't them might possibly be making money somehow.Pete: I think MongoDB is a perfect example of—like, look at their stock price over the last whatever, years. Like, they, I feel like everyone called for the death of MongoDB every time Amazon came out with their new things, yet, they're still a multi-billion dollar company because I can just—give me an API endpoint and you scale the database. There's is—Corey: Look at all the high-profile hires that Mongo was making out of AWS, and I can't shake the feeling they're sitting there going, “Yeah, who's losing important things out of production now?” It's, everyone is exodus-ing there. I did one of those ridiculous graphics of the naming all the people that went over there, and in—with the hurricane evacuation traffic picture, and there's one car going the other way that I just labeled with, “Re:Invent sponsorship check,” because yeah, they have a top tier sponsorship and it was great. I've got to say I've been pretty down on MongoDB for a while, for a variety of excellent reasons based upon, more or less, how they treated customers who were in pain. And I'd mostly written it off.I don't do that anymore. Not because I inherently believe the technology has changed, though I'm told it has, but by the number of people who I deeply respect who are going over there and telling me, no, no, this is good. Congratulations. I have often said you cannot buy authenticity, and I don't think that they are, but the people who are working there, I do not believe that these people are, “Yeah, well, you bought my opinion. You can buy their attention, not their opinion.” If someone changes their opinion, based upon where they work, I kind of question everything they're telling me is, like, “Oh, you're just here to sell something you don't believe in? Welcome aboard.”Pete: Right. Yeah, there's an interview question I like to ask, which is, “What's something that you used to believe in very strongly that you've more recently changed your mind on?” And out of politeness because usually throws people back a little bit, and they're like, “Oh, wow. Like, let me think about that.” And I'm like, “Okay, while you think about that I want to give you mine.”Which is in the past, my strongly held belief was we had to run everything ourselves. “You own your availability,” was the line. “No, I'm not buying Datadog. I can build my own metric stack just fine, thank you very much.” Like, “No, I'm not going to use these outsourced load balancers or databases because I need to own my availability.”And what I realized is that all of those decisions lead to actually delivering and focusing on things that were not the core product. And so now, like, I've really flipped 180, that, if any—anything that you're building that does not directly relate to the core product, i.e. How your business makes money, should one hundred percent be outsourced to an expert that is better than you. Mongo knows how to run Mongo better than you.Corey: “What does your company do?” “Oh, we handle expense reports.” “Oh, what are you working on this month?” “I'm building a load balancer.” It's like that doesn't add the value. Don't do that.Pete: Right. Exactly. And so it's so interesting, I think, to hear Werner say that, you know, we're just building primitives, and you asked for this. And I think that concept maybe would work years ago, when you had a lot of builders who needed tools, but I don't think we have any, like, we don't have as many builders as before. Like, I think we have people who need more complete solutions. And that's probably why all these businesses are being super successful against Amazon.Corey: I'm wondering if it comes down to a cloud economic story, specifically that my cloud bill is always going to be variable and it's difficult to predict, whereas if I just use EC2 instances, and I build load balancers or whatnot, myself, well, yeah, it's a lot more work, but I can predict accurately what my staff compensation costs are more effectively, that I can predict what a CapEx charge would be or what the AWS bill is going to be. I'm wondering if that might in some way shape it?Pete: Well, I feel like the how people get better in managing their costs, right, you'll eventually move to a world where, like, “Yep, okay, first, we turned off waste,” right? Like, step one is waste. Step two is, like, understanding your spend better to optimize but, like, step three, like, the galaxy brain meme of Amazon cost stuff is all, like, unit economics stuff, where trying to better understand the actual cost deliver an actual feature. And yeah, I think that actually gets really hard when you give—kind of spread your product across, like, a slew of services that have varying levels of costs, varying levels of tagging, so you can attribute it. Like, it's really hard. Honestly, it's pretty easy if I have 1000 EC2 servers with very specific tags, I can very easily figure out what it costs to deliver product. But if I have—Corey: Yeah, if I have Corey build it, I know what Corey is going to cost, and I know how many servers he's going to use. Great, if I have Pete it, Pete's good at things, it'll cut that server bill in half because he actually knows how to wind up being efficient with things. Okay, great. You can start calculating things out that way. I don't think that's an intentional choice that companies are making, but I feel like that might be a natural outgrowth of it.Pete: Yeah. And there's still I think a lot of the, like, old school mentality of, like, the, “Not invented here,” the, “We have to own our availability.” You can still own your availability by using these other vendors. And honestly, it's really heartening to see so many companies realize that and realize that I don't need to get everything from Amazon. And honestly, like, in some things, like I look at a cloud Amazon bill, and I think to myself, it would be easier if you just did everything from Amazon versus having these ten other vendors, but those ten other vendors are going to be a lot better at running the product that they build, right, that as a service, then you probably will be running it yourself. Or even Amazon's, like, you know, interpretation of that product.Corey: A few other things that came out that I thought were interesting, at least the direction they're going in. The changes to S3 intelligent tiering are great, with instant retrieval on Glacier. I feel like that honestly was—they talk a good story, but I feel like that was competitive response to Google offering the same thing. That smacks of a large company with its use case saying, “You got two choices here.” And they're like, “Well, okay. Crap. We're going to build it then.”Or alternately, they're looking at the changes that they're making to intelligent tiering, they're now shifting that to being the default that as far as recommendations go. There are a couple of drawbacks to it, but not many, and it's getting easier now to not have the mental overhead of trying to figure out exactly what your lifecycle policies are. Yeah, there are some corner cases where, okay, if I adjust this just so, then I could save 10% on that monitoring fee or whatnot. Yeah, but look how much work that's going to take you to curate and make sure that you're not doing something silly. That feels like it is such an in the margins issue. It's like, “How much data you're storing?” “Four exabytes.” Okay, yeah. You probably want some people doing exactly that, but that's not most of us.Pete: Right. Well, there's absolutely savings to be had. Like, if I had an exabyte of data on S3—which there are a lot of people who have that level of data—then it would make sense for me to have an engineering team whose sole purpose is purely an optimizing our data lifecycle for that data. Until a point, right? Until you've optimized the 80%, basically. You optimize the first 80, that's probably, air-quote, “Easy.” The last 20 is going to be incredibly hard, maybe you never even do that.But at lower levels of scale, I don't think the economics actually work out to have a team managing your data lifecycle of S3. But the fact that now AWS can largely do it for you in the background—now, there's so many things you have to think about and, like, you know, understand even what your data is there because, like, not all data is the same. And since S3 is basically like a big giant database you can query, you got to really think about some of that stuff. But honestly, what I—I don't know if—I have no idea if this is even be worked on, but what I would love to see—you know, hashtag #AWSwishlist—is, now we have countless tiers of EBS volumes, EBS volumes that can be dynamically modified without touching, you know, the physical host. Meaning with an API call, you can change from the gp2 to gp3, or io whatever, right?Corey: Or back again if it doesn't pan out.Pete: Or back again, right? And so for companies with large amounts of spend, you know, economics makes sense that you should have a team that is analyzing your volumes usage and modifying that daily, right? Like, you could modify that daily, and I don't know if there's anyone out there that's actually doing it at that level. And they probably should. Like, if you got millions of dollars in EBS, like, there's legit savings that you're probably leaving on the table without doing that. But that's what I'm waiting for Amazon to do for me, right? I want intelligent tiering for EBS because if you're telling me I can API call and you'll move my data and make that better, make that [crosstalk 00:17:46] better [crosstalk 00:17:47]—Corey: Yeah it could be like their auto-scaling for DynamoDB, for example. Gives you the capacity you need 20 minutes after you needed it. But fine, whatever because if I can schedule stuff like that, great, I know what time of day, the runs are going to kick off that beat up the disks. I know when end-of-month reporting fires off. I know what my usage pattern is going to be, by and large.Yeah, part of the problem too, is that I look at this stuff, and I get excited about it with the intelligent tiering… at The Duckbill Group we've got a few hundred S3 buckets lurking around. I'm thinking, “All right, I've got to go through and do some changes on this and implement all of that.” Our S3 bill's something like 50 bucks a month or something ridiculous like that. It's a no, that really isn't a thing. Like, I have a screenshot bucket that I have an app installed—I think called Dropshare—that hooks up to anytime I drag—I hit a shortcut, I drag with the mouse to select whatever I want and boom, it's up there and the URL is not copied to my clipboard, I can paste that wherever I want.And I'm thinking like, yeah, there's no cleanup on that. There's no lifecycle policy that's turning into anything. I should really go back and age some of it out and do the rest and start doing some lifecycle management. It—I've been using this thing for years and I think it's now a whopping, what, 20 cents a month for that bucket. It's—I just don't—Pete: [laugh].Corey: —I just don't care, other than voice in the back of my mind, “That's an unbounded growth problem.” Cool. When it hits 20 bucks a month, then I'll consider it. But until then I just don't. It does not matter.Pete: Yeah, I think yeah, scale changes everything. Start adding some zeros and percentages turned into meaningful numbers. And honestly, back on the EBS thing, the one thing that really changed my perspective of EBS, in general, is—especially coming from the early days, right? One terabyte volume, it was a hard drive in a thing. It was a virtual LUN on a SAN somewhere, probably.Nowadays, and even, like, many years after those original EBS volumes, like all the limits you get in EBS, those are actually artificial limits, right? If you're like, “My EBS volume is too slow,” it's not because, like, the hard drive it's on is too slow. That's an artificial limit that is likely put in place due to your volume choice. And so, like, once you realize that in your head, then your concept of how you store data on EBS should change dramatically.Corey: Oh, AWS had a blog post recently talking about, like, with io2 and the limits and everything, and there was architecture thinking, okay. “So, let's say this is insufficient and the quarter-million IOPS a second that you're able to get is not there.” And I'm sitting there thinking, “That is just ludicrous data volume and data interactivity model.” And it's one of those, like, I'm sitting here trying to think about, like, I haven't had to deal with a problem like that decade, just because it's, “Huh. Turns out getting these one thing that's super fast is kind of expensive.” If you paralyze it out, that's usually the right answer, and that's how the internet is mostly evolved. But there are use cases for which that doesn't work, and I'm excited to see it. I don't want to pay for it in my view, but it's nice to see it.Pete: Yeah, it's kind of fun to go into the Amazon calculator and price out one of the, like, io2 volumes and, like, maxed out. It's like, I don't know, like $50,000 a month or a hun—like, it's some just absolutely absurd number. But the beauty of it is that if you needed that value for an hour to run some intensive data processing task, you can have it for an hour and then just kill it when you're done, right? Like, that is what is most impressive.Corey: I copied 130 gigs of data to an EFS volume, which was—[unintelligible 00:21:05] EFS has gone from “This is a piece of junk,” to one of my favorite services. It really is, just because of its utility and different ways of doing things. I didn't have the foresight, just use a second EFS volume for this. So, I was unzipping a whole bunch of small files onto it. Great.It took a long time for me to go through it. All right, now that I'm done with that I want to clean all this up. My answer was to ultimately spin up a compute node and wind up running a whole bunch of—like, 400, simultaneous rm-rf on that long thing. And it was just, like, this feels foolish and dumb, but here we are. And I'm looking at the stats on it because the instance was—all right, at that point, the load average [on the instance 00:21:41] was like 200, or something like that, and the EFS volume was like, “Ohh, wow, you're really churning on this. I'm now at, like, 5% of the limit.” Like, okay, great. It turns out I'm really bad at computers.Pete: Yeah, well, that's really the trick is, like, yeah, sure, you can have a quarter-million IOPS per second, but, like, what's going to break before you even hit that limit? Probably many other things.Corey: Oh, yeah. Like, feels like on some level if something gets to that point, it a misconfiguration somewhere. But honestly, that's the thing I find weirdest about the world in which we live is that at a small-scale—if I have a bill in my $5 a month shitposting account, great. If I screw something up and cost myself a couple hundred bucks in misconfiguration it's going to stand out. At large scale, it doesn't matter if—you're spending $50 million a year or $500 million a year on AWS and someone leaks your creds, and someone spins up a whole bunch of Bitcoin miners somewhere else, you're going to see that on your bill until they're mining basically all the Bitcoin. It just gets lost in the background.Pete: I'm waiting for those—I'm actually waiting for the next level of them to get smarter because maybe you have, like, an aggressive tagging system and you're monitoring for untagged instances, but the move here would be, first get the creds and query for, like, the most used tags and start applying those tags to your Bitcoin mining instances. My God, it'll take—Corey: Just clone a bunch of tags. Congratulations, you now have a second BI Elasticsearch cluster that you're running yourself. Good work.Pete: Yeah. Yeah, that people won't find that until someone comes along after the fact that. Like, “Why do we have two have these things?” And you're like—[laugh].Corey: “Must be a DR thing.”Pete: It's maxed-out CPU. Yeah, exactly.Corey: [laugh].Pete: Oh, the terrible ideas—please, please, hackers don't take are terrible ideas.Corey: I had a, kind of, whole thing I did on Twitter years ago, talking about how I would wind up using the AWS Marketplace for an embezzlement scheme. Namely, I would just wind up spinning up something that had, like, a five-cent an hour charge or whatnot on just, like, basically rebadge the CentOS Community AMI or whatnot. Great. And then write a blog post, not attached to me, that explains how to do a thing that I'm going to be doing in production in a week or two anyway. Like, “How to build an auto-scaling group,” and reference that AMI.Then if it ever comes out, like, “Wow, why are we having all these marketplace charges on this?” “I just followed the blog post like it said here.” And it's like, “Oh, okay. You're a dumbass. The end.”That's the way to do it. A month goes by and suddenly it came out that someone had done something similarly. They wound up rebadging these community things on the marketplace and charging big money for it, and I'm sitting there going like that was a joke. It wasn't a how-to. But yeah, every time I make these jokes, I worry someone's going to do it.Pete: “Welcome to large-scale fraud with Corey Quinn.”Corey: Oh, yeah, it's fraud at scale is really the important thing here.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: I still remember a year ago now at re:Invent 2021 was it, or was it 2020? Whatever they came out with, I want to say it wasn't gp3, or maybe it was, regardless, there was a new EBS volume type that came out that you were playing with to see how it worked and you experimented with it—Pete: Oh, yes.Corey: —and the next morning, you looked at the—I checked Slack and you're like well, my experiments yesterday cost us $5,000. And at first, like, the—my response is instructive on this because, first, it was, “Oh, my God. What's going to happen now?” And it's like, first, hang on a second.First off, that seems suspect but assume it's real. I assumed it was real at the outset. It's “Oh, right. This is not my personal $5-a-month toybox account. We are a company; we can absolutely pay that.” Because it's like, I could absolutely reach out, call it a favor. “I made a mistake, and I need a favor on the bill, please,” to AWS.And I would never live it down, let's be clear. For a $7,000 mistake, I would almost certainly eat it. As opposed to having to prostrate myself like that in front of Amazon. I'm like, no, no, no. I want one of those like—if it's like, “Okay, you're going to, like, set back the company roadmap by six months if you have to pay this. Do you want to do it?” Like, [groans] “Fine, I'll eat some crow.”But okay. And then followed immediately by, wow, if Pete of all people can mess this up, customers are going to be doomed here. We should figure out what happened. And I'm doing the math. Like, Pete, “What did you actually do?” And you're sitting there and you're saying, “Well, I had like a 20 gig volume that I did this.” And I'm doing the numbers, and it's like—Pete: Something's wrong.Corey: “How sure are you when you say ‘gigabyte,' that you were—that actually means what you think it did? Like, were you off by a lot? Like, did you mean exabytes?” Like, what's the deal here?Pete: Like, multiple factors.Corey: Yeah. How much—“How many IOPS did you give that thing, buddy?” And it turned out what happened was that when they launched this, they had mispriced it in the system by a factor of a million. So, it was fun. I think by the end of it, all of your experimentation was somewhere between five to seven cents. Which—Pete: Yeah. It was a—Corey: Which is why you don't work here anymore because no one cost me seven cents of money to give to Amazon—Pete: How dare you?Corey: —on my watch. Get out.Pete: How dare you, sir?Corey: Exactly.Pete: Yeah, that [laugh] was amazing to see, as someone who has done—definitely maid screw-ups that have cost real money—you know, S3 list requests are always a fun one at scale—but that one was supremely fun to see the—Corey: That was a scary one because another one they'd done previously was they had messed up Lightsail pricing, where people would log in, and, like, “Okay, so what is my Lightsail instance going to cost?” And I swear to you, this is true, it was saying—this was back in 2017 or so—the answer was, like, “$4.3 billion.” Because when you see that you just start laughing because you know it's a mistake. You know, that they're not going to actually demand that you spend $4.3 billion for a single instance—unless it's running SAP—and great.It's just, it's a laugh. It's clearly a mispriced, and it's clearly a bug that's going to get—it's going to get fixed. I just spun up this new EBS volume that no one fully understands yet and it cost me thousands of dollars. That's the sort of thing that no, no, I could actually see that happening. There are instances now that cost something like 100 bucks an hour or whatnot to run. I can see spinning up the wrong thing by mistake and getting bitten by it. There's a bunch of fun configuration mistakes you can make that will, “Hee, hee, hee. Why can I see that bill spike from orbit?” And that's the scary thing.Pete: Well, it's the original CI and CD problem of the per-hour billing, right? That was super common of, like, yeah, like, an i3, you know, 16XL server is pretty cheap per hour, but if you're charged per hour and you spin up a bunch for five minutes. Like, it—you will be shocked [laugh] by what you see there. So—Corey: Yeah. Mistakes will show. And I get it. It's also people as individuals are very different psychologically than companies are. With companies it's one of those, “Great we're optimizing to bring in more revenue and we don't really care about saving money at all costs.”Whereas people generally have something that looks a lot like a fixed income in the form of a salary or whatnot, so it's it is easier for us to cut spend than it is for us to go out and make more money. Like, I don't want to get a second job, or pitch my boss on stuff, and yeah. So, all and all, routing out the rest of what happened at re:Invent, they—this is the problem is that they have a bunch of minor things like SageMaker Inference Recommender. Yeah, I don't care. Anything—Pete: [laugh].Corey: —[crosstalk 00:28:47] SageMaker I mostly tend to ignore, for safety. I did like the way they described Amplify Studio because they made it sound like a WYSIWYG drag and drop, build a React app. It's not it. It basically—you can do that in Figma and then it can hook it up to some things in some cases. It's not what I want it to be, which is Honeycode, except good. But we'll get there some year. Maybe.Pete: There's a lot of stuff that was—you know, it's the classic, like, preview, which sure, like, from a product standpoint, it's great. You know, they have a level of scale where they can say, “Here's this thing we're building,” which could be just a twinkle in a product managers, call it preview, and get thousands of people who would be happy to test it out and give you feedback, and it's a, it's great that you have that capability. But I often look at so much stuff and, like, that's really cool, but, like, can I, can I have it now? Right? Like—or you can't even get into the preview plan, even though, like, you have that specific problem. And it's largely just because either, like, your scale isn't big enough, or you don't have a good enough relationship with your account manager, or I don't know, countless other reasons.Corey: The thing that really throws me, too, is the pre-announcements that come a year or so in advance, like, the Outpost smaller ones are finally available, but it feels like when they do too many pre-announcements or no big marquee service announcements, as much as they talk about, “We're getting back to fundamentals,” no, you have a bunch of teams that blew the deadline. That's really what it is; let's not call it anything else. Another one that I think is causing trouble for folks—I'm fortunate in that I don't do much work with Oracle databases, or Microsoft SQL databases—but they extended RDS Custom to Microsoft SQL at the [unintelligible 00:30:27] SQL server at re:Invent this year, which means this comes down to things I actually use, we're going to have a problem because historically, the lesson has always been if I want to run my own databases and tweak everything, I do it on top of an EC2 instance. If I want to managed database, relational database service, great, I use RDS. RDS Custom basically gives you root into the RDS instance. Which means among other things, yes, you can now use RDS to run containers.But it lets you do a lot of things that are right in between. So, how do you position this? When should I use RDS Custom? Can you give me an easy answer to that question? And they used a lot of words to say, no, they cannot. It's basically completely blowing apart the messaging and positioning of both of those services in some unfortunate ways. We'll learn as we go.Pete: Yeah. Honestly, it's like why, like, why would I use this? Or how would I use this? And this is I think, fundamentally, what's hard when you just say yes to everything. It's like, they in many cases, I don't think, like, I don't want to say they don't understand why they're doing this, but if it's not like there's a visionary who's like, this fits into this multi-year roadmap.That roadmap is largely—if that roadmap is largely generated by the customers asking for it, then it's not like, oh, we're building towards this Northstar of RDS being whatever. You might say that, but your roadmap's probably getting moved all over the place because, you know, this company that pays you a billion dollars a year is saying, “I would give you $2 billion a year for all of my Oracle databases, but I need this specific thing.” I can't imagine a scenario that they would say, “Oh, well, we're building towards this Northstar, and that's not on the way there.” Right? They'd be like, “New Northstar. Another billion dollars, please.”Corey: Yep. Probably the worst release of re:Invent, from my perspective, is RUM, Real User Monitoring, for CloudWatch. And I, to be clear, I wrote a shitposting Twitter threading client called Last Tweet in AWS. Go to lasttweetinaws.com. You can all use it. It's free; I just built this for my own purposes. And I've instrumented it with RUM. Now, Real User Monitoring is something that a lot of monitoring vendors use, and also CloudWatch now. And what that is, is it embeds a listener into the JavaScript that runs on client load, and it winds up looking at what's going on loading times, et cetera, so you can see when users are unhappy. I have no problem with this. Other than that, you know, liking users? What's up with that?Pete: Crazy.Corey: But then, okay, now, what this does is unlike every other RUM tool out there, which charges per session, meaning I am going to be… doing a web page load, it charges per data item, which includes HTTP errors, or JavaScript errors, et cetera. Which means that if you have a high transaction volume site and suddenly your CDN takes a nap like Fastly did for an hour last year, suddenly your bill is stratospheric for this because errors abound and cascade, and you can have thousands of errors on a single page load for these things, and it is going to be visible from orbit, at least with a per session basis thing, when you start to go viral, you understand that, “Okay, this is probably going to cost me some more on these things, and oops, I guess I should write less compelling content.” Fine. This is one of those one misconfiguration away and you are wailing and gnashing teeth. Now, this is a new service. I believe that they will waive these surprise bills in the event that things like that happen. But it's going to take a while and you're going to be worrying the whole time if you've rolled this out naively. So it's—Pete: Well and—Corey: —I just don't like the pricing.Pete: —how many people will actively avoid that service, right? And honestly, choose a competitor because the competitor could be—the competitor could be five times more expensive, right, on face value, but it's the certainty of it. It's the uncertainty of what Amazon will charge you. Like, no one wants a surprise bill. “Well, a vendor is saying that they'll give us this contract for $10,000. I'm going to pay $10,000, even though RUM might be a fraction of that price.”It's honestly, a lot of these, like, product analytics tools and monitoring tools, you'll often see they price be a, like, you know, MAU, Monthly Active User, you know, or some sort of user-based pricing, like, the number of people coming to your site. You know, and I feel like at least then, if you are trying to optimize for lots of users on your site, and more users means more revenue, then you know, if your spend is going up, but your revenue is also going up, that's a win-win. But if it's like someone—you know, your third-party vendor dies and you're spewing out errors, or someone, you know, upgraded something and it spews out errors. That no one would normally see; that's the thing. Like, unless you're popping open that JavaScript console, you're not seeing any of those errors, yet somehow it's like directly impacting your bottom line? Like that doesn't feel [crosstalk 00:35:06].Corey: Well, there is something vaguely Machiavellian about that. Like, “How do I get my developers to care about errors on consoles?” Like, how about we make it extortionately expensive for them not to. It's, “Oh, all right, then. Here we go.”Pete: And then talk about now you're in a scenario where you're working on things that don't directly impact the product. You're basically just sweeping up the floor and then trying to remove errors that maybe don't actually affect it and they're not actually an error.Corey: Yeah. I really do wonder what the right answer is going to be. We'll find out. Again, we live, we learn. But it's also, how long does it take a service that has bad pricing at launch, or an unfortunate story around it to outrun that reputation?People are still scared of Glacier because of its original restore pricing, which was non-deterministic for any sensible human being, and in some cases lead to I'm used to spending 20 to 30 bucks a month on this. Why was I just charged two grand?Pete: Right.Corey: Scare people like that, they don't come back.Pete: I'm trying to actually remember which service it is that basically gave you an estimate, right? Like, turn it on for a month, and it would give you an estimate of how much this was going to cost you when billing started.Corey: It was either Detective or GuardDuty.Pete: Yeah, it was—yeah, that's exactly right. It was one of those two. And honestly, that was unbelievably refreshing to see. You know, like, listen, you have the data, Amazon. You know what this is going to cost me, so when I, like, don't make me spend all this time to go and figure out the cost. If you have all this data already, just tell me, right?And if I look at it and go, “Yeah, wow. Like, turning this on in my environment is going to cost me X dollars. Like, yeah, that's a trade-off I want to make, I'll spend that.” But you know, with some of the—and that—a little bit of a worry on some of the intelligent tiering on S3 is that the recommendation is likely going to be everything goes to intelligent tiering first, right? It's the gp3 story. Put everything on gp3, then move it to the proper volume, move it to an sc or an st or an io. Like, gp3 is where you start. And I wonder if that's going to be [crosstalk 00:37:08].Corey: Except I went through a wizard yesterday to launch an EC2 instance and its default on the free tier gp2.Pete: Yeah. Interesting.Corey: Which does not thrill me. I also still don't understand for the life of me why in some regions, the free tier is a t2 instance, when t3 is available.Pete: They're uh… my guess is that they've got some free t—they got a bunch of t2s lying around. [laugh].Corey: Well, one of the most notable announcements at re:Invent that most people didn't pay attention to is their ability now to run legacy instance types on top of Nitro, which really speaks to what's going on behind the scenes of we can get rid of all that old hardware and emulate the old m1 on modern equipment. So, because—you can still have that legacy, ancient instance, but now you're going—now we're able to wind up greening our data centers, which is part of their big sustainability push, with their ‘Sustainability Pillar' for the well-architected framework. They're talking more about what the green choices in cloud are. Which is super handy, not just because of the economic impact because we could use this pretty directly to reverse engineer their various margins on a per-service or per-offering basis. Which I'm not sure they're aware of yet, but oh, they're going to be.And that really winds up being a win for the planet, obviously, but also something that is—that I guess puts a little bit of choice on customers. The challenge I've got is, with my serverless stuff that I build out, if I spend—the Google search I make to figure out what the most economic, most sustainable way to do that is, is going to have a bigger carbon impact on the app itself. That seems to be something that is important at scale, but if you're not at scale, it's one of those, don't worry about it. Because let's face it, the cloud providers—all of them—are going to have a better sustainability story than you are running this in your own data centers, or on a Raspberry Pi that's always plugged into the wall.Pete: Yeah, I mean, you got to remember, Amazon builds their own power plants to power their data centers. Like, that's the level they play, right? There, their economies of scale are so entirely—they're so entirely different than anything that you could possibly even imagine. So, it's something that, like, I'm sure people will want to choose for. But, you know, if I would honestly say, like, if we really cared about our computing costs and the carbon footprint of it, I would love to actually know the carbon footprint of all of the JavaScript trackers that when I go to various news sites, and it loads, you know, the whatever thousands of trackers and tracking the all over, like, what is the carbon impact of some of those choices that I actually could control, like, as a either a consumer or business person?Corey: I really hope that it turns into something that makes a meaningful difference, and it's not just greenwashing. But we'll see. In the fullness of time, we're going to figure that out. Oh, they're also launching some mainframe stuff. They—like that's great.Pete: Yeah, those are still a thing.Corey: I don't deal with a lot of customers that are doing things with that in any meaningful sense. There is no AWS/400, so all right.Pete: [laugh]. Yeah, I think honestly, like, I did talk to a friend of mine who's in a big old enterprise and has a mainframe, and they're actually replacing their mainframe with Lambda. Like they're peeling off—which is, like, a great move—taking the monolith, right, and peeling off the individual components of what it can do into these discrete Lambda functions. Which I thought was really fascinating. Again, it's a five-year-long journey to do something like that. And not everyone wants to wait five years, especially if their support's about to run out for that giant box in the, you know, giant warehouse.Corey: The thing that I also noticed—and this is probably the—I guess, one of the—talk about swing and a miss on pricing—they have a—what is it?—there's a VPC IP Address Manager, which tracks the the IP addresses assigned to your VPCs that are allocated versus not, and it's 20 cents a month per IP address. It's like, “Okay. So, you're competing against a Google Sheet or an Excel spreadsheet”—which is what people are using for these things now—“Only you're making it extortionately expensive?”Pete: What kind of value does that provide for 20—I mean, like, again—Corey: I think Infoblox or someone like that offers it where they become more cost-effective as soon as you hit 500 IP addresses. And it's just—like, this is what I'm talking about. I know it does not cost AWS that kind of money to store an IP address. You can store that in a Route 53 TXT record for less money, for God's sake. And that's one of those, like, “Ah, we could extract some value pricing here.”Like, I don't know if it's a good product or not. Given its pricing, I don't give a shit because it's going to be too expensive for anything beyond trivial usage. So, it's a swing and a miss from that perspective. It's just, looking at that, I laugh, and I don't look at it again.Pete: See I feel—Corey: I'm not usually price sensitive. I want to be clear on that. It's just, that is just Looney Tunes, clown shoes pricing.Pete: Yeah. It's honestly, like, in many cases, I think the thing that I have seen, you know, in the past few years is, in many cases, it can honestly feel like Amazon is nickel-and-diming their customers in so many ways. You know, the explosion of making it easy to create multiple Amazon accounts has a direct impact to waste in the cloud because there's a lot of stuff you have to have her account. And the more accounts you have, those costs grow exponentially as you have these different places. Like, you kind of lose out on the economies of scale when you have a smaller number of accounts.And yeah, it's hard to optimize for that. Like, if you're trying to reduce your spend, it's challenging to say, “Well, by making a change here, we'll save, you know, $10,000 in this account.” “That doesn't seem like a lot when we're spending millions.” “Well, hold on a second. You'll save $10,000 per account, and you have 500 accounts,” or, “You have 1000 accounts,” or something like that.Or almost cost avoidance of this cost is growing unbounded in all of your accounts. It's tiny right now. So, like, now would be the time you want to do something with it. But like, again, for a lot of companies that have adopted the practice of endless Amazon accounts, they've almost gone, like, it's the classic, like, you know, I've got 8000 GitHub repositories for my source code. Like, that feels just as bad as having one GitHub repository for your repo. I don't know what the balance is there, but anytime these different types of services come out, it feels like, “Oh, wow. Like, I'm going to get nickeled and dimed for it.”Corey: This ties into the re:Post launch, which is a rebranding of their forums, where, okay, great, it was a little crufty and it need modernize, but it still ties your identity to an IAM account, or the root email address for an Amazon account, which is great. This is completely worthless because as soon as I change jobs, I lose my identity, my history, the rest, on this forum. I'm not using it. It shows that there's a lack of awareness that everyone is going to have multiple accounts with which they interact, and that people are going to deal with the platform longer than any individual account will. It's just a continual swing and a miss on things like that.And it gets back to the billing question of, “Okay. When I spin up an account, do I want them to just continue billing me—because don't turn this off; this is important—or do I want there to be a hard boundary where if you're about to charge me, turn it off. Turn off the thing that's about to cost me money.” And people hem and haw like this is an insurmountable problem, but I think the way to solve it is, let me specify that intent when I provision the account. Where it's, “This is a production account for a bank. I really don't want you turning it off.” Versus, “I'm a student learner who thinks that a Managed NAT Gateway might be a good thing. Yeah, I want you to turn off my demo Hello World app that will teach me what's going on, rather than surprising me with a five-figure bill at the end of the month.”Pete: Yeah. It shouldn't be that hard. I mean, but again, I guess everything's hard at scale.Corey: Oh, yeah. Oh yeah.Pete: But still, I feel like every time I log into Cost Explorer and I look at—and this is years it's still not fixed. Not that it's even possible to fix—but on the first day of the month, you look at Cost Explorer, and look at what Amazon is estimating your monthly bill is going to be. It's like because of your, you know—Corey: Your support fees, and your RI purchases, and savings plans purchases.Pete: [laugh]. All those things happened, right? First of the month, and it's like, yeah, “Your bill's going to be $800,000 this year.” And it's like, “Shouldn't be, like, $1,000?” Like, you know, it's the little things like that, that always—Corey: The one-off charges, like, “Oh, your Route 53 zone,” and all the stuff that gets charged on a monthly cadence, which fine, whatever. I mean, I'm okay with it, but it's also the, like, be careful when that happen—I feel like there's a way to make that user experience less jarring.Pete: Yeah because that problem—I mean, in my scenario, companies that I've worked at, there's been multiple times that a non-technical person will look at that data and go into immediate freakout mode, right? And that's never something that you want to have happen because now that's just adding a lot of stress and anxiety into a company that is—with inaccurate data. Like, the data—like, the answer you're giving someone is just wrong. Perhaps you shouldn't even give it to them if it's that wrong. [laugh].Corey: Yeah, I'm looking forward to seeing what happens this coming year. We're already seeing promising stuff. They—give people a timeline on how long in advance these things record—late last night, AWS released a new console experience. When you log into the AWS console now, there's a new beta thing. And I gave it some grief on Twitter because I'm still me, but like the direction it's going. It lets you customize your view with widgets and whatnot.And until they start selling widgets on marketplace or having sponsored widgets, you can't remove I like it, which is no guarantee at some point. But it shows things like, I can move the cost stuff, I can move the outage stuff up around, I can have the things that are going on in my account—but who I am means I can shift this around. If I'm a finance manager, cool. I can remove all the stuff that's like, “Hey, you want to get started spinning up an EC2 instance?” “Absolutely not. Do I want to get told, like, how to get certified? Probably not. Do I want to know what the current bill is and whether—and my list of favorites that I've pinned, whatever services there? Yeah, absolutely do.” This is starting to get there.Pete: Yeah, I wonder if it really is a way to start almost hedging on organizations having a wider group of people accessing AWS. I mean, in previous companies, I absolutely gave access to the console for tools like QuickSight, for tools like Athena, for the DataBrew stuff, the Glue DataBrew. Giving, you know, non-technical people access to be able to do these, like, you know, UI ETL tasks, you know, a wider group of a company is getting access into Amazon. So, I think anything that Amazon does to improve that experience for, you know, the non-SREs, like the people who would traditionally log in, like, that is an investment definitely worth making.Corey: “Well, what could non-engineering types possibly be doing in the AWS console?” “I don't know, jackhole, maybe paying the bill? Just a thought here.” It's the, there are people who look at these things from a variety of different places, and you have such sprawl in the AWS world that there are different personas by a landslide. If I'm building Twitter for Pets, you probably don't want to be pitching your mainframe migration services to me the same way that you would if I were a 200-year-old insurance company.Pete: Yeah, exactly. And the number of those products are going to grow, the number of personas are going to grow, and, yeah, they'll have to do something that they want to actually, you know, maintain that experience so that every person can have, kind of, the experience that they want, and not be distracted, you know? “Oh, what's this? Let me go test this out.” And it's like, you know, one-time charge for $10,000 because, like, that's how it's charged. You know, that's not an experience that people like.Corey: No. They really don't. Pete, I want to thank you for spending the time to chat with me again, as is our tradition. I'm hoping we can do it in person this year, when we go at the end of 2022, to re:Invent again. Or that no one goes in person. But this hybrid nonsense is for the birds.Pete: Yeah. I very much would love to get back to another one, and yeah, like, I think there could be an interesting kind of merging here of our annual re:Invent recap slash live brunch, you know, stream you know, hot takes after a long week. [laugh].Corey: Oh, yeah. The real way that you know that it's a good joke is when one of us says something, the other one sprays scrambled eggs out of their nose. Yeah, that's the way to do it.Pete: Exactly. Exactly.Corey: Pete, thank you so much. If people want to learn more about what you're up to—hopefully, you know, come back. We miss you, but you're unaffiliated, you're a startup advisor. Where can people find you to learn more, if they for some unforgivable reason don't know who or what a Pete Cheslock is?Pete: Yeah. I think the easiest place to find me is always on Twitter. I'm just at @petecheslock. My DMs are always open and I'm always down to expand my network and chat with folks.And yeah, right, now, I'm just, as I jokingly say, professionally unaffiliated. I do some startup advisory work and have been largely just kind of—honestly checking out the state of the economy. Like, there's a lot of really interesting companies out there, and some interesting problems to solve. And, you know, trying to spend some of my time learning more about what companies are up to nowadays. So yeah, if you got some interesting problems, you know, you can follow my Twitter or go to LinkedIn if you want some great, you know, business hot takes about, you know, shitposting basically.Corey: Same thing. Pete, thanks so much for joining me, I appreciate it.Pete: Thanks for having me.Corey: Pete Cheslock, startup advisor, professionally unaffiliated, and recurring re:Invent analyst pal of mine. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment calling me a jackass because do I know how long it took you personally to price CloudWatch RUM?Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About JesseJesse is a seasoned operations engineer with a deep passion for understanding complex technical and organizational systems. He's spent his career helping Engineering teams achieve their business goals by improving how they interact with their technical systems, and with each other. He's currently a Cloud Economist with Duckbill Group, guiding organizations along their journey of cloud cost optimization and management.Links: The Duckbill Group: https://www.duckbillgroup.com/ Jesse's Twitter: https://twitter.com/jesse_derose AWS Morning Brief: https://www.lastweekinaws.com/podcast/aws-morning-brief/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Your company might be stuck in the middle of a DevOps revolution without even realizing it. Lucky you! Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy in on DevOps practices? Well, download the 2021 State of DevOps report brought to you annually by Puppet since 2011 to explore the trends and blockers keeping evolution firms stuck in the middle of their DevOps evolution. Because they fail to evolve or die like dinosaurs. The significance of organizational buy in, and oh it is significant indeed, and why team identities and interaction models matter. Not to mention weither the use of automation and the cloud translate to DevOps success. All that and more awaits you. Visit: www.puppet.com to download your copy of the report now!Corey: Up next we've got the latest hits from Veem. Its climbing charts everywhere and soon its going to climb right into your heart. Here it is!Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Jesse DeRose, my colleague, and cloud economist at The Duckbill Group. Jesse, thank you for joining me, even though when I asked you, it isn't exactly like you felt you had much of a choice.Jesse: [laugh]. I appreciate being on this podcast with you. I think you've had an opportunity to talk to a few other folks from our organization so far, so I'm just happy to be included. I'm hoping that I get the Members Only jacket after this recording.Corey: Oh, absolutely. The swag that goes out to guests is secret and wonderful, all at the same time. So, let's start at the very beginning. It turns out that despite the prevalent narrative that we put out there, are clouds' economist did not just spring fully-formed from the forehead of some God, and appear—fully-formed—ready to slice AWS bills to ribbons. There's a process and there's always an origin story. Where do you come from? Where were you before this?Jesse: So, my background is mixed. I started with a management of information systems degree, which is inherently interdisciplinary. You're linking, sort of, the importance of both technical systems and business goals and outcomes into a single degree. And when I graduated with this degree, I went out into the working world and said, “Okay, I'm here. I'm ready. Here's my degree. Here's what I'm interested in. Let's do this.”And nobody knew what to do with me, Corey. Nobody knew what management of information systems was. They simultaneously said, “You know, you don't have a computer science or a computer engineering degree, so we don't believe that you've got programmer chops.” Especially since this is the golden age of boot camps and quickstart programming books, and this movement that anybody can be a developer, which makes it even stranger that I'm not spending my weekends learning every programming language under the sun and automating the little tasks.Corey: Oh, absolutely. In fact, I got the exact same feedback; I don't have a degree, and, “Oh, you couldn't possibly be a good programmer because you don't have the degree, you didn't go to the right schools, you didn't go to a boot camp. And when I asked you to write some sample code to demonstrate how to program, you just started crying instead.” Because yeah, it turns out, I'm actually not a good programmer, I just, sort of, brute force my way through it. And so far, so good.Unfortunately, people aren't paying me to program these days, for excellent reason. In fact, I strongly suspect some people are paying me not to. But yeah, now it's funny to laugh about, but back when you're getting started, and going out in that space, in the world of operations, which we both came up in, and looking at it through a lens of the SRE movement, suddenly, “Hey, you used to be really, really good at all these Linux things and working on systems and keeping them up. Great. Now, learn to code.” And that was a big lift, at least for me.Jesse: Absolutely. I struggled with that so much because every company that I interviewed with, every company that I even talked to, just assumed that I had some kind of programming experience and didn't want to talk to me if I didn't have programming experience. And to me, looking back, I think I really look at it like I may not have the programming chops to be the software engineer that is going to write the code for you, day in and day out, but I am the person who knows enough that I can have the conversation with the software engineers. I can be the SRE, I can be the ops person that has the conversation with your software engineers and knows the things that they're talking about, but also knows when to get out of their way and let them be the expert and do what they do best.Corey: There's something to be said for valuing expertise in areas that are—how to put this—not the thing that you think you're looking for. I mean, back when I was getting jobs, before The Duckbill Group and I would be the first ops hire into a team of developers—which happened a few times—the process was always the same, where you'd have a bunch of developers asking what they thought were ops questions, or just giving up on that entirely and trying to figure out how decent have an ops person I would be by how badly I programmed.Or, “Oh, okay, cool. You're an ops person. Great. Can you invert a binary tree on a whiteboard?” It's, “No, but I can invert a rack in your data center. I'll go rage-flip the rack. Why not?” And it takes time. You have to guide those interviews and those conversations. But it's always weird.] interviews are always weird because you're being judged on a skill set that only matters when you're interviewing for a job.Jesse: Yes. This is one of the things that I struggled with the most because I knew that most of the people who were interviewing me were either business people themselves—so they assumed that because their engineering team thought a certain way and acted a certain way that I should act that way—specifically, too—the same way as everybody else on the team. Or they were software engineers themselves, so they said, “Okay, I know how to invert a binary tree”—to your point, Corey—“So, do how to invert a binary tree. If you know how to do that, then sure, you can be part of my team because you know how to do these things and think about these things the same way I do.” Whereas because I was coming from this operations space, I knew other things that were equally as important but weren't part of the conversations that they were used to having, day-to-day.So, they didn't understand that just because I didn't have the same engineering chops as them, that I didn't have important information to share and wasn't able to stand on my own two feet in other ways. And that was one of the things that I really struggled with when I was starting out in the industry because I was thinking to myself, well I have such passion to be part of these conversations, to have that conversation between the business side of the organization and the engineering side of the organization, from an ops perspective, from a business perspective, from a technical perspective, and if I can't convince these people of my own volition, of my own passion for the good of the company, maybe data will help. Maybe there's something that I can find from a scientific research perspective. Maybe there's something that—I'm sure somebody else has already researched this topic or found the same problems that I have in this space, and maybe they're already talking about it, and maybe I can ride their coattails, so to speak, or follow in their footsteps and use the information that other people are talking about the industry to help me not just land these jobs, but ultimately better sell myself and help these companies move forward.Corey: That is fundamentally an encapsulation of what I believe the ops role to be of, make things better, and move them forward. But man, do we get stuck in an awful lot of weird and strange places. And interviewing itself is a skill. Giving an interview, very often, it's a, “I know a bunch of things that are trivia, but I know them. And if I know them, everyone must know them, therefore, if you don't know them, you must be bad at things.” And it turns out—for better or worse—being able to memorize the documentation and spit out answers is not indicative of whether someone is a good ops person or a terrible one.Jesse: Absolutely. I think that is one of the biggest problems that I have faced and one of the biggest problems in the interviewing space today because it's not just about, can you regurgitate this information, but it's about how do you think? How do you look at problems? How do you communicate to the rest of your team, and within the rest of the organization? Those technical skills are ultimately important because you do need to understand some amount of technical information to have those conversations, but the soft skills are also super, super important to be able to communicate effectively, to be able to think collaboratively, and help everybody, not just yourself, but help the team build that shared purpose and move forward together.Corey: We're talking right now, so far, about traditional ops roles. Then we have what we do here, which is beyond the rest of all of that, where all of what we just said is necessary but not sufficient. Then it comes down to great, okay. So, you understand how systems work together; we found that, for what we do and how we do it, you need to be a competent ops person as a fundamental tenet.Otherwise, learning what all these AWS services do will occupy you for the next three years. So okay, we start off there. Then on top of that is, okay, there are consulting skills that it turns out are possible to teach, but incredibly challenging and time-consuming because a lot of them boil down to, can you be in a meeting with stakeholders of various levels? Can you deliver bad news in a way that they don't hate you? Because they don't really want to pay you just say yes to whatever they think.And can you do that in such a way without, you know, actively insulting them, which sounds like a strange thing until you realize, oh, wait, that's right. I do that, too. So ooh, yeah. Corey is going to have that problem, isn't he? Yeah. And that's part of the beautiful part about this place is that finally, I'm able to hire people like you.You were the first cloud economist here, which meant suddenly I didn't have to do it all myself and my mouth slowly stopped getting me in trouble in consulting engagements, so I could spend more time having my mouth getting me in trouble on Twitter.Jesse: Yeah, I have to tell you, Corey, when I originally spoke with you and Mike about this role, I had just taken another operations position with a tech startup, and I was about two months into the role. And Mike sat down with me for coffee one morning and said, “Hey, we're thinking about doing this thing. Are you interested?” And I said, “Yes, but I just started this other operations gig. I can't up and leave them; I really care about the team, I really care about the company. And it would look really bad on me if I just, you know, two months left.”Because—unless they were a really, really awful employer, which they weren't. So, I said, “Sure. I'm interested in doing some kind of part-time work.” And that's ultimately where I started with you and your business partner, Mike. And I have to admit to when Mike originally approached me and said, “Hey, this is what we are thinking about; this is what we're doing,” I didn't really think twice about the opportunity because I wanted to work with you and Mike again.But the way that Mike described the work, just didn't stick with me. It didn't resonate with me, it was more about, “Hey, I would love to work with Mike and Corey again,” than, “Oh, my God. This sounds like the dream role that I want to be a part of.” And then, when I came back to Mike, probably, I don't know, a month or two later, after I had started working part-time with both of you, I said, you know, “Mike, I don't think I really made myself clear. I want to make sure that I help you understand, ultimately, the things that I want to do are having these conversations, being that bridge with the business side, and being able to talk tech with the tech side, and being able to talk business, and make sure that both sides of the conversation are aligned.”And he just looked at me and said, “What do you think we're doing? What do you think we sold you on?” And it was that aha moment where I thought, “Oh, my god, yes.” I had already said yes; I was already working with both of you part-time, but that was the moment that really solidified it for me of, this is what I've kind of been moving towards. I've been wanting to be that person that can speak both languages and have a conversation with both sides of the table, and speak to multiple different audiences, and now I'm finally getting the chance to do that. I'm getting the chance to grow both skill sets, which I think is extremely rare in a lot of the smaller tech spaces that we see today.Corey: You've hit on one of the secrets of The Duckbill Group if I can be so grandiose as to claim that. And it's true because we take a look at people that we bring in, and things that they're good at, and things that we do—the things we do publicly and the things that we do, sort of, behind the scenes and there's no reason we don't talk about them publicly, but there's no real reason for us to do so. Easy example, and what I want to talk to you about next is, you are deep into improving understanding of complex systems, both technical—okay, great people expect that—and organizational, which sometimes throws people for a loop. And it sounds like a weird thing to focus on here because we fix the AWS bill. We do not bill ourselves as management consultants, we do not bill ourselves as coming in and we will restructure your organization because that sounds patently ridiculous, and no one in their right mind is going to buy that thing.I wouldn't buy it, at least not for me. My God, there are large consultancies that specialize in these sorts of things. I don't know how they do it because I certainly don't. We're not here to sell that, though. Fixing the AWS bill—I mean actually fixing it. Fixing the business problem tied to it mandates an understanding of those complex systems. And your expertise and interest in that area is incredibly helpful here. Tell me more about it.Jesse: This is one of the things that I've been really fascinated by ever since I joined Duckbill Group. I think everybody in Duckbill Group has a superpower or has a really passionate hobby to some extent, which makes each of us really interesting, unique individuals that can focus in different areas of a client's bill or a client's pain points when it comes to cloud cost management and help, and find the parts that are frustrating, find the levers that can be moved, and point them out and say, “Okay, this is ultimately where you want to pull this lever or not pull this lever to make these changes.” And to your point, Corey, the one that is most interesting and passionate for me is that organizational development space. It is really understanding, not just the small things that we can do today to help you save money on your AWS bill, but how we, and collectively how our clients can think about costs long term to save money on their AWS bill. And I know that sounds really, really broad, and that's part of why I think that there is a lot of nuance in this space, to your point about other organizations or other vendors that are providing these consulting services, and I think is also something that is also difficult to sell, which is why it's not our expertise in terms of what we are on the cover trying to sell to any of our clients.But I definitely think that there is opportunity to have some of those conversations within each of our clients' spaces to talk about some of the pain points that we see that may ultimately lead to better cost management practices long term, things that ultimately might help the engineering teams communicate better with finance on a long term basis, help the finance team and any of the leadership team more collaboratively talk with the engineering teams about understanding how much money is the product costing us? How much can we continue to spend on this product? Or how much can we discount one of our products for our customers before we are losing shares, losing money? What are the fine lines that we understand, based on how much money we're ultimately spending on these features, on these products, that will help us make better data-driven decisions about other parts of the company?Corey: I really love installing, upgrading, and fixing security agents in my cloud estate! Why do I say that? Because I sell things, because I sell things for a company that deploys an agent, there's no other reason. Because let's face it. Agents can be a real headache. Well, now Orca Security gives you a single tool that detects basically every risk in your cloud environment -- and that's as easy to install and maintain as a smartphone app. It is agentless, or my intro would've gotten me into trouble here, but  it can still see deep into your AWS workloads, while guaranteeing 100% coverage. With Orca Security, there are no overlooked assets, no DevOps headaches, and believe me you will hear from those people if you cause them headaches. and no performance hits on live environments. Connect your first cloud account in minutes and see for yourself at orca.security. Thats “Orca” as in whale, “dot” security as in that things you company claims to care about but doesn't until right after it really should have.Corey: And I want to call out that this is something that we are comparatively enthusiastic amateurs around. It's valuable; it's important; it's an awful lot of deep work, but I'm not sure that we go more than three working days without referencing Dr. Nicole Forsgren's work, internally, as we think about these things. So, if you're hearing this, and you think that okay, AWS bill, fine, whatever. We really want to talk about organizational challenge and improvement, oh, my God, talk to Dr. Forsgren. Holy crap. She's been on this show, at least I think, three times now, and every time I feel like I'm lucky to get her. Most weeks, you know, I'm stuck with people like you. My God, Jesse.Jesse: [laugh].Corey: But no, her work is seminal in this space. And in seriousness, every time I start to question the value of expertise, I look at how deep she goes on all of these things and the level both of understanding that's baked into this, and the amount of sheer work that it takes for her to take all of that very deep, penetrating analysis, and make it accessible and understandable. But every time I look at her work, I come away more impressed than I started, and that wasn't a low bar, to begin with.Jesse: Yes. And this gets back to my earlier comment about, I just want to be here to help. And in a lot of cases, when I was starting out, folks didn't know what to do with me because I didn't have data, I didn't have any information. But we have folks in the industry like Dr. Nicole Forsgren, and other folks who are doing the research, who are knowledgeable in this space, who are putting in the effort to run these studies, to analyze this data, to share the results—and to your comment, Corey—to share the results in a way that makes sense to everybody, that's easy to read, it's approachable, it's understandable.And I am so thankful to have folks in the industry who are doing that work because that is not my expertise. But that means that I get to say to the folks that I'm working with—internally and with our clients—“Hey, don't take my word for it. There are other folks who have done research, and here's what the data says, and here's how we can help you apply this work, or you can apply this work within your own organization.”Corey: And this is the challenge in some cases, too, where there's a lot of organizational theory, and that is being advanced heavily, and in ways that makes teams more effective is super helpful. The challenge, of course, is that sitting here and talking about the theoretical layout of teams and how to improve functioning as an organization is all well and good, but we're brought in by our clients to help them with their AWS billing situation, so at some point the conversation has to evolve beyond, “Okay, so here's what you could do in theory, in a vacuum, assuming spherical cows, et cetera, et cetera.” And their response is, “That's great. You actually going to fix the bill or just pontificate for a while here?” So, for better or worse, we don't really get to sit there and have deep organizational conversations at length with our clients, just because that's not the problem we're there to solve. Everyone's busy and we want to make sure that we're respectful of their time.Jesse: Yeah, one of the things that I've learned through my time with Duckbill Group and through other similar roles in the past, is that I may have a strong passion, I may have this strong guiding light in my head, but it's not the same guiding light that our clients or our customers have. And that's fine because we don't need to necessarily have the same goal in sight, but that means that I, to best serve our clients or best serve our customers, need to make sure that I am aligning, that Duckbill Group is aligning with the clients that we're working with, with the organizations that we're working with. So, I may be thinking to myself, “Oh, my gosh, I would love to come in with this long list of organizational development practices and share a million different things,” but that's not ultimately what they need. Maybe that's something that they're going to want long-term, but it's not what we're here for today. And it's more important to help serve the client that is in front of me that is asking for things now, today, than try to educate them on quote-unquote, “What their problem is,” and then sell them on a solution.Corey: The thing that I think gets lost as well, whenever I start talking in-depth about what we do on Twitter, for example, is generally from other engineers whose response is, “Okay, yeah, sounds great. You come in and say that it will save a bunch of money before you rearchitect our application. But that's an awful lot of engineering time, so I bet engineers most hate you.” And the honest answer is, “Yeah. We know that you would save some significant money if you rearchitected your application, but we also pay attention to organizational dynamics, and we know you're not going to do it because there's no business justification for doing it. So, we're not going to bring it up, other than, possibly, in passing, just so people are aware of the relative benefit if they want to bake that in.”But we don't go in and suggest nonsense that is abhorrent. We all started as engineers ourselves. We are sensitive to engineering time, both in terms of what engineers enjoy working on—which is more important than people think—as well as the sheer cost of engineering time. People get concerned about the AWS bill, but it's invariably pale compared to the cost of the people working on the AWS infrastructure. You want to optimize the right things, and then you want to get back to doing what your company does, not continue to iterate forward and spend thousands of dollars to save tens of dollars.Jesse: Yeah, it's an extremely difficult balance to find. And it's really important to think about, is the ROI on this change worth the change? How much money am I ultimately going to save for the amount of engineering effort that I'm going to put in? Because we don't want to run in and tell your engineering teams, “Rearchitect everything,” if it's going to maybe save them tens of dollars. We want to make it very clear that here are the different levers that you can pull to affect change within your AWS bill, to optimize your AWS bill.It's up to you which ones you want to pull. We are just giving you that guiding path, and then you have the option to say, “Yes, I want to spend the engineering effort to get this kind of ROI.” Or, “You know what? I don't think that's the priority for us right now.” One of the things that I've noticed with a number of our clients is that balance of, do we focus more time on building new features which brings in new customers, or do we spend time on the existing infrastructure and making changes to the existing workloads that we have?And it's this delicate balance of internal work—the things that you have put on the backburner over time that you ultimately say, “Well, I'll come back to that,” versus the new things that are ultimately going to bring in new customers, bring in new users, potentially bring in more money. Because both are important, but there needs to be a delicate balance of both, and I feel like that's one of the biggest challenges that we face when managing an AWS bill and trying to optimize, and organize, and better manage cloud costs.Corey: And that's fundamentally what it comes down to what is the best outcome for the client? And the answer to ‘what is best?' Varies based upon their constraints, what they're focused on, what they're trying to achieve. You could look at the end result of one of our analyses for a customer, and take issue with it in a vacuum of but they're spending all this money on things that they shouldn't be doing, or don't need to be. Why didn't you suggest this, or this, or this, or this?And the answer is because, based upon our conversations, we knew that they weren't going to do it, and suggesting things that we know they're not going to do is one of the best ways to erode trust. We're there to deliver an outcome. We're not naive software that is just running a pile of tools on an Amazon bill and saying, “Here you go. Have fun.” We're not the billing equivalent of a Nessus scan that someone slaps their logo on top of, drops off, then it's 700 pages long. “Have a good one. Check, please.” It doesn't work. Not well, anyway. It doesn't drive to lasting change.Jesse: Yeah. And I think that's ultimately part of where we come in best because we ultimately want to be that bridge between the engineering teams, and finance, and leadership, and essentially the business side of the business. We want to give both sides the information that they need to be able to speak effectively and collaboratively with the other side. We want to make sure that the finance side understands enough tech that they can work with the engineering teams to guide them in terms of what goals are important for finance, in terms of managing budget, in terms of forecasting spend. And then from the flip side, we want to make sure that engineering understands that the business collaboratively wants to manage these things, and also help build the organization, and engineering has great, great potential to do little things, take little steps to help the organization get the data that they need to make these decisions.And that scratches that itch for me. That scratches that itch of how can I really be both sides of the conversation? How can I flex the business lingo and also flex the tech lingo, maybe not in the same conversation but with the same client over time, to really help both sides understand that both sides are important, and both sides need to understand each other in order to help the business succeed?Corey: And that's ultimately what it comes down to. Jesse, thanks for taking the time to speak with me. If people want to hear more about what you have to say, and how you like to say it, where can they find you, other than go into The Duckbill Group and get me a consulting engagement underway?Jesse: So, there's two places that folks can find me. The first is on Twitter at @jesse_derose. And then the second is our other podcast, the AWS Morning Brief podcasts, on the mornings where we don't get to hear your melodious voice, Corey.My colleague, Pete Cheslock and I are talking about all of the interesting things that we've seen on AWS, from client-specific situations to things that we've seen on the job in previous organizations that we use to work at, to new features that AWS is releasing. There's a whole slew of interesting things that we get to talk about from a more practical perspective. Because there's so many releases coming out day-to-day that you're obviously helping us stay on top of, Corey, but there's so many other things that we want to make sure that we are talking about from the real-world applicable perspective.Corey: And we will, of course, put links to these things in the [show notes 00:27:48], but you should already be aware of most of them, at least the ones that are on the company side. Jesse, thank you so much for taking the time to speak with me.Jesse: Thank you for having me.Corey: Jesse DeRose, cloud economist here at The Duckbill Group. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an insulting comment telling me that organizational dynamics really aren't that hard and you could solve it better than Dr. Forsgren does, in a weekend.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.This has been a HumblePod production. Stay humble.

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you’re looking to lower your AWS bill or negotiate a new The post AWS with Pete Cheslock appeared first on Software Engineering Daily.

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you’re looking to lower your AWS bill or negotiate a new The post AWS with Pete Cheslock appeared first on Software Engineering Daily.

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you’re looking to lower your AWS bill or negotiate a new

Corey Quinn is guest hosting on Software Engineering Daily this week, presenting a Tour of the Cloud. Corey Quinn is the Chief Cloud Economist at The Duckbill Group, where he helps companies fix their AWS bill by making it smaller and less horrifying. If you're looking to lower your AWS bill or negotiate a new The post AWS with Pete Cheslock appeared first on Software Engineering Daily.

TranscriptCorey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Today, on a very special episode of AWS Morning Brief: Fridays From the Field, we say our goodbyes to Pete Cheslock.Amy: Oh, no. Did the ops bus finally get him?Jesse: No. Wait, what? What? No. No, he’s not—Amy: You know, the ops bus, the one that takes out all of the ops people, which is why you need data recovery plans.Jesse: [laugh]. I mean, I have plans for other reasons, but no. No, Pete, Pete’s not dead. He’s just—I mean, he’s dead to me, but he’s just not going to be here anymore.Amy: Only on the inside.Jesse: Welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.Amy: I’m Amy Arumbulo Negrette.Pete: I am Pete Cheslock. I’m here for one last, beautiful, glorious time.Jesse: I feel like this is going to be like Breakfast Club but in the data center server room.Pete: Yeah. A little bit. I think so. We will all sit cross-legged on the floor in a circle, share our thoughts and feelings. And maybe some sushi. There were sushi in that movie. And that was, like, really advanced back then in the ’80s.Jesse: Yeah, I like that. So Pete, you want to give us a little bit of background about why you will be moving on from this podcast?Pete: Moving on to a whole new world. Yes. Sadly, I am not dead. The ops bus did not get me, and I was not eaten by my smoker, my meat smoker.Jesse: [laugh]. Although at this point, it’s probably overdue.Pete: You know, the odds of all three of those are pretty high out, to be really perfectly honest, given this pandemic and everything else going on in this world.Amy: Isn’t that how it works? You eventually become the smoked meat.Pete: Yeah, yeah.Jesse: [laugh].Pete: All the time. You know, you are what you eat. And if you eat junk and whatnot—so I eat smoked meats, eventually, I’m just going to become, you know, smoked meats, I guess. But no, I am moving on from The Duckbill Group. Just bittersweet is the best word I can come up with. Very sad, but also very excited.I’m moving on to a new role at a new company that was just kind of an opportunity that I couldn’t pass up. And I’m really excited for something new, but really sad because I don’t get to work with two of my three favorite cloud economists, Jesse, and Amy. Yeah, Corey is one, too, and yes, it’s fun to work with him. But it’s also fun to rag on him a little bit as well.Jesse: I’m pretty sure you still have the opportunity to rag on him no matter where you go.Pete: Yeah, that’s true. I mean, we’re Twitter connected. So, I can just slide into his DMs as needed. Yeah.Amy: And really, what else is Twitter for—Pete: Exactly.Jesse: [laugh].Amy: —than roasting former coworkers and bosses?Pete: Yeah, I expect a constant stream of Twitter DMs every time you find something, some little fun nugget that I’ve left behind.Jesse: I feel like that’s appropriate. So today, Pete, I have two questions for you now that you will be moving on from Duckbill Group, moving on from this podcast, I want to know, looking back at your time here working with Duckbill Group, what did you learn? What are the things that surprised you, that you didn’t expect? And what would you say to somebody who wanted to start working in this space, maybe start a career in cloud economics on their own?Pete: Yeah, so this kind of feels like an exit interview a little bit.Jesse: [laugh]. And a very public exit interview at that. So, make sure that we bleep all the swear words.Pete: I think it’s in Duckbill fashion to do a public—a very public-facing exit interview, right? That is Duckbill in a nutshell.Jesse: I think the only thing more public is if Corey asks you to hold the exit interview on Twitter.Amy: Exactly.Pete: [laugh]. I mean, we might have to do that, now. I like that idea. Yeah, so I think those are great questions, and I love the opportunity to talk about it. Because Duckbill is a fantastic company, and coming into Duckbill last year was totally by luck.Not really—no, not—luck is maybe not the right word. But I had been doing some consulting on my own, and the pandemic and some other forces caused a bunch of my consulting work to dry up really quickly. And I was sitting at home and I’m like, “Wow, I should get a real job.” And I saw a tweet from Mike on Twitter that was like, “Oh, we’re growing The Duckbill Group.” And Mike and Corey and I have known each other for such a long time.We’ve always said it’d be great to work together at some point in the future, but it’s so hard [laugh] to do. You know, to kind of work with your friends, and timing, and circumstance, and schedule, and everything else. And so when I saw that, I was like, wow, like that might be a lot of fun working with that crew. And I’ve got a lot of experience in AWS and I’ve—my title at one of my previous companies was Captain COGS—for Cost Of Goods Sold—because I was so diligent with the Amazon bill. So, it’s kind of one of those things where I felt like I could be useful and helpful to the organization, and talking with Mike and Corey, it just made a ton of sense.And so, it was a lot of fun to come on board. So, but then once you’re kind of in, and you start doing this type of work—and you know, Amy and Jesse, you’ve both experienced this—I think no matter how much knowledge you have of Amazon, very, very quickly, you realize that you actually don’t know as much as you really think you did, right?Jesse: Yeah.Pete: Because it’s so—there’s just so much.Amy: And it changes once every five minutes.Pete: [laugh].Jesse: Oh, yeah.Amy: Literally if you—well, just keep an eye on that changelog, you can watch your day get ruined as time goes on.Jesse: [laugh].Pete: [laugh]. It’s—yeah, it’s a real-time day ruining. And that’s the new. It’s like Amazon Kinesis: It’s all real-time.Jesse: [laugh].Pete: Yeah, it’s so true. And I think the reason behind it is, you know, one of the first things I kind of realized is that when you are working inside of a business and you’re trying to understand, like, an Amazon service, you don’t necessarily go that deep because you’ve got a real job and other stuff to do. And when you’re finally, like—let’s say you’re in Cost Explorer; this is actually my favorite one because learning this took us a while. The documents aren’t very good. But in Cost Explorer, there’s a dropdown box that can show you your charges in different ways: unblended view, blended view, amortized view—if I’m saying that word really incorrectly—net-amortized view, net-unblended view. Like, what do all these mean?Most people just are like, unblended, move on with their lives. But at some point, you kind of need to know and answer that question, and then understand the impact, and all those things, and spending more hours than I care to count trying to correlate the bill and Cost Explorer to look the same. Something that simple, why is that so hard? You know, it’s things like that.Amy: Why is that so hard? I do not understand it. It is exhausting. [laugh].Jesse: It drives me absolutely crazy, and it’s something that in previous roles, you could just say, “Well, this isn’t my responsibility, so I’m not going to worry about it.” But now we’ve got clients who are asking us these questions because it is our responsibility and we do need to worry about it.Pete: Yeah, exactly. So, I think that’s just, kind of, one example. Now, there was a ton that I learned. I mean, just in how discounts might be applied when you look at charges in an account whether if you have an enterprise discount program, or private pricing in some way. I think one of my favorite ones—and this is actually something that catches a lot of people up—is especially in Cost Explorer, there’s kind of two ways that you can view a charge.So, let’s say you’re looking at S3, and you are trying to find your usage by the usage type. Like, I want to compare standard storage to maybe data transfer or something like that. And you go and group by usage type, and they’ll show you, “Hey, for your S3 for this month or day or whatever, you’ll have some spend associated storage and data transfer,” and you’re like, “That’s neat.” And then you say to yourself, “Now, I want to look at it by API.” And maybe you’ll see, wow, there’s a ton of spend associated with GETs or PUTs.And you’ll think that that is actually a request charge. And it’s totally not. It’s like, when you group by API, it’s the API that started the charge, not the charge itself. So, you could have a PUT that started the charge, but the charge itself is actually storage. It’s the little things like that, where you might glance at it in your account and go, “Oh, okay.” But then when you actually need to get down to the per penny on spend and share it with a client, you go even further down the rabbit hole.Jesse: Because why would all of the billing information across different sources be accurate?Amy: And also, why would things be named the same between the bill, and Cost Explorer, and the curve? Having those names be the same, that would just make it too easy, and just streamline the process too much, and be too logical. No, let’s work for it. We have to work for it. It’s a pillar of excellence; we have to work for it.Pete: [laugh]. Exactly. So yeah, I think it’s those types of things that you just start seeing the edge cases. But because of, kind of, the work we do, we keep going. We’re not just, “Oh, wow. Haha, silly Amazon.”But then we keep diving in deeper and deeper to figure out the why. And the reason for that really just comes down to the fact that we’ll need to communicate that in some effective way to the client to get them to understand it. And actually, that kind of leads me to the other thing that I think is probably the most important skill of being a cloud economist, of being in finops, is your ability to write long-form writing, being able to write clear, concise information explaining why the spend is what it is, explaining all of these edge cases, all these interesting parts of cloud cost management, and being able to write that down in such a way that anyone could read it; like a CFO could understand how the charges are happening, just like a head of engineering, who has maybe more impact to the spend.Jesse: Being able to communicate, the differences between different AWS services, between different billing modes, to different audiences is so critical to the work that we do because we’re ultimately going to be working with different people with different backgrounds at every single client that we work with. So, we need to be able to speak the language of different audiences.Amy: And it’s really different, how different C Suites, different departments, their goals are going to be different, too, because they have requirements that they have to fulfill. Finance is very concerned about the literal cost of things, while engineering is—they understand that their architecture comes at a price, and so long as they have the budget for it, they’re cool with it. And you just have to align what those goals are, and have that translate as like, into the document as, “They built it this way for this reason, which was fine at that stage. But as you grow, you need to make sure that it also fulfills these other external expectations.”Corey: Let’s be honest—the past year has been a nightmare for cloud financial management. The pandemic forced us to move workloads to the cloud sooner than anticipated, and we all know what that means—surprises on the cloud bill and headaches for anyone trying to figure out what caused them. The CloudLIVE 2021 virtual conference is your chance to connect with FinOps and cloud financial management practitioners and get a behind-the-scenes look into proven strategies that have helped organizations like yours adapt to the realities of the past year. Hosted by CloudHealth by VMware on May 20th, the CloudLIVE 2021 conference will be 100% virtual and 100% free to attend, so you have no excuses for missing out on this opportunity to connect with the cloud management community. Visit cloudlive.com/corey to learn more and save your virtual seat today. That’s cloud-l-i-v-e.com/corey to register.Pete: Yeah, that’s exactly right. I mean, it’s just—and can you imagine, you have some knowledge you want to share around something as complex as the Amazon bill. I mean we ask for a PDF of your bill when you start working with Duckbill Group. That could be hundreds of pages long, and you’re trying to distill that down into something that, really, anyone can understand. It’s a true superpower to be able to write long-form content like that really well.And I never used to like writing. I was never—never really enjoyed it that much and over the last year, that muscle that you’re working out, now, the ability to write many, many pages around this type of content, just it comes so much more easily. So, I think that’s another big aspect, right? The more you work on it, obviously the easier it gets.Jesse: I don’t know about you, but now that I have focused more on flexing that writing and communication muscle, I’ve noticed it more in both everyone that I work with day-to-day with Duckbill Group and also in my daily life, just watching how people communicate with each other, and how effectively people communicate with each other; it’s both amazing and nerve-wracking all at the same time.Pete: [laugh]. I know. And even—not to say that whenever we sit down to write our reports that we give to our clients, we don’t go through the wave of emotions between the back and forth of, like, “I don’t know what to write,” and then, “Oh, I know of a lot of stuff to write. Let me just get something down.” And then you can’t stop writing. It’s just—it’s this emotional roller coaster that I feel like no matter how many times we need to write a lot of detailed information down, everyone always goes through.Amy: And we really do have a highly collaborative process here, too, where we’re all in the same document, writing, and the person who owns any given report will always have the same stage at the end when all of the sections are filled out, where they go to one of the other people on the team and go, “Every word I put down is absolute garbage. Please help me trim it down, take it out. I don’t even care anymore. Just look at it and tell me that I wrote down words that are in some kind of human language.” [laugh].Jesse: [laugh].Pete: [laugh]. Oh, the plight of the writer. It’s, like, the imposter syndrome that affects the writer. It’s like, “Okay. I wrote a bunch of stuff. I think it’s terrible.” And then you sleep on it, you come back the next day, and you’re like, “Actually, this is pretty good.” [laugh].Amy: I explained concepts. It was fine. I didn’t use a single comma for three pages, but it’s probably fine. [laugh].Jesse: [laugh].Pete: You can take one of mine. Usually, all of my draft documents are commas and M-dashes, just all over the place. Yeah, so I think that’s honestly a big superpower. And I think the last two things that—this is actually something that I’ve looked for in people that I’ve wanted to work with, and people I was hiring, and I see it here as well as these, kind of, two concepts of intellectual curiosity and aptitude to learn, where if you have a base knowledge around Amazon and you have those other attributes—that curiosity and truly enjoying learning—you can accelerate your ability to understand this so incredibly quickly because there’s such a wealth of information out there, and there’s so many documents, there’s so much stuff. It just requires someone who really cares enough to dive in and really want to understand.That’s something that I think we’ve seen here is that the folks who are most successful are just—they want to know why, and they’re not satisfied until they can explain it in a simple way to someone else. That’s the key, right? The attribute of a true expert is someone who can explain something very difficult in a simple way. And I think that’s something that would be critical if you were joining Duckbill, if you were building your own finops or cloud finance team, it is so complex. It’s the intersection of technical architecture and cost, and it touches almost the entire business. So, I think those are some other attributes that I think are just incredibly helpful.Jesse: We’re also usually not entirely satisfied until we’ve either opened a support case with AWS, responded to one of their feedback icons in the AWS documentation—the public AWS documentation—or trolled somebody on Twitter saying, “Shame on you, AWS, for writing documentation that doesn’t make sense.”Amy: It’ll be fine. Someone in your mentions will go, “Did you check the region?” And you would have, and then it’ll still be wrong.Jesse: [laugh].Amy: And it’ll be fine. [laugh]. Eventually, we’ll fix it.Pete: That one—Jesse: Too soon.Pete: —that one still hurts, when we—oh, I’m just like, “Why do the numbers not line up?” And then someone was like—Amy: It's a thing I check for, even if it’s like, “It’s a global resource.” I don’t care. Just tell me. Just tell me it’s fine. [laugh].Pete: “Are you in the right region?” Like—“Dammit, no, I’m not. Oh.” [laugh]. Yeah, that happens to the best of us.Amy: I did, unfortunately, burn so many hours, I think it was last week trying to find out where someone had put their resources. It’s like, “Oh, not us-west-2. It’s us-west-1. Of course.” [laugh].Jesse: So, annoying. Well, I would just like to say, Pete, it has been a joy and a pleasure working with you, it has been a joy and a pleasure complaining about AWS with you, on this podcast, so thank you for your time. That sounded really… really, really standoffish. I didn’t mean it quite as bad as it came off there. [laugh].Pete: Well, you know, I think we need to thank Corey for having a child and thus needing to offload some of his podcast duties over to us, and then the fact that we just never gave him the podcast back, and we just took it over.Jesse: Well, if you’ve got questions that you’d like us to answer, you can go to lastweekinaws.com/QA. And if you’ve enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us what qualities you’re looking for when building out your cloud finance team.Pete: Thanks for coming in.Announcer: This has been a HumblePod production. Stay humble.

Links: Cloud FinOps: https://www.amazon.com/Cloud-FinOps-Collaborative-Real-Time-Management/dp/1492054623 FinOps Foundation: https://www.Finops.org/ AWS cost management blog: https://aws.amazon.com/blogs/aws-cost-management/ Mastering AWS Cost Optimization: https://www.amazon.com/Mastering-AWS-Cost-Optimization-operational/dp/965572803X TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I am Pete Cheslock.Jesse: I’m Jesse DeRose.Pete: Wow, we’re back again. And guess what? We have even more questions. I am… I am… I don’t even know. I have so many emotions right now that are conflicting between a pandemic and non-pandemic that I just—I’m just so happy. I’m just so happy that you listen, all of you out there, all you wonderful humans out there are listening. But more importantly, you are going into lastweekinaws.com/QA and you’re sending us some really great questions.Jesse: Yeah.Pete: And we’re going to answer some more questions today. We’re having so much fun with this, that we’re just going to keep the good times rolling. So, if you also want to keep these good times rolling, send us your questions, and we’ll just—yeah, we’ll just roll with it. Right, Jesse?Jesse: Absolutely. We’re happy to answer more questions on air, happy to let you pick our brains.Pete: All right. Well, we got a couple more questions. Let’s kick it off, Jesse.Jesse: Yeah. So, the first question today is from Barry. Thank you, Barry. “New friend of the pod here.” Always happy to have friends of the pod. Although I do feel like that starts to get, like, Children of the Corn, kind of. I think we started that, and I also am excited about it, and also upset with myself for starting that.Pete: That’s all right. Friend of the pod. Friend of the pod.Jesse: “New friend of the pod here. I work in strategic sourcing and procurement and I was curious if there are any ways that you recommend to get up to speed with managing cloud spend. This is usually closely monitored by finance or different groups in product, but I can see a significant potential value for a sourcing professional to help, also.” And that’s from Barry, thank you, Barry.Pete: Well, I’m struggling not to laugh. “This is usually closely monitored by finance or different groups in product.”Jesse: Yeah…Pete: But I mean, let’s be honest, it’s not monitored by anyone. It’s just running up a meter in a taxi going 100 miles an hour.Jesse: Yeah, that’s the hardest part. I want everybody to be involved in the cloud cost management practice, but there’s that same idea of if it’s everyone’s responsibility, it’s no one’s responsibility. And so this usually ends up at a point where you’ve got the CFO walking over to the head of engineering saying, “Why did the spend go up?” And that’s never a good conversation to have.Pete: No, never a good one. Well, Barry because you’re a friend of the pod, we will answer this question for you. And honestly, I think it’s a great question, which is, we actually have been working with a lot of larger enterprises and these enterprises still have their classic sourcing and procurement teams. That’s not an expertise that is going away anytime soon, but like most teams within the company that are adopting cloud, it’s obviously going to evolve as people are moving away from, kind of, capital intensive purchases and into, honestly, more complex, multi-year OpEx style purchases, with cloud services and all the different vendors that come with it. It’s going to just get a lot harder.I mean, it’s probably already a lot harder for those types of teams. And so there’s a bunch of places I think that you can go that can help level up your skills around cloud spend. And I would say the first place that I personally got to dive in a little bit more—I mean, my history has been using Amazon cloud and being a person who cared about how much my company spent on it, but when you—joining Duckbill, you need to dive into other areas around the FinOps world. And the book, the O’Reilly book, Cloud FinOps is actually a really great resource.Yeah, I think it’s really well written and there’s a lot of great chapters within there that you can kind of pick and choose based on what you’re most interested in learning about. If you’re trying to learn more about unit economics, or you’re trying to learn more about how to monitor and track things like that, it’s a great book to dive into, and becomes a really great reference that you can leverage as you’re trying to level up this expertise within yourself or your team.Jesse: It’s a really, really great resource. The other thing to think about is any kind of collaborative social spaces where you can be with like-minded individuals who also care about cloud costs. Now, there’s a number of meetups that exist under the FinOps title that may be worth looking into. Obviously, we’re recording this during the pandemic so I don’t recommend doing those in person. But as you are able to, there may be opportunities for in-person meetups and smaller local groups focusing on cloud cost management strategies together. But also check out the FinOps Foundation. They have a Slack space that I would love to tell you more about, but unfortunately, we’re not allowed to join. So—Pete: Yep.Jesse: —I can’t really say more about it than that. I would hope that you’re allowed to join, but they have some strict guidelines. So, I mean, the worst that can happen is they say no; it’s definitely worth signing up.Pete: Yeah, and they have to us. [laugh].Jesse: Yeah.Pete: I think when you get into the FinOps Foundation, you should angrily say that we should have more FinOps experts in here like the great Jesse DeRose should be a member of this one because right now, he’s just framed his rejection notice from there, and—Jesse: Oh, yeah.Pete: —while it looks beautiful on the wall, while I’m on a Zoom with him, I want more for you, Jesse.Jesse: I want more for me, too. I’m not going to lie.Pete: So, I don’t know this might sound a little ridiculous that I’m going to say something nice about AWS, but they have a fantastic cost management blog. This is a really fantastic resource, really incredible resource, with a lot more content more recently. They seem to be doing some great work on the recruiting side and bringing on some real fantastic experts around cost management.I mean, just recently within the past few months they talk about unit economics: How to select a unit metric that might support your business, talking more about unit metrics in practice. They start at the basics, too. I mean, obviously, we deal a lot in unit economics and unit metrics; they will start you off with something very basic and say, “Well, what even is this thing?” And talk to you more about cost reporting using AWS organizations for some of this. It’s a really fantastic resource.It’s all free, too, which is—it’s weird to say that something from AWS is free. So, anytime that you can find a free resource from Amazon, I say, highly recommend it. But there are a lot of blogs on the AWS site, but again, the Cost Management Blog, great resource. I read it religiously; I think what they’re writing is some of, really, the best content on the blog in general.Jesse: There’s one other book that I want to recommend called Mastering AWS Cost Optimization and we’ll throw links to all these in the [show notes 00:07:30], but I, unfortunately, have not read this book yet, so I can’t give strong recommendations for it, but it is very similar in style and vein to the Cloud FinOps book that we just mentioned, so might be another great resource to pick up to give you some spot learning of different components of the cloud cost management workflow and style.Pete: Awesome. Yeah, definitely agree. I’d love to see, again, more content out here. There’s a lot of stuff that exists. And even A Cloud Guru has come up with cost management training sessions.Again, we’d like to see more and more of this. I’d love to see more of this come from Amazon. I’d love to see—you know, they have a certification path in all these different areas; I’d love to see more of that in the cost management world because I think it’s going to become more complex, and having that knowledge, there is so much knowledge, it’s spread so far across AWS, helping more people get up to speed on it will be just critical for businesses who want to better understand what their spend is doing. So, really great question, Barry, friend of the pod. We should get some pins for that, right? Friend of the pod pins?Jesse: Oh, yeah.Pete: And yeah, really great question. Really appreciate you sending it and hopefully that helps you. And if not, guess what? You can go to lastweekinaws.com/QA, and just ask us a follow-up question, Barry. Because you’re a friend of the pod. So, we’ll hopefully hear from you again soon.Jesse: Thanks, Barry.Pete: Thanks.Announcer: If your mean time to WTF for a security alert is more than a minute, it’s time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you’re building a secure business on AWS with compliance requirements, you don’t really have time to choose between antivirus or firewall companies to help you secure your stack. That’s why Lacework is built from the ground up for the cloud: Low effort, high visibility, and detection. To learn more, visit lacework.com.Pete: All right, we have one more question. Jesse, what is it?Jesse: “All right, most tech execs I speak with have already chosen a destination hyperscaler of choice. They ask me to take them there. I can either print out a map they can follow, procedural style, or I can be their Uber driver. I could be declarative. I prefer the latter for flexibility reasons, but having said that, where does one actually start?Do you start with Infrastructure as a Service and some RDS to rid them of that pesky expensive Oracle bill? Do we start with a greenfield? I mean, having a massive legacy footprint, it takes a while to move things over, and integrating becomes a costly affair. There’s definitely a chicken and egg scenario here. How do I ultimately find the best path forward?” That question is from Marsellus Wallace? Thank you, Marsellus.Pete: Great question. And I’m not just saying that. I guess I have a question. Or at least, maybe we have different answers based on what this really looks like. Is this a legacy data center migration?The solution here is basically lift-and-shift. Do it quickly. And most importantly, don’t forget to refactor and clean up after you shut down your old data center. Don’t leave old technical debt behind. And, yeah, you’re going to spend a lot, you’re going to look at your bill and go, “Holy hell, what just happened here?”But it’s not going to stay that way. That’s probably—if you do it right—the highest your bill is going to be because lift-and-shift means basically just moving compute from one location to another. And if you’re—as we spoken about probably a million times, Jesse and I, if you just run everything on EC2 like a data center, it’s the most expensive way to do the cloud stuff. So, you’re going to then refactor and bring in ephemerality and tiering of data and all those fun things that we talk about. Now, is this a hybrid cloud world?That’s a little bit different because that means you’re not technically going to get rid of, maybe, physical locations or physical data centers, so where do you start? It’s my personal opinion—and Jesse has his own opinion, too, and guess what it’s our podcast and we’re going to tell it like it is.Jesse: [laugh].Pete: [laugh]. You know, my belief is, starting with storage is honestly a great way to get into cloud. Specifically S3. Maybe even your corporate file systems, using a tool like FSX. It’s honestly why many businesses start their cloud journey, by moving corporate email and file systems into the cloud.I mean, as a former Microsoft Exchange administrator, I am thoroughly happy that you don’t have to manage that, really, anymore and you can push that in the cloud. So, I think storage is honestly a great way to get started within there: Get S3 going, move your file systems in there, move your email in there if you haven’t yet. That’s a really great way to do it. Now, the next one that I would move probably just as aggressively into and, Marsellus, you mentioned it: RDS, right? “Should we move into RDS, get rid of expensive Oracle bills?”Yeah, anytime you can pay ol’ Uncle Larry less money is better in my mindset. Databases are, again, another really great way of getting into AWS. They work so well, RDS is just such a great service, but don’t forget about DMS, the database migration service. This is the most underrated cloud service that Amazon has in there, it will help you migrate your workloads into RDS, into Amazon Aurora. But one thing I do want to call out before you start migrating data in there, talk to your account manager—you have one even if you don’t think you have one—before starting anything, and have them help you identify if there are any current programs that exist to help you migrate that data in.Again, Amazon will incentivize you to do it, they will provide you credits, like map credits or other investment credits, maybe even professional services that can help you migrate this data from an on-premise Oracle into AWS, I think you will be very pleasantly surprised with how aggressive that they can be to help you get into there. The last thing that I would say is another great thing to move in our data projects. So, let’s say you want to do a greenfield one, greenfield type of project into Amazon, data projects are a really great way to move in there. I’m talking things like EMR, Databricks, Qubole, you get to take advantage of Spot Fleets with EMR, but also Databricks and Qubole can manage Spot infrastructure and really take advantage of cloud ephemerality. So if, like I said, you started by pushing all your data into S3, you’re already halfway there on a really solid data engineering project, and now you get to leverage a lot of these other ancillary services like Glue, Glue DataBrew, Athena, Redshift.I mean, once the data is in S3, you have a lot of flexibility. So, that’s my personal opinion on where to get started there. But Jesse, I know you always have a different take on these, so where do you think that they should start?Jesse: Yeah, I think all of the recommendations you just made are really, really great options. I always like to look at this from the perspective of the theory side or the strategy side. What ultimately do these tech execs want to accomplish? Is it getting out of data centers? Is it better cost visibility?Is it optimizing spend? Is it better opportunity to move fast, get new R&D things that you can’t get in a data center? What do these tech execs ultimately want to accomplish? And ask them. Start by asking them.Prioritize the work that they want to accomplish first, and work with teams to change their behaviors to accomplish their goals. One of the biggest themes that we see in the space moving from data centers into cloud providers or even just growing within a given cloud provider is cost visibility. Do teams know why their spend is what it is? Do they know why it went up or down month-over-month? Can they tell you the influences and the drivers that cause their spend to go up or down?Can they specifically call out which teams or product usage increased or decreased, and what ultimately led to your spending changing? Make sure that every team has an architecture diagram and they can explain how they use AWS, how data moves from one service to another, both within their product and to other products. Because there’s definitely going to be sharp edges with data transfer between accounts. We’ve seen this happen to a number of clients before; I’ve gotten bit by this bullet. So, talk to your teams, or talk to your tech executives and have those tech executives talk to their teams to understand what do they ultimately want to accomplish?Can they tie all of what they’re trying to accomplish back to business metrics? Maybe a spike in user logins generated more usage? If you’re a photo storage company, did a world event prompt a lot of users to upload photos prompting higher storage costs? Are you able to pull out these specific insights? That’s ultimately the big question here. Can you boil it down to a business KPI that changed, that ultimately impacted your AWS spend?Pete: I think this is a scenario of where you get started. Why not both? Just maybe do both of these things that we’re saying.Jesse: Yeah.Pete: And honestly, I think you’ll end up in a pretty great place. So, let us know how that works out, Marsellus, and thank you for the question. Again, you also can send us your questions, and we will maybe answer these on a future episode; lastweekinaws.com/QA, drop a question in there, put your name, or not or a fake name, or even a joke. That’s fine, too. I don’t know what the text limit is on the name, Jesse. Can you put a joke there? I don’t know. You know what? Test that out for us. It’s not slash QA for nothing. So, give that a little QA, or a question and answer and [unintelligible 00:17:29]. All right. Well, thanks, Jesse, for helping me out answering more questions.Jesse: Thanks, everybody for the awesome questions.Pete: If you enjoyed this podcast, please go to lastweekinaws.com/review, give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review and give it a five-star rating on your podcast platform of choice and tell us, what would be the last thing that you would move to AWS? It’s QuickSight, isn’t it?Jesse: [laugh].Pete: Thanks, everyone. Bye-bye.Announcer: This has been a HumblePod production. Stay humble.

Links:Unconventional Guide: https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/ TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. We’re back again, my name is Pete Cheslock.Jesse: I’m Jesse DeRose. So, happy to be back in the studio after our whirlwind tour of the Unconventional Guide that I feel like we’ve been on for roughly as long as the pandemic’s been going on at this point; probably a little bit less. But lots of really great content there that we were happy to talk about, and I’m happy to be moving on to some other topics.Pete: Yeah, absolutely. And the topics, we actually get to move on to some of our favorite topics, which are answering your questions. And it turns out, Jesse, there’s more than two people that listen to us. There’s a lot of you; there are dozens of you out there, and we love it.Jesse: You like me. You really like me.Pete: So, great. So, great to see. We’ve been getting tons of fantastic questions, a few of which we’re going to answer right now. You can also have your question answered by going over to the lastweekinaws.com/QA and enter in your question there. You can enter in your name, or you can leave it blank, or you could just put something funny there. Anything works. We’re happy to dive in deeper on any particular topic, again, whether it’s about this recent Unconventional Guide series or just something you’re curious about in your day-to-day in your cost management life.Jesse: Today’s questions are really great because they ultimately get at the practical side of all of our recommendations. Because I feel like every single time I subscribe to one of those self-help books or blogs and I read all these really great short, sweet tidbits, I think to myself, “This is perfect. I’ll go apply this to everything in my life.” But then doing the actual work part is so much harder. Where do you even start with that first step once you’ve got the big picture grand idea? So, today we’ve got some really, really great questions, focusing on the best ways to get started on your cloud cost management journey. So, let’s start off with these questions.First question is, “Could you cover some practical approaches to applying some of your Cost Management Guide? A lot of your suggestions sound simple on paper, but in practice, they become quite complicated.” So, true. Absolutely, absolutely a concern. “I’ve had some success pulling in a small group of subject matter experts together for short periods of time focusing on low risk, easy things to do. How have you approached actually doing this? What meetings do you set up? What do you take for notes? How do you document your savings? How do you find new opportunities?” That’s from Brian O. Brian O., That’s a really, really great question.The other one that I want to add to this: “We’re a big AWS shop, and I’ve spent some time inside the AWS beast in the past, and I still struggle with multi-account multi-region data transfer in general, but specifically analyzing cost and usage. There are examples specifically like if data transfer out goes up $25,000 last month, how do you attribute that? How do you know where to apply that? How do you know what ultimately prompted that spend? Love how you work through these types of challenges. What is relatively easy at a single account level gets exponentially more complex with every account and region we function in.” So, true. And that’s from Todd. Thank you, Todd. In both cases, absolutely true.There’s this really great idea of we can give you the really short and sweet things to think about, but taking those first steps for practically applying these ideas is tough, and it needs to scale over time. And not every practice does.Pete: Yeah, these are great questions. I, kind of, am remembering that meme that was around for a while, which was, how to draw an owl. “First, draw two circles, and then, you know, you draw the rest of the owl.”Jesse: Yeah.Pete: And honestly, oftentimes, some of the stuff even that we say, Jesse, feels that way, and it doesn’t intend to come across that way. It’s just, we could bore you all on a multi-hour long recording of some of these topics. I mean, we do this with our clients, and our clients pay for this pleasure [laugh] for us to put them to sleep with our soft tones of the cloud cost management world. But I think the reality is that it is complex and there are probably unlikely to be quick wins in a lot of these places. One thing that we found is honestly, monitoring, visibility, I think all the cool kids are calling it observability now—Jesse: [laugh].Pete: —you know, I can’t believe I’m going to say this, but CloudWatch is actually probably one of the best cloud cost reduction tools that exist out there. There are so many services within AWS that you’re probably using today, that by default, report data to CloudWatch. And those statistics are potentially a huge place to identify resources that are over-provisioned and underused, idle resources, things like that. I can’t tell you how many times that I will go into a client account, and one of the first places I go to is—after Cost Explorer—is probably CloudWatch. So, monitoring spend and monitoring what’s happening there is kind of a great way to get started on that cloud cost idea because you’re getting charged for everything that happens, so knowing what’s happening, and knowing how it’s changing over time is a great way to start understanding and reducing it.Jesse: Yeah. And I think AWS is probably also using some of those CloudWatch metrics in their optimization recommendations that they make within their own optimization tooling. And it’s probably just not clearly defined or clearly outlined for AWS customers to be able to use the same metrics. So, I feel like if my Compute Optimizer could quickly load or link to a graph that showed me low CPU utilization across a number of instances, that’s a really handy way for me to start using more of CloudWatch’s metrics.Pete: Yeah, I think Compute Optimizer is honestly, criminally underused out there. I don’t know why. Then honestly, one of the other complaints is like, “Well, you can’t get memory statistics unless you have a CloudWatch Agent.” Yes. So honestly, install the CloudWatch agent; have it report up, the, like, one or two memory metrics that Compute Optimizer needs to make a recommendation and the cost will more than pay for itself.And now you can even output those statistics to S3 and do some fun programmatic stuff with it. Put those outputs in front of the engineers that own those resources and be like, “Hey, yo. This thing says, change your i3.24xl. Could you move it to something a little bit more useful, like a t3.small?”Jesse: And these are just some practical applications for some of the specific metrics we’re talking about, but this is a practice that you might want to turn into a process, you might want to turn into an ongoing amount of work. And in a lot of cases, we’ve seen this start as one engineer who’s really interested in understanding AWS, really passionate about the bill, maybe isn’t in a leadership or management role so maybe they don’t have a direct business requirement to optimize their spend, but they’re really, really interested in this work and they grow into a role where they are taking on more and more of this work. And that’s not scalable; that engineer is going to get burned out very, very quickly if they have a day-to-day role that is focused in development and doing all of this optimization work, cost cloud management work on the side. We generally recommend at least one dedicated individual who starts building these dashboards, starts looking at some of these metrics, starts these conversations with teams, and ultimately grow that into a full team.Pete: Right. I think that’s the biggest thing that we’re seeing in the industry is actual cloud finance teams coming into existence at companies. It’s such a critical role and it’s sad to see when people are like, “Arg, spend is out of control. We’re doubling year over year on spend and no one really seems to know why.” And honestly, it’s because no one cares about it. You don’t have any ownership on it. And, you know, we see it a lot, right? It’s like, “Well, everyone owns the Amazon bill.” That’s code for, “No one owns the Amazon bill.”Jesse: Yeah.Pete: But these cloud finance teams, and even the term cloud economist, as silly as it is, it’s centered in reality, which is we create financial models to understand spend and we dive into those numbers to make the usage makes sense to folks like CFOs inside of companies. Yeah, there’s a couple of ways that we have seen some of this done at scale. In one case using kind of active monitoring, and actively monitoring the spend based on really granular budgets, and reporting it as such. So, maybe you’re breaking these budgets out to be product-specific, or team specific, or business unit, or things like that, and then basically reaching out to these engineering teams. Because you are actively monitoring the spend on a recurring basis, you can reach out to those teams, when their spend goes over a given threshold that you’ve put in place, or when you, maybe, find some optimization opportunities.You’re probably thinking to yourself, “Wow, I don’t have the time for that.” Yeah, but you need to create the time or you need to create the team for this. The companies that we work with who have a dedicated team around this are the ones that do the best. In some cases, we’ve seen having a Dedicated Cloud finance team causes the bill to actually decrease over time, which, you’re thinking to yourself, “Wow. An Amazon bill that goes down? We so rarely see that.”Even for us, our clients come to us, we help them find optimizations. They’ll make those optimizations, but then they replace that spend with other investments. Usually, it’s new projects and new spend. But actually seeing the bill go down because of a dedicated effort of a team is still, again, amazing to see. The other side is we’ve seen maybe more of a passive monitoring or something around the background of things where you have a cloud platform team that provides abstractions and guardrails to the user.So, you’re not really trying to actively stand in the way of users and what they’re able to do and reaching out to them in an ongoing way, but you’re abstracting away the kind of complexity of the cloud and letting them basically live in a safe space that you are controlling for them. And that’s another way that you can kind of build in some of this cloud financial knowledge where teams can get that visibility into what they’re spending and know, is this too high? Is it going out of a boundary? Is there a number that I need to keep inside of? I think these are important things that level of visibility around cost and that team’s actual charges get people to start thinking, “Well, hold on a second. We’re above budget.” Even though maybe it’s not a real budget, “We’re above a spend by 20%. We need to bring that down.” And you give them the tools they need and the dashboards to effect that change on their own.Jesse: This idea of passive monitoring is really all about making the right thing the easy thing to do. If you, as a member of the cloud platform team or as a member of leadership who cares about cloud spend, wants to make sure that teams are managing their spend in some capacity, maybe not actively directly, but at some level make sure that there are these guide rails in place that keep them within the boundaries of what they’re ultimately able to do, or what you ultimately want them to work on. This makes it a lot easier for them to not spin up an i3 instance that they don’t ultimately need; it makes it a lot easier for them to not deploy resources that are missing tags. Put as many guardrails in place that keep the teams independently able to work within the space that they are building, and developing, and functioning, but ultimately gives them the opportunity to continue being independent and really thrive within whatever work they’re doing.Pete: Yeah, the next thing that we recommend to everyone. And actually, we recommend it before even engaging with The Duckbill Group, you’ll get an onboarding document of things to do, and the thing we always recommend is turn on the Cost and Usage Report. If you’re listening to this and you’re like, “What’s the Cost and Usage Report?” Well, boy are you going to have a fun learning because it is a highly granular usage report of everything that you’ve ever done within Amazon, and it’s extremely powerful. The downside is that it can be hard to navigate; it takes a little time to learn.But go turn it on; the cost is minimal; it’s the cost of storing this data in S3. Preferably when you turn this on, turn it on into Parquet format because it’ll allow you to query it with tools like Athena, or Tableau, or Looker or—God forbid—SageMaker. And this tool, this Cost and Usage Report, lets you dive in at an extremely granular level, down to the resource visibility—per hour, per resource visibility. And it’s something you have to enable, but again, highly recommend it to enable that resource level usage. Because now you can go and find out, well, for SageMaker I’m seeing a growth in spend.Well, which resource is it within SageMaker? You can break that down really granularly. So, Cost and Usage Report is another place that, again, if you’re not using this today, if you don’t have at least a SageMaker dashboard, which costs basically nothing—a couple of dollars a month—pointed at your Cost and Usage Report, you’re missing out on some really great ways to understand the changes in spend over time.Announcer: If your mean time to WTF for a security alert is more than a minute, it’s time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you’re building a secure business on AWS with compliance requirements, you don’t really have time to choose between antivirus or firewall companies to help you secure your stack. That’s why Lacework is built from the ground up for the Cloud: low effort, high visibility, and detection. To learn more, visit lacework.com.Jesse: Another couple of really great options are the AWS Cost Anomaly Detection service and AWS Budgets. Both are free, which is absolutely fantastic. I highly recommend checking them out. AWS Cost Anomaly Detection, once enabled, will actually look for anomalies in your spend across different AWS services, across different cost attribution tags, across different cost categories. There’s a lot of opportunities here for you to see anomalous spend and act on it.This can be shared with teams as soon as the anomaly occurs, through Slack notification or an email, or maybe you get email notifications on a weekly basis, or a monthly basis, or some kind of recurring basis, for all of the anomalies that you saw within a given time period. We recorded an episode about Cost Anomaly Detection a while back; highly recommend checking that episode out. It’s got a lot of really great features and recommendations for getting started.The other one I mentioned is AWS Budgets. Again, if you’re not really sure where to start, try creating some budgets for your teams. Maybe look at the last six months of spend for each team, maybe look at spend across different tags, or team units, or business units, whatever makes the most sense for the way that your organization is set up, and create some budgets for those groups. These budgets could be for specific AWS services if you are a single team running within a single AWS account, it could be as complex as multiple business units across multiple different accounts across different parts of the organization. There’s lots of great opportunities here for you to start to better understand your spend, get better visibility into your cloud spend.Pete: Yeah, absolutely. I think all of those are great tools that can really help you. And, Jesse, I know we’ve talked about this before. Even just monitoring your tagging, not like, “Oh, are we tagging 50% of our resources?” But you want to monitor for your untagged by spend. So, if 95% of your spend is tagged, you’re crushing it. That’s amazing. But that may only be 50% of your things.So, I guess, care less about how many of your resources are tagged—because some of them just can’t be tagged, or are tagged in a painful way—but focus more on the money aspect of it. And that will lead you into the ability to start creating some governance strategies. And that term governance, it just—Jesse: Oof.Pete: —makes me feel gross. Yeah. Oh god, terrible word. But the [laugh] sad state of the world is, that’s what most companies we talk to need; they just don’t have it. When the companies that we talk to who are like, “Our spend is going up, and we’re not sure why.” Or, “How do we get our engineers to care about cost savings?” And things like that. You know, having a governance strategy, a way to react to those changes in spend in a, hopefully, automated way, is critical to helping control that spend.Jesse: This really gets to the heart of why is cloud cost management important? It could be important for different reasons for different parts of the organization. Account structure, tagging, all of these different things can be important for different parts of the organization for different reasons. And that’s fine. The important thing is to socialize those reasons why to all the different parts of the company so that everybody understands what’s at stake.Everybody understands how they can collaborate and create these best practices together. This really dives into the idea of behaviors and systems. I know it sounds a little bit not within the vein of engineering work, and finance, and cloud cost management, but what kind of behaviors do you ultimately want to see within your teams? What kind of actions do you want to see your engineers taking? Do you want them to start thinking about cost in all of their architecture discussions?Do you want them to review the budgets that you’ve created for them every month? Every week? During stand-up meetings? What kind of things do you ultimately want to see them doing on a regular basis that maybe they aren’t doing right now, that maybe would ultimately help the company succeed with all of this cloud cost management work that you’re creating? And again, going back to the idea of making the right thing the easy thing to do, how can you improve the existing technical systems that you have within your organization to make the right thing the easy thing to do?How can you change your CI/CD pipelines? How can you change the tools that you’re using for cost visibility, like Looker, or Tableau, or SageMaker, or something else, such that teams can quickly and easily self-service the information that they need to make their decisions to go about their days, go about their work more easily?Pete: So, Jesse, you’re saying that it’s a mixture of software and culture? Kind of sounds like DevOps a little bit, doesn’t it? [laugh].Jesse: Yeah. Yeah, it kind of does.Pete: Yeah, it kind of does. So, you know, I think all of that is to say, it’s hard work, it’s not going to come easy, but how would we get started? Like, when we enter into an engagement with one of our clients, we’re coming in from total outsiders and we’re trying to navigate through a company with complex communication structures, and maybe teams that are entrenched in different ways. How do we get started? Well, we dive in; we start with big numbers, right?What are your top ten places your money goes, just by service? I’ll answer it for you. It’s probably EC2, S3, RDS, and then dealer’s choice for the last ones, maybe data transfer, maybe Lambda, if you’re really weird. And if Lambda is in your top five, you should absolutely give us a call because that should not be the case. [laugh].But start with those big numbers, understand where the money is going. But then go to the next level in. Okay, within EC2, where is your spend going? Or the dastardly EC2 ‘other cost’ category; okay’s the money going? Is it in regional data transfer, which is also what’s called cross-AZ data transfer? Is it in your NAT gateways? Why?That’s the next question. Why is the spend high in that area? You may not be able to understand because it may not be tagged—we find that a lot—but start asking questions. And that’s what we do: We start reaching out to technical folks within the company. We’d say, “Hey, we see you’ve got a high amount of usage on EMR, but they’re all clusters that are running 24/7. They’re not scaling up and down as the jobs are happening. Who knows more about EMR?” And we just start asking questions. And we’re asking them, “Well, are you doing anything on the cost optimization side? Have you tried to do anything cost optimization-wise to reduce it, and you haven’t been able to? How does this infrastructure scale? Does it scale linearly with the number of users? Does it scale in a different way? Who are the consumers?”And then you kind of even go another level down to see, do you find anything that just looks odd? I saw on one account for a client we were working with, VPC costs were just extremely high, much higher than I’ve ever seen before. What was interesting is that the cost was not data-transfer related; it was the pure number of endpoints that they had created that that cost far outweighed any other costs to data transfer; there was just a piece of technical debt that they were aware of, but the structure of their multi-accounts, they just couldn’t do anything about it. But again, you’re looking for things like that. And you know that you are doing a good job if, essentially, you can get to the end of this process—which could take months and it could take years depending on your scale—is if you can answer this question that if your customers, or users, or consumers of the applications on your cloud service if they increased by 200%, 500%, 1,000%. What would happen to your cloud spend? How would it change? That’s the end game you’re trying to get to. That’s the unit economics, the unit economic model and forecasting, and now you’re a superhero because now you can answer that question that not a lot of people are able to answer with their cloud usage.Jesse: I also want to add that, as you’re asking questions, you’re going to find teams that specifically will tell you, “We created this infrastructure in this way because security told us to,” or, “Because our business requirements say that we have an SLA that means we need to keep data for this amount of time at this level of availability.” And that’s totally fine. That doesn’t mean that you need to necessarily change those requirements. But now you might have a dollar amount for those business decisions. Now, you might ultimately be able to say, okay, our product SLA may say that we need to keep data for 90 days, but keeping data for 90 days, that business decision is costing us hundreds of thousands of dollars every month because of the sheer volume of data that we now have to keep. Is that something that we ultimately are okay with? And are we okay spending that much money every month to keep this business decision, or do we need to revisit that business decision? And that’s only something that you and your teams can decide for yourselves.Pete: Awesome. These are great questions. You could also send us a question lastweekinaws.com/QA. We would love to spend some time diving into it and just helping you out and helping you get through your day. That’s what we’re here for.If you enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review. Give it a five-star review on your podcast platform of choice, and tell us what is your favorite EC2 instance to turn off for your engineers.Jesse: [laugh].Pete: Thanks, everyone.Announcer: This has been a HumblePod production. Stay humble.

Pete Cheslock is a cloud economist at The Duckbill Group and an advisor and consultant who helps startups with product strategy, messaging, and other go-to-market needs. Prior to these positions, he worked at a slew of tech companies, holding positions such as VP of Products at ChaosSearch, VP of Technical Operations at Threat Stack, Inc., Director of DevTools at Dyn, and Director of Technical and Cloud Operations at Sonian. Pete holds a masters in business administration from Babson and a bachelors in communications from Michigan State University. Join Corey and Pete as they talk about the virtual edition of re:Invent, what it was like to make fun of companies in a virtual expo hall, why vendors were aggressive in following up with leads from re:Invent, how virtual booth pricing at re:Invent didn’t really make any sense, what Corey and Pete like so much about the expo hall, how Pete enjoyed not having to spend a week in Vegas and come home sick this year, how people don’t follow AWS events like folks follow rock bands and why that’s a good thing, how re:Invent has evolved over time and how that evolution continues today, and more.

Links: Unconventional Guide to AWS Cost Management: https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/ Migrate from Oracle to Amazon Aurora: https://aws.amazon.com/getting-started/hands-on/migrate-oracle-to-amazon-aurora/ TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I am Pete Cheslock.Jesse: I’m Jesse DeRose.Pete: We’re coming at you again with some more listener questions from the Unconventional Guide to AWS Cost Management. I’m excited. People are listening to us, Jesse.Jesse: This is fantastic. I’m really excited that we have one fan. I’ve always wanted one fan.Pete: Well, two fans now. Maybe even more because we keep getting questions. And you can also be one of our Friends of the Pod by going to lastweekinaws.com/QA. And you can give us some feedback, you can give us a question and, like, will totally answer it because we like Friends of the Pod.Jesse: We may or may not enter you into a raffle to get a Members Only jacket that’s branded with ‘Friends with the Pod.’Pete: We should get some pins made, maybe.Jesse: Ohh…Pete: I think that's a good idea.Jesse: Yeah.Pete: So, what are we answering today, or attempting to answer for our listener, Jesse?Jesse: So today, we’ve got a really great question from [Godwin 00:01:20]. Thank you, Godwin, Godwin writes, “I truly believe that the system that I support is, like, a data hoarder. We do a lot of data ingestion, we recently did a lift-and-shift of the system to AWS, we use an Oracle database. The question is, how do I segregate the data and start thinking about moving it out of traditional relational databases and into other types of databases? Presently, our method is all types of data goes into a quote-unquote, ‘all-purpose database,’ and the database is growing quite fast. Where should I get started?”Pete: Well, I just want to commend you for a lift-and-shift into Amazon. That’s a Herculean feat, no matter what you’re lifting and shifting over. Hopefully, you have maybe started to decommission those original data centers and you don’t just have more data in twice as many locations.Jesse: [laugh]. But I also want to call out well done for thinking about not just the lift-and-shift, but the next step. I feel like that’s the thing that a lot of people forget about. They think about the lift-and-shift, and then they go, “Awesome. We’re hybrid. We’re in AWS, now. We’re in our data center. We’re good. Case closed.” And they forget that there’s a lot more work to do to modernize all those workloads in AWS, once you’ve lifted and shifted. And this is part of that conversation.Pete: Yeah, that’s a really good point because I know we’ve talked about this in the past, the lift-and-shift shot clock: when you don’t start migrating, start modernizing those applications to take advantage of things that are more cloud-native, the technical debt is really going to start piling up, and the folks that are going to manage that are going to get more burnt out, and it really is going to end poorly. So, the fact you’re starting to think about this now is a great thing. Also, what is available to you now that you’re on AWS is huge compared to a traditional data center.Jesse: Yeah.Pete: And that’s not just talking about the—I don’t even know if I’ve ever counted how many different databases exist on Amazon. I mean, they have a database for, at this point, every type of data. I mean, is there a type of data that they’re going to create, just so that they can create a database to put it into?Jesse: Wouldn’t surprise me at this point.Pete: They’ll find a way [laugh] to come up with that charge on your bill. But when it comes to Oracle, specifically Oracle databases, there’s obviously a big problem in not only the cost of the engine, running the database on a RDS or something to that effect, but you have licensing costs that are added into it as well. Maybe you have a bring-your-own-license or maybe you’re just using the off-the-shelf, but the off-the-shelf, kind of, ‘retail on-demand pricing’ RDS—I’m using air quotes for all these things, but you can’t see that—they will just have the licensing costs baked in as well. So, you’re paying for it—kind of—either way.Jesse: And I think this is something also to think about that we’ll dive into in a minute, but one of the things that a lot of people forget about when they move into AWS says that you’re not just paying for data sitting on a piece of hardware in a data center that’s depreciating, now. You’re paying for storage, you’re paying for I/O costs, you’re paying for data transfer, to Pete’s point, you’re also paying for some of the license as well, potentially. So, there’s lots of different costs associated with keeping an Oracle Database running in AWS. So, that’s actually probably the best place to start thinking about this next step about where to get started. Think about the usage patterns of your data.And this may be something that you need to involve engineering, maybe involve product for if they’re part of these conversations for storage of your product or your feature sets. Think about what are the usage patterns of your data?Pete: Yeah, exactly. Now, you may say to yourself, “Well, we’re on Oracle”—and I’m sure people listening are like, “Well, that’s your problem. You should just move off of Oracle.” And since you can’t go back in time and undo that decision—and the reality is, it probably was a good decision at the time. There’s a lot of businesses, including Amazon, who ran all of their systems on Oracle.And then migrated off of them. Understanding the usage patterns, what type of data is going into Oracle, I think is a big one. Because if you can understand the access patterns of the types of data that are going in, that can help you start peeling off where that data should go. Now, let’s say you’re just pushing all new data created. And we don’t even know what your data is, so we’re going to take some wild assumptions here on what you could possibly do—but more so just giving you homework, really—thinking about the type of data going in, right?If you’re just—“I’m pushing all of my data into this database because someday we might need to query it.” That’s actually a situation where you really want to start thinking of leveraging more of a data warehouse-style approach to it, where you have a large amount of data being created, you don’t know if you’re going to need to query it in the future, but you might want to glean some value out of that. Using S3, which is now available to you outside of your data center world, is going to be super valuable to just very cheaply shove data into S3, to be able to go back in later time. And then you can use things like Athena to ad hoc query that data, or leverage a lot of the ingestion services that exist to suck that data into other databases. But thinking about what’s being created, when it is going into places is a big first step to start understanding, well, how quickly does this data need to come back?Can the query be measured in many seconds? Can it be done ad hoc, like in Athena? Does it need to be measured in milliseconds? What’s the replication that needs to happen? Is this very valuable data that we need to have multiple backups on?Is it queried more than it’s created? Maybe you need to have multiple replica reader databases that are there. So, all these types of things of really understanding just what’s there to begin with, and it’s probably going to be in talking to a lot of engineering teams.Jesse: Yeah, you can think about this project in the same way that you might move from a monolith to a microservice architecture. So, if you’re moving from a monolith to a microservice architecture, you might start peeling away pieces of the monolith, one at a time. Pieces that can easily be turned into microservices that stand on their own within the cloud, even if they’re running on the same underlying infrastructure as the monolith itself within AWS. And then, as you can pull those pieces away, then start thinking about does this need to be in a relational database? Does this need to have the same amount of uptime and availability as the resources that are sitting in my Oracle Database right now?All those things that Pete just mentioned, start thinking about all of those components to figure out where best to pull off the individual components of data, and ultimately put them in different places within AWS. And to be clear, there’s lots of great guides on the internet that talk about moving from your Oracle database into, gosh, just about any database of choice. AWS even has specific instructions for this, and we’ll throw a link in the [show notes 00:09:02].They really, really want you to move this data to RDS Aurora. They go through painstaking detail to talk about using the AWS schema conversion tool to convert your schema over; they talk about the AWS database migration service to migrate the data over, and then they talk about performing post-migration activities such as running SQL queries for validating the object types, object count, things like that. I think that a lot of folks actually don’t know that the database migration service exists, and it’s something worth calling out as a really powerful tool.Pete: Yeah, the Amazon DMS service is honestly I think, a super-underrated service that people just don’t know about. It has the ability to replicate data from both on-premises databases to Amazon databases but also databases already running on Amazon. You could replicate from a database running on EC2 into Aurora. You could replicate that into S3—you know, replicate data into S3 that way, bringing things into sync—replicate that data into S3, and then maybe use it for other purposes. It can replicate data from DocumentDB into other sources.So, they’re clearly doing a big investment in there. And to Jesse’s point, yeah, Amazon really wants this data. So, talk to your account manager as you’re testing out some of these services. Do a small proof of concept, maybe, to see how well it works, if you can understand the queries, or you can point your application over at an Aurora database with some of this data migrated in; that’s a great way to understand how well this could work for your organization. But as Jesse mentioned, they do want that data in Aurora.So, if it turns out that you’re looking at your—you know, migrate some data in there, and it’s starting to work, and you’re kind of getting a feel for the engineering effort to migrate there, stop. Talk to your account manager before you spend any more money on Aurora because it’s very likely that they can put together a program—if a program doesn’t already exist—to incentivize you to move that data over; they can give you subject matter expertise; they can provide you credits to help you migrate that data over. Don’t feel like you have to do this on your own. You have an account team; you should definitely reach out to them, and they will provide you a lot of help to get that data in there. They’ve done it for many of their other clients, and they’re happy to do it for you because they know that, long term, when you move that data to Aurora, it’s going to be very sticky in Aurora.You’re probably not going to move off of there. It’s a long game for them; that’s how they play it. So, check out those services; that could be a really great way to help you get rid of your Oracle addiction.Jesse: Yeah, and if you’re able to, as we talked about earlier, if you’re able to identify workloads that don’t need to run in a relational database, or don’t need to run in, maybe, a database at all, for that matter, stick that data in S3. Call it a day. Put them on lifecycle management policies or different storage tiers, and use Athena for ad hoc queries, or maybe Redshift if you’re doing more data warehouse-style tasks. But if that data doesn’t need to live in a relational database, there are many cheaper options for that data.Pete: Exactly. But one last point I will make is don’t shove it into MongoDB just because you want to have schema-less, or—Jesse: Please.Pete: —think about what you’re going to use it for, think about what the data access patterns because there is a right place for your data. Don’t just jump into no-SQL just ‘cause because you’ll probably end up with a bigger problem. In the long run.Corey: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the Cloud: low effort, high visibility and detection. To learn more, visit lacework.com.Pete: So Jesse, I’m looking at our list of questions. And it turns out, we have another question.Jesse: Ohh.Pete: Two questions came in.Jesse: You like me, you really like me!Pete: It’s so great. Again, you can also send us a question, lastweekinaws.com/QA. You can go there, drop in a question and feel free to put your name. Or not; you can be anonymous, it’s totally fine. We’ll happily answer your question either way. So Jesse, who is our next question from? What is this one about?Jesse: This one’s from [Joseph 00:13:19]. They write in, “Hey, folks. Love the show. Longtime listener, first-time caller.” Thank you. “I would love to know how people manage their costs in AWS Batch. Jobs themselves can’t be tagged for cost allocation, which makes things a bit complicated.” Lord Almighty, yes, it does. “How best should I see if the jobs are right-sized? Are they over-provisioned in terms of memory or compute? What’s the best way to see if EC2 is my better choice, versus Fargate, versus other options? How can I tell if the batch-managed cluster itself is under-utilized?”Pete: Oof. This is a loaded question with a lot of variables.Jesse: Yeah. And so we’re going to break it down because there’s definitely a couple questions here. But I want to start off with what AWS Batch is, just really quick to make sure everybody’s on the same page here. AWS Batch, effectively, is a managed service in AWS that schedules it and runs your batch computing jobs on top of AWS compute resources. Effectively, it does a lot of the heavy lifting configuration for you so you can just focus on analyzing the results of those queries.Pete: Yeah, exactly. And Batch supports a really wide variety of tooling that can operate this, and that’s why it’s hard for us to give, specifically, how you might optimize this, but I think some of the optimizations actually mirror a lot of the optimizations we’ve done with optimizing EMR clusters and things of that nature, where you’re running these distributed jobs. And you want to make sure that if you’re running straight off of EC2 instances, then you want to make sure that they are essentially maxed out. If the CPU is anything less than 100% for an on-demand instance, then there’s wasted, or there’s opportunity for improvement. And so making sure that your jobs are sized appropriately and balancing out memory and CPU so that, effectively, you’re using all of the memory and all of the CPU, that’s a real basic first step.But honestly, a lot of folks kind of miss out on that. They just kind of run a job and go off and do their own thing. They never really go back and look at those graphs. You can go to CloudWatch, they’re all going to be there for you.Jesse: Yeah. And to this point, there’s always an opportunity to make these workloads more ephemeral. If you have the opportunity to make it more ephemeral, please, please, please, please, absolutely do so. Unless your batch job needs to run 24/7. We’ve seen that in a few cases where they have, essentially, clusters that are running 24/7, but they’re not actually utilized regularly; the workloads are only scheduled for a short amount of time.So, if you don’t need those batch jobs running 24/7, please, by all means, move to more ephemeral resources, like Fargate. Fargate on Spot, Spot Instances in general, or even Lambda, which AWS Batch now supports as well.Pete: Yeah, it has some step function support, which is pretty interesting. Yeah, this is a great opportunity to aggressively—aggressively—leverage Spots, if you’re not currently today. The reality is that check out Fargate on Spot if you don’t need, like, a custom operating system, you don’t need a custom EBS volume size. If you do, then EC2 on Spot is probably the best option that you really have. But really do not want to be running anything on on-demand instances. Even on-demand instances with a really good savings plan, you’re still leaving money on the table because Spot Instances are going to be a lot cheaper than even the best savings plan that’s out there.Jesse: And I think that’s a good point, too, Pete, which is if you do need to run these workloads on-demand, 24/7, think about if you can get away with using Spot Instances. If you can’t get away with using Spot Instances, at least purchase a savings plan if you don’t do anything else. If you take nothing else away from this, at least make sure that you have some kind of savings plan in place for these resources so that you’re not paying on-demand costs 24/7. But in most cases, you can likely make them more ephemeral, which is going to save you a lot more money in the long run.Pete: Yeah, exactly. That’s the name of the game. I mean, when we talk to folks on Amazon, the more ephemeral you can make your application—the more you can have it handle interruption—the less expensive it will be to operate. And that goes from everywhere from Spot Instances and how they’re priced, right? If you just get a normal Spot Instance, it will have a really aggressive discount on it if you need zero time in advance before interruption.So, if that instance can just go in at any second, then you’ll get the best discount on that Spot Instance. But if your app needs a little time, or runs for a defined period of time—let’s say your app runs for one hour—you can get a defined duration Spot of one hour, you’ll get a great discount still and you’ll only pay for however long you use it, but you will get that resource for one whole hour, and then you’ll lose it. If that’s still too aggressive, there’s configurable options up to six hours. Again, less discount, but more stability in that resource. So, that’s the trade-off you make when you move over to Spot Instances.Jesse: So, I also want to make sure that we get to the second part of this question, which is about attributing cost to your AWS Batch workloads. According to the AWS Batch documentation, you can tag AWS Batch compute environments, jobs, job definitions, and job queues, but you can’t propagate those tags to the underlying resources that actually run those jobs. Which to me, kind of just defeats the point.Pete: Yeah. [sigh]. Hashtag AWS wishlist here. You know, again, continuing to expand out tagging support for things that don’t support it. I know we’ve seen kind of weird inconsistencies, and just even, like, tagging ECS jobs and where you have to tag them for they’re to apply.So, I know it’s a hard problem, but obviously, it’s something that should be continually worked out on because, yeah, if you’re trying to attribute these costs, you’re left with the only option to run them in separate Amazon accounts, which solves this problem, but again, depending on your organization, could increase just the management overhead of those. But that is the ultimate way. I mean, that is the one way to ensure 100% of costs are encapsulated to a service is to have them run in a dedicated account. The downside being is that if you have a series of different jobs running across a different, maybe, business units, then obviously that’s going to break down super quick.Jesse: Yeah, and it’s also worth calling out that if there’s any batch jobs that need to send data to different places—maybe the batch job belongs to product A, but it needs to send data to product B—there’s going to be some amount of data transfer either across regionally or across accounts in order to share that data, depending on how your organization, how your products are set up. So, keep in mind that there are potentially some minor charges that may appear with this, but ultimately, if you’re talking about the best ways to really attribute costs for your AWS Batch workloads, linked accounts is the way to go.Pete: Yeah. If you need attribution down to the penny—some of our clients absolutely do. For invoicing purposes, they need attribution for business unit down to the penny. And if you’re an organization that needs that, then the only way to get that, effectively, is segmented accounts. So, keep that in mind.Again, until Amazon comes out with the ability to get a little bit more flexible tagging, but also, too, feel free to yell at your account manager—I mean, ask them nicely. They are people, too. But, you know, let them know that you want this. Amazon builds what the customers want, and if you don’t tell them that you want it, they’re not going to prioritize it. I’m not saying if you tell them, you’re going to get it in a couple of months, but you’re never going to get it if you don’t say anything. So, definitely let people know when there’s something that doesn’t work the way you expect it to.Jesse: Absolutely.Pete: Awesome. Wow. Two questions. I feel it’s like Christmas. Except—Jesse: [laugh].Pete: —it’s Christmas in almost springtime. It’s great. Well, again, you, too, can join us by being a Friend of the Pod, which Jesse really loves that one for some reason. [laugh].Jesse: Yeah. Don’t know why, but it’s going to be stuck in my brain.Pete: Exactly. You too can be a Friend of the Pod by going to lastweekinaws.com/QA and you can send us a question. We would love to spend some time in a future episode, answering them for you.If you’ve enjoyed this podcast, please go to lastweekinaws.com/review. Give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review and give it a five-star rating on your podcast platform of choice and tell us why you want to be a Friend of the Pod. Thank you.Announcer: This has been a HumblePod production. Stay humble. 

Links:Unconventional Guide to AWS Cost Management:https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/ TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I am Pete Cheslock.Jesse: I’m Jesse DeRose.Pete: We’re back again. And we’re here. We made it, Jesse.Jesse: I was worried. This was a journey. Thank you, everybody, for coming on this journey with us.Pete: It was quite an experience going through the Unconventional Guide to AWS Cost Savings. We’ve made it. I just can’t believe we’re here.Jesse: Yeah.Pete: So, what are we talking about today for the culmination of our magnum opus of cost savings optimizations?Jesse: This is a fun one. And I know I keep saying that this is my favorite about everyone, but I have to admit that this one, this topic today probably is my absolute favorite. This one I get really nerdy over. Today, we’re talking about how to predict your future and make your CFO happy. No—spoiler alert—there are not any crystal balls involved in this one. There’s no stock market conversations.This is talking about how you can use all of the different things that we’ve talked about throughout the course of this Unconventional Guide to really bring it all together into a couple ideas that will help you better understand your cloud costs, and really better understand your business, I think.Pete: Yeah. All of the things we talked about really lead up to this one, which is the clients of ours that are the most mature, who are incredibly optimized in their Amazon usage, are the ones who have adopted a majority of these specific items. They all lead to this last one, that ability to predict your future usage based on something that’s happening internally, or if a salesperson comes to you and says, “Hey, we’re about to close this deal, but I need to discount our service.” People are going to start wanting to know well, what is the cheapest that you could sell your service for and still have a positive gross margin?Jesse: Yeah. So, if you’ve done a lot of the things that we’ve talked about in the last couple episodes—I apologize, I know homework’s not the best for a podcast—but if you’ve had the opportunity to work on some of those things, you should have a ton of valuable insights into your spend. We’re talking about tagging, and showback models in particular, maybe even a chargeback model. But you can ultimately use all of this data to better understand what is your forecasted spend is going to look like with a new potential customer coming onto the platform? Or if you get into the topic that we’re going to talk about today, which is mostly unit economics, you can really understand how much can I discount my service and still make a profit, like Pete mentioned?Pete: Yeah, I mean, imagine there’s a global pandemic that happens, and it causes your usage to spike by 500% within the course of a month. How did your spend change? Do you know where it changed? And did it change in ways that you were expecting it to? Like, my databases grew by a lot, and this other thing didn’t grow by very much.Like, that would be expected. But also another thing that—a question that we actually like to ask a lot of our clients, if your sales just doubled overnight, okay would your spend change? Where are the places that are most expensive to operate your service? And again, this is kind of generic. I’ve worked in a lot of SaaS services, so I always think of sales, but just think of whether you’re using the cloud for a SaaS service that you provide and sell, like, B2C, things like that, or B2B, you still have users.They might be internal users. Well, what if your users doubled overnight? What if half the company was using your internal service and now the whole company is? How does that change your usage?Jesse: And it’s also important to think about not just your AWS usage, but all of the other services that you use that support your overall business model: things like monitoring and observability tools, logging vendors, maybe third-party sim tools. All of these are affecting your overall total infrastructure cost and are all part of this conversation. So, it’s really important to start thinking about those architecture diagrams. Remember, when we said, way, way back at the beginning of this conversation, to overlay costs on top of your architecture diagram, understanding that, understanding what parts of your product or what parts of your architecture are the most expensive will really help you understand what’s going to change?Pete: Yeah, let’s say you’ve got a six-figure bill to Datadog or one of the big log management vendors out there, but inside of that bill, is that all just evenly spread across the whole business? What if your log vendor was—the entire spend was all by one service that some developer left the debug logging enabled for? You know, you’d want a way of understanding that maybe that spend was concentrated in maybe a non-production aspect of your account. Because then again, that wouldn’t grow, right? That wouldn’t affect your growth in your sales the same way as if maybe all of your services were equally sending logs of a certain volume over.So, all of those extra services, they all add up, and we see it more and more, as more of our clients start adopting more than just Amazon services: they might be adopting a Snowflake, they might be adopting third-party services running databases running in other services, or EMR type workloads that are not on EMR, and they’re running on Qubole or things like that. There’s just a lot of these services that more and more people are consuming from that fall outside of just the AWS invoice.Jesse: And this also gets back to not just architecture diagrams, but also tagging and showback models, cost visibility, really understanding where your spend is going. And this is fantastic to understand where your spend is going, but finance is probably going to want something a little bit more than this. It’s not just about how much are we spending, or where are we spending it, and maybe it’s not even a finance question. Maybe this is a sales conversation, assuming that you’re a SaaS company. Maybe this is, as Pete mentioned before, “Hey, we want to understand where can we provide discounts? What services can we ultimately discount to negotiate getting new customers on the platform?”Pete: So, Jesse, we hear a lot of these terms a lot, and I’d love a ‘explain like I am a five-year-old’ version of it, but we hear a chargeback. And we hear showback, and honestly, I’ve never worked at those massive companies where you might implement these things, but can you give us just a real quick—for all the listeners out there, when we say showback, what does that mean? And when we say chargeback, what does that mean?Jesse: So, a showback model essentially takes all of your cloud costs, all of your total infrastructure spend, your AWS spend, all the third-party spend, and it shows every team, every product, every microservice, maybe, depending—or maybe even business unit, depending on how your organization is split up—it shows each one of those units, how much they are actually spending, how much they’re actually using these different cloud vendors. So, this is where tagging comes in super handy because if you’ve tagged all of your taggable resources, and properly attributed all of your cloud costs with tagging and linked accounts, you have a very clear idea of who’s spending what. You know very clearly, maybe 70 to 80% of your total infrastructure spend is related to one particular product because all the cost is attributed to one particular product. And maybe that’s something you didn’t know before. Maybe now you know okay, maybe that product needs to be a little bit more expensive so that we can make sure that we are making money off of it, or profiting off of it, whereas other services can be discounted because they’re not as expensive.Whereas in a chargeback model, you are ultimately not just showing each of these teams, hey, here’s how much you spent on AWS and Datadog usage and all these other vendors every month, you’re actually charging them for that usage. You’re actually pulling their cloud costs from their budget.Pete: Yeah. They might actually have a budget of money. It’s all—if you want to really explain like I’m five, it’d be like, I give my child their—they get $1 for all of the tasks that they do throughout the week, I don’t actually give them the money because I usually have to subtract out their, like, Roblox spend of the week [laugh] and things like that. It’s all virtual, but at the end of the day, you know, we’re kind of virtually giving this business unit some money, and then, kind of, virtually charging them for their services within.Jesse: Yeah. And this is mature. We don’t see a lot of companies doing this. This is hard because you have to take other steps first to get here. And so this is why we harp so much on cost attribution through tagging and through linked accounts.This is why we harp so much on cost visibility and overlaying those cloud costs on your architecture diagrams to understand all of this data to lead to this point, which is understanding, where, how much is my primary product actually costing us? How much is my secondary product actually costing us? Or maybe how much is this business unit costing us in terms of total infrastructure spend?Pete: Yeah, I mean, I can kind of share my history with this at previous companies is that, again, eventually someone in the financial department is going to say, “What was our cost for Amazon?” They specifically will want to know the production cost because that figures into a term called gross margin, which you often hear at SaaS businesses. Gross margin is basically you take all the revenue that came in and you subtract away what it took to support that revenue. And mostly, that is just the Amazon bill and these other vendors, perhaps, and you end up with a percentage. And hopefully, it’s a positive percentage.It means you’re theoretically making money at a gross, I mean, obviously, before you pay salaries and all those other items, but that being kind of beside the point for now, that number, you’re probably going to get asked for. So, you wouldn’t want to give like your straight Amazon bill, like, “Oh, well, we spent $100,000 last month,” because some of that spend was probably in research and development; it was probably in a development account or a QA account. You really just want your product spent. So, at a previous company, the first step we took was break out our spend via production and development, just two criteria. Now, for us because we started with just a handful of accounts—this was before a lot of accounts were more prevalent, before organizations—before it was easy to handle a lot of accounts—we had a Prod and a Dev. Super easy. Look at Prod, look at Dev. There’s the two bills.But then as time went on, we needed to get more granular. We were running some development workloads, testing out new databases at scale in kind of a hidden dark deployed mode, in production. Well, we want to subtract that spend from there. And that requires tagging. I mean, that’s why we really harped on tagging for a couple of episodes because tagging is the only way you’re going to be able to do that.Now, we see more often a lot of our clients do maybe an account per product, or account per business unit. Those are, again, really effective ways to corral your spend to make it really easy to break it out and add it up. It’s really just trying to break it down to the most reasonable spend unit possible that you can then play around with and adjust. Mostly to go back to your CFO when they asked you, “Hey, I need to know this specific answer.” You’ve got it hopefully somewhat available.Jesse: Okay, so this is where we’re going to start talking about unit economics. And hopefully, your eyes will not glaze over. I want to make sure that—this is important, this is really actually beneficial. It’s not just a specific economic thing that you learned back in Econ 101. This is actually going to be useful and beneficial for your organization.So, unit economics describes your product in terms of revenues and costs in relation to a unit KPI—that’s where the ‘unit’ term comes from in ‘unit economics’—that tracks closely with customer demand. So, that’s a really gross definition, I know, and I apologize.Pete: You know, and we can even extend that a little bit further and give some good examples. Like, maybe if you are a website that provides eLearning services, your unit might be the number of daily active users or thousands of daily active users, right, could be a thing. That could be a unit that you’re selling. I actually worked at a SaaS company where we sold a piece of software that would run per server, and we broke our unit down to the servers—the things that we sold—down to that level.Jesse: Yeah, if you are in the airline industry, for example, your unit would probably be every passenger. How many passengers are you able to sell tickets to on every plane? What do those costs look like?Corey: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the Cloud: low effort, high visibility and detection. To learn more, visit lacework.com.Pete: And you don’t need just, like, one unit. Maybe you have one unit for your whole platform like the whole gross production spend breaks down into one specific unit, you could do that. But you could also have units at a per-service level because maybe it’s like you’re processing a lot of documents. I worked for an email archiving company, forever ago, and our unit was the amount of emails that were indexed and archived so we could figure out, we might have one customer who just didn’t generate a lot of emails, but they had tons of users. Well, one of our units was the volume of emails that we were indexing and archiving for that customer, whereas on the flip side, if maybe our spend was driven more by user count, and not document count, maybe that’s what we want our unit to be, is per user.Jesse: Yeah. It’s really important to call out that you might have a single easy-to-define unit; you might have a more complex relationship that’s weighted with a couple different factors of different components of the architecture. But ultimately, your unit KPI and how you break out your costs to support your customers will be unique to your overall business.Pete: Exactly. And this is where you’re only going to find this answer out with a lot of conversations, internally. It could come to you pretty easily, you know, just based on how your business is. But I think for a lot of folks using Amazon, especially if you’re just in a specific business unit inside of a broader business, it could be a little bit more challenging to figure out. But what you’re really trying to do is just figure out, when X changes, our spend changes, and we spend more or we spend less. Try to solve for X. That’s really what you’re trying to do.Jesse: Okay. So, now we’ve covered the unit KPI part of this conversation. Awesome. So, we’re done, Pete, right? We just take our AWS bill and then—Pete: Yeah.Jesse: —divide it by the unit and we’re done.Pete: So, easy. I’ve got my unit. I’ve got my bill. I got an iPhone that can do a calculator. Good to go.Jesse: [laugh]. Good. We’re done. Well, wait. What about if I have multiple AWS accounts? Wait, what if I have multiple different products?Pete: Yeah, that’s… I mean, I kind of calculator. I mean, I might be here all day, but…Jesse: [laugh]. I’ve got a whiteboard. We’ve got some time.Pete: Yeah, we got time. That’s a great point, though. Again, what if you do have things that are just spread all over the place? What if you’ve got two different products, two different services inside the same account? Because of course, you would. That’s a super normal thing. I’m not even saying that sarcastically. That’s a super normal thing.Jesse: Absolutely.Pete: Well, how do you handle this? How do you handle shared services?Jesse: Yeah. I mean I—Pete: We could go on for too long on that one, but these are questions you really want to start asking.Jesse: Yeah. And remember that you’re potentially going to have different unit KPIs for different products, for different business units. That’s fine. That’s expected. But make sure that you are measuring appropriately for each of those.The incoming costs, the incoming revenues, and costs for each of these isn’t going to change. That’s coming from your tagged usage and your linked accounts, but maybe the unit that you’re dividing that spend by is going to change, and that’s fine. This is where a spreadsheet comes in super, super handy. I love my Excel spreadsheets for this. Very, very easy to just bring in all of the bill data across different accounts, and really clearly attribute this spend is for the service, or the spend is for this product, or the spend is for this business unit, and then divide that by the unit that we have in question to get your actual unit KPI, to get your unit economic metric.Pete: Yeah, and this is where the superpower comes in. Once you have this number, now you can better understand and make product-level decisions. Again, whether you’re a SaaS product with a product you’re selling to external customers or building an internal tool, your product is the thing that the internal users consume. Your product decisions can now be driven by this. I mean, I have recollections of conversations with product teams, where they would talk about certain services internally, how they wanted to expand and do all the stuff with it.And I said, “Well, right now that one service represents one-third of our total spend, right? Our gross margin, that is one-third. But we looked at the users, and it’s only being used by one percent.” When you have these big numbers and saying, “Wow, the company spends a third of their money on something one percent of people use,” then maybe that’s not the place we want to be investing product decisions into. Maybe it is, but you don’t know enough to have that conversation unless you have this data.Jesse: Absolutely. I think there’s one other small caveat that we haven’t touched on that I do want to call out, and this comes back to your conversation about tagging. We have noticed that a lot of teams want to tag to a certain extent, and then start building their showback models immediately, which is great that you’ve got investment, you’ve got energy, you really want to get to that showback model, get to that chargeback model, that unit economic model space. But if your usage or cloud usage is not thoroughly tagged and accurately tagged, your resulting data is not going to be accurate either. So, we think about this in terms of a cost margin or a cost error.So, for example, if your production spend or your production usage is only 60% tagged, that means you’ve got 40% error in that data that’s coming in; your cloud spend for production has 40% error margin, which is huge.Pete: Yeah, exactly. Track your untagged spend, as well as your tagged spend. I mean, make sure you have a story for the things that are not tagged. That includes things like data transfer and things that maybe are not as taggable within AWS. That’s an important aspect of this that you’ll want to make sure you’re at least not forgetting about.Even if you can’t tag it, you don’t have a solution for it, make sure it’s in the back of your head that this is maybe not as accurate of a forecast because we’re just taking data transfer and dividing it by product versus actually looking at which product uses the most to transfer.Jesse: Yeah, and this is a tough concept, so don’t feel bad if you listen to this episode, again, don’t feel bad if you go download the Unconventional Guide from the Duckbill Group website—we’ll have the link in the [show notes 00:20:12]—this one is a tough concept because it brings in a lot of other moving parts to ultimately get at this one unifying really, really important idea. This is one that we see a lot of clients and potential clients struggle with. So, if you’re taking some time to understand this concept, you’re not alone.Pete: Exactly. This is the goal of all of the previous work, and this is something that you would measure in just a multi-year commitment in most businesses. And the larger the business is, the longer that work is going to take because it’s hard, there are a lot of moving pieces, and so many things need to be done in advance of all of this. And again, realize you’re not doing this work in a vacuum. There’s things that are moving and shifting as it’s all happening. So, don’t beat yourself up if you’re looking at this and thinking to yourself, “This is just a huge task. I’m never going to get this done.” It’s just not something that’s going to happen overnight.All right, well, hey, if you’ve enjoyed this podcast, if you’ve enjoyed this series, please go to lastweekinaws.com/reviewand give it a five-star rating on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review give it a five-star rating, but then tell us what’s the next series you want to have if you didn’t like this one.Also, don’t forget, you can give us your feedback and any questions that we’ll be continuing to answer in future episodes lastweekinaws.com/QA. You don’t need to put your name, can be totally anonymous. Give us your question. We’d love to dive into some of those topics.And finally, you can download our Unconventional Guide, the whole PDF of everything we’ve talked about at the Duckbill Group website. We’ll include that link in our [show notes 00:21:51] and you can head over there. Thanks again.Announcer: This has been a HumblePod production. Stay humble.

Jessica and Matt spend a little time with Corey Quinn and Pete Cheslock of the Duckbill Group to dig into the mysteries of AWS billing, why product names are all terrible, and what exactly is a "cloud economist" anyway?

Jessica and Matt spend a little time with Corey Quinn and Pete Cheslock of the Duckbill Group to dig into the mysteries of AWS billing, why product names are all terrible, and what exactly is a "cloud economist" anyway?

Pete Cheslock is an advisor and consultant who helps startups with product strategy, messaging, and other go-to-market needs. Prior to going out on his own, he worked at a slew of tech companies, holding positions such as VP of Products at CHAOSSEARCH, VP of Technical Operations at Threat Stack, Inc., Director of DevTools at Dyn, and Director of Technical and Cloud Operations at Sonian. Pete holds a master's in business administration from Babson and a bachelor's in communications from Michigan State University. Join Corey and Pete as they discuss the differences between CHAOSSEARCH and Elasticsearch, proper etiquette for the conference badge-scanning experience, how tech can be a bubble and not everyone might know the tools you do, the increasingly prominent roles artificial intelligence and machine learning play in the AWS ecosystem, why the re:Invent experience is like a marathon, what it’s like listening to a talk on a pair of headphones, which re:Invent announcement made the least waves, why diversity amongst chip manufacturers is a good thing, and more.

Would you like access to unlimited retention of your data within your Amazon S3, which costs far less than online storage on disc? Well, the next time you’re at re:Invent, visit CHAOSSEARCH’s booth. Today, we’re talking to Pete Cheslock, vice president of products at CHAOSSEARCH and former vice president of operations at Threat Stack. CHAOSSEARCH helps people get access to their login event data using Amazon S3. Some of the highlights of the show include: re:Invent - Year of the Pin: People go nuts for conference swag and were collecting pins as if they were gold Scan Your Badge and Drip Emails: Annoying and passive-aggressive marketing trends meant to be spontaneous and interesting Need a job? Corey’s looking to hire a “Quinntern” to use a tag email address to gather conference swag at the next re:invent; if interested, contact him    Corey and Pete’s Swag Rules: Something you want or can use, continues to be valuable, no sizes, no socks Densify Drama: Conference flyer to generate leads failed, created complaints Track and analyze data, but don’t use it to invade privacy or become creepy Las Vegas: Right place for conferences, such as re:Invent? Rather than focusing on going to conference sessions, make meeting and talking to people doing interesting things your priority Midnight Madness Event: Only place Corey could do stand-up Cloud comedy re:Invent 2019: Plan appropriately, identify what you want to get out of it, register ASAP to get a nearby hotel, and schedule meetings with AWS staff Links: Pete Cheslock on Twitter Pete Cheslock on LinkedIn CHAOSSEARCH Threat Stack AWS Amazon S3 Amazon Elasticsearch re:Invent Corey Quinn’s Newsletter Corey Quinn on Twitter Corey Quinn’s Email Sonian Acloud.guru Densify Oracle Apache Cassandra DigitalOcean AWS re:Invent 2018 - Keynote with Andy Jassy AWS re:Invent 2018 - Keynote with Werner Vogels AWS re:Inforce VMware Dreamforce Kubernetes Datadog

Pete Cheslock has appeared on DevOps Chat as his career has spanned several leading DevOps enabled companies. It seems that where there is DevOps, there is Pete Cheslock. Pete has joined Chaos Search as VP of product. Pete tells us why he joined and what excites him about Chaos Search and DevOps.

Elasticsearch is a powerful tool for storing and analyzing data, but when using it for logs and other time oriented information it can become problematic to keep all of your history. Chaos Search was started to make it easy for you to keep all of your data and make it usable in S3, so that you can have the best of both worlds. In this episode the CTO, Thomas Hazel, and VP of Product, Pete Cheslock, describe how they have built a platform to let you keep all of your history, save money, and reduce your operational overhead. They also explain some of the types of data that you can use with Chaos Search, how to load it into S3, and when you might want to choose it over Amazon Athena for our serverless data analysis.

Do you need data captured that let you know when things don’t look quite right? Need to identify issues before they become major problems for your organization? Turn to Threat Stack, which has Cloud issues of its own, and helps its customers with their Cloud issues. Today, I’m talking to Pete Cheslock, who runs technical operations at Threat Stack, which handles security monitoring, alerting, and remediation. The company uses Amazon Web Services (AWS), but its customer base can run anywhere.   Some of the highlights of the show include: Challenges Threat Stack experienced with AWS and how it dealt with them Threat Stack helps companies improve their security posture in AWS Security shouldn’t be an issue, if providers do their job; shared responsibility Education is needed about what matters regarding security, avoiding mistakes Cloud is still so new; not many people have abroad experience managing it Scanning customer accounts against best practices to identify risks Threat Stack’s scanning tool is worthwhile, but most tools lack judgement and perspective Threat Stack offers context between host- and Cloud-based events; tying data together is the secret sauce You shouldn’t have to pay a bunch of money to have a robust security system Good operations is good security; update, patch, track, and perform other tasks Lack of validation about what services are going to be a successful or not Vendor Lock-in: Understand your choices when building your system Pervasiveness and challenge of containerization and Kubernetes Cloud reduces cycle time and effort to bring a product to market Amazon is a game changer with what it allows you to do and solve problems Links: Pete Cheslock Digital Ocean Threat Stack AWS re:Invent Kubernetes

Cloud computing and ubiquitous virtualization have changed the ways that our applications are built and deployed. This new environment requires a new way of tracking and addressing the security of our systems. ThreatStack is a platform that collects all of the data that your servers generate and monitors for unexpected anomalies in behavior that would indicate a breach and notifies you in near-realtime. In this episode ThreatStack's director of operations, Pete Cheslock, and senior infrastructure security engineer, Patrick Cable, discuss the data infrastructure that supports their platform, how they capture and process the data from client systems, and how that information can be used to keep your systems safe from attackers.

Pete Cheslock (http://www.codemonkey.fm/guests/pete-cheslock) joins us to discuss working at Threat Stack and the latest WikiLeaks Vault 7, Github's permissive IP ownership for employees, Google Cloud Spanner. Special Guest: Pete Cheslock.

I had a chance to sit down with Pete Cheslock and Chris Gervais of Threat Stack to talk about DevOps and Security. Pete and Chris are two sharp people and so it was a great discussion. Hope you enjoy another DevOps Chat from DevOps.com

Who owns your availability? Recent events in the npm community have rekindled the perennial discussion about dependency management and controlling points of potential failure. Long-time operations professionals Charity Majors (Hound) and Pete Cheslock (Threat Stack) join the ADO crew to discuss.

Who owns your availability? Recent events in the npm community have rekindled the perennial discussion about dependency management and controlling points of potential failure. Long-time operations professionals Charity Majors (Hound) and Pete Cheslock (Threat Stack) join the ADO crew to discuss.

It's been a year of ADO, with topics from CI to security, panels of devs and of ops, and even Pete Cheslock. For the last episode of the year, we thought it would be fun to revisit Matt and Trevor chatting, sans guests - which hasn't happened since the first episode!

It’s been a year of ADO, with topics from CI to security, panels of devs and of ops, and even Pete Cheslock. For the last episode of the year, we thought it would be fun to revisit Matt and Trevor chatting, sans guests - which hasn't happened since the first episode!

Conferences - some people are addicted, and others have never been. What is the point of conferences? What's an unconference? Why shouldn't I just stay home and watch the livestreams? Jason Dixon (Monitorama), Bridget Kromhout (devopsdays Minneapolis), and Pete Cheslock (devopsdays Boston) give their two cents (and more) about why conferences are the place to be.

Conferences - some people are addicted, and others have never been. What is the point of conferences? What's an unconference? Why shouldn't I just stay home and watch the livestreams? Jason Dixon (Monitorama), Bridget Kromhout (devopsdays Minneapolis), and Pete Cheslock (devopsdays Boston) give their two cents (and more) about why conferences are the place to be.

DevOps 'Thought Leaders' Pete Cheslock, Nathen Harvey, and Randi Harper help us understand all the things you can do wrong when 'doing the DevOps'.

DevOps 'Thought Leaders' Pete Cheslock, Nathen Harvey, and Randi Harper help us understand all the things you can do wrong when 'doing the DevOps'.

DevOps Delicacy - Monitorama:  Pete Cheslock on Culture with @petecheslock