Podcasts about humblepod

Chris Hill, owner of HumblePod and host of the We Built This Brand podcast, joins Corey on Screaming in the Cloud to discuss the future of podcasting and the role emerging technologies will play in the podcasting space. Chris describes why AI is struggling to make a big impact in the world of podcasting, and also emphasizes the importance of authenticity and finding a niche when producing a show. Corey and Chris discuss where video podcasting works and where it doesn't, and why it's more important to focus on the content of your podcast than the technical specs of your gear. Chris also shares insight on how to gauge the health of your podcast audience with his Podcast Listener Lifecycle evaluation tool.About ChrisChris Hill is a Knoxville, TN native and owner of the podcast production company, HumblePod. He helps his customers create, develop, and produce podcasts and is working with clients in Knoxville as well as startups and entrepreneurs across the United States, Silicon Valley, and the world.In addition to producing podcasts for nationally-recognized thought leaders, Chris is the co-host and producer of the award-winning Our Humble Beer Podcast and the host of the newly-launched We Built This Brand podcast. He also lectures at the University of Tennessee, where he leads non-credit courses on podcasts and marketing.  He received his undergraduate degree in business at the University of Tennessee at Chattanooga where he majored in Marketing & Entrepreneurship, and he later received his MBA from King University.Chris currently serves his community as the President of the American Marketing Association in Knoxville. In his spare time, he enjoys hanging out with the local craft beer community, international travel, exploring the great outdoors, and his many creative pursuits.Links Referenced: HumblePod: https://www.humblepod.com/ HumblePod Quick Edit: https://humblepod.com/services/quick-edit Podcast Listener Lifecycle: https://www.humblepod.com/podcast/grow-your-podcast-with-the-listener-lifecycle/ Twitter: https://twitter.com/christopholies Transcript:Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Are you navigating the complex web of API management, microservices, and Kubernetes in your organization? Solo.io is here to be your guide to connectivity in the cloud-native universe!Solo.io, the powerhouse behind Istio, is revolutionizing cloud-native application networking. They brought you Gloo Gateway, the lightweight and ultra-fast gateway built for modern API management, and Gloo Mesh Core, a necessary step to secure, support, and operate your Istio environment.Why struggle with the nuts and bolts of infrastructure when you can focus on what truly matters - your application. Solo.io's got your back with networking for applications, not infrastructure. Embrace zero trust security, GitOps automation, and seamless multi-cloud networking, all with Solo.io.And here's the real game-changer: a common interface for every connection, in every direction, all with one API. It's the future of connectivity, and it's called Gloo by Solo.io.DevOps and Platform Engineers, your journey to a seamless cloud-native experience starts here. Visit solo.io/screaminginthecloud today and level up your networking game.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My returning guest probably knows more about this podcast than I do. Chris Hill is not only the CEO of HumblePod, but he's also the producer of a lot of my various media endeavors, ranging from the psychotic music videos that I wind up putting out to mock executives on their birthdays to more normal videos that I wind up recording when I'm forced into the studio and can't escape because they bar the back exits, to this show. Chris, thank you for joining me, it's nice to see you step into the light.Chris: It's a pleasure to be here, Corey.Corey: So, you have been, effectively, producing this entire podcast after I migrated off of a previous vendor, what four years ago? Five?Chris: About four or five years ago now, yeah. It's been a while.Corey: Time is a flat circle. It's hard to keep track of all of that. But it's weird that you and I don't get to talk nearly as much as we used to, just because, frankly, the process is working and therefore, you disappear into the background.Chris: Yeah.Corey: One of the dangerous parts of that is that the only time I ever wind up talking to you is when something has gone wrong somewhere and frankly, that does not happen anymore. Which means we don't talk.Chris: Yeah. And I'm okay with that. I'm just kidding. I love talking to you, Corey.Corey: Oh, I tolerate you. And every once in a while, you irritate me massively, which is why I'm punishing you this year by—Chris: [laugh].Corey: Making you tag along for re:Invent.Chris: I'm really excited about that one. It's going to be fun to be there with you and Jeremy and Mike and everybody. Looking forward to it.Corey: You know how I can tell that you've never been to re:Invent before?Chris: “I'm looking forward to it.”Corey: Exactly. You still have life in your eyes and a spark in your step. And yeah… that'll change. That'll change. So, a lot of this show is indirectly your fault because this is a weird thing for a podcaster to admit, but I genuinely don't listen to podcasts. I did when I was younger, back when I had what the kids today call ‘commute' or ‘RTO' as they start slipping into the office, but I started working from home almost a decade ago, and there aren't too many podcasts that fit into the walk from the kitchen to my home office. Like great, give me everything you want me to know in about three-and-a-half seconds. Go… and we're done. It doesn't work. So, I'm a producer, but I don't consume my own content, which I think generally is something you only otherwise see in, you know, drug dealers.Chris: Yeah. Well, and I mean, I think a lot of professional media, like, you get to a point where you're so busy and you're creating so much content that it's hard to sit down and review your own stuff. I mean, even at HumblePod, I'm in a place where we're producing our own show now called We Built This Brand, and I end up in a place where some weeks I'm like, “I can't review this. I approve it. You send it out, I trust you.” So, Corey, I'm starting to echo you in a lot of ways and it's just—it makes me laugh from time to time.Corey: Somewhat recently, I wound up yet again, having to do a check on, “Hey, you use HumblePod for your podcasting work. Do you like them?” And it's fun. It's almost like when someone reaches out about someone you used to work with. Like, “We're debating hiring this person. Should we?” And I love being able to give the default response for the people I've worked with for this long, which is, “Shut up and hire them. Why are you talking to me and not hiring them faster? Get on with it.”Because I'm a difficult customer. I know that. The expectations I have are at times unreasonably high. And the fact that I don't talk to you nearly as much as I used to shows that this all has been working. Because there was a time we talked multiple times a day back—Chris: Mm-hm.Corey: When I had no idea what I was doing. Now, 500-some-odd episodes in, I still have no idea what I'm doing, but by God, I've gotten it down to a science.Chris: Absolutely you have. And you know, technically we're over 1000 episodes together, I think, at this point because if you combine what you're doing with Screaming in the Cloud, with Last Week in AWS slash AWS Morning Brief, yeah, we've done a lot with you. But yes, you've come a long way.Corey: Yes, I have become the very whitest of guys. It works out well. It's like, one podcast isn't enough. We're going to have two of them. But it's easy to talk about the past. Let's talk instead about the future a little bit. What does the future of podcasting look like? I mean, one easy direction to go in with this, as you just mentioned, there's over 1000 episodes of me flapping my gums in the breeze. That feels like it's more than enough data to train an AI model to basically be me without all the hard work, but somehow I kind of don't see it happening anytime soon.Chris: Yeah, I think listeners still value authenticity a lot and I think that's one of the hard things you're seeing in podcasting as a whole is that these organizations come in and they're like, “We're going to be the new podcast killer,” or, “We're going to be the next thing for podcasting,” and if it's too overproduced, too polished, like, I think people can detect that and see that inauthenticity, which is why, like, AI coming in and taking over people's voices is so crazy. One of the things that's happening right now at Spotify is that they are beta testing translation software so that Screaming in the Cloud could automatically be in Spanish or Last Week in AWS could automatically be in French or what have you. It's just so surreal to me that they're doing this, but they're doing exactly what you said. It's language learning models that understand what the host is saying and then they're translating it into another language.The problem is, what if that automation gets that word wrong? You know how bad one wrong word could be, translating from Spanish or French or any other language from English. So, there's a lot of challenges to be met there. And then, of course, you know, once they've got your voice, what do they do with it? There's a lot of risk there.Corey: The puns don't translate very well, most of the time, either.Chris: Oh, yes.Corey: Especially when I mis-intentionally mispronounce words like Ku-BER-netees.Chris: Exactly. I mean, it's going to be auto-translated into text at some point before it's then put out as, you know, an audio source, and so if you say something wrong, it's going to be an issue. And Ku-BER-netees or Chat-Gippity or any of those great terms that you have, they're going to also be translated wrong as well, and that creates its own can of worms so to speak.Corey: Well, let me ask you something because you have always been one to embrace emerging technologies. It's one of the things I appreciate about you; you generally don't recommend solutions from the Dark Ages when it comes to what equipment should I have and how should I maintain it and the rest. But there are a lot of services out there that will now do automatic transcription and the service that you use at the moment remains a woman named Cecilia, who's remarkably good at what she does. But why have you not replaced her with a robot?Chris: [laugh]. Very simply put, I mean, it kind of goes back to what I was just saying about language translation. AI does not understand context for human words as well as humans do, and so words are wrong a lot of times in auto transcription. I mean, I can remember a time when, you know, we first started working with you all were, if there was one thing wrong in a transcript, an executive at AWS would potentially make fun of you on Twitter for it. And so, we knew we had to be on our A-game when it came to that, so finding someone who had that niche expertise of being able to translate not just words and understand words, but also understand tech terminology, you know, I think that that's, that's its own animal and its own challenge. So yeah, I mean, you could easily get away with something—Corey: Especially with my attentional mispronunciation where she's, “I don't quite know what you're saying here, and neither does the entire rest of the industry.” Like, “Postgres-squ—do you mean Postgres? Who the hell calls it Postgres-squeal?” I do. I call it that. Two warring pronunciations, I will unify them by coming up with a third that is far worse. It's kind of my shtick. The problem is, at some point, it becomes too inside-jokey when I have 15 words that I'm doing that too, and suddenly no one knows what the hell I'm talking about and the joke gets old quickly.Chris: Yep.Corey: So, I've tried to scale that back. But there are still a few that I… I can't help but play with.Chris: Yeah. And it's always fun bringing someone new in to work on—work with you all because they're always like, “What is he saying? Does he mean this?” And [laugh] it's always an adventure.Corey: It keeps life fun though.Chris: Absolutely.Corey: So, one thing that you did for a while, back when I was starting out, it almost felt like you were in cahoots with Big Microphone because once I would wind up getting a setup all working and ready for the recording, like, “Great. Everything working terrifically? Cool, throw it away. It's time for generation three of this.” I think I'm on, like, gen six, or gen seven now, but it's been relatively static for the past few years. Are the checks not as big as they used to be? I mean, if we hit a point of equilibrium? What's going on?Chris: Yeah, unfortunately, Big Microphone isn't paying what they used to. The economy and interest rates and all that, it's just making it hard. But once you get to a certain level of gear, it's going to be more important that you have good content than better and better gear. Could we keep going? Sure. If you wanted to buy a studio and you wanted to get Neumann microphones or something like that, we could keep going. But again, Big Microphone is not paying what they used to.Corey: When people reach out because they're debating starting a podcast and they ask me for advice, other than hire HumblePod, the next question they usually get around to is gear. And I don't think that they are expecting my answer, which is, it does not matter. Because if the content is good, the listeners will forgive an awful lot. You could record it into your iPhone in a quiet room and they will put up with that. Whereas if the content isn't good, it doesn't matter what the production value is because people are constantly being offered better things to do with their time. You've got to grab them, you have to be compelling to your target audience or the rest of it does not matter.Chris: Yeah. And I think that's the big challenge with audio is a lot of people get excited, especially I find this true of people in the tech industry of like, “Okay, I want to learn all the tech stuff, I love all the cool tech stuff, and so I'm going to go out and buy all this equipment first.” And then they spend $5,000 on equipment and they never record a single episode because they put all their time and energy into researching and buying gear and never thought about the content of the show. The truth is, you could start with your iPhone and that's it. And while I don't necessarily advise that, you'd be surprised at the quality of audio on an iPhone.I've had a client have to re-record something while they were traveling remotely and I said, “You just need to get your iPhone out.” They took their AirPods, plugged them in and, I said, “No. Take them out, use the microphone on the iPhone.” And you can start with something as simple as that. Now, once you want to start making it better, sure, that's a great way to grow and that does influence people staying with your podcast over time, but I think in the long run, content trumps all.Corey: One of the problems I keep seeing is that people also want to record a podcast because they have a great idea for a few episodes. My rule of thumb—because I've gotten this wrong before—is, okay, if you want to do a whole new podcast, come up with the first 12 episodes. Because two, three, four, of course, you've got your ideas. And then by the—you'll find in many cases, you're going to have a problem by the end of it. Years ago, I did a mini-series inside of AMB called “Networking in the Cloud” where it was sponsored by, at the time, ThousandEyes, before Cisco bought them and froze them in amber for all eternity.But it was fun for the first six episodes and then I realized I'd said all I needed to say about networking, and I was on the hook for six more. And Ivan Pepeinjak, who's his own influencer type in the BGP IP space was like, “This is why you should stay in your lane. He's terrible. He got it all wrong.” Like, “Great. Come on and tell me exactly how I got it wrong,” because I was trying to approach it from a very surface topical area, but BGP is one of those areas where I get very wrapped around my own axle just because I've never used it in anger. Being able to pivot the show format is what saved me on that. But if I had started doing this as its own individual podcast and launched, it would have died on the vine, just because it would not have had enough staying power and I didn't have the interest to continue working on it. Could someone else come up with a networking-in-the-cloud podcast that had hundreds of episodes? Absolutely, but those people are what we call competent and good at things in a way that I very much am not.Chris: Yep. And I completely agree. I mean, 12 is my default number, so—I'm not going to take credit for your saying 12, but I know we've talked about that before. And—Corey: It was a 12-episode miniseries is why. And I remember by ten, I had completely scraped the bottom of the barrel. Then Ivan saved me on one of them, and then I did, I think, a mini-series-in-review, which is cheating but worked.Chris: Yeah. I remember that, the trials and travails of giving that out. It was fun, though. But with that, yeah, like, 12 is a good number because, like, to your point, if you have 12 and you want to do a monthly show, you've got a year's worth of content, if you do bi-weekly, that's six months, and if it's a weekly show, it's at least a quarter's worth of content. So, it does help you think through and at least come up with any potential roadblocks you might have by at least listing out, here's what episodes one, two, three, four, five and so on would be. And so, I do think that's a great approach.Corey: And don't be an idiot like I was and launch a newsletter and then podcast that focus on last week's news because you can't work ahead on that. If you can, why are you not a multi-billionaire for playing the markets? If you can predict the future, there's a more lucrative career for you than podcasting, I promise. But that means that I have to be on the treadmill on some level. I've gotten it down to a point where I can stretch it to ten days. I can take ten days off if I preload, do it as early as I possibly can beforehand and then as late as I possibly can when I return. Anything more than that, I'm either skipping a week or delaying the show or have to get a guest author or artist in.Chris: Yeah. And you definitely need that time off, and so that's the one big challenge, I think with podcasting, too, is like you create this treadmill for yourself that you constantly have to fill content after content after content. I think that's one of the big challenges in podcasting and one of the reasons we see so many podcasts fade out. I don't know if you're familiar, but there is a term called podfade, which is just that: people burning out, fading out in their excitement for a podcast. And most podcasters fade out by episode seven or eight, somewhere in that range, so to see someone go for say, like, you have 500 episodes plus, we're talking about a ton of good content. You've found your rhythm, you've found your groove. That can do it. But yeah, it's always, always a challenge staying motivated.Corey: One thing that consistently surprises me is that the things I care about as the creator and the things the audience cares about are not the same. And you have to be respectful of your audience's time. I've done the numbers on the shows that I put out and it's something on the order of over a year of human time for every episode that I put out. If I'm going to take a year from humanity's collective lifetimes in order to say my inane thoughts, then I have to be respectful of the audience's time. Which means, “Oh, I'm going to have a robot do it so I don't have to put the work in.” It doesn't work that way. That's not how you sustain.Chris: Right. In and again, it takes out that humanity that makes podcasting so special and makes that connection with even the listener so special. And I'm sure you've experienced this too. When you go to re:Invent, like, we're going to have here in just a few short months, people know you, and they probably say things and bring up things that you haven't even thought about. And you're like, “Where did you even learn that I did that?” And then you realize, “Oh, I said that on a podcast episode.”Corey: Yeah. What's weird is I don't get much feedback online for it, but people will talk to me in depth about the show. They'll come up to me near constantly and talk about it. They don't reach out the same way, which I guess makes sense. There are a couple of podcasts that I've really admired and listened to on and off in the car for years, but I've never reached out to the creators because I feel like I would sound ridiculous. It's not true. I know intellectually it's not true, but it feels weird to do it.Chris: One of the ways I got into podcasting was a podcast that just invited me to—you know, invited their listeners to sign up and engage with them. And I think that's something in the medium that does make it interesting is once you do engage, you find out that these creators respond. And where else do you get that, you know? If you're watching a big TV show and you tweet at somebody online that you admire in the show, the chance of them even liking what you said about them online is very slim to none. But with podcasting, there's just a different level of accessibility I find with most productions and most shows that makes it really something special.Corey: One thing that still surprises me—and I don't think I've ever been this explicit about it on the show, but why the hell not I have nothing to hide—Thursday evening, 5 p.m. Pacific time. That's when the automation fires and rotates everything for the newsletter and the AWS Morning Brief. Anything that comes in after that, unless I manually do an override, will not be in the next week's issue; it'll be the week after.That applies to Security as well, which means 5 p.m. on Thursday, it seals it, I write and record it and it goes ou—that particular one goes out Thursday morning the following week. And no one has ever said anything about this seems awfully late. Occasionally, there's been news the day before and someone said, “Oh, why didn't you include this?”And it's because, believe it or not, I don't just type this in and hit the send button. There's a bit more to it than that these days. But people don't need the sense of immediacy. This idea of striving to be first is not sustainable and it leads to terrible outcomes. My entire philosophy has not been to have the first take but rather the best take.Chris: Mm-hm.Corey: Sometimes I even get it right.Chris: And I mean in podcasting, too. Like, it's about, you serve a certain niche, right? Like, the people who are interested in AWS services and in this world of cloud computing listen to what you say, listen to the people you interview, and really enjoy those conversations. But that's not everybody in the world. That's not a very broad audience. And so, I think that those niches really serve a purpose.And the way I've always thought about it is, like, if you go to the grocery store, you know how you always have that rack of magazines with the most random interests? That's essentially what podcasting is. It's like each podcast is a different magazine that serves someone's random—and hyper-specific sometimes—niche interest in things. I mean, the number of things you can find podcasts on is just ridiculous. And I think the same is true for this. But the people who do follow, they're very serious, they're very dedicated, they do listen, and yeah, I think it's just a fascinating, fascinating thing.Corey: The way that I see it has been that I've been learning more from the audience and the things that people say that most people would believe, but… I make a lot of mistakes doing this, but talking to people does tend to shine a light on a lot of this. But enough about the past. Most of my episodes are about things that have previously happened. What does the future of podcasting look like? Where's it going from here?Chris: Oh, man. Well, I think the big question on everybody's mind is, do I need a video podcast? And I think that for most people, that's where the big question lies right now. I get a lot of questions about it, I get people reaching out, and I think the short answer to that is… not really. Or to answer a question I know you love, Corey, it depends.And the reason for that is, there's a lot with the tech of podcasting that just isn't going to distribute to everywhere, all at once anymore. The beauty of podcasting is that it's all based on an RSS feed. If you build an RSS feed and you put it in Apple Podcasts and Spotify, that RSS feed will distribute everywhere and it will distribute your audio everywhere. And what we see happening right now, and really one of the bigger challenges in podcasting, is that the RSS feed only provides audio. Technically, that's not accurate, but it does for most services.So, YouTube has recently come out and said that they are going to start integrating RSS feeds, so you'll be able to do those audiogram-esque things that a lot of people have done through apps like Headliner and stuff for a long time, or even their podcast host may automatically translate a version of their audio podcast into a video and just do, like, a waveform. They're going to have that in YouTube. TikTok is taking a similar approach. And they're both importing just the audio. And the reason I said earlier, that's technically not accurate is because RSS feeds can also support MP4s, but neither service is going to accept that or ingest it directly into their service from what you provide outbound.So, it's a very interesting time because it feels like we're getting there with video, but we're still not there, and we're still probably several years off from it. So, there's a lot of interest in video and I think the future is going to be video, but I think it's going to be a combination, too, with audio because who wants to sit and watch something for an hour-and-a-half when you're used to listening to it your commute or while you do the dishes or any number of other things that don't involve having your eyeballs directly on the content.Corey: We've tried it with this show. I found that it made the recording process a bit more onerous because everyone is suddenly freaking out about how they look and I can't indulge my constant nose-picking habit. Kidding. So, it was more work, I had to gussy myself up a bit more than dressing like a slob like I do some mornings because I do have young children and a deadline to get them to school by. But I never saw the audience to materialize there and be worth it.Because watching a video of two people talking with each other, it feels too much like a Zoom call that you can't participate in, so what's the point?Chris: Right.Corey: So, there's that. There's the fact that I also have very intentionally built most of what I do around newsletters and podcasts because at least so far, those are not dependent upon algorithmic discovery in the same way. I don't have to bias for things that YouTube likes this month. Instead, I can focus on the content that people originally signed up to hear me put out and I don't have to worry about it in the same way. Email predates me, it'll be here long after I'm gone, and that seems to make sense.I also look at how I have consumed podcasts, and times when I do, it's almost always while I'm doing something else. And if I have to watch a screen, that becomes significantly more distracting, and harder for me to find the time to do it with.Chris: I think what you're seeing is that, like, there's some avenues to where video podcasting is really good and really interesting, and I think the real place where that works best right now is in-person interviews. So, Corey, if you went out and interviewed Andy Jassy in person in Seattle, that to me would be something that would warrant bringing the cameras out for and putting online because people would want to see you in the office interacting with him. That would be interesting. To your point, during the Zoom calls and things like that, you end up in a place where people just aren't as interested in sitting and watching the Zoom call. And I think that's something that is a clear distinction to make.Entertainment, comedy, doing things in person, I think that's where the real interest in video is and that's why I don't think video will be for everybody all the time. The thing that is starting to come up as well is discoverability, and that has always been a challenge, but as we get into—and we probably don't want to go down this rabbit hole, but you know, what's happened to Twitter and X, like, discoverability is becoming more of a challenge because they're limiting access to that platform. They're limiting discoverability if you're not willing to pay for a blue checkmark. They're doing all these things to make it harder for small independent podcasts to grow.And the places that are opening up for it to grow are places like YouTube, places like TikTok, that have the ability to not only just put your full podcasts online now, but you can actually do, like, YouTube shorts or highlighted clips, and directly link those back to the long-form content that you're producing. So, there is some value in it, there is a technology and a future there for it, but it's just a very complicated time to be in podcasting and figuring out where to go to grow. That's probably the biggest challenge that we face and I think ultimately, that just comes down to developing an audience outside of these social media channels.Corey: One thing that you were talking about a while back in a conversation that I don't think I've ever followed up with you on—and there's no time like in front of a bunch of public people to do that—Chris: [laugh].Corey: You were talking to me about something that you were calling the Podcast Listener Lifecycle.Chris: Yes.Corey: What's your point on that?Chris: So, the Listener Lifecycle is something I developed, just to be frank, working with you guys, learning from you all, and also my background in marketing, and in building audiences and things, from my own podcasts and other things that I did prior to building HumblePod, led me to a place of going, how can we best explain to a client where their podcast is? How does it exist? Where does it exist? All that good stuff. And basically, the Listener Lifecycle is just that.It's a design—and we'll have links to it because I actually did a whole podcast season on the Listener Lifecycle from beginning to end, so that's probably the easiest way to talk about it. But essentially, it's the idea of, you're curious about a show, and how do you go from being curious about a show to exploring a podcast, to then becoming a follower of the podcast, literally clicking the Follow button. What does it take to get through each one of those stages? How can you identify where your audience is? And basically, it's a tool you can use to say, “Well, this is where my listener is in the stages.” And then once they get to be a follower, how do I build them into something more?Well, get them to be a subscriber, subscribe to a newsletter, subscribe to a Patreon or Substack or whatever that subscription service is that you prefer to use, and get them off of just being on social media and following you there and following you in a podcast audio form. Because things can happen: your podcast host could break and you'd lose your audience, right? We've seen Twitter, which we may have thought years ago that it would never go away, and now we don't know how long it's going to be there. It could be gone by the time we're done with this conversation for all we know. I've got all my notifications turned off, so we're basically in a liminal space at this point.But with that said, there's a lot of risk in audiences changing and things like that, so audience portability is really important. So, the more you can collect email addresses, collect contact information, and communicate with that group of people, the better your audience is going to be. And so, that's what it's about is helping people get to that stage where they can do that so that they don't lose audiences and so that they can even build and grow audiences beyond that to the point where they get to the last phase, which is the ‘true fan' phase. And that's where you get people who love your show, retweet everything you do, repost everything you do, and share it with all their friends every time you're creating new content. And that's ultimately what you want: those die-hard people that come up to you and know everything about you at re:Invent, those are the people that you want to create more of because they're going to help you grow your show and your audience, ultimately. So, that's what it's about. I know that's a lot. But again, like, we'll have a link in the show notes to where you can learn more about it.Corey: Indeed, we will. Normally I'm the one that says, “And we'll include a link to that in the show notes.” But you're the one that has to actually make all that happen. Here's another glimpse behind the curtain. I have a Calendly link that I pass out to people to book time on the show. They fill out the form, which is relatively straightforward and low effort by design, and the next time I think about it is ten minutes beforehand when it pops up with, “Hey, you have a recording to go to.” Great. I book an hour for a half-hour recording. I wind up going through this entire conversation. When we're done, we close out the episode, we chat a bit, I close the tab, and I don't think about it again, it's passed off to you folks entirely. It is the very whitest of white glove treatments. Because I, once again, am the very whitest of white guys.Chris: We aim to please [laugh].Corey: Exactly. Because I remember before this, I used to have things delayed by months because I would forget to copy the freaking file into Dropbox, of all things. And that was just wild to me.Chris: And we stay on you about that because we want to make sure that your show gets out and—Corey: And now it automatically transfers and I—when the automation works—I don't have to think about it again. What is fun to me is despite all the time that I spend in enterprise cloud services, we still use things that are prosumer, like Dropbox and other things that are human-centric because for some reason, most of your team are not also highly competent cloud developers. And I still think it is such a miss that something like S3, which would be perfect for this, requires that level of engineering. And I have more self-respect than that. I'd have to build some stuff in order to make that work effectively on my end, let alone folks who have actual jobs that don't involve messing around with cloud services all day.But it blows my mind that there's still such this gulf between things that sound like you would have one of your aging parents deal with versus something that is extraordinarily capable and state-of-the-art. I know they're launching a bunch of things like Amazon's IVS, which is a streaming offering, a lot of their elemental offerings for media packaging, but I look at it, it's like wow, not only is this expensive, it doesn't solve any problems that we actually have and would add significant extra steps to every part of it. Thanks, but no thanks. And sure, maybe we're not the target market, but I can't shake the feeling that there are an awful lot of people like us that fit that profile.Chris: Yeah. And I mean, you bring up a good point about not using S3, things like that. It has occurred to me as well that, hey, maybe we should find somebody to help us develop a technology like this to make it easier on us on the back end to do all the recording and the production in one place, one database, and be able to move on. So, at some point I would love to get there. That's probably a conversation for after the podcast, Corey, but definitely is something that we've been thinking about at HumblePod is, how do we reach that next step of making it even easier on our clients?Corey: Well, it is certainly appreciated. But again, remember, your task is to continue to perform the service excellently, not be the poster child for cloud services with dumb names.Chris: [laugh]. Yes, yes. And I'm sure we could come up with a bunch.Corey: One last question before we wind up calling in an episode. I know that I've been emphasizing the white glove treatment that I get—and let's be clear, you are not inexpensive, but you're also well worth it; you deliver value extraordinarily for our needs—do you offer things that are not quite as, we'll call it, high-touch and comprehensive?Chris: Yes, we do actually. We just recently launched a new service called Quick Edit and it's just that. It's still humans touching the service, so it's not a bunch of automated, hey, we're just running this through an AI program and it's going to spit it out on the other end. We actually have a human that touches your audio, cleans it up, and sends it back. And yeah, we're there to make sure that we can clean things up quickly and easily and affordably for those folks that are just in a pinch.Maybe you edit most weeks and you're just tired of doing the editing, maybe you're close to podfading and you just want an extra boost to see if you can keep the show going. That's what we have the Quick Edit service for. And that starts at $150 an episode and we'll edit up to 45 minutes of audio for you within that. And yeah, there's some other options available as well if you start to add more stuff, but just come check us out. You can go to humblepod.com/services/quick-edit and find that there.Corey: And we will, of course, put links to that in the show notes. Or at least you will. I certainly won't.Chris: [laugh].Corey: Chris, thank you so much for taking the time to speak with me. If people want to learn more, other than hunting you down at re:Invent, which they absolutely should do, where's the best place for them to find you?Chris: I mean@HumblePod anywhere is the quickest, easiest way to find me anywhere—or at least find the business—and you can find me at @christopholies. And we'll have a link to that in the show notes for sure because it's not worth spelling out on the podcast.Corey: I would have pronounced it chris-to-files, but that's all right. That's how it works.Chris: [laugh].Corey: Thank you so much, Chris for everything that you do, as well as suffering my nonsensical slings and arrows for the last half hour. We'll talk soon.Chris: You're welcome, Corey.Corey: Chris Hill, CEO at HumblePod. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you hated this episode, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment that I'm sure Chris or one of his colleagues will spend time hunting down from all corners of the internet to put into a delightful report, which I will then never read.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Richard Seroter, Director of Outbound Product Management at Google, joins Corey on Screaming in the Cloud to discuss what's new at Google. Corey and Richard discuss how AI can move from a novelty to truly providing value, as well as the importance of people maintaining their skills and abilities rather than using AI as a black box solution. Richard also discusses how he views the DevRel function, and why he feels it's so critical to communicate expectations for product launches with customers. About RichardRichard Seroter is Director of Outbound Product Management at Google Cloud. He's also an instructor at Pluralsight, a frequent public speaker, and the author of multiple books on software design and development. Richard maintains a regularly updated blog (seroter.com) on topics of architecture and solution design and can be found on Twitter as @rseroter. Links Referenced: Google Cloud: https://cloud.google.com Personal website: https://seroter.com Twitter: https://twitter.com/rseroter LinkedIn: https://www.linkedin.com/in/seroter/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Human-scale teams use Tailscale to build trusted networks. Tailscale Funnel is a great way to share a local service with your team for collaboration, testing, and experimentation.  Funnel securely exposes your dev environment at a stable URL, complete with auto-provisioned TLS certificates. Use it from the command line or the new VS Code extensions. In a few keystrokes, you can securely expose a local port to the internet, right from the IDE.I did this in a talk I gave at Tailscale Up, their first inaugural developer conference. I used it to present my slides and only revealed that that's what I was doing at the end of it. It's awesome, it works! Check it out!Their free plan now includes 3 users & 100 devices. Try it at snark.cloud/tailscalescream Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. We have returning guest Richard Seroter here who has apparently been collecting words to add to his job title over the years that we've been talking to him. Richard, you are now the Director of Product Management and Developer Relations at Google Cloud. Do I have all those words in the correct order and I haven't forgotten any along the way?Richard: I think that's all right. I think my first job was at Anderson Consulting as an analyst, so my goal is to really just add more words to whatever these titles—Corey: It's an adjective collection, really. That's what a career turns into. It's really the length of a career and success is measured not by accomplishments but by word count on your resume.Richard: If your business card requires a comma, success.Corey: So, it's been about a year or so since we last chatted here. What have you been up to?Richard: Yeah, plenty of things here, still, at Google Cloud as we took on developer relations. And, but you know, Google Cloud proper, I think AI has—I don't know if you've noticed, AI has kind of taken off with some folks who's spending a lot the last year… juicing up services and getting things ready there. And you know, myself and the team kind of remaking DevRel for a 2023 sort of worldview. So, yeah we spent the last year just scaling and growing and in covering some new areas like AI, which has been fun.Corey: You became profitable, which is awesome. I imagined at some point, someone wound up, like, basically realizing that you need to, like, patch the hole in the pipe and suddenly the water bill is no longer $8 billion a quarter. And hey, that works super well. Like, wow, that explains our utility bill and a few other things as well. I imagine the actual cause is slightly more complex than that, but I am a simple creature.Richard: Yeah. I think we made more than YouTube last quarter, which was a good milestone when you think of—I don't think anybody who says Google Cloud is a fun side project of Google is talking seriously anymore.Corey: I misunderstood you at first. I thought you said that you're pretty sure you made more than I did last year. It's like, well, yes, if a multi-billion dollar company's hyperscale cloud doesn't make more than I personally do, then I have many questions. And if I make more than that, I have a bunch of different questions, all of which could be terrifying to someone.Richard: You're killing it. Yeah.Corey: I'm working on it. So, over the last year, another trend that's emerged has been a pivot away—thankfully—from all of the Web3 nonsense and instead embracing the sprinkle some AI on it. And I'm not—people are about to listen to this and think, wait a minute, is he subtweeting my company? No, I'm subtweeting everyone's company because it seems to be a universal phenomenon. What's your take on it?Richard: I mean, it's countercultural now to not start every conversation with let me tell you about our AI story. And hopefully, we're going to get past this cycle. I think the AI stuff is here to stay. This does not feel like a hype trend to me overall. Like, this is legit tech with real user interest. I think that's awesome.I don't think a year from now, we're going to be competing over who has the biggest model anymore. Nobody cares. I don't know if we're going to hopefully lead with AI the same way as much as, what is it doing for me? What is my experience? Is it better? Can I do this job better? Did you eliminate this complex piece of toil from my day two stuff? That's what we should be talking about. But right now it's new and it's interesting. So, we all have to rub some AI on it.Corey: I think that there is also a bit of a passing of the buck going on when it comes to AI where I've talked to companies that are super excited about how they have this new AI story that's going to be great. And, “Well, what does it do?” “It lets you query our interface to get an answer.” Okay, is this just cover for being bad UX?Richard: [laugh]. That can be true in some cases. In other cases, this will fix UXes that will always be hard. Like, do we need to keep changing… I don't know, I'm sure if you and I go to our favorite cloud providers and go through their documentation, it's hard to have docs for 200 services and millions of pages. Maybe AI will fix some of that and make it easier to discover stuff.So in some cases, UIs are just hard at scale. But yes, I think in some cases, this papers over other things not happening by just rubbing some AI on it. Hopefully, for most everybody else, it's actually interesting, new value. But yeah, that's a… every week it's a new press release from somebody saying they're about to launch some AI stuff. I don't know how any normal human is keeping up with it.Corey: I certainly don't know. I'm curious to see what happens but it's kind of wild, too, because there you're right. There is something real there where you ask it to draw you a picture of a pony or something and it does, or give me a bunch of random analysis of this. I asked one recently to go ahead and rank the US presidents by absorbency and with a straight face, it did it, which is kind of amazing. I feel like there's a lack of imagination in the way that people talk about these things and a certain lack of awareness that you can make this a lot of fun, and in some ways, make that a better showcase of the business value than trying to do the straight-laced thing of having it explain Microsoft Excel to you.Richard: I think that's fair. I don't know how much sometimes whimsy and enterprise mix. Sometimes that can be a tricky part of the value prop. But I'm with you this some of this is hopefully returns to some more creativity of things. I mean, I personally use things like Bard or what have you that, “Hey, I'm trying to think of this idea. Can you give me some suggestions?” Or—I just did a couple weeks ago—“I need sample data for my app.”I could spend the next ten minutes coming up with Seinfeld and Bob's Burgers characters, or just give me the list in two seconds in JSON. Like that's great. So, I'm hoping we get to use this for more fun stuff. I'll be fascinated to see if when I write the keynote for—I'm working on the keynote for Next, if I can really inject something completely off the wall. I guess you're challenging me and I respect that.Corey: Oh, I absolutely am. And one of the things that I believe firmly is that we lose sight of the fact that people are inherently multifaceted. Just because you are a C-level executive at an enterprise does not mean that you're not also a human being with a sense of creativity and a bit of whimsy as well. Everyone is going to compete to wind up boring you to death with PowerPoint. Find something that sparks the imagination and sparks joy.Because yes, you're going to find the boring business case on your own without too much in the way of prodding for that, but isn't it great to imagine what if? What if we could have fun with some of these things? At least to me, that's always been the goal is to get people's attention. Humor has been my path, but there are others.Richard: I'm with you. I think there's a lot to that. And the question will be… yeah, I mean, again, to me, you and I talked about this before we started recording, this is the first trend for me in a while that feels purely organic where our customers, now—and I'll tell our internal folks—our customers have much better ideas than we do. And it's because they're doing all kinds of wild things. They're trying new scenarios, they're building apps purely based on prompts, and they're trying to, you know, do this.And it's better than what we just come up with, which is awesome. That's how it should be, versus just some vendor-led hype initiative where it is just boring corporate stuff. So, I like the fact that this isn't just us talking; it's the whole industry talking. It's people talking to my non-technical family members, giving me ideas for what they're using this stuff for. I think that's awesome. So yeah, but I'm with you, I think companies can also look for more creative angles than just what's another way to left-align something in a cell.Corey: I mean, some of the expressions on this are wild to me. The Photoshop beta with its generative AI play has just been phenomenal. Because it's weird stuff, like, things that, yeah, I'm never going to be a great artist, let's be clear, but being able to say remove this person from the background, and it does it, as best I can tell, seamlessly is stuff where yeah, that would have taken me ages to find someone who knows what the hell they're doing on the internet somewhere and then pay them to do it. Or basically stumble my way through it for two hours and it somehow looks worse afterwards than before I started. It's the baseline stuff of, I'm never going to be able to have it—to my understanding—go ahead just build me a whole banner ad that does this and hit these tones and the rest, but it is going to help me refine something in that direction, until I can then, you know, hand it to a professional who can take it from my chicken scratching into something real.Richard: If it will. I think that's my only concern personally with some of this is I don't want this to erase expertise or us to think we can just get lazy. I think that I get nervous, like, can I just tell it to do stuff and I don't even check the output, or I don't do whatever. So, I think that's when you go back to, again, enterprise use cases. If this is generating code or instructions or documentation or what have you, I need to trust that output in some way.Or more importantly, I still need to retain the skills necessary to check it. So, I'm hoping people like you and me and all our —every—all the users out there of this stuff, don't just offload responsibility to the machine. Like, just always treat it like a kind of slightly drunk friend sitting next to you with good advice and always check it out.Corey: It's critical. I think that there's a lot of concern—and I'm not saying that people are wrong on this—but that people are now going to let it take over their jobs, it's going to wind up destroying industries. No, I think it's going to continue to automate things that previously required human intervention. But this has been true since the Industrial Revolution, where opportunities arise and old jobs that used to be critical are no longer centered in quite the same way. The one aspect that does concern me is not that kids are going to be used to cheat on essays like, okay, great, whatever. That seems to be floated mostly by academics who are concerned about the appropriate structure of academia.For me, the problem is, is there's a reason that we have people go through 12 years of English class in the United States and that is, it's not to dissect of the work of long-dead authors. It's to understand how to write and how to tell us a story and how to frame ideas cohesively. And, “The computer will do that for me,” I feel like that potentially might not serve people particularly well. But as a counterpoint, I was told when I was going to school my entire life that you're never going to have a calculator in your pocket all the time that you need one. No, but I can also speak now to the open air, ask it any math problem I can imagine, and get a correct answer spoken back to me. That also wasn't really in the bingo card that I had back then either, so I am a hesitant to try and predict the future.Richard: Yeah, that's fair. I think it's still important for a kid that I know how to make change or do certain things. I don't want to just offload to calculators or—I want to be able to understand, as you say, literature or things, not just ever print me out a book report. But that happens with us professionals, too, right? Like, I don't want to just atrophy all of my programming skills because all I'm doing is accepting suggestions from the machine, or that it's writing my emails for me. Like, that still weirds me out a little bit. I like to write an email or send a tweet or do a summary. To me, I enjoy those things still. I don't want to—that's not toil to me. So, I'm hoping that we just use this to make ourselves better and we don't just use it to make ourselves lazier.Corey: You mentioned a few minutes ago that you are currently working on writing your keynote for Next, so I'm going to pretend, through a vicious character attack here, that this is—you know, it's 11 o'clock at night, the day before the Next keynote and you found new and exciting ways to procrastinate, like recording a podcast episode with me. My question for you is, how is this Next going to be different than previous Nexts?Richard: Hmm. Yeah, I mean, for the first time in a while it's in person, which is wonderful. So, we'll have a bunch of folks at Moscone in San Francisco, which is tremendous. And I [unintelligible 00:11:56] it, too, I definitely have online events fatigue. So—because absolutely no one has ever just watched the screen entirely for a 15 or 30 or 60-minute keynote. We're all tabbing over to something else and multitasking. And at least when I'm in the room, I can at least pretend I'll be paying attention the whole time. The medium is different. So, first off, I'm just excited—Corey: Right. It feels a lot ruder to get up and walk out of the front row in the middle of someone's talk. Now, don't get me wrong, I'll still do it because I'm a jerk, but I'll feel bad about it as I do. I kid, I kid. But yeah, a tab away is always a thing. And we seem to have taken the same structure that works in those events and tried to force it into more or less a non-interactive Zoom call, and I feel like that is just very hard to distinguish.I will say that Google did a phenomenal job of online events, given the constraints it was operating under. Production value is great, the fact that you took advantage of being in different facilities was awesome. But yeah, it'll be good to be back in person again. I will be there with bells on in Moscone myself, mostly yelling at people, but you know, that's what I do.Richard: It's what you do. But we missed that hallway track. You missed this sort of bump into people. Do hands-on labs, purposely have nothing to do where you just walk around the show floor. Like we have been missing, I think, society-wise, a little bit of just that intentional boredom. And so, sometimes you need at conference events, too, where you're like, “I'm going to skip that next talk and just see what's going on around here.” That's awesome. You should do that more often.So, we're going to have a lot of spaces for just, like, go—like, 6000 square feet of even just going and looking at demos or doing hands-on stuff or talking with other people. Like that's just the fun, awesome part. And yeah, you're going to hear a lot about AI, but plenty about other stuff, too. Tons of announcements. But the key is that to me, community stuff, learn from each other stuff, that energy in person, you can't replicate that online.Corey: So, an area that you have expanded into has been DevRel, where you've always been involved with it, let's be clear, but it's becoming a bit more pronounced. And as an outsider, I look at Google Cloud's DevRel presence and I don't see as much of it as your staffing levels would indicate, to the naive approach. And let's be clear, that means from my perspective, all public-facing humorous, probably performative content in different ways, where you have zany music videos that, you know, maybe, I don't know, parody popular songs do celebrate some exec's birthday they didn't know was coming—[fake coughing]. Or creative nonsense on social media. And the the lack of seeing a lot of that could in part be explained by the fact that social media is wildly fracturing into a bunch of different islands which, on balance, is probably a good thing for the internet, but I also suspect it comes down to a common misunderstanding of what DevRel actually is.It turns out that, contrary to what many people wanted to believe in the before times, it is not getting paid as much as an engineer, spending three times that amount of money on travel expenses every year to travel to exotic places, get on stage, party with your friends, and then give a 45-minute talk that spends two minutes mentioning where you work and 45 minutes talking about, I don't know, how to pick the right standing desk. That has, in many cases, been the perception of DevRel and I don't think that's particularly defensible in our current macroeconomic climate. So, what are all those DevRel people doing?Richard: [laugh]. That's such a good loaded question.Corey: It's always good to be given a question where the answers are very clear there are right answers and wrong answers, and oh, wow. It's a fun minefield. Have fun. Go catch.Richard: Yeah. No, that's terrific. Yeah, and your first part, we do have a pretty well-distributed team globally, who does a lot of things. Our YouTube channel has, you know, we just crossed a million subscribers who are getting this stuff regularly. It's more than Amazon and Azure combined on YouTube. So, in terms of like that, audience—Corey: Counterpoint, you definitionally are YouTube. But that's neither here nor there, either. I don't believe you're juicing the stats, but it's also somehow… not as awesome if, say, I were to do it, which I'm working on it, but I have a face for radio and it shows.Richard: [laugh]. Yeah, but a lot of this has been… the quality and quantity. Like, you look at the quantity of video, it overwhelms everyone else because we spend a lot of time, we have a specific media team within my DevRel team that does the studio work, that does the production, that does all that stuff. And it's a concerted effort. That team's amazing. They do really awesome work.But, you know, a lot of DevRel as you say, [sigh] I don't know about you, I don't think I've ever truly believed in the sort of halo effect of if super smart person works at X company, even if they don't even talk about that company, that somehow presents good vibes and business benefits to that company. I don't think we've ever proven that's really true. Maybe you've seen counterpoints, where [crosstalk 00:16:34]—Corey: I can think of anecdata examples of it. Often though, on some level, for me at least, it's been okay someone I tremendously respect to the industry has gone to work at a company that I've never heard of. I will be paying attention to what that company does as a direct result. Conversely, when someone who is super well known, and has been working at a company for a while leaves and then either trashes the company on the way out or doesn't talk about it, it's a question of, what's going on? Did something horrible happen there? Should we no longer like that company? Are we not friends anymore? It's—and I don't know if that's necessarily constructive, either, but it also, on some level, feels like it can shorthand to oh, to be working DevRel, you have to be an influencer, which frankly, I find terrifying.Richard: Yeah. Yeah. I just—the modern DevRel, hopefully, is doing a little more of product-led growth style work. They're focusing specifically on how are we helping developers discover, engage, scale, become advocates themselves in the platform, increasing that flywheel through usage, but that has very discreet metrics, it has very specific ownership. Again, personally, I don't even think DevRel should do as much with sales teams because sales teams have hundreds and sometimes thousands of sales engineers and sales reps. It's amazing. They have exactly what they need.I don't think DevRel is a drop in the bucket to that team. I'd rather talk directly to developers, focus on people who are self-service signups, people who are developers in those big accounts. So, I think the modern DevRel team is doing more in that respect. But when I look at—I just look, Corey, this morning at what my team did last week—so the average DevRel team, I look at what advocacy does, teams writing code labs, they're building tutorials. Yes, they're doing some in person events. They wrote some blog posts, published some videos, shipped a couple open-source projects that they contribute to in, like gaming sector, we ship—we have a couple projects there.They're actually usually customer zero in the product. They use the product before it ships, provides bugs and feedback to the team, we run DORA workshops—because again, we're the DevOps Research and Assessment gang—we actually run the tutorial and Docs platform for Google Cloud. We have people who write code samples and reference apps. So, sometimes you see things publicly, but you don't see the 20,000 code samples in the docs, many written by our team. So, a lot of the times, DevRel is doing work to just enable on some of these different properties, whether that's blogs or docs, whether that's guest articles or event series, but all of this should be in service of having that credible relationship to help devs use the platform easier. And I love watching this team do that.But I think there's more to it now than years ago, where maybe it was just, let's do some amazing work and try to have some second, third-order effect. I think DevRel teams that can have very discrete metrics around leading indicators of long-term cloud consumption. And if you can't measure that successfully, you've probably got to rethink the team.[midroll 00:19:20]Corey: That's probably fair. I think that there's a tremendous series of… I want to call it thankless work. Like having done some of those ridiculous parody videos myself, people look at it and they chuckle and they wind up, that was clever and funny, and they move on to the next one. And they don't see the fact that, you know, behind the scenes for that three-minute video, there was a five-figure budget to pull all that together with a lot of people doing a bunch of disparate work. Done right, a lot of this stuff looks like it was easy or that there was no work at all.I mean, at some level, I'm as guilty of that as anyone. We're recording a podcast now that is going to be handed over to the folks at HumblePod. They are going to produce this into something that sounds coherent, they're going to fix audio issues, all kinds of other stuff across the board, a full transcript, and the rest. And all of that is invisible to me. It's like AI; it's the magic box I drop a file into and get podcast out the other side.And that does a disservice to those people who are actively working in that space to make things better. Because the good stuff that they do never gets attention, but then the company makes an interesting blunder in some way or another and suddenly, everyone's out there screaming and wondering why these people aren't responding on Twitter in 20 seconds when they're finding out about this stuff for the first time.Richard: Mm-hm. Yeah, that's fair. You know, different internal, external expectations of even DevRel. We've recently launched—I don't know if you caught it—something called Jump Start Solutions, which were executable reference architectures. You can come into the Google Cloud Console or hit one of our pages and go, “Hey, I want to do a multi-tier web app.” “Hey, I want to do a data processing pipeline.” Like, use cases.One click, we blow out the entire thing in the platform, use it, mess around with it, turn it off with one click. Most of those are built by DevRel. Like, my engineers have gone and built that. Tons of work behind the scenes. Really, like, production-grade quality type architectures, really, really great work. There's going to be—there's a dozen of these. We'll GA them at Next—but really, really cool work. That's DevRel. Now, that's behind-the-scenes work, but as engineering work.That can be some of the thankless work of setting up projects, deployment architectures, Terraform, all of them also dropped into GitHub, ton of work documenting those. But yeah, that looks like behind-the-scenes work. But that's what—I mean, most of DevRel is engineers. These are folks often just building the things that then devs can use to learn the platforms. Is it the flashy work? No. Is it the most important work? Probably.Corey: I do have a question I'd be remiss not to ask. Since the last time we spoke, relatively recently from this recording, Google—well, I'd say ‘Google announced,' but they kind of didn't—Squarespace announced that they'd be taking over Google domains. And there was a lot of silence, which I interpret, to be clear, as people at Google being caught by surprise, by large companies, communication is challenging. And that's fine, but I don't think it was anything necessarily nefarious.And then it came out further in time with an FAQ that Google published on their site, that Google Cloud domains was a part of this as well. And that took a lot of people aback, in the sense—not that it's hard to migrate a domain from one provider to another, but it brought up the old question of, if you're building something in cloud, how do you pick what to trust? And I want to be clear before you answer that, I know you work there. I know that there are constraints on what you can or cannot say.And for people who are wondering why I'm not hitting you harder on this, I want to be very explicit, I can ask you a whole bunch of questions that I already know the answer to, and that answer is that you can't comment. That's not constructive or creative. So, I don't want people to think that I'm not intentionally asking the hard questions, but I also know that I'm not going to get an answer and all I'll do is make you uncomfortable. But I think it's fair to ask, how do you evaluate what services or providers or other resources you're using when you're building in cloud that are going to be around, that you can trust building on top of?Richard: It's a fair question. Not everyone's on… let's update our software on a weekly basis and I can just swap things in left. You know, there's a reason that even Red Hat is so popular with Linux because as a government employee, I can use that Linux and know it's backwards compatible for 15 years. And they sell that. Like, that's the value, that this thing works forever.And Microsoft does the same with a lot of their server products. Like, you know, for better or for worse, [laugh] they will always kind of work with a component you wrote 15 years ago in SharePoint and somehow it runs today. I don't even know how that's possible. Love it. That's impressive.Now, there's a cost to that. There's a giant tax in the vendor space to make that work. But yeah, there's certain times where even with us, look, we are trying to get better and better at things like comms. And last year we announced—I checked them recently—you know, we have 185 Cloud products in our enterprise APIs. Meaning they have a very, very tight way we would deprecate with very, very long notice, they've got certain expectations on guarantees of how long you can use them, quality of service, all the SLAs.And so, for me, like, I would bank on, first off, for every cloud provider, whether they're anchor services. Build on those right? You know, S3 is not going anywhere from Amazon. Rock solid service. BigQuery Goodness gracious, it's the center of Google Cloud.And you look at a lot of services: what can you bet on that are the anchors? And then you can take bets on things that sit around it. There's times to be edgy and say, “Hey, I'll use Service Weaver,” which we open-sourced earlier this year. It's kind of a cool framework for building apps and we'll deconstruct it into microservices at deploy time. That's cool.Would I literally build my whole business on it? No, I don't think so. It's early stuff. Now, would I maybe use it also with some really boring VMs and boring API Gateway and boring storage? Totally. Those are going to be around forever.I think for me, personally, I try to think of how do I isolate things that have some variability to them. Now, to your point, sometimes you don't know there's variability. You would have just thought that service might be around forever. So, how are you supposed to know that that thing could go away at some point? And that's totally fair. I get that.Which is why we have to keep being better at comms, making sure more things are in our enterprise APIs, which is almost everything. So, you have some assurances, when I build this thing, I've got a multi-year runway if anything ever changes. Nothing's going to stay the same forever, but nothing should change tomorrow on a dime. We need more trust than that.Corey: Absolutely. And I agree. And the problem, too, is hidden dependencies. Let's say what is something very simple. I want to log in to [unintelligible 00:25:34] brand new AWS account and spin of a single EC2 instance. The end. Well, I can trust that EC2 is going to be there. Great. That's not one service you need to go through that critical path. It is a bare minimum six, possibly as many as twelve, depending upon what it is exactly you're doing.And it's the, you find out after the fact that oh, there was that hidden dependency in there that I wasn't fully aware of. That is a tricky and delicate balance to strike. And, again, no one is going to ever congratulate you—at all—on the decision to maintain a service that is internally painful and engineering-ly expensive to keep going, but as soon as you kill something, even it's for this thing doesn't have any customers, the narrative becomes, “They're screwing over their customers.” It's—they just said that it didn't have any. What's the concern here?It's a messaging problem; it is a reputation problem. Conversely, everyone knows that Amazon does not kill AWS services. Full stop. Yeah, that turns out everyone's wrong. By my count, they've killed ten, full-on AWS services and counting at the moment. But that is not the reputation that they have.Conversely, I think that the reputation that Google is going to kill everything that it touches is probably not accurate, though I don't know that I'd want to have them over to babysit either. So, I don't know. But it is something that it feels like you're swimming uphill on in many respects, just due to not even deprecation decisions, historically, so much as poor communication around them.Richard: Mm-hm. I mean, communication can always get better, you know. And that's, it's not our customers' problem to make sure that they can track every weird thing we feel like doing. It's not their challenge. If our business model changes or our strategy changes, that's not technically the customer's problem. So, it's always our job to make this as easy as possible. Anytime we don't, we have made a mistake.So, you know, even DevRel, hey, look, it puts teams in a tough spot. We want our customers to trust us. We have to earn that; you will never just give it to us. At the same time, as you say, “Hey, we're profitable. It's great. We're growing like weeds,” it's amazing to see how many people are using this platform. I mean, even services, you don't talk about having—I mean, doing really, really well. But I got to earn that. And you got to earn, more importantly, the scale. I don't want you to just kick the tires on Google Cloud; I want you to bet on it. But we're only going to earn that with really good support, really good price, stability, really good feeling like these services are rock solid. Have we totally earned that? We're getting there, but not as mature as we'd like to get yet, but I like where we're going.Corey: I agree. And reputations are tricky. I mean, recently InfluxDB deprecated two regions and wound up turning them off and deleting data. And they wound up getting massive blowback for this, which, to their credit, their co-founder and CTO, Paul Dix—who has been on the show before—wound up talking about and saying, “Yeah, that was us. We're taking ownership of this.”But the public announcement said that they had—that data in AWS was not recoverable and they're reaching out to see if the data in GCP was still available. At which point, I took the wrong impression from this. Like, whoa, whoa, whoa. Hang on. Hold the phone here. Does that mean that data that I delete from a Google Cloud account isn't really deleted?Because I have a whole bunch of regulators that would like a word if so. And Paul jumped onto that with, “No, no, no, no, no. I want to be clear, we have a backup system internally that we were using that has that set up. And we deleted the backups on the AWS side; we don't believe we did on the Google Cloud side. It's purely us, not a cloud provider problem.” It's like, “Okay, first, sorry for causing a fire drill.” Secondly, “Okay, that's great.” But the reason I jumped in that direction was just because it becomes so easy when a narrative gets out there to believe the worst about companies that you don't even realize you're doing it.Richard: No, I understand. It's reflexive. And I get it. And look, B2B is not B2C, you know? In B2B, it's not, “Build it and they will come.” I think we have the best cloud infrastructure, the best security posture, and the most sophisticated managed services. I believe that I use all the clouds. I think that's true. But it doesn't matter unless you also do the things around it, around support, security, you know, usability, trust, you have to go sell these things and bring them to people. You can't just sit back and say, “It's amazing. Everyone's going to use it.” You've got to earn that. And so, that's something that we're still on the journey of, but our foundation is terrific. We just got to do a better job on some of these intangibles around it.Corey: I agree with you, when you s—I think there's a spirited debate you could have on any of those things you said that you believe that Google Cloud is the best at, with the exception of security, where I think that is unquestionably. I think that is a lot less variable than the others. The others are more or less, “Who has the best cloud infrastructure?” Well, depends on who had what for breakfast today. But the simplicity and the approach you take to security is head and shoulders above the competition.And I want to make sure I give credit where due: it is because of that simplicity and default posturing that customers wind up better for it as a result. Otherwise, you wind up in this hell of, “You must have at least this much security training to responsibly secure your environment.” And that is never going to happen. People read far less than we wish they would. I want to make very clear that Google deserves the credit for that security posture.Richard: Yeah, and the other thing, look, I'll say that, from my observation, where we do something that feels a little special and different is we do think in platforms, we think in both how we build and how we operate and how the console is built by a platform team, you—singularly. How—[is 00:30:51] we're doing Duet AI that we've pre-announced at I/O and are shipping. That is a full platform experience covering a dozen services. That is really hard to do if you have a lot of isolation. So, we've done a really cool job thinking in platforms and giving that simplicity at that platform level. Hard to do, but again, we have to bring people to it. You're not going to discover it by accident.Corey: Richard, I will let you get back to your tear-filled late-night writing of tomorrow's Next keynote, but if people want to learn more—once the dust settles—where's the best place for them to find you?Richard: Yeah, hopefully, they continue to hang out at cloud.google.com and using all the free stuff, which is great. You can always find me at seroter.com. I read a bunch every day and then I've read a blog post every day about what I read, so if you ever want to tune in on that, just see what wacky things I'm checking out in tech, that is good. And I still hang out on different social networks, Twitter at @rseroter and LinkedIn and things like that. But yeah, join in and yell at me about anything I said.Corey: I did not realize you had a daily reading list of what you put up there. That is news to me and I will definitely track in, and then of course, yell at you from the cheap seats when I disagree with anything that you've chosen to include. Thank you so much for taking the time to speak with me and suffer the uncomfortable questions.Richard: Hey, I love it. If people aren't talking about us, then we don't matter, so I would much rather we'd be yelling about us than the opposite there.Corey: [laugh]. As always, it's been a pleasure. Richard Seroter, Director of Product Management and Developer Relations at Google Cloud. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you had an AI system write for you because you never learned how to structure a sentence.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Humble Happenstance. The things we never expected, the threads that weave throughout experiences, and the moments that lead us to the career path we love. Let's see where Happenstance takes us...Chris Hill, a Knoxville native, is the owner of podcast production company HumblePod, assisting clients worldwide in creating and developing podcasts. In addition to producing podcasts for renowned thought leaders, he co-hosts and produces the award-winning Our Humble Beer Podcast. As a University of Tennessee lecturer, Chris leads non-credit courses on podcasts and marketing. He holds an undergraduate degree in Marketing & Entrepreneurship from the University of Tennessee at Chattanooga and an MBA from King University. Chris serves as the President-Elect of the American Marketing Association in Knoxville and enjoys spending time with his family, international travel, outdoor exploration, and engaging in creative pursuits.Let's Connect:@HappenstanceThePodcast@CareerCoachCassie

Emily Gorcenski, Data & AI Service Line Lead at Thoughtworks, joins Corey on Screaming in the Cloud to discuss how big data is changing our lives - both for the better, and the challenges that come with it. Emily explains how data is only important if you know what to do with it and have a plan to work with it, and why it's crucial to understand the use-by date on your data. Corey and Emily also discuss how big data problems aren't universal problems for the rest of the data community, how to address the ethics around AI, and the barriers to entry when pursuing a career in data. About EmilyEmily Gorcenski is a principal data scientist and the Data & AI Service Line Lead of ThoughtWorks Germany. Her background in computational mathematics and control systems engineering has given her the opportunity to work on data analysis and signal processing problems from a variety of complex and data intensive industries. In addition, she is a renowned data activist and has contributed to award-winning journalism through her use of data to combat extremist violence and terrorism. The opinions expressed are solely her own.Links Referenced: ThoughtWorks: https://www.thoughtworks.com/ Personal website: https://emilygorcenski.com Twitter: https://twitter.com/EmilyGorcenski Mastodon: https://mastodon.green/@emilygorcenski@indieweb.social TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is Emily Gorcenski, who is the Data and AI Service Line Lead over at ThoughtWorks. Emily, thank you so much for joining me today. I appreciate it.Emily: Thank you for having me. I'm happy to be here.Corey: What is it you do, exactly? Take it away.Emily: Yeah, so I run the data side of our business at ThoughtWorks, Germany. That means data engineering work, data platform work, data science work. I'm a data scientist by training. And you know, we're a consulting company, so I'm working with clients and trying to help them through the, sort of, morphing landscape that data is these days. You know, should we be migrating to the cloud with our data? What can we migrate to the cloud with our data? Where should we be doing with our data scientists and how do we make our data analysts' lives easier? So, it's a lot of questions like that and trying to figure out the strategy and all of those things.Corey: You might be one of the most perfectly positioned people to ask this question to because one of the challenges that I've run into consistently and persistently—because I watch a lot of AWS keynotes—is that they always come up with the same talking point, that data is effectively the modern gold. And data is what unlocks value to your busin—“Every business agrees,” because someone who's dressed in what they think is a nice suit on stage is saying that it's, “Okay, you're trying to sell me something. What's the deal here?” Then I check my email and I discover that Amazon has sent me the same email about the same problem for every region I've deployed things to in AWS. And, “Oh, you deploy this to one of the Japanese regions. We're going to send that to you in Japanese as a result.”And it's like, okay, for a company that says data is important, they have no idea who any of their customers are at this point, is that is the takeaway here. How real is, “Data is important,” versus, “We charge by the gigabyte so you should save all of your data and then run expensive things on top of it.”Emily: I think data is very important, if you know what you're going to do with it and if you have a plan for how to work with it. I think if you look at the history of computing, of technology, if you go back 20 years to maybe the early days of the big data era, right? Everyone's like, “Oh, we've got big data. Data is going to be big.” And for some reason, we never questioned why, like, we were thinking that the ‘big' in ‘big data' meant big is in volume and not ‘big' as in ‘big pharma.'This sort of revolution never really happened for most companies. Sure, some companies got a lot of value from the, sort of, data mining and just gather everything and collect everything and if you hit it with a big computational hammer, insights will come out and somehow there's insights will make you money through magic. The reality is much more prosaic. If you want to make money with data, you have to have a plan for what you're going to do with data. You have to know what you're looking for and you have to know exactly what you're going to get when you look at your data and when you try to answer questions with it.And so, when we see somebody like Amazon not being able to correlate that the fact that you're the account owner for all of these different accounts and that the language should be English and all of these things, that's part of the operational problem because it's annoying, to try to do joins across multiple tables in multiple regions and all of those things, but it's also part—you know, nobody has figured out how this adds value for them to do that, right? There's a part of it where it's like, this is just professionalism, but there's a part of it, where it's also like… whatever. You've got Google Translate. Figure out yourself. We're just going to get through it.I think that… as time has evolved from the initial waves of the big data era into the data science era, and now we're in, you know, all sorts of different architectures and principles and all of these things, most companies still haven't figured out what to do with data, right? They're still investing a ton of money to answer the same analytics questions that they were answering 20 years ago. And for me, I think that's a disappointment in some regards because we do have better tools now. We can do so many more interesting things if you give people the opportunity.Corey: One of the things that always seemed a little odd was, back when I wielded root credentials in anger—anger,' of course, being my name for the production environment, as opposed to, “Theory,” which is what I call staging because it works in theory, but not in production. I digress—it always felt like I was getting constant pushback from folks of, “You can't delete that data. It's incredibly important because one day, we're going to find a way to unlock the magic of it.” And it's, “These are web server logs that are 15 years old, and 98% of them by volume are load balancer health checks because it turns out that back in those days, baby seals got more hits than our website did, so that's not really a thing that we wind up—that's going to add much value to it.” And then from my perspective, at least, given that I tend to live, eat, sleep, breathe cloud these days, AWS did something that was refreshingly customer-obsessed when they came out with Glacier Deep Archive.Because the economics of that are if you want to store a petabyte of data, with a 12-hour latency on request for things like archival logs and whatnot, it's $1,000 a month per petabyte, which is okay, you have now hit a price point where it is no longer worth my time to argue with you. We're just not going to delete anything ever again. Problem solved. Then came GDPR, which is neither here nor there and we actually want to get rid of those things for a variety of excellent legal reasons. And the dance continues.But my argument against getting rid of data because it's super expensive no longer holds water in the way that it wants did for anything remotely resembling a reasonable amount of data. Then again, that's getting reinvented all the time. I used to be very, I guess we'll call it, I guess, a data minimalist. I don't want to store a bunch of data, mostly because I'm not a data person. I am very bad thinking in that way.I consider SQL to be the chests of the programming world and I'm not particularly great at it. And I also unlucky and have an aura, so if I destroy a bunch of stateless web servers, okay, we can all laugh about that, but let's keep me the hell away from the data warehouse if we still want a company tomorrow morning. And that was sort of my experience. And I understand my bias in that direction. But I'm starting to see magic get unlocked.Emily: Yeah, I think, you know, you said earlier, there's, like, this mindset, like, data is the new gold or data is new oil or whatever. And I think it's actually more true that data is the new milk, right? It goes bad if you don't use it, you know, before a certain point in time. And at a certain point in time, it's not going to be very offensive if you just leave it locked in the jug, but as soon as you try to open it, you're going to have a lot of problems. Data is very, very cheap to store these days. It's very easy to hold data; it's very expensive to process data.And I think that's where the shift has gone, right? There's sort of this, like, Oracle DBA legacy of, like, “Don't let the software developers touch the prod database.” And they've kind of kept their, like, arcane witchcraft to themselves, and that mindset has persisted. But now it's sort of shifted into all of these other architectural patterns that are just abstractions on top of this, don't let the software engineers touch the data store, right? So, we have these, like, streaming-first architectures, which are great. They're great for software devs. They're great for software devs. And they're great for data engineers who like to play with big powerful technology.They're terrible if you want to answer a question, like, “How many customers that I have yesterday?” And these are the things that I think are some of the central challenges, right? A Kappa architecture—you know, streaming-first architecture—is amazing if you want to improve your application developer throughput. And it's amazing if you want to build real-time analytics or streaming analytics into your platform. But it's terrible if you want your data lake to be navigable. It's terrible if you want to find the right data that makes sense to do the more complex things. And it becomes very expensive to try to process it.Corey: One of the problems I think I have that is that if I take a look at the data volumes that I work with in my day-to-day job, I'm dealing with AWS billing data as spit out by the AWS billing system. And there isn't really a big data problem here. If you take a look at some of the larger clients, okay, maybe I'm trying to consume a CSV that's ten gigabytes. Yes, Excel is going to violently scream itself to death if I try to wind up loading it there, and then my computer smells like burning metal all afternoon. But if it fits in RAM, it doesn't really feel like it's a big data problem, on some level.And it just feels that when I look at the landscape of all the different tools you can use for things like this, they just feel like it's more or less, hmm, “I have a loose thread on my shirt. Could you pass me that chainsaw for a second?” It just seems like stupendous overkill for anything that I'm working with. Counterpoint; that the clients I'm working with have massive data farms and my default response when I meet someone who's very good at an area that I don't do a lot of work in is—counterintuitively to what a lot of people apparently do on Twitter—is not the default assumption of oh, “I don't know anything about that space. It must be worthless and they must be dumb.”No. That is not the default approach to take anything, from my perspective. So, it's clear there's something very much there that I just don't see slash understand. That is a very roundabout way of saying what could be uncharitably distilled down to, “So, is your entire career bullshit?” But no, it is clearly not.There is value being extracted from this and it's powerful. I just think that there's been an industry-wide, relatively poor job done of explaining that value in ways that don't come across as contrived or profoundly disturbing.Emily: Yeah, I think there's a ton of value in doing things right. It gets very complicated to try to explain the nuances of when and how data can actually be useful, right? Oftentimes, your historical data, you know, it really only tells you about what happened in the past. And you can throw some great mathematics at it and try to use it to predict the future in some sense, but it's not necessarily great at what happens when you hit really hard changes, right?For example, when the Coronavirus pandemic hit and purchaser and consumer behavior changed overnight. There was no data in the data set that explained that consumer behavior. And so, what you saw is a lot of these things like supply chain issues, which are very heavily data-driven on a normal circumstance, there was nothing in that data that allowed those algorithms to optimize for the reality that we were seeing at that scale, right? Even if you look at advanced logistics companies, they know what to do when there's a hurricane coming or when there's been an earthquake or things like that. They have disaster scenarios.But nobody has ever done anything like this at the global scale, right? And so, what we saw was this hard reset that we're still feeling the repercussions of today. Yes, there were people who couldn't work and we had lockdowns and all that stuff, but we also have an effect from the impact of the way that we built the systems to work with the data that we need to shuffle around. And so, I think that there is value in being able to process these really, really large datasets, but I think that actually, there's also a lot of value in being able to solve smaller, simpler problems, right? Not everything is a big data problem, not everything requires a ton of data to solve.It's more about the mindset that you use to look at the data, to explore the data, and what you're doing with it. And I think the challenge here is that, you know, everyone wants to believe that they have a big data problem because it feels like you have to have a big data problem if you—Corey: All the cool kids are having this kind of problem.Emily: You have to have big data to sit at the grownup's table. And so, what's happened is we've optimized a lot of tools around solving big data problems and oftentimes, these tools are really poor at solving normal data problems. And there's a lot of money being spent in a lot of overkill engineering in the data space.Corey: On some level, it feels like there has been a dramatic misrepresentation of this. I had an article that went out last year where I called machine-learning selling pickaxes into a digital gold rush. And someone I know at AWS responded to that and probably the best way possible—she works over on their machine-learning group—she sent me a foam Minecraft pickaxe that now is hanging on my office wall. And that gets more commentary than anything, including the customized oil painting I have of Billy the Platypus fighting an AWS Billing Dragon. No, people want to talk about the Minecraft pickaxe.It's amazing. It's first, where is this creativity in any of the marketing that this department is putting out? But two it's clearly not accurate. And what it took for me to see that was a couple of things that I built myself. I built a Twitter thread client that would create Twitter threads, back when Twitter was a place that wasn't overrun by some of the worst people in the world and turned into BirdChan.But that was great. It would automatically do OCR on images that I uploaded, it would describe the image to you using Azure's Cognitive Vision API. And that was magic. And now I see things like ChatGPT, and that's magic. But you take a look at the way that the cloud companies have been describing the power of machine learning in AI, they wind up getting someone with a doctorate whose first language is math getting on stage for 45 minutes and just yelling at you in Star Trek technobabble to the point where you have no idea what the hell they're saying.And occasionally other data scientists say, “Yeah, I think he's just shining everyone on at this point. But yeah, okay.” It still becomes unclear. It takes seeing the value of it for it to finally click. People make fun of it, but the Hot Dog, Not A Hot Dog app is the kind of valuable breakthrough that suddenly makes this intangible thing very real for people.Emily: I think there's a lot of impressive stuff and ChatGPT is fantastically impressive. I actually used ChatGPT to write a letter to some German government agency to deal with some bureaucracy. It was amazing. It did it, was grammatically correct, it got me what I needed, and it saved me a ton of time. I think that these tools are really, really powerful.Now, the thing is, not every company needs to build its own ChatGPT. Maybe they need to integrate it, maybe there's an application for it somewhere in their landscape of product, in their landscape of services, in the landscape of their interim internal tooling. And I would be thrilled actually to see some of that be brought into reality in the next couple of years. But you also have to remember that ChatGPT is not something that came because we have, like, a really great breakthrough in AI last year or something like that. It stacked upon 40 years of research.We've gone through three new waves of neural networking in that time to get to this point, and it solves one class of problem, which is honestly a fairly narrow class of problem. And so, what I see is a lot of companies that have much more mundane problems, but where data can actually still really help them. Like how do you process Cambodian driver's licenses with OCR, right? These are the types of things that if you had a training data set that was every Cambodian person's driver's license for the last ten years, you're still not going to get the data volumes that even a day worth of Amazon's marketplace generates, right? And so, you need to be able to solve these problems still with data without resorting to the cudgel that is a big data solution, right?So, there's still a niche, a valuable niche, for solving problems with data without having to necessarily resort to, we have to load the entire internet into our stream and throw GPUs at it all day long and spend hundreds of—tens of millions of dollars in training. I don't know, maybe hundreds of millions; however much ChatGPT just raised. There's an in-between that I think is vastly underserved by what people are talking about these days.Corey: There is so much attention being given to this and it feels almost like there has been a concerted and defined effort to almost talk in circles and remove people from the humanity and the human consequences of what it is that they're doing. When I was younger, in my more reckless years, I was never much of a fan of the idea of government regulation. But now it has become abundantly clear that our industry, regardless of how you want to define industry, how—describe a society—cannot self-regulate when it comes to data that has the potential to ruin people's lives. I mean, I spent a fair bit of my time in my career working in financial services in a bunch of different ways. And at least in those jobs, it was only money.The scariest thing I ever dealt with, from a data perspective is when I did a brief stint at Grindr because that was the sort of problem where if that data gets out, people will die. And I have not had to think about things like that have that level of import before or since, for which I'm eternally grateful. “It's only money,” which is a weird thing for a guy who fixes cloud bills for a living to say. And if I say that in a client call, it's not going to go very well. But it's the truth. Money is one of those things that can be fixed. It can be addressed in due course. There are always opportunities there. Someone just been outed to their friends, family, and they feel their life is now in shambles around them, you can't unring that particular bell.Emily: Yeah. And in some countries, it can lead to imprisonment, or—Corey: It can lead to death sentences, yes. It's absolutely not acceptable.Emily: There's a lot to say about the ethics of where we are. And I think that as a lot of these high profile, you know, AI tools have come out over the last year or so, so you know, Stable Diffusion and ChatGPT and all of this stuff, there's been a lot of conversation that is sort of trying to put some counterbalance on what we're seeing. And I don't know that it's going to be successful. I think that, you know, I've been speaking about ethics and technology for a long time and I think that we need to mature and get to the next level of actually addressing the ethical problems in technology. Because it's so far beyond things like, “Oh, you know, if there's a biased training data set and therefore the algorithm is biased,” right?Everyone knows that by now, right? And the people who don't know that, don't care. We need to get much beyond where, you know, these conversations about ethics and technology are going because it's a manifold problem. We have issues with the people labeling this data are paid, you know, pennies per hour to deal with some of the most horrific content you've ever seen. I mean, I'm somebody who has immersed myself in a lot of horrific content for some of the work that I have done, and this is, you know, so far beyond what I've had to deal with in my life that I can't even imagine it. You couldn't pay me enough money to do it and we're paying people in developing nations, you know, a buck-thirty-five an hour to do this. I think—Corey: But you must understand, Emily, that given the standard of living where they are, that that is perfectly normal and we wouldn't want to distort local market dynamics. So, if they make a buck-fifty a day, we are going to be generous gods and pay them a whopping dollar-seventy a day, and now we feel good about ourselves. And no, it's not about exploitation. It's about raising up an emerging market. And other happy horseshit that lies people tell themselves.Emily: Yes, it is. Yes, it is. And we've built—you know, the industry has built its back on that. It's raised itself up on this type of labor. It's raised itself up on taking texts and images without permission of the creators. And, you know, there's—I'm not a lawyer and I'm not going to play one, but I do know that derivative use is something that at least under American law, is something that can be safely done. It would be a bad world if derivative use was not something that we had freely available, I think, and on the balance.But our laws, the thing is, our laws don't account for the scale. Our laws about things like fair use, derivative use, are for if you see a picture and you want to take your own interpretation, or if you see an image and you want to make a parody, right? It's a one-to-one thing. You can't make 5 million parody images based on somebody's art, yourself. These laws were never built for this scale.And so, I think that where AI is exploiting society is it's exploiting a set of ethics, a set of laws, and a set of morals that are built around a set of behavior that is designed around normal human interaction scales, you know, one person standing in front of a lecture hall or friends talking with each other or things like that. The world was not meant for a single person to be able to speak to hundreds of thousands of people or to manipulate hundreds of thousands of images per day. It's actually—I find it terrifying. Like, the fact that me, a normal person, has a Twitter following that, you know, if I wanted to, I can have 50 million impressions in a month. This is not a normal thing for a normal human being to have.And so, I think that as we build this technology, we have to also say, we're changing the landscape of human ethics by our ability to act at scale. And yes, you're right. Regulation is possibly one way that can help this, but I think that we also need to embed cultural values in how we're using the technology and how we're shaping our businesses to use the technology. It can be used responsibly. I mean, like I said, ChatGPT helped me with a visa issue, sending an email to the immigration office in Berlin. That's a fantastic thing. That's a net positive for me; hopefully, for humanity. I wasn't about to pay a lawyer to do it. But where's the balance, right? And it's a complex topic.Corey: It is. It absolutely is. There is one last topic that I would like to talk to you about that's a little less heavy. And I've got to be direct with you that I'm not trying to be unkind, but you've disappointed me. Because you mentioned to me at one point, when I asked how things were going in your AWS universe, you said, “Well, aside from the bank heist, reasonably well.”And I thought that you were blessed as with something I always look for, which is the gift of glorious metaphor. Unfortunately, as I said, you've disappointed me. It was not a metaphor; it was the literal truth. What the hell kind of bank heist could possibly affect an AWS account? This sounds like something out of a movie. Hit me with it.Emily: Yeah, you know, I think in the SRE world, we tell people to focus on the high probability, low impact things because that's where it's going to really hurt your business, and let the experts deal with the black swan events because they're pretty unlikely. You know, a normal business doesn't have to worry about terrorists breaking into the Google data center or a gang of thieves breaking into a bank vault. Apparently, that is something that I have to worry about because I have some data in my personal life that I needed to protect, like all other people. And I decided, like a reasonable and secure and smart human being who has a little bit of extra spending cash that I would do the safer thing and take my backup hard drive and my old phones and put them in a safety deposit box at an old private bank that has, you know, a vault that's behind the meter-and-a-half thick steel door and has two guards all the time, cameras everywhere. And I said, “What is the safest possible thing that you can do to store your backups?” Obviously, you put it in a secure storage location, right? And then, you know, I don't use my AWS account, my personal AWS account so much anymore. I have work accounts. I have test accounts—Corey: Oh, yeah. It's honestly the best way to have an AWS account is just having someone else having a payment instrument attached to it because otherwise oh God, you're on the hook for that yourself and nobody wants that.Emily: Absolutely. And you know, creating new email addresses for new trial accounts is really just a pain in the ass. So, you know, I have my phone, you know, from five years ago, sitting in this bank vault and I figured that was pretty secure. Until I got an email [laugh] from the Berlin Polizei saying, “There has been a break-in.” And I went and I looked at the news and apparently, a gang of thieves has pulled off the most epic heist in recent European history.This is barely in the news. Like, unless you speak German, you're probably not going to find any news about this. But a gang of thieves broke into this bank vault and broke open the safety deposit boxes. And it turns out that this vault was also the location where a luxury watch consigner had been storing his watches. So, they made off with some, like, tens of millions of dollars of luxury watches. And then also the phone that had my 2FA for my Amazon account. So, the total value, you know, potential theft of this was probably somewhere in the $500 million range if they set up a SageMaker instance on my account, perhaps.Corey: This episode is sponsored in part by Honeycomb. I'm not going to dance around the problem. Your. Engineers. Are. Burned. Out. They're tired from pagers waking them up at 2 am for something that could have waited until after their morning coffee. Ring Ring, Who's There? It's Nagios, the original call of duty! They're fed up with relying on two or three different “monitoring tools” that still require them to manually trudge through logs to decipher what might be wrong. Simply put, there's a better way. Observability tools like Honeycomb (and very little else becau se they do admittedly set the bar) show you the patterns and outliers of how users experience your code in complex and unpredictable environments so you can spend less time firefighting and more time innovating. It's great for your business, great for your engineers, and, most importantly, great for your customers. Try FREE today at honeycomb.io/screaminginthecloud. That's honeycomb.io/screaminginthecloud.Corey: The really annoying part that you are going to kick yourself on about this—and I'm not kidding—is, I've looked up the news articles on this event and it happened, something like two or three days after AWS put out the best release of last years, or any other re:Invent—past, present, future—which is finally allowing multiple MFA devices on root accounts. So finally, we can stop having safes with these things or you can have two devices or you can have multiple people in Covid times out of remote sides of different parts of the world and still get into the thing. But until then, nope. It's either no MFA or you have to store it somewhere ridiculous like that and access becomes a freaking problem in the event that the device is lost, or in this case stolen.Emily: [laugh]. I will just beg the thieves, if you're out there, if you're secretly actually a bunch of cloud engineers who needed to break into a luxury watch consignment storage vault so that you can pay your cloud bills, please have mercy on my poor AWS account. But also I'll tell you that the credit card attached to it is expired so you won't have any luck.Corey: Yeah. Really sad part. Despite having the unexpired credit card, it just means that the charge won't go through. They're still going to hold you responsible for it. It's the worst advice I see people—Emily: [laugh].Corey: Well, intentioned—giving each other on places like Reddit where the other children hang out. And it's, “Oh, just use a prepaid gift card so it can only charge you so much.” It's yeah, and then you get exploited like someone recently was and start accruing $60,000 a day in Lambda charges on an otherwise idle account and Amazon will come after you with a straight face after a week. And, like, “Yes, we'd like our $360,000, please.”Emily: Yes.Corey: “We tried to charge the credit card and wouldn't you know, it expired. Could you get on that please? We'd like our money faster if you wouldn't mind.” And then you wind up in absolute hell. Now, credit where due, they in every case I am aware of that is not looking like fraud's close cousin, they have made it right, on some level. But it takes three weeks of back and forth and interminable waiting.And you're sitting there freaking out, especially if you're someone who does not have a spare half-million dollars sitting around. Imagine who—“You sound poor. Have you tried not being that?” And I'm firmly convinced that it a matter of time until someone does something truly tragic because they don't understand that it takes forever, but it will go away. And from my perspective, there's no bigger problem that AWS needs to fix than surprise lifelong earnings bills to some poor freaking student who is just trying to stand up a website as part of a class.Emily: All of the clouds have these missing stairs in them. And it's really easy because they make it—one of the things that a lot of the cloud providers do is they make it really easy for you to spin up things to test them. And they make it really, really hard to find where it is to shut it all down. The data science is awful at this. As a data scientist, I work with a lot of data science tools, and every cloud has, like, the spin up your magical data science computing environment so that your data scientist can, like, bang on the data with you know, high-performance compute for a while.And you know, it's one click of a button and you type in a couple of na—you know, a couple of things name, your service or whatever, name your resource. You click a couple buttons and you spin it up, but behind the scenes, it's setting up a Kubernetes cluster and it's setting up some storage bucket and it's setting up some data pipelines and it's setting up some monitoring stuff and it's setting up a VM in order to run all of this stuff. And the next thing that you know, you're burning 100, 200 euro a day, just to, like, to figure out if you can load a CSV into pandas using a Jupyter Notebook. And you're like—when you try to shut it all down, you can't. It's you have to figure, oh, there is a networking thing set up. Well, nobody told me there's a networking thing set up. You know? How do I delete that?Corey: You didn't say please, so here you go. Without for me, it's not even the giant bill going from $4 a month in S3 charges to half a million bucks because that is pretty obvious from the outside just what the hell's been happening. It's the little stuff. I am still—since last summer—waiting for a refund on $260 of ‘because we said so' SageMaker credits because of a change of their billing system, for a 45-minute experiment I had done eight months before that.Emily: Yep.Corey: Wild stuff. Wild stuff. And I have no tolerance for people saying, “Oh, you should just read the pricing page and understand it better.” Yeah, listen, jackhole. I do this for a living. If I can fall victim to it, anyone can. I promise. It is not that I don't know how the billing system works and what to do to avoid unexpected charges.And I'm just luck—because if I hadn't caught it with my systems three days into the month, it would have been a $2,000 surprise. And yeah, I run a company. I can live with that. I wouldn't be happy, but whatever. It is immaterial compared to, you know, payroll.Emily: I think it's kind of a rite of passage, you know, to have the $150 surprise Redshift bill at the end of the month from your personal test account. And it's sad, you know? I think that there's so much better that they can do and that they should do. Sort of as a tangent, one of the challenges that I see in the data space is that it's so hard to break into data because the tooling is so complex and it requires so much extra knowledge, right? If you want to become a software developer, you can develop a microservice on your machine, you can build a web app on your machine, you can set up Ruby on Rails, or Flask, or you know, .NET, or whatever you want. And you can do all of that locally.And you can learn everything you need to know about React, or Terraform, or whatever, running locally. You can't do that with data stuff. You can't do that with BigQuery. You can't do that with Redshift. The only way that you can learn this stuff is if you have an account with that setup and you're paying the money to execute on it. And that makes it a really high barrier for entry for anyone to get into this space. It makes it really hard to learn. Because if you want to learn anything by doing, like many of us in the industry have done, it's going to cost you a ton of money just to [BLEEP] around and find out.Corey: Yes. And no one likes the find out part of those stories.Emily: Nobody likes to find out when it comes to your bill.Corey: And to tie it back to the data story of it, it is clearly some form of batch processing because it tries to be an eight-hour consistency model. Yeah, I assume for everything, it's 72. But what that means is that you are significantly far removed from doing a thing and finding out what that thing costs. And that's the direct charges. There's always the oh, I'm going to set things up and it isn't going to screw you over on the bill. You're just planting a beautiful landmine you're going to stumble blindly into in three months when you do something else and didn't realize what that means.And the worst part is it feels victim-blamey. I mean, this is my pro—I guess this is one of the reasons I guess I'm so down on data, even now. It's because I contextualize it in a sense of the AWS bill. No one's happy dealing with that. You ever met a happy accountant? You have not.Emily: Nope. Nope [laugh]. Especially when it comes to clouds stuff.Corey: Oh yeah.Emily: Especially these days, when we're all looking to save energy, save money in the cloud.Corey: Ideally, save the planet. Sustainability and saving money align on the axis of ‘turn that shit off.' It's great. We can hope for a brighter tomorrow.Emily: Yep.Corey: I really want to thank you for being so generous with your time. If people want to learn more, where can they find you? Apparently filing police reports after bank heists, which you know, it's a great place to meet people.Emily: Yeah. You know, the largest criminal act in Berlin is certainly a place you want to go to get your cloud advice. You can find me, I have a website. It's my name, emilygorcenski.com.You can find me on Twitter, but I don't really post there anymore. And I'm on Mastodon at some place because Mastodon is weird and kind of a mess. But if you search me, I'm really not that hard to find. My name is harder to spell, but you'll see it in the podcast description.Corey: And we will, of course, put links to all of this in the show notes. Thank you so much for your time. I really appreciate it.Emily: Thank you for having me.Corey: Emily Gorcenski, Data and AI Service Line Lead at ThoughtWorks. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insipid, insulting comment, talking about why data doesn't actually matter at all. And then the comment will disappear into the ether because your podcast platform of choice feels the same way about your crappy comment.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About CatherineCatharine brings more than fifteen years of experience building global networks and large scale data center infrastructure to the challenge of scaling quickly and safely.  She loves building engaged and curious teams, providing insightful forecasting tools, and thinking about how to build to scale in a sustainable way to preserve a humane quality of life on this swiftly tilting planet. When not trying to predict the future as a capacity planner, she's often knitting extremely complicated sweaters and coming up with ridiculous puns.TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Tailscale SSH is a new, and arguably better way to SSH. Once you've enabled Tailscale SSH on your server and user devices, Tailscale takes care of the rest. So you don't need to manage, rotate, or distribute new SSH keys every time someone on your team leaves. Pretty cool, right? Tailscale gives each device in your network a node key to connect to your VPN, and uses that same key for SSH authorization and encryption. So basically you're SSHing the same way that you're already managing your network.So what's the benefit? Well, built-in key rotation, the ability to manage permissions as code, connectivity between any two devices, and reduced latency. You can even ask users to re-authenticate SSH connections for that extra bit of security to keep the compliance folks happy. Try Tailscale now - it's free forever for personal use.Corey: Kentik provides Cloud and NetOps teams with complete visibility into hybrid and multi-cloud networks. Ensure an amazing customer experience, reduce cloud and network costs, and optimize performance at scale — from internet to data center to container to cloud. Learn how you can get control of complex cloud networks at www.kentik.com, and see why companies like Zoom, Twitch, New Relic, Box, Ebay, Viasat, GoDaddy, booking.com, and many, many more choose Kentik as their network observability platform. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. As a cloud economist, I wind up talking to an awful lot of folks about optimizing their AWS bills. That is what it says on the tent. It's what I do. Increasingly, I'm having discussions around the idea of sustainability because the number-one rule of cloud economics is also the number-one rule for sustainability. Step one, turn that shit off. If you're not using it, turn that shit off. If it doesn't add value commensurate to what it costs, turn that shit off. Because the best way to optimize something is to get rid of it. Today, to go into bit more depth on that, my guest is Catharine Strauss. Catharine, thank you for joining me.Catharine: Thank you. I'm excited.Corey: So, you have a long and storied career of effectively running global-scale network operations in terms of capacity planning, in terms of building out world-spanning networks, and logistics of doing that. You know, the stuff that's completely invisible to most people, except when it breaks. So, it's more or less a digital plumbing-type of role. How did you go from there to thinking about sustainability in a networking context?Catharine: Yeah. Thank you. I got dropped into networking as a career option, completely from the physical side, building out global networks. And all of the constraints that we were dealing with, were largely physical, logistical, or legal. So, we would do things like ship things through customs and have items stopped because they were miscategorized as munitions because they were lasers, “Pew, pew.” We had things like contract negotiations for data centers to do trenching into them that needed easements with the railroad. Like, just weird stuff that you don't normally think of as a cloud-project constraint. So, all of these physical constraints made it just more interesting to me because they were just so tactile.Corey: There's so much that is out there in the world that is completely divorced from anything that you have to think about in terms of building out networks and software. Until, suddenly, it's very much there, and you're learning that there's an entire universe/industry/ecosystem that you know nothing about that you now need to get into. Railroad easements are a terrific example of that. It's, “Wait, what, we're building the cloud here. What the hell does the railroad have to do—is there actually a robber baron I need to go fight somewhere? How does this work?” The old saw about the cloud just being someone else's computer is not particularly helpful, but it is true. There's a tremendous amount of work that goes into building out the physical footprint for a data center—let alone a hyper-scale cloud provider's data center—that does not have to be something the vast majority of us need to think about anymore. And that's, kind of, glorious and magical. But it does mean that there are people who very much need to think about that.Increasingly, we're seeing the sustainability and climate story of cloud extend beyond those folks. There are no carbon-footprint tools and dashboards in all the major cloud providers that I'm aware of. Well, I'd say it's a good start, but in some cases, it's barely that. It feels like this is something that people are at least starting to take semi-seriously in the context of cloud. How have you seen that evolving?Catharine: So, when I think about a data center, I see it as a factory where you take heavy metals and electricity—Corey: And turn them into YAML. Sorry, sorry. Go ahead.Catharine: [laugh] you turn them into spreadsheets, cat videos, and waste heat, right? So, when I'm looking at, you know, this tremendous global network, I started to look into what's the environmental cost of that. And what I found was, kind of, surprising. Like, three percent of our total global emissions, is coming from computing and the internet, and all of these things that I spent my career building. And I started to have waves of regret. And looking at that in the context of: how can we make things better? How can we make things more efficient, and how can we operate better with the physical constraints of electricity and energy grids, and what they are struggling with doing to provide us with what we need or managing this beast of an internet?Corey: Right now, it feels like there's an awful lot of—I don't know what the term is, greenwashing, cloud washing—basically, making your problem someone else's problem. I feel like the cloud providers are in a position where they have to walk something of a tightrope. Because on the one hand, yeah, there are choices I can make as a customer that will absolutely improve the carbon footprint of what it is that I'm doing. On the other, they never invite me to have conversations to negotiate with their energy providers around a lot of these things. So, it feels like, “Oh, yeah. Make sure that the cloud you're using is green enough.” “Wasn't that what I'm paying you for?” That feels like it's a really weird dichotomy that I'm still struggling to reconcile exactly how to approach.Catharine: Yeah. I, you know, I looked at the Amazon Sustainability platform, and they've got those two parts of it. They've got sustainability in the cloud, sustainability of the cloud. And, you know, I've worked with enough Google SREs to know that they and Amazon Data Center providers and Azure, they all have a vested interest in making it as cheap as possible to operate their data centers. And that goes far beyond individual server performance. It goes to the way that they do cooling. And, like, the innovations there are tremendous.But they're not doing that out of the goodness of their heart; they're doing that because it makes business sense for them. It reduces the cost for them to provide these services. And, you know, in some cases, it really obscures things because they will sign energy contracts and then keep them super-secret. There's very little transparency because these are industry secrets, and they don't want to damage their negotiation positions for the next deal that they sign. So, Amazon, you know, will put PR releases out there about all of their solar farms that they are sustaining in Virginia. But they don't talk about what percent that is of their total energy consumption, and they don't talk about, you know, what the total footprint is because that is considered either a security risk or an economic risk if people were to find out, you know, exactly how much energy they're pulling.Corey: I am, somewhat, sympathetic, but only to the reality that the more carbon transparency that a cloud provider gives around the relative greenness of a given service that they offer in a given region, the closer they get to exposing a significant component on their per-service margins. And they're, understandably, extraordinarily reluctant to that because then people will do things like figure out exactly how much are they up-charging things like data egress and ongoing per-hour session charges for some sage-maker nonsense.There's an awful lot out there that I don't think they want to have out there just for, on the one hand, the small one that's easy to deal with is the customer uprising. But more so, they don't want to expose this to their competitors.Catharine: Yeah, I don't know that I have a ton of sympathy. If the service is cheaper because they're running off of green energy, as we have increasingly seen in the market that solar and wind are just the cheapest alternative. If it's cheaper for Amazon and Google, I, kind of, feel like they should convey that, so that people can take advantage of those savings.We've got a demand issue, where, I think, the demand for these renewable energy sources is outstripping supply. But they're planning for the next five years where that decreasingly becomes an issue. So, why not let people operate according to their values, or even, you know, their own best interests in choosing data centers that are emitting fewer emissions into the world?Corey: There seems to be a singular focus between all of these providers in what they're displaying through their tools. And that is on carbon footprint, and it is also suspiciously, tightly bounded to what looks like compute. There're a lot of other climate-impacting effects of large-scale cloud providers. It has significant disruption to local waterways. There are tremendous questions around the sustainability around manufacturing of the various components that get turned into equipment that gets sold to these providers then integrating into other things. There's an awful lot of downstream effects. And I can't shake the feeling that focusing on how renewable the energy is to power the compute, focuses on a very small part of the story. How do you land on that one?Catharine: I would agree with that. I think people will often say, “Oh, what you should do if you're managing,” you know, “Your data center resources is for efficiency, you should be updating your hardware once a year or putting out the resources that are the most powerful.” The tipping point might be later than you actually think because what happens to those resources when they go back out into the environment when you decommission them? It's so hard to resell them, especially, globally. The reuse of gear is becoming harder and harder, and so the lifetime of that gear, that equipment, those servers, routers, whatnot, all of that is becoming harder and harder to do. And the disposal of those materials has a tremendous impact.So, I do think the energy is a big part of it, and it feels like the thing that we can control the most. But, like, if you really want to change the world, go work on carbon-neutral cement or batteries made out of rust and sand to store solar energy. You know, go work on low-heat steel. Those are the things where you're really making an impact. What we need to do in the market is really transform our notion of the cloud as this infinite nebulous, weightless item into something that is physical and has a physical impact on our lives.So, when you're trying to decide what your retention policy is for your data in your company when you're trying to decide where to replicate data, how long to hold it in active storage, you're really thinking about the megawatts that it takes, and the impact of that on the full picture.Corey: Well, a question that I've had as I look across my customer base of large companies doing interesting and exciting things with cloud, is I would love—absolutely love—to see a comparative analysis done by each provider that in very human terms, says what the relative climate impact is of taking all of their different storage services, on a per-petabyte basis, where I say, “Okay, if I want to store this in their object storage, or if I want to put this on disc volumes, or I want to use their deep-archive storage that looks an awful lot like tape, I don't care so much about the cost of those things, but I want to know what is the climate impact of this,” because I think that would be revelatory on a whole bunch of different levels. But it seems it's computes where they tend to focus instead.Catharine: Yeah, it would be really nice if as businesses, we started to look at the fuller impact of our actions. And it isn't just about the money saved. But my genuine belief is that it will get cheaper to do the right thing. And it is getting cheaper every day to use fewer resources. But the market has not caught up to that, and you can see that in how many companies are still giving away free, unlimited storage, right? You know, how many Go-Pro videos of someone's backyard, how many hours of that kind of footage is there out there in the world that's never going to get viewed again, but is sitting out there taking up energy that, at the same time, that we're having brownouts, and people are suffering and having to turn off their air conditioning?Corey: I think that we would do well as a society to get rid of a heck of a lot more data just because it sits there; it burns energy; it costs money, and I'm sorry, you're going to really have to reach to convince me that the web server access logs from 2012 are in any way business valuable or relevant to, basically, anyone out there.But I want to take it one step further because now that we know that we're definitely burning the planet to wind up storing a petabyte of data here, I'm very curious as to the climate footprint of then going into your world, taking that data, and throwing it somewhere else across the internet. Because I can tell you, almost to the penny, what that's going to cost, and it's an astonishingly large number because yeah, egress fees are what they are, but I couldn't tell you what the climate footprint of that is.Catharine: Yeah. When I was working at Fastly, we did a lot of optimizations across our network to avoid peak traffic because that was how we were built. You know, we had to build out to a certain network capacity, and then we could build, essentially, the area under our diurnal curve, we can build that out. But we don't have to, necessarily, serve it from the absolute closest data center. If we could serve it from a nearby data center or a provider that was three milliseconds of ‘wait and see more,' we could potentially use resources that we have elsewhere in the cloud to serve that request more efficiently.And I think we have an opportunity to do that with data centers scattered around the globe. Why aren't we load balancing so that we're pushing traffic from the data centers that are off-peak—you know, have energy to spare to accommodate for the data centers that are reaching capacity and don't have enough energy on the grid—why aren't we using these resources more efficiently?Corey: I've often lamented, from an economic perspective, that if I want to spend less money and optimize things, I can wind up trading out my instance types. Okay, I have a super-fast, high-end processor that costs a lot of money. I can get shittier compute by spending less. The same story with storage. I can get slower storage for less money that's a lot less performing, and it has some latencies added, but, “Great,” but I can make that decision.With networking, it's all of its nothing. It's there is no option for me to say, “I want to pay half of what the normal data rates are, but in return, I really only care that this data gets to where it's going by next Tuesday.” I don't need it done in sub-second latency speeds. There's no way to turn that off or to make that election. Increasingly, I really am coming around to the idea that cloud economics and sustainability are one in the same.Catharine: Yeah. For me, it makes a lot of sense. And, you know, when I look at people in their careers, focusing on cloud economics feels like a very, very easy win if you also care about sustainability. And it feels like once you have the data and the reporting tools—and, you know, we talked about the big gaps there—but if you're reporting on both your costs and the carbon footprint, you're developing a plan for how to optimize on both of them at the same time, and you're bringing that back to your management, bringing that back to your teammates, and really making sustainability an active value in your organization.I feel like there's not only a benefit to you, the finances of your company, and your personal career, but there's also a social impact where, you know, maybe you can feel a little less guilty about eating that steak. Maybe you can offset some travel that is increasing your carbon footprint; maybe you can do a trade-off; maybe you can do everything in little bursts across a broad scope, instead of us needing, you know, some big solution that's going to save us. There's no one solution.I think that's the main thing I've discovered in my education on sustainability is it has to be 50,000 small things, the ‘magic buckshot' rather than the ‘magic bullet,' is the term that I see used a lot. Carbon removal from the sky is coming, but while we wait for it, we got to slow the pace of digging the hole, and really give our solutions a chance to work.Corey: I despair at times at the lack of corporate will, I suppose, to wind up pursuing cloud sustainability as a customer of one of the cloud providers. I get people reaching out to me, pretty frequently, to help optimize the cost of their AWS bill. That is, definitely, what I do for a living. If I don't have people reaching out on that, something is going wrong somewhere. And even then, there have been months that have been relatively slow in recent years. Because well, it turns out when money is free, you don't really care that much about saving money. Now, people are tightening their belts and have to think about it a lot more, but that is a direct incentive of if you go ahead and optimize your cloud-spend bill, you will have more money.That is, sort of, what our capitalist system is supposed to optimize for in many respects. “Great,” you can have more money. But it's still not exciting for folks, and it's not what they really wind up chasing after. I despair at getting them to think larger than money because that's the only thing that companies generally tend to think about in the abstract, and start worrying about the future and climate and to invest significant effort in doing climate optimization. I don't know that there is a business today in greening your cloud workloads that could be started the way that I have for fixing the AWS bill.Catharine: Yeah, I don't think there's a business in it; I think it's a movement. It's like accessibility; it's like security; it's like a lot of other movements that have happened recently in tech where it becomes everybody's job. And it's important to people. And it becomes part of your company's brand, and you use it for recruitment; you use it for advancing your own career; you use it for making people feel like they're making a better decision.When I look at the three big cloud providers, and I look at the ways that they are marketing their sustainability, it is so slick. You go to their sustainability page and it's all, you know, beautiful, flashy graphics and information on all these feel-good things. Because they know, if they don't do it, they're going to be passed over because somebody is going to bring this up when they're evaluating their choices. Because we want it; we all want it. We just don't quite know how to get there. And until recently, it was more expensive, and you did have a green tax made the sustainable options more expensive. We're turning the page on that. Solar is cheaper than coal. And that's all you really—all you have to say to justify some of these advancements. It's all going to flow out of that simple fact.Corey: Cloud native just means you've got more components or microservices than anyone (even a mythical 10x engineer) can keep track of. With OpsLevel, you can build a catalog in minutes and forget needing that mythical 10x engineer. Now, you'll have a 10x service catalog to accompany your 10x service count. Visit OpsLevel.com to learn how easy it is to build and manage your service catalog. Connect to your git provider and you're off to the races with service import, repo ownership, tech docs, and more. Corey: I think that there's a tremendous opportunity here to think about this. And I think you're right. It absolutely takes on aspects of looking like a movement to do that. I'm optimistic about that. The counterpoint is that individuals are often not tremendously effective at altering the behavior of trillion-dollar companies, or even the relatively small ‘only' 50 billion-dollar companies out there.I can see where it starts, and I can see the outcome that you want there. I just have no idea what it looks like in between. It's, like, “Step two, we'll figure this part out later. Step three, climate.”Catharine: Yeah. If I were going to do it at my company, I would go to HR. And I would say, “I would like to form an employee-resource group around sustainability. Do you know anyone on the executive team who is interested in sustainability?” Get them to sponsor it; talk to that sponsor, and say, “We're the co-benefits here. What do you see as things that we absolutely need to do from a corporate-strategy standpoint that are aligned with this?”And then start having meetings—open meetings—where you invite people concerned about climate change, and you start to talk cross-functionally about, “What can we do? Can we change our retention policies? Can we change the way that we bill for services? Can we individually delete data on our Wiki that hasn't been accurate for seven years?” And, you know, start to talk and share successes. Then take it out to the larger industry and start giving talks because people want to be able to do something. Climate despair is real, but we, as cloud technologists, are so powerful in the resources that we have stewardship over. But I have to think that there is a possibility of making real change here.Corey: There's a certain point of scale at which point, having a sustainability conversation becomes productive. There are further points of scale where it becomes mandatory, let's be clear here. But when I'm building something in the off hours—mostly for shit-posting purposes—it generally tends to wind up costing maybe seven cents or so, when all is said and done because I'm using Lambda functions and other things that don't take a whole lot of computer resources out there. Googling what the most climate-effective way to implement that would be, is one of those exercises where the google search has a bigger carbon footprint than the entire start-to-finish of what it is that I'm building. It's not worth me looking into that.There is some inflection point between that, and we run 500,000 servers around the world or 500,000 instances where, yeah, there's a definite on-ramp where you need to start thinking about these things. What is that, I guess, that first initial point of, “I should be thinking about this,” for a given workload?Catharine: So, I've been trying to get data on this, and my best calculation is that an average server in a hyperscale data center, where you're using the whole thing for an entire year, is one to two tons of CO2 per year. So, I think when you start to look at other initiatives that you're seeing, I think the tipping point is around ten tons per year. And for some people, that's a lot; that's a lot of resources that you need to get up to that point.Corey: That feels directionally right. I think that is absolutely around where it starts to make sense. I mean, right now, I'm also in the uncomfortable creeping-awareness position of I've run a medium-sized EC2 instance persistently. That is my developer environment. I have it running all the time because having a Linux box is, sort of, handy. And whether I need it or not, it's there. If I were to turn it off when I go to sleep at night, for example, I do not believe that would have any climate impact whatsoever from the perspective of this is a medium-size instance. There are a bunch of those on any individual server.Amazon is not going to turn off Iraq right now because my instance is there or it's not. It is well within the margin of error for anything they have as far as provisioning or de-provisioning something. So, then someone, like, steals it to the term you used of climate despair a few minutes ago, that's what this feels like. It's one of those, “Well, okay. So, if it makes no actual difference if I were to spend instrumenting that thing to turn itself off at night and turn itself back on in the morning, it doesn't change a damn thing. I'm just doing something that is effectively meaningless in order to make myself feel better.”The enormity of the problem and the task, and doing it at scale, well, I'm not going to convince customers to do that. And for some cases, maybe that's for the better; maybe it's not. But I feel like for whatever I do, there's nothing I can do to make a difference in that sense, in my small-scale personal environment.Catharine: Yeah, yeah. I definitely appreciate that. This feel to me like the same concept of—I don't know, a couple of months ago, if you remember, California had a heat wave, and there were rolling brownouts. And we got a text that said, “Energy is at a high right now. Please turn off any unnecessary devices,” trying to avoid additional impact to the energy grid. And if you go and you look at the graph, there was an immediate decrease of 1500 megawatts in that moment because enough people got the text and took a small action, and it had the necessary impact. We avoided the brownouts, and the power, generally, kept flowing because it's such a big system.You know, if we're talking about three percent of global emissions, we're talking about, you know, power that's the size of the aviation industry. We're talking about power that's, roughly, the size of Switzerland just on data centers. You, as an individual, are not going to be able to make an impact; you, as an individual talking about this to as many people as possible—as we're doing right now—that starts to move the needle. And the thing I like about forming a grass roots group inside of your company is that it's not just about the data centers. Maybe, it's also about the service that comes in and brings you food and uses disposable containers; maybe, it's about people talking about their electric cars; maybe, it's about installing a heat pump; maybe, it's about talking about solutions instead of just talking about creeping dread all the time.Like, my move into sustainability has been largely in response to I can't keep doom-scrolling. I have to find the people who are making the solutions happen. And I just got out of a program with Climatebase where that is what I did for nine weeks is talk about the solutions. And all of the people in the companies that are actually doing something, they're so much more optimistic than the people I talk about who are just reading the headlines.Corey: Doing something absolutely feels better than sitting here helplessly and more less doom-scrolling about it. I absolutely empathize there. I think the trick is to get people to start taking action on this. I am curious, getting a little bit back to where you come from, something you alluded to at one point, was how energy markets are akin to network throughput. And I definitely wanted to dive into that. What do you mean? I'm not disagreeing, but I also have a really hard time seeing that. Help?Catharine: Yeah. So, I used to do capacity planning for Fastly. And so, we would spend all day staring at the diurnal curve of our network throughput because we had to plan for the peak. Whatever our traffic throughput was, our global network needed to be able to handle it. And every day—maybe we got close to that peak; maybe we didn't—but every day it would dip down into just the doldrums as people went to sleep and weren't using the internet.So, when I moved into looking at energy markets, specifically smart grids, and the way that renewables affect the available supply of electricity, I saw that same electricity curve; it's called the duck curve in electricity markets where you have this diurnal pattern and a point every day, where the grid has electricity available but no demand.So, when I was managing costs for our network, we would be trying, as much as possible, to fill that trough every day because it was free for us because we had already built out the infrastructure to fulfill that demand.And the energy markets are same way. We have built out the infrastructure. We just need the demand to meet the timing of the day. Put another way, you have to think fourth-dimensionally. It's like Doc Brown in Back to the Future III. Marty says, “If we continue along this track, the bridge isn't built yet. We're going to plunge into the canyon and die.” And Doc Brown says, “No, no, no. You're not thinking fourth-dimensionally. When we travel through time, we will be in the future, and the bridge will be there.” So, if we can shift the load from one region where energy is being consumed at its peak and move the traffic over to a region in the Pacific Northwest or a different time zone where they haven't yet hit their energy-consumption peak, we can more efficiently use the infrastructure that is already been built out.Corey: I really wish things were a lot easier to move around in that context. Data transfer fees make that very challenging, even if you can get around the latency challenges—which for many workloads is fine; that is not a prohibitive challenge. It's the moving things around; moving data to those other regions, especially, in the sense of, “But, okay. You're making it worse because now you have the data living in two different places instead of only one. You've doubled the carbon footprint of it, too.”For some workloads, it absolutely has significant merit. I just don't know exactly what that's going to look like—actually, I take that back—the more I think about that, the more I realize that in some level, that's what SDNs do already where, “Great, if this has to be built into something; if I hit an AWS endpoint or an API Gateway or something, I want to have an option when I'm building that out to be able to have that do more or less a follow-the-sun style pattern where it's honed out of wherever energy markets are inexpensive.” And that certainly is going to break things for a lot of workloads, but not all of them, not by far.Catharine: Yeah, and I think that is where my context is coming from. You know, working at Fastly, that was the notion, you know, “We're caching your data close to your end-users, so you don't have to operate resources in that area.” And we have a certain amount of leeway to how we serve that traffic. But it is a more global-distributed model and spinning up servers only when you need them is also a model that takes advantage of not having idle services around just in case you need them, actually responding to demand in real-time.If you look at what the future holds for, you know, smart grids, energy networks, there's this tremendous ability—and I would be very surprised if the big providers are not working on this—to integrate the two—so that electricity availability and how our network traffic is served, is just built into the big providers.Corey: I really hope that one of these big providers leads the way on that. That's the kind of thing that they should really want to see come out of these folks. We are recording this before AWS reinvents. So, if they did come out with something like this, good for them, and also, I have no idea, at the time of this recording, whether they are or not. So, if I got it right, no, I'm not breaking any confidentiality agreements. I feel I need to call that out explicitly because everyone assumes that I—that I have magic insight into everything they're going to come out with. Not really; usually it's all after the fact.Catharine: What I'm really hoping is that by the time this airs, Amazon has already released version two of their carbon footprint tool, where they have per data center visibility where it's no longer three months in arrears, so that you can actually do experimentation and see how differences in the way you implement your cloud impact your carbon footprint. Rather than just, like, sort of, the receipt of, “Yep, here's your carbon footprint.” Like, “No, no, no; I want to make it better. How do I make it better?”So, I'm very much hoping they make an announcement of that kind, and then I'll come back.Corey: You're welcome to come back if and when there's anything that any of these providers release that materially changes the trajectory we're currently on. I want to thank you for being so generous with your time. If people want to learn more, where's the best place for them to find you?Catharine: Yeah. You can find me on my website, Summerstir.com. And also, I hang out an awful lot with some very smart people on ClimateAction.tech. Their Slack is a great repository for people concerned about exactly these issues.Corey: And we will, of course, put links to that in the [show notes 00:37:21]. Thank you so much for being so generous with your time. I appreciate it.Catharine: This has been delightful. Thank you.Corey: Catharine Strauss, budding digital sustainability consultant. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that also includes the cloud sustainability metrics for that podcast platform of choice.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About AerinAerin is a Cloud Sustainability Advocate and neurodiverse founder in tech on a mission to help developers understand the real impact that cloud computing has on the world and reduce their carbon emissions in the cloud. Did you know that internet and cloud computing contribute over 4% of annual carbon emissions? Twice that of the airline industry!Aerin also hosts "Public Cloud for Public Good," a podcast targeted towards developers and senior leaders in tech. Every episode, they also donate £500 to charities and highlight organisations that are working towards a better future. Listen and learn how you can contribute towards making the world a better place through the use of public cloud services.Links Referenced: Twitter: https://twitter.com/aerincloud LinkedIn: https://www.linkedin.com/in/aerinb/ Public Cloud for Public Good: https://publicgood.cloud/ duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Uptycs, because they believe that many of you are looking to bolster your security posture with CNAPP and XDR solutions. They offer both cloud and endpoint security in a single UI and data model. Listeners can get Uptycs for up to 1,000 assets through the end of 2023 (that is next year) for $1. But this offer is only available for a limited time on UptycsSecretMenu.com. That's U-P-T-Y-C-S Secret Menu dot com.Corey: Cloud native just means you've got more components or microservices than anyone (even a mythical 10x engineer) can keep track of. With OpsLevel, you can build a catalog in minutes and forget needing that mythical 10x engineer. Now, you'll have a 10x service catalog to accompany your 10x service count. Visit OpsLevel.com to learn how easy it is to build and manage your service catalog. Connect to your git provider and you're off to the races with service import, repo ownership, tech docs, and more. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn and I am joined what feels like roughly a year later by a returning guest, Aerin Booth. How long have you been?Aerin: I've been really great. You know, it's been a journey of a year, I think, since we sort of did this podcast even, like, you know, a year and a bit since we met, and, like, I'm doing so much and I think it's doing, like, a big difference. And yeah, I can't wait for everything else. It's just yeah, a lot of work right now, but I'm really enjoying it. So, I'm really well, thank you.Corey: Normally, I like to introduce people by giving their job title and the company in which they work because again, that's a big deal for an awful lot of people. But a year ago, you were independent. And now you still are. And back when I was doing my own consulting independently, it felt very weird to do that, so I'm just going to call you the Ted Lasso of cloud at this point.Aerin: [laugh].Corey: You've got the mustache, you've got the, I would say, obnoxiously sunny disposition. It's really, there's a certain affinity right there. So, there we go. I feel like that is the best descriptor for what you have become.Aerin: I—do know what, I only just watched Ted Lasso over Christmas and I really found it so motivational in some ways because wow, like, it's not just who we'd want to be in a lot of ways? And I think, you know, for the work that I do, which is focused on sustainability, like, I want to present a positive future, I want to encourage people to achieve more and collaborate, and yeah, basically work on all these problems that we need to be worked on. And yeah, I think that's [laugh] [crosstalk 00:02:02]—Corey: One of the challenges of talking to you sometimes is you talk about these depressing things, but there's such a—you take such an upbeat, positive approach to it that I, by comparison, invariably come away from our conversations during, like, I'm Surly McBastard over here.Aerin: [laugh]. Yeah, you can be the bad cop of cloud computing and I'll try and be the good cop. Do you know, you say that the stuff I talk about is depressing, and it is true and people do worry about climate change. Like I did an online conference recently, it's focused on FinOps, and we had a survey, “Do you worry about climate change?” 70% of the people that responded said they worry about it.So, we all know, it's something we worry about and we care about. And, you know, I guess what I'm really trying to do is encourage people to care a bit more and start taking action and look after yourself. Because you know, when you do start taking action towards it, when you join those communities that are also working on it, it is good, it is helpful. And, you know, I've gone through some ups and downs and some of this, like, just do I throw in the towel because no one cares about it? Like, we spoke last year; I had attended re:Invent for the first time.This year, I was able to speak at re:Invent. So, I did a talk on being ethical in tech. And it was fun, it was good. I enjoyed what I delivered, but I had about 35 people sign up to that. I'm pretty sure if I talked about serverless or the next Web3 blockchain product, I would have got hundreds more. But what I'm starting to realize is that I think people just aren't ready to, sort of, want to do this yet. And yeah, I'm hoping that'll change.Corey: Let's first talk about, I guess, something that is more temporally pressing than some other things. Not that it is more important than climate change, mind you, but it feels like it's on a shorter timeline which is, relatively soon after this recording, there is a conference that you are kicking off called The State of Open. Ajar, Aerin. The State of Open is ajar. What is this conference? Is it in person? Is it virtual? Is it something where you and three friends are going to show up and basically talk to each other? How big? How small? What is it? What's it about? Tell me more, please. I'm riveted.Aerin: So, State of Open conference is a conference that's been in the works now for maybe about two weeks, a little bit longer in the planning, but the work we've been putting in over the last two weeks. It'll be on the seventh and eighth of February in London as a physical event in the QEII Conference Centre, but it will also be available online. And you know, when we talk about the State of Open, it's that question: what is the State of Open? The state of open-source, the state of open hardware, and the state of open data. And it is going to be probably the first and hopefully the biggest open-source conference in the UK.We already have over 100 confirmed guest speakers from Jimmy Wales, the co-founder of Wikipedia, to many of our great guests and headliners who haven't even announced yet for the plenary. So, I'm really excited. And the reason why I wanted to get involved with this is because one of the coolest things about this conference—compared to some others like re:Invent, for example—is that sustainability and diversity run through every single thing that we do. So, as the content director, I reviewed every single CFP for both of these things. I mean, you couldn't get a better person than someone like me, who's the queer person who won't shut up about sustainability to sort of do this thing.So, you know, I looked after those scorings for the CFPs in support of the CFP chairs. And now, as I'm working with those individual speakers on their content and making sure that diversity is included in the content. It's not just the diversity of the speaker, for example it's, who were the other people whose voice you're raising? What other people if you worked on this? Are there anyone that you've mentored, like, you know, actually, you know, let's have this as a wider conversation?Corey: Thank God. I thought you were about to say diversity of thought, and I was about to reach through the screen to strangle you.Aerin: [laugh]. No, no. I mean, we're doing really well, so of the announced speakers online, we are 40% non-male and about 18% non-white, which to be honest, for a fair sheer conference, when we didn't really do that much to specifically call this out, but I would probably raise this to Amanda Brock, who is the CEO of OpenUK, you know, she has built a community in the UK and around the world over the last few years which has been putting women forward and building these links. And that's why we've had such a great response for our first-year conferences, the work she's put in. It's hard.Like, this isn't easy. You know, we've had to do a lot of work to make sure that it is representative, at least better than other conferences, at least. So, I'm really excited. And like, there's so much, like, open-source is probably going to be the thing that saves the world. If we're going to end up looking at two different futures with monopolies and closed systems and all the money going towards cloud providers versus a fair and equitable society, open-source is the thing that's going to get us closer to that. So yeah, this conference will be a great event.Corey: Is it all in person? Is it being live-streamed as well? What is the deal here?Aerin: So, in person, we have loads of different things going on, but what will be streamed online if you sign up for virtual ticket is five different tracks. So, our platform engineering track, our security track, government law and policy, open data, and open hardware. And of course, the keynote and plenaries. But one of the things I'm also really proud about this conference is that we're really focusing on the developer experience, like, you know, what is your experience at the conference? So, we also have an unconference, we have a sub-conference run by Sustain OSS focused on workshops related to climate change and sustainability.We have loads of developer experience halls in the event itself. And throughout the day, over the two days, we have two one-hour blocks with no speaking content at all so that we can really make sure that people have that hardware track and are out there meeting each other and having a good time. And obviously, of course, like any good conference, the all-hands party on the first night. So, it really is a conference that's doing things differently from diversity to sustainability to that experience. So, it's awesome.Corey: One of the challenges that I've seen historically around things aiming at the idea of open conferences—and when we talk open-source, et cetera, et cetera—open' seems like it is a direction parallel to, we haven't any money, where it's, “Yes, we're a free software foundation,” and it turns out conferences themselves are not free. And you wind up with a whole bunch of folks showing up to it who are, in many cases, around the fringes of things. There are individual hobbyists who are very passionate about a thing but do not have the position in the corporate world. I'm looking through the lengthy list of speakers you have here and that is very much not this. These are serious people at serious companies. Not that there are not folks who are individual practitioners and passionate advocates and hobbyists than the rest. This is, by virtually any way you look at it, a remarkably diverse conference.Aerin: Mmm. You know, you are right about, like, that problem in open-source. It's like, you know, we look at open and whether we want to do open and we just go, “Well, it won't make me any money. I can't do that. I don't have the time. I need to bring in some money.”And one of the really unique things, again, about this conference is—I have not even mentioned it yet—we have an entrepreneurship room. So, we have 20 tables filled with entrepreneurs and CEOs and founders of open-source companies throughout the two days where you can book in time to sit at that table and have conversations with them. Ask them the questions that you want to ask about, whether it's something that you want to work on, or a company you want to found, and you'll be able to get that time. I had a very similar experience in some ways. It was re:Invent.I was a peer talk expert and you know, I had 15 or so conversations with some really interesting people just because they were able put that time in and they were able to find me on the website. So, that's something we are replicating to get those 20 also entrepreneurs and co-founders out to everyone else. They want to be able to help you and support you.Corey: That is an excellent segue if I do say so myself. Let's talk about re:Invent. It's the one time of the year you and I get to spend time in the same room. One thing that I got wrong is that I overbooked myself as I often do, and I didn't have time to do anything on their peer talk expert program, which is, you more or less a way that any rando can book time to sit down and chat with you. Now, in my case, I have assassination concerns because it turns out Amazon employees can read that thing too and some of them might work on billing. One wonders.So yeah, I have to be a little careful for personal reasons but for most people, it's a non-issue. I didn't get as much time as I wanted to talk to folks in the community. That is not going to repeat itself at the end of this year. But what was your take on re:Invent, because I was in meetings for most of them?Aerin: So, comparing this re:Invent to the re:Invent I went to, my first re:Invent when we met in 2021, you know, that was the re:Invent that inspired me to get into sustainability. They'd announced stuff to do with the shared responsibility model. A few months later, they released their carbon calculator, and I was like, “Yeah, this is the problem. This is the thing I want to work on and it will make me happy.” And a lot of that goes into, you know, finding a passion that keeps me motivated when things aren't that great.When maybe not a lot of money is coming in, at least I know, I'm doing everything I can to help save the world. So, re:Invent 2021 really inspired me to get involved with sustainability. When I look at re:Invent 2022, you might have Adam Selipsky on the main stage saying that sustainability is the problem of our generation, but that is just talk and bluster compared to what they were putting out in terms of content and their experience of, like, let's say the sustainability—I don't know what to call it—tiny little square in the back of the MGM Grand compared to the paid hall in the expo. Like, you know, that's the sort of thing where you can already see the prioritization of money. Let's put the biggest sponsors and all the money that we can bring it in the big hall where everyone is, and then put the thing we care about the most, apparently—sustainability—in the back of the MGM.And that in itself was annoying, but then you get there in the content, and it was like a massive Rivian van, like, an advert for, “Oh, Amazon has done all this to electrify Rivian and deliver you Prime.” But where was the people working on sustainability in the cloud? You know, we had a couple of teams who were talking about the customer carbon footprint tool, but there was just not much. And I spoke to a lot of people and they were saying similar things, like, “Where are the announcements? Where are the actual interesting things?” Rather than just—which is kind of what I'm starting to realize is that a lot of the conversations about sustainability is about selling yourself as sustainable.Use me rather than my competitors because we're 88% more, kind of, carbon neutral when it comes to traditional data centers, not because we are really going to solve these problems. And not to say that Amazon isn't doing innovative, amazing things that no one else can't do, because that is true, and cloud as part of the solution, but you know, sustainability shouldn't be about making more sales and growing your business, it should be about making the world a better place, not just in terms of carbon emissions, but you know, our life, the tech that we can access. Three billion people on this planet have never accessed the internet. And as we continue to grow all of our services like AI and machine learning and new Web3, bloody managed services come online, that's going to be more carbon, more compute power going towards the already rich and the already westernized people, rather than solving the problems we need to solve in the face of climate change.So, I was a little bit disappointed. And I did put a tweet thread out about it afterwards. And I just hope it can be different next year and I hope more people will start to ask for this. And that also what I'm starting to realize is that until more Amazon customers put this as their number one priority and say, “I'm not going to do business with you because of this issue,” or, you know, “This is what we really care about,” they're not going to make a change. Unless it starts to impact their bottom lines and people start to choose other cloud providers, they're not going to prioritize it.And I think up until this point, we're not seeing that from customers. We're kind of getting some people like me shouting about it, but across the board, sustainability isn't the number one priority right now. It's, like what Amazon says, security or resiliency or something else.Corey: And I think that, at least from where I set, the challenge is that if you asked me what I got out of re:Invent, and what the conversations I had—going into it, what are my expectations, and what do I hope to get and how's it going to end up, and then you ask you that same question—though maybe you are a poor example of this—and then you ask someone who works out as an engineer at a company that uses AWS and their two or three years into their career, why don't you talk to a manager or director or someone else? And the problem is if you start polling the entire audience, you'll find that this becomes—you're going to wind up with 20 different answers, at least. The conference doesn't seem like it has any idea of what it wants to be and to whom and in that vacuum, it tries to be all things to all people. And surprise, just like the shooting multifunction printer some of us have in our homes, it doesn't do well with any of those things because it's trying to stand in too many worlds at the same time.Aerin: You know, let's not, like, look at this from a way that you know, re:Invent is crap and, like, do all the work that everyone puts it is wasted because it is a really great event for a lot of different things for a lot of different people. And to be honest, the work that the Amazon staff put into it is pretty out of this world. I feel sorry though because you know, the rush for AWS sell more and do this massive event, they put people through the grinder. And I feel like, I don't know, we could see the cracks in some of that, the way that works. But, you know, there's so many people that I speak to who were like, “Yeah, I'm definitely not going again. I'm not even going to go anywhere near submitting a talk.”And, sort of, the thing is, like, I can imagine if the conference was something different; it was focused at sustainability at number one, it was about making the world a better place from everything that they do, it was about bringing diverse communities together. Like, you know, bringing these things up the list would make the whole thing a lot better. And to be honest, it would probably make it a lot more enjoyable [laugh] for the Amazon staff who end up talking at it. Because, you know, I guess it can feel a bit soulless over time is all you're doing is making money for someone else and selling more things. And, yeah, I think there's a lot more… different things we can do and a lot more things we can talk about if people just start to talk about, like you know, if you care about this as well and you work at Amazon, then start saying that as well.It'll really make a difference if you say we want re:Invent to look different. I mean, even Amazon staff, [laugh] and we've not even mentioned this one because I got Covid straight after re:Invent, nine days and staring at a wall in hotel room in Vegas was not my idea of a good time post-conference. So, that was a horrible, horrible experience. But, you know, I've had people call it re:Infect. Like, where are the Covid support?Like, there was hardly any conversation about that. It was sort of like, “Don't mention it because oh, s”—whatever else. But imagine if you just did something a little bit differently to look like you care about your customers. Just say, “We recommend people mask or take a test,” or even provide tests and masks. Like, even if it's not mandatory, they could have done a lot more to make it safer for everyone. Because, yeah, imagine having the reputation of re:Infect rather than re:Invent?Corey: I can only imagine how that would play out.Aerin: Only imagine.Corey: Yeah, it's it feels like we're all collectively decided to pretend that the pandemic is over. Because yeah, that's a bummer. I don't want to think about it. You know, kind of like we approach climate change.Aerin: Yeah. At the end of the day, like, and I keep coming across this more and more, you know, my thinking has changed over the last year because, like, you know, initially it was like a hyperactive puppy. Why are we caring about this? Like, yeah, if I say it, people will come, but the reality is, we have to blinker ourselves in order to deal with a lot of this stuff. We can't always worry about all of this stuff all of the time. And that's fine. That's acceptable. We do that in so many different parts of our life.But there comes to a point when you kind of think, “How much do I care about this?” And for a lot of people, it's because they have kids. Like, anyone who has kids right now must have to think, “Wow, what's the future going to look like?” And if you worry about what the future is going to look like, make sure you're taking steps to make the world a better place and make it the future you want it to look like. You know, I made the decision a long time ago not to have kids because I don't think I'd want to bring anyone into the world on what it might actually end up being, but you know, when I speak to people who are older in the 60s and they're like, “Oh, you've got 100 years. You don't need to worry about it.” Like, “Maybe you can say that because you're closer to dying than I am.” But yeah, I have to worry about this now because I'll still be eighty when all this shit is kicking off [laugh].Corey: This episode is sponsored in part by our friends at Strata. Are you struggling to keep up with the demands of managing and securing identity in your distributed enterprise IT environment? You're not alone, but you shouldn't let that hold you back. With Strata's Identity Orchestration Platform, you can secure all your apps on any cloud with any IDP, so your IT teams will never have to refactor for identity again. Imagine modernizing app identity in minutes instead of months, deploying passwordless on any tricky old app, and achieving business resilience with always-on identity, all from one lightweight and flexible platform.Want to see it in action? Share your identity challenge with them on a discovery call and they'll hook you up with a complimentary pair of AirPods Pro. Don't miss out, visit Strata.io/ScreamingCloud. That's Strata dot io slash ScreamingCloud.Corey: That I guess is one of the big fears I have—and I think it's somewhat unfounded—is that every year starts to look too much like the year before it. Because it's one of those ideas where we start to see the pace of innovation is slowing at AWS—and I'm not saying that to piss people at Amazon off and have them come after me with pitchforks and torches again—but they're not launching new services at the rate they once did, which is good for customers, but it starts to feel like oh, have we hit peak cloud this is what it's going to look like? Absolutely not. I don't get the sense that the world is like, “Well, everything's been invented. Time to shut down the patent office,” anytime soon.And in the short term, it feels like oh, there's not a lot exciting going on, but you look back the last five years even and look at how far we've come even in that period of time and—what is it? “The days are long, but the years are short.” It becomes a very macro thing of as things ebb and flow, you start to see the differences but the micro basis on a year-to-year perspective, it seems harder to detect. So longer term, I think we're going to see what the story looks like. And it's going to be satisfying one. Just right now, it's like, well, this wasn't as entertaining as I would have hoped, so I'm annoyed. Which I am because it wasn't, but that's not the biggest problem in the world.Aerin: It's not. And, you know, you look at okay, cool, there wasn't all these new flashy services. There was a few things are announced, I mean, hopefully that are going to contribute towards climate change. One of them is called AWS Supply Chain. And the irony of seeing sort of like AWS Supply Chain where a company that already has issues with data and conversations around competition, saying to everyone, “Hey, trust us and give all of your supply chain information and put it into one of our AWS products,” while at the same time their customer carbon footprint tool won't even show the full scope for their emissions of their own supply chain is not lost on me.And you do say, “Maybe we should start seeing things at a macro level,” but unless Amazon and other cloud hyperscalers start pulling the finger out and showing us how they have got a vision between now and 2040, and now in 2050, of how they're going to get there, it kind of just feels like they're saying, “It'll all be fine as long as we continue to grow, as long as we keep sucking up the market.” And, you know, an interesting thing that just kicked off in the UK back in November was the Competition and Markets Authority have started an investigation into the cloud providers on how they are basically sucking up all these markets, and how the growth of things that are not hyperscale is going. So, in the UK, the percentage of cloud has obviously gone up—more and more cloud spending has gone up—but kind of usage across non-hyperscalers has gone down over that same period. And they really are at risk of sucking up the world. Like, I have got involved in a lot of different things.I'm an AWS community builder; like, I do promote AWS. And, you know, the reason why I promote cloud, for example is serverless. We need serverless as the way we run our IT because that's the only way we'll do things like time shifting or demand shifting. So, when we look at renewable energy on the grid if that really high, the wind is blowing and the sun is shining, we want more workloads to be running then and when they're tiny, and they're [unintelligible 00:21:03], and what's the call it serverless generally, uh—Corey: Hype?Aerin: Function as a Code?Corey: Function—yeah, Function as a Service and all kinds of other nonsense. But I have to ask, when you're talking about serverless, in this context, is a necessary prerequisite of serverless that scale to zero when it's [unintelligible 00:21:19].Aerin: [laugh]. I kind of go back to marketing. What Amazon releasing these days when it relates to serverless that isn't just marketing and saying, “Oh, it's serverless.” Because yeah, there was a few products this year that is not scaled to zero is it? It's a 100-pound minimum. And when you're looking at number of accounts that you have, that can add up really quickly and it excludes people from using it.Corey: It's worse than that because it's not number of accounts. I consider DynamoDB to be serverless, by any definition of the term. Because it is. And what I like about it is I can have a separate table for every developer, for every service or microservice or project that they have, and in fact, each branch can have its own stuff like that. I look at some of the stuff that I build with multi-branch testing and whatnot, and, “Oh, wow. That would cost more than the engineer if they were to do that with some of the serverless offerings that AWS has put out.”Which makes that entire philosophy a complete non-starter, which means that invariably as soon as you start developing down that path, you are making significant trade-offs. That's just from a economics slash developer ergonomics slash best practices point of view. But there's a sustainability story to it as well.Aerin: Yeah. I mean, this sustainability thing is like, if you're not going to encourage this new way of working, like, if you're not going to move everyone to this point of view and this is how we need to do things, then you kind of just propagating the old world, putting it into your data center. For every managed service that VMware migrated piece of crap, just that land in the cloud, it's not making a real difference in the world because that's still going to exist. And we mentioned this just before the podcast and, you know, a lot of focus these days and for a lot of people is, “Okay, green energy is the problem. We need to solve green energy.”And Amazon is the biggest purchaser of power purchase agreements in renewable energy around the world, more than most governments. Or I think that the biggest corporate purchaser of it anyway. And that all might sound great, like, “Oh, the cloud is going to solve this problem for me and Amazon is going to solve it for me even better because they're bigger.” But at the end of the day, when we think about a data center, it exists in the real world.It's made of concrete. You know, when you pour concrete and when you make concrete, it releases CO2. It's got racks of servers that all are running. So, those individual servers had to be made by whoever it is in Asia or mined from rare earth metals and end up in the supply chain and then transported into the data centers in us-east-1. And then things go wrong. You have to repair you have to replace and you have to maintain them.Unless we get these circular economies going in a closed system, we can't just continue to grow like this. Because carbon emissions related to Scope 3, all those things I've just been talking about, basically anything that isn't the energy, is about 80 to 90% of all the carbon emissions. So, when Amazon says, “Oh, we're going to go green and get energy done by 2030”—which is seven years away—they've then got ten years to solve 90% of the problem. And we cannot all just continue to grow and think of tech as neutral and better for the world if we still got that 90% problem, which we do right now. And it really frustrates me when you look at the world and the way we've jumped on technology just go on, “Oh, it must be good.”Like Bitcoin, for example. Bitcoin has released 200 million metric tons of CO2 since its inception. And for something that is basically a glorified Ponzi scheme, I can't see how that is making the world a better place. So, when cloud providers are making managed services for Web3 and for blockchain, and they're selling more and more AI and machine learning, basically so they can keep on selling GPU access, I do worry about whether our path to infinite growth with all of these hyperscalers is probably the wrong way of looking at things. So, linking back to, you know, the conference, open-source and, you know, thinking about things differently is really important in tech right now.And not just for your own well-being and being able to sleep at night, but this is how we're going to solve our problems. When all companies on the planet want people to be sustainable and we have to start tackling this because there's a financial cost related to it, then you're going to be in the vogue. If you're really good developer, thinking about things differently can be efficient, then yeah, you're the developer that's going to win in the future. You might be assisted by ChatGPT three or whatever else, but yeah, sustainability and efficiency can really be the number one priority because it's a win, win, win. We save the world, we make ourselves better, we sleep better at night, and you just become a better developer.I keep monologuing at this point, but you know, when it comes to stuff like games design, we look at things like Quake and Pokemon and all these things when there's like, “How did they get these amazing games and these amazing experiences in such small sizes,” they had boundaries. They had boundaries to innovate within because they had to. They couldn't release the game if they couldn't fit into the cartridge, therefore, they made it work. When the cloud is sold as infinitely scalable and horizontally scalable and no one needs to worry about this stuff because you can get your credit card out, people stop caring about being innovative and being more efficient. So yeah, let's get some more boundaries in the cloud.Corey: What I find that is super helpful, has been, like, if I can, like, descri—like, Instagram is down. Describe your lunch to me style meme description, like, the epic handshake where you have two people clasping hands, and one side is labeled in this case, ‘sustainability advocates,' and the other side should be labeled ‘cloud economists,' and in the middle, it's, “Turn that shit off.” Because it's not burning carbon if it's not running, and it's not costing you anything—ideally—if it's not running, so it's one of those ideas where we meet in the middle. And that's important, not just because it makes both of us independently happy because it's both good for the world and you'll get companies on board with this because, “Wait. We can do this thing and it saves us money?” Suddenly, you're getting them aligned because that is their religion.If companies could be said to have a religion, it is money. That's the way it works. So, you have to make it worth money for them to do the right thing or you're always going to be swimming upstream like a depressed salmon.Aerin: I mean, look at why [unintelligible 00:27:11] security is near the top: because there's so many big fines related to security breaches. It will cost them money not to be secure. Right now, it doesn't cost companies money to be inefficient or to release all this carbon, so they get away with it or they choose to do it. And I think that's going to change. We see in regulations across you're coming out.So, you know, if you work for a big multinational that operates in Europe, by next year, you'll have to report on all of your Scope 3 carbon emissions. If you're a customer of AWS right now, you have no ability to do that. So, you know, this is going to be crunch time over the next 18 months to two years for a lot of big businesses, for Amazon and the other hyperscalers, to really start demonstrating that they can do this. And I guess that's my big push. And, you know, I want to work with anyone, and it's funny because I have been running this business for about, you know, a couple of years now, it's been going really well, I did my podcast, I'm on this path.But I did, last year, take some time, and I applied into AWS. And you know, I was like, “Okay, maybe I'll apply for this big tech company and help Amazon out.” And because I'll take that salary and I'll do something really good with it afterwards, I'll do my time for three years and attend re:Invent and deliver 12 talks and never sleep, but you know, at the end of it, I'll say, “Okay, I've done that and now I can do something really good.” Unfortunately, I didn't get the role—or fortunately—but you know, when I applied for that role, what I said to them is, “I really care about sustainability. I want to make the world a better place. I want to help your customers be more sustainable.”And they didn't want me to join. So, I'm just going to continue doing that but from the outside. And whether that means working with politicians or developers or anyone else to try and make the world better and to kind of help fight against climate change, then, yeah, that's definitely what I'm doing.Corey: So, one last question before we wind up calling it an episode. How do we get there? What is the best next step that folks can take? Because it's easy to look at this as a grand problem and realize it's too big to solve. Well, great. You don't need to solve the entire problem. You need take the first step. What is that first step?Aerin: Individuals, I would say it's just realizing that you do care about it and you want to take action. And you're going to say to yourself, “Even if I do little things, I'm going to move forward towards that point.” So, if that is being a more sustainable engineer or getting more conversations about climate change or even just doing other things in your community to make the world a better place than it is, taking that action. But one thing that I can definitely help about and talk a bit more of is that at the conference itself, I'll be running a panel with some great experts called the, “Next Generation of Cloud Education.” So, I really think we need to—like I said earlier in the podcast—to think differently about the cloud and IT.So, I am doing this panel and I'm bringing together someone like Simon Wardley to help people do Wardley Mapping. Like, that is a tool that allows you to see the landscape that you're operating in. You know, if you use that sort of tool to understand the real-world impact of what you're doing, then you can start caring about it a bit more. I'm bringing in somebody called Anne Currie, who is a tech ethicist and speaker and lecturer, and she's actually written some [laugh] really great nonfiction books, which I'd recommend everyone reads. It starts with Utopia Five.And that's about asking, “Well, is this ethical? Can we continue to do these things?” Can't—talks about things about sustainability. If it's not sustainable for everyone, it's not ethical. So, when I mentioned 3 billion people currently don't use the internet, it's like, can we continue to just keep on doing things the same way?And then John Booth, who is a data center expert, to help us really understand what the reality is on the ground. What are these data centers really look like? And then Amanda Brock, from OpenUK in the conference will joining as well to talk about, kind of, open-source and how we can make the world kind of a better place by getting involved in these communities. So, that'll be a really great panel.But what I'm also doing is releasing this as an online course. So, for people who want to get involved, it will be very intimate, about 15 seats on each core, so three weeks for you to actually work and talk directly with some of these experts and me to figure out what you want to do in the world of climate change and how you can take those first steps. So, it'll be a journey that even starts with an ecotherapist to help us deal with climate grief and wonder about the things we can do as individuals to feel better ourselves and be happier. So, I think that'd be a really great thing for a lot of people. And, yeah, not only that, but… it'll be great for you, but it also goes towards making the world a better place.So, 50% of the course fees will be donated, 25%, to charity, and 25% supporting open-source projects. So, I think it kind of just win, win, win. And that's the story of sustainability in general. It's a win, win, win for everyone. If you start seeing the world through a lens of sustainability, you'll save money, you'll sleep better at night, you'll get involved with some really great communities, and meet some really great people who care about this as well. And yeah, it'll be a brighter future.Corey: If people want to learn more, where can they find you?Aerin: So, if you want to learn more about what I'm up to, I'm on Twitter under @aerincloud, that A-E-R-I-N cloud. And then you can also find me on LinkedIn. But I also run my own podcast that was inspired by Corey, called Public Cloud for Public Good talking about cloud sustainability and how to make the world a better place for the use of public cloud services.Corey: And we will, of course, put a link to that in the [show notes 00:32:32]. Thank you so much for your time. I appreciate it, as always.Aerin: Thank you.Corey: Aerin Booth, the Ted Lasso of cloud. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this episode, please leave a five-star review on your podcast platform of choice, along with an angry and insulting comment that I will immediately scale to zero in true serverless fashion.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About Chris Chris Farris has been in the IT field since 1994 primarily focused on Linux, networking, and security. For the last 8 years, he has focused on public-cloud and public-cloud security. He has built and evolved multiple cloud security programs for major media companies, focusing on enabling the broader security team's objectives of secure design, incident response and vulnerability management. He has developed cloud security standards and baselines to provide risk-based guidance to development and operations teams. As a practitioner, he's architected and implemented multiple serverless and traditional cloud applications focused on deployment, security, operations, and financial modeling.Chris now does cloud security research for Turbot and evangelizes for the open source tool Steampipe. He is one if the organizers of the fwd:cloudsec conference (https://fwdcloudsec.org) and has given multiple presentations at AWS conferences and BSides events.When not building things with AWS's building blocks, he enjoys building Legos with his kid and figuring out what interesting part of the globe to travel to next. He opines on security and technology on Twitter and his website https://www.chrisfarris.comLinks Referenced: Turbot: https://turbot.com/ fwd:cloudsec: https://fwdcloudsec.org/ Steampipe: https://steampipe.io/ Steampipe block: https://steampipe.io/blog TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Tailscale SSH is a new, and arguably better way to SSH. Once you've enabled Tailscale SSH on your server and user devices, Tailscale takes care of the rest. So you don't need to manage, rotate, or distribute new SSH keys every time someone on your team leaves. Pretty cool, right? Tailscale gives each device in your network a node key to connect to your VPN, and uses that same key for SSH authorization and encryption. So basically you're SSHing the same way that you're already managing your network.So what's the benefit? Well, built-in key rotation, the ability to manage permissions as code, connectivity between any two devices, and reduced latency. You can even ask users to re-authenticate SSH connections for that extra bit of security to keep the compliance folks happy. Try Tailscale now - it's free forever for personal use.Corey: This episode is sponsored by our friends at Logicworks. Getting to the cloud is challenging enough for many places, especially maintaining security, resiliency, cost control, agility, etc, etc, etc. Things break, configurations drift, technology advances, and organizations, frankly, need to evolve. How can you get to the cloud faster and ensure you have the right team in place to maintain success over time? Day 2 matters. Work with a partner who gets it - Logicworks combines the cloud expertise and platform automation to customize solutions to meet your unique requirements. Get started by chatting with a cloud specialist today at snark.cloud/logicworks. That's snark.cloud/logicworksCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is someone that I have been meaning to invite slash drag onto this show for a number of years. We first met at re:Inforce the first year that they had such a thing, Amazon's security conference for cloud, as is Amazon's tradition, named after an email subject line. Chris Farris is a cloud security nerd at Turbot. He's also one of the organizers for fwd:cloudsec, another security conference named after an email subject line with a lot more self-awareness than any of Amazon's stuff. Chris, thank you for joining me.Chris: Oh, thank you for dragging me on. You can let go of my hair now.Corey: Wonderful, wonderful. That's why we're all having the thinning hair going on. People just use it to drag us to and fro, it seems. So, you've been doing something that I'm only going to describe as weird lately because your background—not that dissimilar from mine—is as a practitioner. You've been heavily involved in the security space for a while and lately, I keep seeing an awful lot of things with your name on them getting sucked up by the giant app surveillance apparatus deployed to the internet, looking for basically any mention of AWS that I wind up using to write my newsletter and feed the content grist mill every year. What are you doing and how'd you get there?Chris: So, what am I doing right now is, I'm in marketing. It's kind of a, you know, “Oops, I'm sorry I did that.”Corey: Oh, the running gag is, you work in DevRel; that means, “Oh, you're in marketing, but they're scared to tell you that.” You're self-aware.Chris: Yeah.Corey: Good for you.Chris: I'm willing to address that I'm in marketing now. And I've been a cloud practitioner since probably 2014, cloud security since about 2017. And then just decided, the problem that we have in the cloud security community is a lot of us are just kind of sitting in a corner in our companies and solving problems for our companies, but we're not solving the problems at scale. So, I wanted a job that would allow me to reach a broader audience and help a broader audience. Where I see cloud security having—you know, or cloud in general falling down is Amazon makes it really hard for you to do your side of shared responsibility, and so we need to be out there helping customers understand what they need to be doing. So, I am now at a company called Turbot and we're really trying to promote cloud security.Corey: One of the first promoted guest episodes of this show was David Boeke, your CTO, and one of the things that I regret is that I've sort of lost track of Turbot over the past few years because, yeah, one or two things might have been going on during that timeline as I look back at having kids in the middle of a pandemic and the deadly plague o'er land. And suddenly, every conversation takes place over Zoom, which is like, “Oh, good, it's like a happy hour only instead, now it's just like a conference call for work.” It's like, ‘Conference Calls: The Drinking Game' is never the great direction to go in. But it seems the world is recovering. We're going to be able to spend some time together at re:Invent by all accounts that I'm actively looking forward to.As of this recording, you're relatively new to Turbot, and I figured out that you were going there because, once again, content hits my filters. You wrote a fascinating blog post that hits on an interest of mine that I don't usually talk about much because it's off-putting to some folk, and these days, I don't want to get yelled at and more than I have to about the experience of traveling, I believe it was to an all-hands on the other side of the world.Chris: Yep. So, my first day on the job at Turbot, I was landing in Kuala Lumpur, Malaysia, having left the United States 24 hours—or was it 48? It's hard to tell when you go to the other side of the planet and the time zones have also shifted—and then having left my prior company day before that. But yeah, so Turbot about traditionally has an annual event where we all get together in person. We're a completely remote company, but once a year, we all get together in person in our integrate event.And so, that was my first day on the job. And then you know, it was basically two weeks of reasonably intense hackathons, building out a lot of stuff that hopefully will show up open-source shortly. And then yeah, meeting all of my coworkers. And that was nice.Corey: You've always had a focus through all the time that I've known you and all the public content that you've put out there that has come across my desk that seems to center around security. It's sort of an area that I give a nod to more often than I would like, on some level, but that tends to be your bread and butter. Your focus seems to be almost overwhelmingly on I would call it AWS security. Is that fair to say or is that a mischaracterization of how you view it slash what you actually do? Because, again, we have these parasocial relationships with voices on the internet. And it's like, “Oh, yeah, I know all about that person.” Yeah, you've met them once and all you know other than that is what they put on Twitter.Chris: You follow me on Twitter. Yeah, I would argue that yes, a lot of what I do is AWS-related security because in the past, a lot of what I've been responsible for is cloud security in AWS. But I've always worked for companies that were multi-cloud; it's just that 90% of everything was Amazon and so therefore 90% of my time, 90% of my problems, 90% of my risk was all in AWS. I've been trying to break out of that. I've been trying to understand the other clouds.One of the nice aspects of this role and working on Steampipe is I am now experimenting with other clouds. The whole goal here is to be able to scale our ability as an industry and as security practitioners to support multiple clouds. Because whether we want to or not, we've got it. And so, even though 90% of my spend, 90% of my resources, 90% of my applications may be in AWS, that 10% that I'm ignoring is probably more than 10% of my risk, and we really do need to understand and support major clouds equally.Corey: One post you had recently that I find myself in wholehearted agreement with is on the adoption of Tailscale in the enterprise. I use it for all of my personal nonsense and it is transformative. I like the idea of what that portends for a multi-cloud, or poly-cloud, or whatever the hell we're calling it this week, sort of architectures were historically one of the biggest problems in getting to clouds two speak to one another and manage them in an intelligent way is the security models are different, the user identity stuff is different as well, and the network stuff has always been nightmarish. Well, with Tailscale, you don't have to worry about that in the same way at all. You can, more or less, ignore it, turn on host-based firewalls for everything and just allow Tailscale. And suddenly, okay, I don't really have to think about this in the same way.Chris: Yeah. And you get the micro-segmentation out of it, too, which is really nice. I will agree that I had not looked at Tailscale until I was asked to look at Tailscale, and then it was just like, “Oh, I am completely redoing my home network on that.” But looking at it, it's going to scare some old-school network engineers, it's going to impact their livelihoods and that is going to make them very defensive. And so, what I wanted to do in that post was kind of address, as a practitioner, if I was looking at this with an enterprise lens, what are the concerns you would have on deploying Tailscale in your environment?A lot of those were, you know, around user management. I think the big one that is—it's a new thing in enterprise security, but kind of this host profiling, which is hey, before I let your laptop on the network, I'm going to go make sure that you have antivirus and some kind of EDR, XDR, blah-DR agents so that you know we have a reasonable thing that you're not going to just go and drop [unintelligible 00:09:01] on the network and next thing you know, we're Maersk. Tailscale, that's going to be their biggest thing that they are going to have to figure out is how do they work with some of these enterprise concerns and things along those lines. But I think it's an excellent technology, it was super easy to set up. And the ability to fine-tune and microsegment is great.Corey: Wildly so. They occasionally sponsor my nonsense. I have no earthly idea whether this episode is one of them because we have an editorial firewall—they're not paying me to set any of this stuff, like, “And this is brought to you by whatever.” Yeah, that's the sponsored ad part. This is just, I'm in love with the product.One of the most annoying things about it to me is that I haven't found a reason to give them money yet because the free tier for my personal stuff is very comfortably sized and I don't have a traditional enterprise network or anything like that people would benefit from over here. For one area in cloud security that I think I have potentially been misunderstood around, so I want to take at least this opportunity to clear the air on it a little bit has been that, by all accounts, I've spent the last, mmm, few months or so just absolutely beating the crap out of Azure. Before I wind up adding a little nuance and context to that, I'd love to get your take on what, by all accounts, has been a pretty disastrous year-and-a-half for Azure security.Chris: I think it's been a disastrous year-and-a-half for Azure security. Um—[laugh].Corey: [laugh]. That was something of a leading question, wasn't it?Chris: Yeah, no, I mean, it is. And if you think, though, back, Microsoft's repeatedly had these the ebb and flow of security disasters. You know, Code Red back in whatever the 2000s, NT 4.0 patching back in the '90s. So, I think we're just hitting one of those peaks again, or hopefully, we're hitting the peak and not [laugh] just starting the uptick. A lot of what Azure has built is stuff that they already had, commercial off-the-shelf software, they wrapped multi-tenancy around it, gave it a new SKU under the Azure name, and called is cloud. So, am I super-surprised that somebody figured out how to leverage a Jupyter notebook to find the back-end credentials to drop the firewall tables to go find the next guy over's Cosmos DB? No, I'm not.Corey: I find their failures to be less egregious on a technical basis because let's face it, let's be very clear here, this stuff is hard. I am not pretending for even a slight second that I'm a better security engineer than the very capable, very competent people who work there. This stuff is incredibly hard. And I'm not—Chris: And very well-funded people.Corey: Oh, absolutely, yeah. They make more than I do, presumably. But it's one of those areas where I'm not sitting here trying to dunk on them, their work, their efforts, et cetera, and I don't do a good enough job of clarifying that. My problem is the complete radio silence coming out of Microsoft on this. If AWS had a series of issues like this, I'm hard-pressed to imagine a scenario where they would not have much more transparent communications, they might very well trot out a number of their execs to go on a tour to wind up talking about these things and what they're doing systemically to change it.Because six of these in, it's like, okay, this is now a cultural problem. It's not one rando engineer wandering around the company screwing things up on a rotational basis. It's, what are you going to do? It's unlikely that firing Steven is going to be your fix for these things. So, that is part of it.And then most recently, they wound up having a blog post on the MSRC, the Microsoft Security Resource Center is I believe that acronym? The [mrsth], whatever; and it sounds like a virus you pick up in a hospital—but the problem that I have with it is that they spent most of that being overly defensive and dunking on SOCRadar, the vulnerability researcher who found this and reported it to them. And they had all kinds of quibbles with how it was done, what they did with it, et cetera, et cetera. It's, “Excuse me, you're the ones that left customer data sitting out there in the Azure equivalent of an S3 bucket and you're calling other people out for basically doing your job for you? Excuse me?”Chris: But it wasn't sensitive customer data. It was only the contract information, so therefore it was okay.Corey: Yeah, if I put my contract information out there and try and claim it's not sensitive information, my clients will laugh and laugh as they sue me into the Stone Age.Chris: Yeah well, clearly, you don't have the same level of clickthrough terms that Microsoft is able to negotiate because, you know, [laugh].Corey: It's awful as well, it doesn't even work because, “Oh, it's okay, I lost some of your data, but that's okay because it wasn't particularly sensitive.” Isn't that kind of up to you?Chris: Yes. And if A, I'm actually, you know, a big AWS shop and then I'm looking at Azure and I've got my negotiations in there and Amazon gets wind that I'm negotiating with Azure, that's not going to do well for me and my business. So no, this kind of material is incredibly sensitive. And that was an incredibly tone-deaf response on their part. But you know, to some extent, it was more of a response than we've seen from some of the other Azure multi-tenancy breakdowns.Corey: Yeah, at least they actually said something. I mean, there is that. It's just—it's wild to me. And again, I say this as an Azure customer myself. Their computer vision API is basically just this side of magic, as best I can tell, and none of the other providers have anything like it.That's what I want. But, you know, it almost feels like that service is under NDA because no one talks about it when they're using this service. I did a whole blog post singing its praises and no one from that team reached out to me to say, “Hey, glad you liked it.” Not that they owe me anything, but at the same time it's incredible. Why am I getting shut out? It's like, does this company just have an entire policy of not saying anything ever to anyone at any time? It seems it.Chris: So, a long time ago, I came to this realization that even if you just look at the terminology of the three providers, Amazon has accounts. Why does Amazon have Amazon—or AWS accounts? Because they're a retail company and that's what you signed up with to buy your underwear. Google has projects because they were, I guess, a developer-first thing and that was how they thought about it is, “Oh, you're going to go build something. Here's your project.”What does Microsoft have? Microsoft Azure Subscriptions. Because they are still about the corporate enterprise IT model of it's really about how much we're charging you, not really about what you're getting. So, given that you're not a big enterprise IT customer, you don't—I presume—do lots and lots of golfing at expensive golf resorts, you're probably not fitting their demographic.Corey: You're absolutely not. And that's wild to me. And yet, here we are.Chris: Now, what's scary is they are doing so many interesting things with artificial intelligence… that if… their multi-tenancy boundaries are as bad as we're starting to see, then what else is out there? And more and more, we is carbon-based life forms are relying on Microsoft and other cloud providers to build AI, that's kind of a scary thing. Go watch Satya's keynote at Microsoft Ignite and he's showing you all sorts of ways that AI is going to start replacing the gig economy. You know, it's not just Tesla and self-driving cars at this point. Dali is going to replace the independent graphics designer.They've got things coming out in their office suite that are going to replace the mom-and-pop marketing shops that are generating menus and doing marketing plans for your local restaurants or whatever. There's a whole slew of things where they're really trying to replace people.Corey: That is a wild thing to me. And part of the problem I have in covering AWS is that I have to differentiate in a bunch of different ways between AWS and its Amazon corporate parent. And they have that problem, too, internally. Part of the challenge they have, in many cases, is that perks you give to employees have to scale to one-and-a-half million people, many of them in fulfillment center warehouse things. And that is a different type of problem that a company, like for example, Google, where most of their employees tend to be in office job-style environments.That's a weird thing and I don't know how to even start conceptualizing things operating at that scale. Everything that they do is definitionally a very hard problem when you have to make it scale to that point. What all of the hyperscale cloud providers do is, from where I sit, complete freaking magic. The fact that it works as well as it does is nothing short of a modern-day miracle.Chris: Yeah, and it is more than just throwing hardware at the problem, which was my on-prem solution to most of the things. “Oh, hey. We need higher availability? Okay, we're going to buy two of everything.” We called it the Noah's Ark model, and we have an A side and a B side.And, “Oh, you know what? Just in case we're going to buy some extra capacity and put it in a different city so that, you know, we can just fail from our primary city to our secondary city.” That doesn't work at the cloud provider scale. And really, we haven't seen a major cloud outage—I mean, like, a bad one—in quite a while.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: The outages are always fascinating, just from the way that they are reported in the mainstream media. And again, this is hard, I get it. I am not here to crap on journalists. They, for some ungodly, unknowable reason, have decided not to spend their entire career focusing on the nuances of one very specific, very deep industry. I don't know why.But as [laugh] a result, they wind up getting a lot of their baseline facts wrong about these things. And that's fair. I'm not here to necessarily act as an Amazon spokesperson when these things happen. They have an awful lot of very well-paid people who can do that. But it is interesting just watching the blowback and the reaction of whatever there's an outage, the conversation is never “Does Amazon or Azure or Google suck?” It's, “Does cloud suck as a whole?”That's part of the reason I care so much about Azure getting their act together. If it were just torpedoing Microsoft's reputation, then well, that's sad, but okay. But it extends far beyond that to a point where it's almost where the enterprise groundhog sees the shadow of a data breach and then we get six more years of data center build-outs instead of moving things to a cloud. I spent too many years working in data centers and I have the scars from the cage nuts and crimping patch cables frantically in the middle of the night to prove it. I am thrilled at the fact that I don't believe I will ever again have to frantically drive across town in the middle of the night to replace a hard drive before the rest of the array degrades. Cloud has solved those problems beautifully. I don't want to go back to the Dark Ages.Chris: Yeah, and I think that there's a general potential that we could start seeing this big push towards going back on-prem for effectively sovereign data reasons, whether it's this country has said, “You cannot store your data about our citizens outside of our borders,” and either they're doing that because they do not trust the US Silicon Valley privacy or whatever, or because if it's outside of our borders, then our secret police agents can come knocking on the door at two in the morning to go find out what some dissidents' viewings habits might have been, I see sovereign cloud as this thing that may be a back step from this ubiquitous thing that we have right now in Amazon, Azure, and Google. And so, as we start getting to the point in the history books where we start seeing maps with lots of flags, I think we're going to start seeing a bifurcation of cloud as just a whole thing. We see it already right now. The AWS China partition is not owned by Amazon, it is not run by Amazon, it is not controlled by Amazon. It is controlled by the communist government of China. And nobody is doing business in Russia right now, but if they had not done what they had done earlier this year, we might very well see somebody spinning up a cloud provider that is completely controlled by and in the Russian government.Corey: Well, yes or no, but I want to challenge that assessment for a second because I've had conversations with a number of folks about this where people say, “Okay, great. Like, is the alt-right, for example, going to have better options now that there might be a cloud provider spinning up there?” Or, “Well, okay, what about a new cloud provider to challenge the dominance of the big three?” And there are all these edge cases, either geopolitically or politically based upo—or folks wanting to wind up approaching it from a particular angle, but if we were hired to build out an MVP of a hyperscale cloud provider, like, the budget for that MVP would look like one 100 billion at this point to get started and just get up to a point of critical mass before you could actually see if this thing has legs. And we'd probably burn through almost all of that before doing a single dime in revenue.Chris: Right. And then you're doing that in small markets. Outside of the China partition, these are not massively large markets. I think Oracle is going down an interesting path with its idea of Dedicated Cloud and Oracle Alloy [unintelligible 00:22:52].Corey: I like a lot of what Oracle's doing, and if younger me heard me say that, I don't know how hard I'd hit myself, but here we are. Their free tier for Oracle Cloud is amazing, their data transfer prices are great, and their entire approach of, “We'll build an entire feature complete region in your facility and charge you what, from what I can tell, is a very reasonable amount of money,” works. And it is feature complete, not, “Well, here are the three services that we're going to put in here and everything else is well… it's just sort of a toehold there so you can start migrating it into our big cloud.” No. They're doing it right from that perspective.The biggest problem they've got is the word Oracle at the front end and their, I would say borderline addiction to big-E enterprise markets. I think the future of cloud looks a lot more like cloud-native companies being founded because those big enterprises are starting to describe themselves in similar terminology. And as we've seen in the developer ecosystem, as go startups, so do big companies a few years later. Walk around any big company that's undergoing a digital transformation, you'll see a lot more Macs on desktops, for example. You'll see CI/CD processes in place as opposed to, “Well, oh, you want something new, it's going to be eight weeks to get a server rack downstairs and accounting is going to have 18 pages of forms for you to fill out.” No, it's “click the button,” or—Chris: Don't forget the six months of just getting the financial CapEx approvals.Corey: Exactly.Chris: You have to go through the finance thing before you even get to start talking to techies about when you get your server. I think Oracle is in an interesting place though because it is embracing the fact that it is number four, and so therefore, it's like we are going to work with AWS, we are going to work with Azure, our database can run in AWS or it can run in our cloud, we can interconnect directly, natively, seamlessly with Azure. If I were building a consumer-based thing and I was moving into one of these markets where one of these governments was demanding something like a sovereign cloud, Oracle is a great place to go and throw—okay, all of our front-end consumer whatever is all going to sit in AWS because that's what we do for all other countries. For this one country, we're just going to go and build this thing in Oracle and we're going to leverage Oracle Alloy or whatever, and now suddenly, okay, their data is in their country and it's subject to their laws but I don't have to re-architect to go into one of these, you know, little countries with tin horn dictators.Corey: It's the way to do multi-cloud right, from my perspective. I'll use a component service in a different cloud, I'm under no illusions, though, in doing that I'm increasing my resiliency. I'm not removing single points of failure; I'm adding them. And I make that trade-off on a case-by-case basis, knowingly. But there is a case for some workloads—probably not yours if you're listening to this; assume not, but when you have more context, maybe so—where, okay, we need to be across multiple providers for a variety of strategic or contextual reasons for this workload.That does not mean everything you build needs to be able to do that. It means you're going to make trade-offs for that workload, and understanding the boundaries of where that starts and where that stops is going to be important. That is not the worst idea in the world for a given appropriate workload, that you can optimize stuff into a container and then can run, more or less, anywhere that can take a container. But that is also not the majority of most people's workloads.Chris: Yeah. And I think what that comes back to from the security practitioner standpoint is you have to support not just your primary cloud, your favorite cloud, the one you know, you have to support any cloud. And whether that's, you know, hey, congratulations. Your developers want to use Tailscale because it bypasses a ton of complexity in getting these remote island VPCs from this recent acquisition integrated into your network or because you're going into a new market and you have to support Oracle Cloud in Saudi Arabia, then you as a practitioner have to kind of support any cloud.And so, one of the reasons that I've joined and I'm working on, and so excited about Steampipe is it kind of does give you that. It is a uniform interface to not just AWS, Azure, and Google, but all sorts of clouds, whether it's GitHub or Oracle, or Tailscale. So, that's kind of the message I have for security practitioners at this point is, I tried, I fought, I screamed and yelled and ranted on Twitter, against, you know, doing multi-cloud, but at the end of the day, we were still multi-cloud.Corey: When I see these things evolving, is that, yeah, as a practitioner, we're increasingly having to work across multiple providers, but not to a stupendous depth that's the intimidating thing that scares the hell out of people. I still remember my first time with the AWS console, being so overwhelmed with a number of services, and there were 12. Now, there are hundreds, and I still feel that same sense of being overwhelmed, but I also have the context now to realize that over half of all customer spend globally is on EC2. That's one service. Yes, you need, like, five more to get it to work, but okay.And once you go through learning that to get started, and there's a lot of moving parts around it, like, “Oh, God, I have to do this for every service?” No, take Route 53—my favorite database, but most people use it as a DNS service—you can go start to finish on basically everything that service does that a human being is going to use in less than four hours, and then you're more or less ready to go. Everything is not the hairy beast that is EC2. And most of those services are not for you, whoever you are, whatever you do, most AWS services are not for you. Full stop.Chris: Yes and no. I mean, as a security practitioner, you need to know what your developers are doing, and I've worked in large organizations with lots of things and I would joke that, oh, yeah, I'm sure we're using every service but the IoT, and then I go and I look at our bill, and I was like, “Oh, why are we dropping that much on IoT?” Oh, because they wanted to use the Managed MQTT service.Corey: Ah, I start with the bill because the bill is the source of truth.Chris: Yes, they wanted to use the Managed MQTT service. Okay, great. So, we're now in IoT. But how many of those things have resource policies, how many of those things can be made public, and how many of those things are your CSPM actually checking for and telling you that, hey, a developer has gone out somewhere and made this SageMaker notebook public, or this MQTT topic public. And so, that's where you know, you need to have that level of depth and then you've got to have that level of depth in each cloud. To some extent, if the cloud is just the core basic VMs, object storage, maybe some networking, and a managed relational database, super simple to understand what all you need to do to build a baseline to secure that. As soon as you start adding in on all of the fancy services that AWS has. I re—Corey: Yeah, migrating your Step Functions workflow to other cloud is going to be a living goddamn nightmare. Migrating something that you stuffed into a container and run on EC2 or Fargate is probably going to be a lot simpler. But there are always nuances.Chris: Yep. But the security profile of a Step Function is significantly different. So, you know, there's not much you can do there wrong, yet.Corey: You say that now, but wait for their next security breach, and then we start calling them Stumble Functions instead.Chris: Yeah. I say that. And the next thing, you know, we're going to have something like Lambda [unintelligible 00:30:31] show up and I'm just going to be able to put my Step Function on the internet unauthenticated. Because, you know, that's what Amazon does: they innovate, but they don't necessarily warn security practitioners ahead of their innovation that, hey, you're we're about to release this thing. You might want to prepare for it and adjust your baselines, or talk to your developers, or here's a service control policy that you can drop in place to, you know, like, suppress it for a little bit. No, it's like, “Hey, these things are there,” and by the time you see the tweets or read the documentation, you've got some developer who's put it in production somewhere. And then it becomes a lot more difficult for you as a security practitioner to put the brakes on it.Corey: I really want to thank you for spending so much time talking to me. If people want to learn more and follow your exploits—as they should—where can they find you?Chris: They can find me at steampipe.io/blog. That is where all of my latest rants, raves, research, and how-tos show up.Corey: And we will, of course, put a link to that in the [show notes 00:31:37]. Thank you so much for being so generous with your time. I appreciate it.Chris: Perfect, thank you. You have a good one.Corey: Chris Farris, cloud security nerd at Turbot. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry insulting comment, and be sure to mention exactly which Azure communications team you work on.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About ChrisChris is a mostly-backend mostly-engineer at Remix Labs, working on visual app development. He has been in software startups for ten years, but his first and unrequited love was particle physics.  Before joining Remix Labs, he wrote numerical simulation and analysis tools for the Large Hadron Collider, then co-founded Roobiq, a clean and powerful mobile client for Salesforce back when the official ones were neither.Links Referenced: Remix Labs: https://remixlabs.com/ Twitter: https://twitter.com/chrisvermilion TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Tailscale SSH is a new, and arguably better way to SSH. Once you've enabled Tailscale SSH on your server and user devices, Tailscale takes care of the rest. So you don't need to manage, rotate, or distribute new SSH keys every time someone on your team leaves. Pretty cool, right? Tailscale gives each device in your network a node key to connect to your VPN, and uses that same key for SSH authorization and encryption. So basically you're SSHing the same way that you're already managing your network. So what's the benefit? Well, built-in key rotation, the ability to manage permissions as code, connectivity between any two devices, and reduced latency. You can even ask users to re-authenticate SSH connections for that extra bit of security to keep the compliance folks happy. Try Tailscale now - it's free forever for personal use.Corey: This episode is sponsored by our friends at Logicworks. Getting to the cloud is challenging enough for many places, especially maintaining security, resiliency, cost control, agility, etc, etc, etc. Things break, configurations drift, technology advances, and organizations, frankly, need to evolve. How can you get to the cloud faster and ensure you have the right team in place to maintain success over time? Day 2 matters. Work with a partner who gets it - Logicworks combines the cloud expertise and platform automation to customize solutions to meet your unique requirements. Get started by chatting with a cloud specialist today at snark.cloud/logicworks. That's snark.cloud/logicworksCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. When I was nine years old, one of the worst tragedies that can ever befall a boy happened to me. That's right, my parents moved me to Maine. And I spent the next ten years desperately trying to get out of the state.Once I succeeded and moved to California, I found myself in a position where almost nothing can drag me back there. One of the exceptions—basically, the only exception—is Monktoberfest, a conference put on every year by the fine folks at RedMonk. It is unquestionably the best conference that I have ever been to, and it continually amazes me every time I go. The last time I was out there, I met today's guest. Chris Vermilion is a Senior Software Developer at Remix Labs. Chris, now that I finished insulting the state that you call home, how are you?Chris: I'm great. I'm happy to be in a state that's not California.Corey: I hear you. It's, uh—I talk a lot of smack about Maine. But to be perfectly direct, my problem with it is that I grew up there and that was a difficult time in my life because I, really I guess, never finished growing up according to most people. And all right, we'll accept it. No one can hate a place in the same way that you can hate it if you grew up there and didn't enjoy the experience.So, it's not Maine that's the problem; it's me. I feel like I should clarify that I'm going to get letters and people in Maine will write those letters and then have to ride their horses to Massachusetts to mail them. But we know how that works.Chris: [laugh].Corey: So, what is Remix Labs? Let's start there. Because Remix sounds like… well, it sounds like a term that is overused. I see it everywhere in the business space. I know there was a Remix thing that recently got sold to I think it was at Shopify or Spotify; I keep getting those two confused. And—Chris: One of the two, yeah.Corey: Yeah, exactly one of them plays music and one of them sells me things except now I think they both do both, and everything has gone wonky and confusing. But what do you folks do over there?Chris: So, we work on visual app development for everybody. So, the goal is to have kind of a spreadsheet-on-steroids-like development environment where you can build interactively, you have live coding, you have a responsive experience in building interactive apps, websites, mobile apps, a little bit of everything, and providing an experience where you can build systems of engagement. So tools, mobile apps, that kind of work with whatever back-end resources you're trying to do, you can collaborate across different people, pass things around, and you can do that all with a nice kind of visual app developer, where you can sort of drop nodes around and wire them together and built in a way that's it's hopefully accessible to non-developers, to project managers, to domain experts, to you know, whatever stakeholders are interested in modifying that final product.Corey: I would say that I count as one of those. I use something similar to build the tool that assembles my newsletter every week, and that was solving a difficult problem for me. I can write back-ends reasonably well, using my primary tool, which is sheer brute force. I am not much of a developer, but it turns out that with enough enthusiasm, you can overcome most limitations. And that's great, but I know nothing about front end; it does not make sense to me, it does not click in the way that other things have clicked.So, I was fourth and inches from just retaining a contractor to build out a barely serviceable internal app. And I discovered, oh, use this low-code tool to drag and drop things and that basically was Visual Basic for internal apps. And that was awesome, but they're still positioned squarely in the space of internal apps only. There's no mobile app story, there's—and it works well enough for what I do, but I have other projects, I want to wind up getting out the door that are not strictly for internal use that would benefit from being able to have a serviceable interface slapped onto. It doesn't need to be gorgeous, it doesn't need to win awards, it just needs to be, “Cool, it can display the output of a table in a variety of different ways. It has a button and when I click a button, it does a thing, generally represented as an API call to something.”And doesn't take much, but being able to have something like that, even for an internal app, has been absolutely transformative just for workflow stuff internally, for making things accessible to people that are not otherwise going to be able to do those sorts of things, by which I mean me.Chris: Yeah. I mean, exactly, I think that is the kind of use case that we are aiming for is making this accessible to everybody, building tools that work for people that aren't necessarily software developers, they don't want to dive into code—although they can if they want, it's extensible in that way—that aren't necessarily front-end developers or designers, although it's accessible to designers and if you want to start from that end, you can do it. And it's amenable to collaboration, so you can have somebody that understands the problem build something that works, you can have somebody that understands design build something that works well and looks nice, and you can have somebody that understands the code or is more of a back-end developer, then go back in and maybe fine-tune the API calls because they realize that you're doing the same thing over and over again and so there's a better way to structure the lower parts of things. But you can pass around that experience between all these different stakeholders and you can construct something that everybody can modify to sort of suit their own needs and desires.Corey: Many years ago, Bill Clinton wound up coining the phrase, ‘The Digital Divide' to talk about people who had basically internet access and who didn't—those who got it or did not—and I feel like we have a modern form of that, the technology haves and have nots. Easy example of this for a different part of my workflow here: this podcast, as anyone listening to it is probably aware by now, is sponsored by awesome folks who wind up wanting to tell you about the exciting services or tools or products that they are building. And sometimes some of those sponsors will say things like, “Okay, here's the URL I want you to read into the microphone during the ad read,” and my response is a polite form of, “Are you serious?” It's seven different subdirectories on the web server, followed by a UTM series of tracking codes that, yeah, I promise, none of you are going to type that in. I'm not even going to wind up reading into the microphone because my attention span trips out a third of the way through.So, I needed a URL shortener. So, I set up snark.cloud for this. For a long time, that was relatively straightforward because I just used an S3 bucket with redirect objects inside of it. But then you have sort of the problem being a victim of your own success, to some extent, and I was at a point where, oh, I can have people control some of these things that aren't me; I don't need to be the person that sets up the link redirection work.Yeah, the challenge is now that you have a business user who is extraordinarily good at what he does, but he's also not someone who has deep experience in writing code, and trying to sit here and explain to him, here's how to set up a redirect object in an S3 bucket, like, why didn't I save time and tell him to go screw himself? It's awful. So, I've looked for a lot of different answers for this, and the one that I found lurking on GitHub—and I've talked about it a couple of times, now—runs on Google Cloud Run, and the front-end for that of the business user—which sounds ridiculous, but it's also kind of clever, is a Google Sheet. Because every business user knows how to work a Google Sheet. There's one column labeled ‘slug' and the other one labeled ‘URL' that it points to.And every time someone visits a snark.cloud slash whatever the hell the slug happens to be, it automatically does a redirect. And it's glorious. But I shouldn't have to go digging into the depths of GitHub to find stuff like that. This feels like a perfect use case for a no-code, low-code tool.Chris: Yeah. No, I agree. I mean, that's a cool use case. And I… as always, our competitor is Google Sheets. I think everybody in software development in enterprise software's only real competitor is the spreadsheet.Corey: Oh, God, yes, I wind up fixing AWS bills for a living and my biggest competitor is always Microsoft Excel. It's, “Yeah, we're going to do it ourselves internally,” is what most people do. It seems like no matter what business line I've worked in, I've companies that did Robo-advising for retirement planning; yeah, some people do it themselves in Microsoft Excel. I worked for an expense reporting company; everyone does that in Microsoft Excel. And so, on and so forth.There are really very few verticals where that's not an option. It's like, but what about a dating site? Oh, there are certain people who absolutely will use Microsoft Excel for that. Personally, I think it's a bad idea to hook up where you VLOOKUP but what do I know?Chris: [laugh]. Right, right.Corey: Before you wound up going into the wide world of low-code development over at Remix, you—well, a lot of people have different backstories when I talk to them on this show. Yours is definitely one of the more esoteric because the common case and most people talk about is oh, “I went to Stanford and then became a software engineer.” “Great. What did you study?” “Computer Science,” or something like it. Alternately, they drop out of school and go do things in their backyard. You have a PhD in particle physics, is it?Chris: That's right. Yeah.Corey: Which first, is wild in his own right, but we'll get back to that. How did you get here from there?Chris: Ah. Well, it's kind of the age-old story of academia. So, I started in electrical engineering and ended up double majoring in physics because that you had to take a lot of physics to be an engineer, and I said, you know, this is more fun. This is interesting. Building things is great, but sitting around reading papers is really where my heart's at.And ended up going to graduate school, which is about the best gig you can ever get. You get paid to sit in an office and read and write papers, and occasionally go out drinking with other grad students, and that's really about it.Corey: I only just now for the first time in my life, realized how much some aspects of my career resemble being a [laugh] grad student. Please, continue.Chris: It doesn't pay very well is the catch, you know? It's very hard to support a lifestyle that exists outside of your office, or, you know, involves a family and children, which is certainly one downside. But it's a lot of fun and it's very low stress, as long as you are, let's say, not trying to get a job afterward. Because where this all breaks down is that, you know, as I recall, the time I was a graduate student, there were roughly as many people graduating as graduate students every year as there were professors total in the field of physics, at least in the United States. That was something like the scale of the relationship.And so, if you do the math, and unfortunately, we were relatively good at doing math, you could see, you know, most of us were not going to go on, you know? This was the path to becoming a professor, but—Corey: You look at number of students and the number of professorships available in the industry, I guess we'll call it, and yeah, it's hmm, basic arithmetic does not seem like something that anyone in that department is not capable of doing.Chris: Exactly. So, you're right, we were all I think, more or less qualified to be an academic professor, certainly at research institutions, where the only qualification, really, is to be good at doing research and you have to tolerate teaching students sometimes. But there tends to be very little training on how to do that, or a meaningful evaluation of whether you're doing it well.Corey: I want to dive into that a bit because I think that's something we see a lot in this industry, where there's no training on how to do a lot of different things. Teaching is one very clear example, another one is interviewing people for jobs, so people are making it up as they go along, despite there being decades and decades of longitudinal studies of people figuring out what works and what doesn't, tech his always loved to just sort of throw it all out and start over. It's odd to me that academia would follow in similar patterns around not having a clear structure for, “Oh, so you're a grad student. You're going to be teaching a class. Here's how to be reasonably effective at it.” Given that higher education was not the place for me, I have very little insight into this. Is that how it plays out?Chris: I don't want to be too unfair to academia as a whole, and actually, I was quite lucky, I was a student at the University of Washington and we had a really great physics education group, so we did actually spend a fair amount of time thinking about effective ways to teach undergraduates and doing this great tutorial system they had there. But my sense was in the field as a whole, for people on the track to become professors at research institutions, there was typically not much in the way of training as a teacher, there was not really a lot of thought about pedagogy or the mechanics of delivering lectures. You know, you're sort of given a box full of chalk and a classroom and said, you know, “You have freshman physics this quarter. The last teacher used this textbook and it seems to be okay,” tended to be the sort of preparation that you would get. You know, and I think it varies institution to institution what kind of support you get, you know, the level of graduate students helping you out, but I think in lots of places in academia, the role of professors as teachers was the second thought, you know, if it was indeed thought at all.And similarly, the role of professors as mentors to graduate students, which, you know, if anything, is sort of their primary job is guiding graduate students through their early career. And again, I mean, much like in software, that was all very ad hoc. You know, and I think there are some similarities in terms of how academics and how tech workers think of themselves as sort of inventing the universe, we're at the forefront, the bleeding edge of human knowledge, and therefore because I'm being innovative in this one particular aspect, I can justify being innovative in all of them. I mean, that's the disruptive thing to do, right?Corey: And it's a shame that you're such a nice person because you would be phenomenal at basically being the most condescending person in all of tech if you wanted to. Because think about this, you have people saying, “Oh, what do you do?” “I'm a full-stack engineer.” And then some of the worst people in the world, of which I admit I used to be one, are, “Oh, full-stack. Really? When's the last time you wrote a device driver?”And you can keep on going at that. You work in particle physics, so you're all, “That's adorable. Hold my tea. When's the last time you created matter from energy?” And yeah, and then it becomes this the—it's very hard to wind up beating you in that particular game of [who'd 00:15:07] wore it better.Chris: Right. One of my fond memories of being a student is back when I got to spend more time thinking about these things and actually still remembered them, you know, in my electoral engineering days and physics days, I really had studied all the way down from the particle physics to semiconductor physics to how to lay out silicon chips and, you know, how to build ALUs and CPUs and whatnot from basic transistor gates. Yeah, and then all the way up to, you know, writing compilers and programming languages. And it really did seem like you could understand all those parts. I couldn't tell you how any of those things work anymore. Sadly, that part of my brain has now taken up with Go's lexical scoping rules and borrow checker fights with Rust. But there was a time when I was a smart person and knew those things.Corey: This episode is sponsored in part by our friends at Strata. Are you struggling to keep up with the demands of managing and securing identity in your distributed enterprise IT environment? You're not alone, but you shouldn't let that hold you back. With Strata's Identity Orchestration Platform, you can secure all your apps on any cloud with any IDP, so your IT teams will never have to refactor for identity again. Imagine modernizing app identity in minutes instead of months, deploying passwordless on any tricky old app, and achieving business resilience with always-on identity, all from one lightweight and flexible platform.Want to see it in action? Share your identity challenge with them on a discovery call and they'll hook you up with a complimentary pair of AirPods Pro. Don't miss out, visit Strata.io/ScreamingCloud. That's Strata dot io slash ScreamingCloud.Corey: I want to go back to what sounded like a throwaway joke at the start of the episode. In seriousness, one of the reasons—at least that I told myself at the time—that I left Maine was that it was pretty clear that there was no significant, lasting opportunity in industry when I was in Maine. In fact, the girl that I was dating at the time in college graduated college, and the paper of record for the state, The Maine Sunday Telegram, which during the week is called The Portland Press Herald, did a front-page story on her about how she went to school on a pulp and paper scholarship, she was valedictorian in her chemical engineering class at the University of Maine and had to leave the state to get a job. And every year they would roll out the governor, whoever that happened to be, to the University of Maine to give a commencement speech that's, “Don't leave Maine, don't leave Maine, don't leave Maine,” but without any real answer to, “Well, for what jobs?”Now, that Covid has been this plague o'er the land that has been devastating society for a while, work-from-home has become much more of a cohesive thing. And an awful lot of companies are fully embracing it. How have you seen Maine change based upon that for one, and for another, how have you found that community has been developed in the local sense because there was none of that in Maine when I was there? Even the brief time where I was visiting for a conference for a week, I saw definite signs of a strong local community in the tech space. What happened? I love it.Chris: It's great. Yeah, so I moved to Maine eight years ago, in 2014. And yeah, I was lucky enough to pretty early on, meet up with a few of the local nerds, and we have a long-running Slack group that I just saw was about to turn nine, so I guess I was there in the early days, called Computers Anonymous. It was a spinoff, I think, from a project somebody else had started in a few other cities. The joke was it was a sort of a confessional group of, you know, we're here to commiserate over our relationships with technology, which all of us have our complaints.Corey: Honestly, tech community is more of a support group than most other areas, I think.Chris: Absolutely. All you have to do is just have name and technology and somebody will pipe up. “Okay, you know, I've a horror story about that one.” But it has over the years turned into, you know, a very active Slack group of people that meet up once a month for beers and chats with each other, and you know, we all know each other's kids. And when the pandemic hit, it was absolutely a lifeline that we were all sort of still talking to each other every day and passing tips of, you know, which restaurants were doing takeout, and you know which ones were doing takeout and takeout booze, and all kinds of local knowledge was being spread around that way.So, it was a lucky thing to have when that hit, we had this community. Because it existed already as this community of, you know, people that were remote workers. And I think over the time that I've been here, I've really seen a growth in people coming here to work somewhere else because it's a lovely place to live, it's a much cheaper place to live than almost anywhere else I've ever been, you know, I think it's pretty attractive to the folks come up from Boston or New York or Connecticut for the summer, and they say, “Ah, you know, this doesn't seem so bad to live.” And then they come here for a winter, and then they think, “Well, okay, maybe I was wrong,” and go back. But I've really enjoyed my time here, and the tools for communicating and working remotely, have really taken off.You know, a decade ago, my first startup—actually, you know, in kind of a similar situation, similar story, we were starting a company in Louisville, Kentucky. It was where we happen to live. We had a tech community there that were asking those same questions. “Why is anybody leaving? Why is everybody leaving?”And we started this company, and we did an accelerator in San Francisco, and every single person we talked to—and this is 2012—said, you have to bring the company to San Francisco. It's the only way you'll ever hire anybody, it's the only way you'll ever raise any money, this is the only place in the world that you could ever possibly run a tech company. And you know, we tried and failed.Corey: Oh, we're one of those innovative industries in the world. We've taken a job that can be done from literally anywhere that has internet access and created a land crunch on eight square miles, located in an earthquake zone.Chris: Exactly. We're going to take a ton of VC money and where to spend 90% of it on rent in the Bay Area. The rent paid back to the LPs of our VC funds, and the circle of life continues.Corey: Oh, yeah. When I started this place as an independent consultant six years ago, I looked around, okay, should I rent space in an office so I have a place where I go and work? And I saw how much it costs to sublet even, like, a closed-door office in an existing tech startup's office space, saw the price tag, laughed myself silly, and nope, nope, nope. Instead installed a door on my home office and got this place set up as a—in my spare room now is transformed into my home office slash recording studio. And yeah, “Well, wasn't it expensive to do that kind of stuff?” Not compared to the first three days of rent in a place like that it wasn't. I feel like that's what's driving a lot of the return to office stories is the sort of, I guess, an expression of the sunk cost fallacy.Chris: Exactly. And it's a variation of nobody ever got fired for choosing IBM, you know? Nobody ever got fired for saying we should work in the office. It's the way we've always done things, people are used to it, and there really are difficulties to collaborating effectively remotely, you know? You do lose something with the lack of day-to-day contact, a lack of in-person contact, people really do get kind of burned out on interacting over screens. But I think there are ways around that and the benefits, in my mind, my experience, you know, working remotely for the last ten years or so, tend to outweigh the costs.Corey: Oh, yeah. If I were 20 years younger, I would absolutely have been much more amenable to staying in the state. There's a lot of things that recommend it. I mean, I don't want people listening to this to think I actually hate Maine. It's become a running joke, but it's also, there was remarkably little opportunity in tech back when I lived there.And now globally, I think we're seeing the rise of opportunity. And that is a line I heard in a talk once that stuck with me that talent is evenly distributed, but opportunity isn't. And there are paths forward now for folks who—I'm told—somehow don't live in that same eight-square miles of the world, where they too can build tech companies and do interesting things and work intelligently with other folks. I mean, the thing that always struck me as so odd before the pandemic was this insistence on, “Oh, we don't allow remote work.” It's, “Well, hang on a minute. Aren't we all telecommuting in from wherever offices happen to be to AWS?” Because I've checked thoroughly, they will not let you work from us-east-1. In fact, they're very strict on that rule.Chris: [laugh]. Yeah. And it's remarkable how long I think the attitude persisted that we can solve any problem except how to work somewhere other than SoMa.Corey: Part of the problem too in the startup space, and one of the things I'm so excited about seeing what you're doing over at Remix Labs, is so many of the tech startups for a long time felt like they were built almost entirely around problems that young, usually single men had in their 20s when they worked in tech and didn't want to deal with the inconveniences of having to take care of themselves. Think food delivery, think laundry services, think dating apps, et cetera, et cetera. It feels like now we're getting into an era where there's a lot of development and focus and funding being aimed at things that are a lot more substantial, like how would we make it possible for someone to build an app internally or externally without making them go to through a trial-by-fire hazing ritual of going to a boot camp for a year first?Chris: Yeah. No, I think that's right. I think there's been an evolution toward building tools for broader problems, for building tools that work for everybody. I think there was a definite startup ouroboros in the, kind of, early days of this past tech boom of so much money being thrown at early-stage startups with a couple of young people building them, and they solved a zillion of their own problems. And there was so much money being thrown at them that they were happy to spend lots of money on the problems that they had, and so it looked like there was this huge market for startups to solve those problems.And I think we'll probably see that dry up a little bit. So, it's nice to get back to what are the problems that the rest of us have. You know, or maybe the rest of you. I can't pretend that I'm not one of those startup people that wants on-demand laundry. But.Corey: Yet you wake up one day and realize, oh, yeah. That does change things a bit. Honestly, one of the weirdest things for me about moving to California from Maine was just the sheer level of convenience in different areas.Chris: Yes.Corey: And part of it is city living, true, but Maine is one those places where if you're traveling somewhere, you're taking a car, full stop. And living in a number of cities like San Francisco, it's, oh great, if I want to order food, there's not, “The restaurant that delivers,” it's, I can have basically anything that I want showing up here within the hour. Just that alone was a weird, transformative moment. I know, I still feel like 20 years in, that I'm “Country Boy Discovers City for the First Time; Loses Goddamn Mind.” Like, that is where I still am. It's still magic. I became an urban creature just by not being one for my formative years.Chris: Yeah. No, I mean, absolutely. I grew up in Ann Arbor, which is sort of a smallish college town, and certainly more urban than the areas around it, but visiting the big city of Detroit or Lansing, it was exciting. And, you know, I got older, I really sort of thought of myself as a city person. And I lived in San Francisco for a while and loved it, and Seattle for a while and loved it.Portland has been a great balance of, there's city; it's a five minute drive from my house that has amazing restaurants and concerts and a great art scene and places to eat and roughly 8000 microbreweries, but it's still a relatively small community. I know a lot of the people here. I sort of drive across town from one end to the other in 20 minutes, pick up my kids from school pretty easily. So, it makes for a nice balance here.Corey: I am very enthused on, well, the idea of growing community in localized places. One thing that I think we did lose a bit during the pandemic was, every conference became online, so therefore, every conference becomes the same and it's all the same crappy Zoom-esque experience. It's oh, it's like work with a slightly different topic, and for once the people on this call can't fire me… directly. So, it's one of those areas of just there's not enough differentiation.I didn't realize until I went back to Monktoberfest a month or so ago at the time at this call recording just how much I'd missed that sense of local community.Chris: Yeah.Corey: Because before that, the only conferences I'd been to since the pandemic hit were big corporate affairs, and yeah, you find community there, but it also is very different element to it, it has a different feeling. It's impossible to describe unless you've been to some of these community conferences, I think.Chris: Yeah. I mean, I think a smallish conference like that where you see a lot of the same people every year—credit to Steven, the whole RedMonk team for Monktoberfest—that they put on such a great show that every year, you see lots and lots of faces that you've seen the last several because everybody knows it's such a great conference, they come right back. And so, it becomes kind of a community. As I've gotten older a year between meetings doesn't seem like that long time anymore, so these are the friends I see from time to time, and you know, we have a Slack who chat from time to time. So, finding those ways to sort of cultivate small groups that are in regular contact and have that kind of specific environment and culture to them within the broader industry, I think has been super valuable, I think. To me, certainly.Corey: I really enjoyed so much of what has come out of the pandemic in some ways, which sounds like a weird thing to say, but I'm trying to find the silver linings where I can. I recently met someone who'd worked here with me for a year-and-a-half that I'd never met in person. Other people that I'd spoken to at length for the last few years in various capacity, I finally meet them in person and, “Huh. Somehow it never came up in conversation that they're six foot eight.” Like, “Yeah, okay/ that definitely is one of those things that you notice about them in person.” Ah, but here we are.I really want to thank you for spending as much time as you have to talk about what you're up to, what your experiences have been like. If people want to learn more, where's the best place for them to find you? And please don't say Maine.Chris: [laugh]. Well, as of this recording, you can find me on Twitter at @chrisvermilion, V-E-R-M-I-L-I-O-N. That's probably easiest.Corey: And we will, of course, put links to that in the [show notes 00:28:53]. Thank you so much for being so generous with your time. I appreciate it.Chris: No, thanks for having me on. This was fun.Corey: Chris Vermilion, Senior Software Developer at Remix Labs. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment, and since you're presumably from Maine when writing that comment, be sure to ask a grown-up to help you with the more difficult spellings of some of the words.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About AlyssAlyss Noland is the head of Developer Relations Relations and Product Marketing at Common Room, an intelligent community-led growth platform. She previously led product marketing for Developer Experience at GitHub where she focused on open source community investment and helping engineering teams find success through development metrics and developer-focused research. She's been working in tech since 2012 in various roles from Sales Engineering and Developer Advocacy to Product Marketing with companies such as GitHub, Box, Atlassian, and BigCommerce, as well as being an advisor at Heavybit. Links Referenced: Common Room: https://www.commonroom.io/ Heavybit: https://www.heavybit.com/ Twitter: https://twitter.com/PreciselyAlyss Twitch: https://www.twitch.tv/PreciselyAlyss TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Tailscale SSH is a new, and arguably better way to SSH. Once you've enabled Tailscale SSH on your server and user devices, Tailscale takes care of the rest. So you don't need to manage, rotate, or distribute new SSH keys every time someone on your team leaves. Pretty cool, right? Tailscale gives each device in your network a node key to connect to your VPN, and uses that same key for SSH authorization and encryption. So basically you're SSHing the same way that you're managing your network.So what's the benefit? You'll get built-in key rotation, the ability to manage permissions as code, connectivity between two devices, and reduced latency. You can even ask users to re-authenticate SSH connections for that extra bit of security. Try Tailscale now - it's free forever for personal use forever.Corey: This episode is sponsored by our friends at Logicworks. Getting to the cloud is challenging enough for many places, especially maintaining security, resiliency, cost control, agility, etc, etc, etc. Things break, configurations drift, technology advances, and organizations, frankly, need to evolve. How can you get to the cloud faster and ensure you have the right team in place to maintain success over time? Day 2 matters. Work with a partner who gets it - Logicworks combines the cloud expertise and platform automation to customize solutions to meet your unique requirements. Get started by chatting with a cloud specialist today at snark.cloud/logicworks. That's snark.cloud/logicworksCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I often wonder how to start these conversations, but sometimes it's just handed to me and I don't even have to do a whole lot of work. My guest today is Alyss Noland, who's the Head of Developer Relations Relations and Product Marketing at Common Room. Alyss, thank you for joining me.Alyss: Thanks for having me, Corey. I'm really excited to be here.Corey: So, developer relations relations. It feels like an abstraction that has been forced to be built on top of another abstraction that has gotten too complicated, so as best I can tell, you are walking around as a human equivalent of Kubernetes.Alyss: Oh, gosh, I would really hope not to be a human equivalent of Kubernetes. I think that would make me an octopus. But—Corey: Yeah, “What did you say about me?” Yeah.Alyss: [laugh].Corey: “I didn't come here to be insulted, Quinn.” Yeah.Alyss: No, like listen, I love octopodes. Which [tattoo 00:01:24] is which? So, developer relations relations. Yes, it's an abstraction on an abstraction. A really critical level, it is how do I relate? Can I relate to people that are in the developer relations profession at large?We are at the point at which this is a somewhat poorly-defined area that is continuing to grow. And there's a lot of debates in that space and so I'm really excited to be at an organization that will give me a platform to try and move the industry forward.Corey: Your relatively recent career history is honestly fascinating to me. You spent about a year and a half as a senior developer advocate at Box. And as anyone who's ever tried it knows, it's very hard to beat Box [beatboxing noises]. But you tried and went to GitHub, in which case, you basically transitioned pretty quickly from a Senior Product Marketing Manager to Director of Product Marketing, where you were the go-to-market lead for GitHub Copilot.Alyss: Yeah, that was a really interesting project to be on. I started off at the technical preview back in 2021, launching that too—it ended up being with about a little over a million, two million folks in technical preview. And it's fairly new to the market. There was nothing else—or at the time, there had been nothing else that was using a descendant of GPT-3. There was nothing else using a descendant of GPT-3 to generate suggestions for code to—there were a couple that were using GPT-2, but the amount of language coverage they had was a little bit limited, what they were suggesting was a little bit limited.And it's hard to say, like, highlight of my career, but at that point in time, I would say probably, highlight of my career to be able to work on something with that opportunity for impact.Corey: As someone who was in the technical preview and now tried to be a paying customer of it, but I can't because of my open-source work, it wound up giving it to me for free. I found it to be absolutely transformative. And I know I'm going to get letters and I don't even slightly care because it's not, “I'm going to tab-complete my application.” If a tool can do that, your application is probably not that complex. No, for me, what I find incredibly valuable is the ability to tab-complete through obnoxious boilerplate. CloudFormation, I am not subtweeting you; I am calling you out directly. You are wordy and obnoxious. Fix yourself.And especially in languages that I don't deal with day-to-day—because I'm not a full-time developer—I forget certain parameters or argument order or things like that and being able to effectively tab-complete is awesome for that use case. It's not doing my job; it's automating the crappy part of my job. And I absolutely love it for that.Alyss: Yeah, and was really interesting working on a common portion of product marketing work is that we build messaging houses. We try to identify where's the value to the user, to the organization at large, depending on, like, who it is we're trying to sell to, how does that ladder up from, like, an IoT to a manager. And so, one of the things that I got really excited about as we started to see it—and there's some great work that Dr. Eirini Kallaimvakou has published that I would definitely refer to if you're interested in diving deeper into it—is the way in which Copilot and this, like, ability to improve the boilerplate experience, improve the boring shit—automate the boring shit, if you will—is about developer satisfaction. It's not about making you build your commits faster or about having more lines of code that you like get deployed out; it's about making your jobs suck less.Corey: Well, if you spent, what was it roughly two years, give or take, at GitHub between your various roles—and yes, I'm going to pronounce it ‘GIF-ub' because that's my brand of obnoxious, so I'm going to go for it—you went to Common Room. Let's begin there. What does Common Room do, exactly?Alyss: So, Common Room is an intelligent community-led growth platform. And there's a few things kind of packed into that really short description, but the idea is that we've seen all of these product-lead grows businesses. But at a critical point, and something we've seen at GitHub, which is a product-led growth company, it's something that we've seen at Atlassian, Asana, you name half a dozen different, like, SaaS companies, self-hosted software, open-source, community is at the heart of it. And so, how do you nurture that community? How do you measure that community? How do you prove that the work that you're doing is valuable?And that's what Common Room is setting out to do. And so, when I saw—like, they're not the only person or organization in the market that's doing this, but I think they're doing it exceptionally well, and with really great goals in mind. And so, I'm enthused to try and facilitate that investment in community for more organizations.Corey: One of the challenges that I have seen of products in the community space is it tended, historically, to go in really, I guess I'll call them uncomfortable directions. In the before times, I used to host dinner parties near constantly here, and someone confide into me once—after, you know, six beers or so, because that's when people get the excitingly honest—they mentioned that, “Yeah, I'm supposed to wind up putting these dinners into Salesforce”—or whatever the hell it was—“To track the contacts we have with influencers in this space.” And that made me feel so profoundly uncomfortable. It's, you're invited here to spend time with my friends and my family. You're meeting my kids, it's, yeah, this is just a go-to-market motion and you can [BLEEP] on out of here and never come back.And I did not get that sense to be clear and I'm told the company wound up canceling that horrifying program, but it does feel like it's very easy to turn an authentic relationship into something that feels remarkably sleazy. That said, Common Room has been around for a while and I have yet to hear a single accusation that you folks have come within a thousand miles of doing that. How do you avoid the trap?Alyss: It's a slippery slope, and I can't say that Common Room creates any kind of like enforcement or silos or prevents organizations from falling into this trap. Fundamentally, the way in which community can be abused, the way in which these relationships can be taken advantage of, at least from the perception of the parties that initially built the relationship, is to take the context out of them, to take the empathy out of them, take the people out of them. And so, that is fundamentally left to the organization's principles, it's left to how much authority does community have within the business relative to a sales team. And so first, being able to elevate community in such a way to show that they are having that impact already without having to turn the community into a prospect pool is, I think, one of the critical first steps, and it's something that we've been able to break through initially by connecting things like Slack, Discord, Twitter to show, here's all these people talking about you, here's all the things that they're saying, here's the sentiment analysis, and also, now we're going to push that into Salesforce. So, you can see that this started out in community and it was fostered there. Now, you can see the ROI, you don't need to go hitting up our community contacts to try and sell to them because we're doing it on your behalf in a very real way.Corey: Part of the challenge, I think, is that—and you've talked to me about this in previous conversations we've had—that so much of community is distilled down to a sales motion, which let's be direct, it kind of sucks at, in some levels, because it's okay, great, I'm here to talk to you about how community works. Well, in the AWS community, for example, the reason that formed and is as broad and fast as it is because AWS's documentation is Byzantine and there's a sort of shared suffering that we all get to commiserate over. And whenever AWS tries to take, “Ownership,” quote-unquote, of its community, right, that doesn't actually work that way. They have community watering holes, but to my understanding, the largest AWS-centric Slack team is the Open Guide to AWS's Slack team, which now has, at last count, 15,000 people in it. I'm lucky enough to be the community lead for that project.But it was pre-existing before I got there and it's great to be able to go and talk to people who are using these things. It doesn't feel like it is owned, run, or controlled—because it's not—by AWS themselves. It's clear from the way that your product has evolved, that you feel similarly around that where it's about being aware of the community rather than controlling the community. And that's important.Alyss: Absolutely. And one of the ways in which we, like, highlight this as soon as you're in the product, is being able to show community responsiveness and then what percentage of those responses are coming from my team members. And frankly, as someone who's previously set strategy for developer relations teams, for developer communities, what I want to see is community members responding to each other, community members knowing what's the right place to look, what's the right answer, how am I ensuring that they have the resources that they need, the answers that they need. Because at the end of the day, I can't scale one-to-one; no one can. And so, the community being able to support itself is at the heart of the definition of community.Corey: One of the other problems that I've seen historically, and I'll call it the Chef problem because Chef had an incredibly strong community, and as someone who is deep in the configuration management space myself, but never use Chef, it was the one that I avoided for a variety of reasons at the time, it was phenomenal. I wound up going to ChefConf, despite not being a Chef user, just to spend time with some of the great people that were involved. The blunder that they made before they were acquired into irrelevance by progress—and to be fair, the industry changed direction toward immutable infrastructure in ways that were hard to foresee—but the problem is, they made was hiring their entire community. And it doesn't sound like that would be a bad thing, but suddenly, everyone who was talking about the product had a Chef email address, and that hits very differently.Alyss: It does. And it goes back to that point of trying to maintain those authentic relationships. And if we're to step outside of tech, I have a background prior to tech in the video game industry, and that was a similar problem. Nearly every single community-made application, extension ends up getting acquired by some organization, like Curse, and then piped full of ads, or the person that you thought you could ask or to see build some other better experience of version control software, or a Git client ends up getting consumed into a large business and then the project never sees the light of day. And frankly, that's not how you run community in my estimation.My estimation is, if the community is doing things better than you are, take notes. Product management, pay attention. That's something that is another aspect of doing developer relations is about checking in with those teams, about showing them evidence. And like, it so often ends up being qualitative in a way that doesn't change people's minds or their feelings, where people want to see quantitative numbers in order to say, “Oh, this is the business justification. Like, this is the ROI. This proves that this is the thing we should invest in.” And frankly, no. Like, sometimes it is a little bit more about stepping back and letting the organic empathy and participation happen without having to own it.Corey: There's a sense, I think that a lot of companies feel the need to own every conversation that happens around them, their product, et cetera, and you can't. You just can't, unless—to be direct—your company is failing. Just because if no one's talking about you, then great, you're the only ones talking about you. And you can see this from time to time and it's depressing as hell when you have people who work for a company all tweeting the same cookie-cutter statement, and they get zero interaction except from a bot account. It's sad.Alyss: Yeah. And I've unfortunately seen this more times than I can count in community Slacks where people just, like, copy-paste whatever marketing handed to them, and I would be shocked if they got any engagement at all. Because that's… cool. What do I know about you? Why do I care about this event? Have you personalized it to me?And yeah, you don't want the organization to be the only one talking about you. If you are then you've already failed in this, you know, product-led growth motion. You've kind of—if we want to get into the murky water of NPS, like, nobody's going and telling their friends about your product [laugh]. And the thing that's so valuable is the authentic voice. It's the, “I'm excited to talk about this and I like it enough to tell you what I like about it.” I like it enough to tell you about this use case that might never seen the light of day, but because we're having a conversation between ourselves, it can all be personalized. It can all be about what's going on between us and about our shared experiences. And that is ten times more powerful than most Twitter-promoted ads you'll ever see.Corey: So, I want to unpack a little bit about not developer relations as such, but developer relations relations because I can mostly understand—badly—what product marketing is, but developer relations relations—or as you'd like to call it developer relations squared—that's something new. I've always called DevRel to be devrelopers, and people get annoyed enough at that. What is that newfound layer of abstraction on top of it?Alyss: Well, there's several things that I'm going to end up—and I say end up; I'm six weeks into the role, so I have a lot of high hopes for where I hope this goes. And one of those is things, like, we don't have a very shared understanding and shared definition of what developer advocacy even is, what is developer relations? Does developer marketing belong under that umbrella? How should organizations approach developer relations? How should they value it? Where should it, you know, belong in terms of business strategy?And there's an opportunity for a company whose business it is to elevate this industry, this career path, if you will, where we can spend the time, we can spend the money to say, here's what success looks like. We've interviewed all these groups, we've talked with the leaders in this space that are making it their jobs to think about this. Here's a set of group-developed recommendations for how the industry should mature. Or here's an open-source set of job descriptions and requirements. And like, let's get to some level of shared understanding.So, as an example of, kind of, where I'm leading to with all of this, and some of the challenges that developer relations faces is the State of Developer Relations report that just came out. There's a significant number of people that are coming into developer advocate, developer relations roles for the first time, they have one to two years of experience, they're coming into programs that have been around for one to two years, and so what does that tell you? That tells you you're bringing in people with no experience to try to establish brand new programs, that they're being asked to by their business, and they don't have the vocabulary, the tools, the frameworks in which to establish that for themselves. And so, they're going to be swayed by, you know, the tides of business, by the influences of their leadership without having their own pre-built notions. And so, how do we give them that equipment and how do we elevate the practice?Corey: Cloud native just means you've got more components or microservices than anyone (even a mythical 10x engineer) can keep track of. With OpsLevel, you can build a catalog in minutes and forget needing that mythical 10x engineer. Now, you'll have a 10x service catalog to accompany your 10x service count. Visit OpsLevel.com to learn how easy it is to build and manage your service catalog. Connect to your git provider and you're off to the races with service import, repo ownership, tech docs, and more. Corey: It feels like so much of the DevRel discourse has turned into, one, we define it by what is not, and two, it doesn't matter how you're measuring it, you're measuring it wrong. I feel like that is, I guess we'll call it counterproductive, for lack of a better descriptor. It feels like there's such a short-sighted perspective on all of this, but at the same time, you've absolutely got to find ways to articulate the value of DevRel slash community to the business otherwise, it turns into a really uncomfortable moment when, okay, time to cut costs. Why should we keep your function over a different function? If there's not a revenue or upside or time to market or some form of value story tied to that, that the business can understand that isn't just touchy-feely, it's a very difficult path forward from there. How do you see it?Alyss: I agree with you and I've, frankly, run into this problem several times in my career, and every time I've been a developer advocate. It's, you know—and where I've found the most success is not in saying, “Here's exactly the numbers that I'm going to be constantly looking at. I'm going to try to produce this many pieces of content, or I'm absolutely not speaking at events. And that's not my job. Or I'm not writing code. That's not my job.”It's about understanding what is driving the business forward. Who do I need participation and buy-in from and where am I hoping to go? Like, what does a year out from this look like? What does three years out from this look like? At Box, we do not want to be the API governance standard. That is not our job. That's not where we sit within engineering.That's frankly, if you really want to get into it, internal developer advocacy because it can influence the impact on the community. It is not the core focus and there are probably people better equipped and better educated on the core application. Big commerce, platform ecosystem, platform flywheel developers are fundamentally a part of continuing to grow the business and how do I go make that point to sales, how do I go make that point to partners, how do I go make that point to customer success, so that I can build a function that has more than one person. And so, I think to kind of bring it back to the larger question, that is where I see our greatest challenge is that we haven't given ourselves the vocabulary or the framework to understand the level of complexity that DevRel has become in being across so many industries, and being in B2B, and being in business to developer, and being in business to consumer. No one size fits all and we need to stop trying to treat it as though it can be.Corey: I think that there is a, how to put it, a problem in terms of how Twitter views a lot of these things. Someone wound up finally distilling it down for me in relatively recent times with a very resonant quote, which was simply put, that Twitter is not where you go for nuance. Twitter is where you go to be righteous. And I realized, oh, my God, that describes a good 80% of the things I've put up there. Like when I talk about how when companies do this thing to their staff and it's crappy, I am not necessarily for a nuanced debate, although of course there's always nuance and edge cases in the rest.As a counterpoint, whenever I wind up talking about things on Twitter and speak in generalities, I get a whole bunch of people pushing back with a, “Well, what about this edge case? That renders your entire point invalid.” And, ugh, not really. It feels like one of the casualties of the pandemic has been a sense of community in a sense of humans relating to other humans. I think we're all tired of the Zoom calls from hell I got to see you a couple of weeks before this recording at Monktoberfest in Portland, Maine, and oh, my God, dealing with people face to face, it was so much richer, at least from my perspective, compared to everything that we've been able to do during the pandemic. Am I alone on that? Are you seeing this across the board? Where companies are talking about this?Alyss: I will say with confidence, you're not alone in this. Whether or not companies are talking about it is also across the board. How rich are those understandings? How rich are those conversations? Because trying to step back as a brand is not really a way.Like, having nuance, being real, been community members, like that's not a way in which I think companies can participate in a way that feels truly authentic. That's why you need faces. That's why you need people. That's why you need folks whose job it is to do this. But in terms of things are lost, like, Twitter is not the right place to be having these conversations. It's not the right place in which to necessarily relate to people, absolutely.When you get distilled down all of your interactions into oh, I've got a notification. Oh, I have a checkmark, and so I have, like, better moderation tools. Oh, like, I made a statement and I don't want to hear a solution for it. We get all of these, uncurated experiences that are so dissatisfying that it does make us miss being around people who can read body language, that can understand my immediate relationship to them in spaces that we choose to be in, whereas Twitter is this big panopticon where we can just get yelled at and yell at each other. And it loves to amplify those conversations far more than any of the touchy-feely, good news success stories.Corey: When you take a look across the entire landscape of managing DevRel programs and ensuring that companies are receiving value for it, and—by which I mean, nurturing the long-term health of communities because yes, I am much more interested in that than I am in next quarter's numbers, how do you see that evolving, particularly with the recent economic recession or correction or drawback or everything's on fire, depending upon who it is you talk to? How do you see that evolving?Alyss: It goes back to what I said earlier about, I can speak in generalities, there will be specifics to various organizations, but at a fundamental part, like, I'll kind of take a step back and maybe make some very strong statements about what I think DevRel is, in a regard, which is, without documentation, without support, you don't have a product. And if you don't have folks going out and understanding what it is your customers need, and especially when those customers are maybe all the time or sometimes developers, and understanding what it is that they're saying and truly how having empathy for what's going on in their day-to-day, what task are they trying to complete, how relevant is this to them, if you don't invest in that, when that happens, you've lost the plot. And so, in those instances, unfortunately, that's a conversation with leadership team. Your leadership doesn't fundamentally understand the value and maybe it's worth it to make the argument in favor of to illustrate that without this feedback loop, without this investment in the educational journey of developers, without the investment in what is going on in our product, and where have we allowed ourselves to remain ignorant of what is happening in the day-to-day of our users. We need those folks.Product managers are in sprints, they're in standups. They're doing, like, strategic planning and their yearly planning. We need a group who is rewarded to care about this but also is innately driven to do so as well. And that's not something that you can make. And it's not something that we otherwise see. It's part of why we have such an absence in good developer marketing is because marketers aren't paid well enough to ever have learned the skills to be developers, and so there's no skills transfer.Corey: One last topic that I want to get into something you've only been doing for a short while, but you've become an advisor at Heavybit, which is a VC firm. How did that come about and what do you do?Alyss: So currently, I—I'll do the super-high level. What I do right now is I host office hours with seed startups and Series A that are in the dev tool space. And we generally talk about developer relations, a little bit in developer marketing go-to-market strategies. And it's super enriching for me because I love hearing about different experiences and problems and, like, areas of practice. But it was really interesting, and a little bit of a make-your-own-luck-and-opportunity type deal.Where I live in Austin, Texas; I do not live in the Bay Area, I don't have all those connections, I've been a bit distant from it. And I saw someone who had accepted a role that I had interviewed for, end up in some of their content. And I was like, “They're doing a great job. They definitely deserve to be there, but I also had similar qualifications, so why should I also be there?” And I found someone, his name's Tim, on LinkedIn, who runs their events. And I reached out and I said, “Hey, Tim, how would you like a new advisor?” And so, Tim responded back and we—Corey: Knock knock. Who's there? It's me.Alyss: Yeah, exactly. It's—and it was just, I want this thing to happen. How do I make it happen? I ask.Corey: And what does it day-to-day that look like? How much time does it take? What do you do exactly?Alyss: Yeah. I mean, right now, it's about five hours every quarter. So, I spend anywhere between 30 minutes to an hour with various organizations that are a part of Heavybit's portfolio, talking with them through their motion to go general availability, or they want to start participating in events, or they want to discover what are the right events for them to—or, like, DevOpsDays, should we participate in that? Should we hire a DevRel person? Should we hire a product marketing person? Just helping them sort wheat from chaff in terms of, like, how to proceed.And so, it's relatively, for me, lightweight. And Heavybit also gives us the opportunity to contribute back in blog posts, participate in podcasts and be able to have some of those richer conversations. So, I have a set of bookmarks, so there's over 100, bookmarks long, that is fully curated across several different categories. That was my first blog post was diving into a few of those where I think are critical areas of developer relations. What are some of the conversations on DevRel metrics? How do I think about setting a DevRel strategy for the first time? How do I do my first DevRel hire? And so, I wouldn't even call it a second job. It's more of a getting to, again, enrich my own experience, see a wider variety of different problems in this space and expand my own understanding.Corey: I really want to thank you for being so generous with your time. If people want to learn more about what you're up to, how you view the world, and basically just come along for the ride as you continue to demonstrate a side of tech that I don't think we get to see very often, where can they find you?Alyss: I am@PreciselyAlyss on Twitter, as well as Twitch. Aside from that, I would not recommend looking for me.Corey: Excellent. Always a good decision. I will put links to that in the [show notes 00:30:00]. Thank you so much for your time. I appreciate it.Alyss: Thanks, Corey.Corey: Alyss Noland, Head of Developer Relations Relations and Product Marketing at Common Room. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment belittling community and letting the rest of us know by observation just why you've been thrown out of every community to which you've ever been a part.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About PetePete is currently the Head of Growth And Community for AppMap, the open source dynamic runtime code analyzer. Pete also works with early stage startups, helping them navigate the complex world of early stage new product development.Pete also fully acknowledges his profile pic is slightly out of date, but has been too lazy to update it to reflect current hair growth trends.Links:AppMap: https://appmap.io/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: If you asked me to rank which cloud provider has the best developer experience, I'd be hard-pressed to choose a platform that isn't Google Cloud. Their developer experience is unparalleled and, in the early stages of building something great, that translates directly into velocity. Try it yourself with the Google for Startups Cloud Program over at cloud.google.com/startup. It'll give you up to $100k a year for each of the first two years in Google Cloud credits for companies that range from bootstrapped all the way on up to Series A. Go build something, and then tell me about it. My thanks to Google Cloud for sponsoring this ridiculous podcast.Corey: Cloud native just means you've got more components or microservices than anyone (even a mythical 10x engineer) can keep track of. With OpsLevel, you can build a catalog in minutes and forget needing that mythical 10x engineer. Now, you'll have a 10x service catalog to accompany your 10x service count. Visit OpsLevel.com to learn how easy it is to build and manage your service catalog. Connect to your git provider and you're off to the races with service import, repo ownership, tech docs, and more. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn and this is probably my favorite recurring episode slash tradition, every year. I drag Pete Cheslock on who talks with me about his experience at re:Invent. Last year, Pete, you didn't go. The year before, none of us went because it was all virtual, but it feels like we're finally getting back into the swing of things. How are you, Pete?Pete: I am doing great. It is always a pleasure. It was amazing to see other humans in person at a industry event. As weird as it sounds to say that, you know, it was great to be in Vegas [laugh], it was mostly great, just because there were other humans there too that I wanted to see.Corey: Because this is going to confuse folks who haven't been following our various adventures, these days, you are the Head of Growth and Community at AppMap. But you and I have been talking for years and you did a stint working at The Duckbill Group here with us as a cloud economist. Ah, I miss those days. It was fun working with you and being able to bother you every day as opposed to just on special occasions like this.Pete: Yeah, I know. I got to slide into your Slack DMs in addition, and then when I didn't get a response, I would slide into your Twitter DMs. It worked out perfectly. So yeah, it's been a wild ride. I mean, I took an interlude from my startup journey by continually working at tech startups.And yeah, I got to join onboard the Duckbill and have, you know, a really wonderful time cutting bills and diving into all of the amazing parts of people's Amazon usage. But I am also equally broken in my brain, and continually said to myself, “Maybe I'll do another startup.” [laugh].Corey: Right. And it turns out that we're not a startup. Everyone likes to think we are. It's like, oh, okay—like Amazon, for example, has us historically in their startup division as far as how they—the buckets as they put different accounts into. And if you look at us through that lens, it's yeah, we're a specific kind of startups, specifically a failing startup—or failed—because to us growth is maybe we'll hire one or two people next year, as opposed to, “Oh, yeah, we're going to TEDx this place.” No, yeah, we're building a lifestyle business very much by design.Pete: I'd be very curious how many account managers actually Duckbill has kind of churned through because usually, you get to keep your account manager if you're growing at a pretty incredible clip. And it's kind of a bellwether for, like, how fast are we—are we growing so fast that we have kept our account manager for multiple years?Corey: Your timing is apt. We're a six-year-old company and I just met our fourth account manager last week.Pete: [laugh].Corey: No it's, honestly, what happens with AWS account managers is the only way you get to keep them is if your spend trajectory on AWS matches their career trajectory inside of AWS. Because if you outpace them, they'll give you to someone that they view as being more senior, whereas if they outpace you, they're going to stop dealing with the small accounts and move on to the bigger ones. Honestly, at this point, I've mostly stopped dealing with my account managers. I had one that was just spectacular. It was sad to see him get promoted; good for him.But I get tired of trying to explain the lunacy that is me to someone on the AWS side every year. It just doesn't make sense because my accounts are super weird and when they try and suggest the usual things that work for 99.995% of AWS customers and things they care about, it falls to custard when it comes to me specifically. And that's not on them; it's because I'm weird and broken.Pete: I'm remembering now one of the best account managers that I ever worked with at a startup, years and years ago. She was with us for a couple of years, pretty solidly. And then, you know, because careers are long and jobs are short, when I was at The Duckbill Group again, doing work, turns out she was the account manager on this other thing, you know? Which, like, looking at the company she was account manager for was like 500x [laugh] my previous company, so I was like, “Oh, yeah. You're clearly moving up in the world because my company did not 500x.” So, sometimes you got to chase the ones who are.Corey: So, let's talk about re:Invent. This felt like the first re:Invent post-pandemic. And let's be clear, I wound up getting Covid by the end, so I don't recommend that to everyone. But let's be clear, this was not a re:Invent were anyone officially accepted that Covid existed. I was one of the only people wearing masks to most of the events I was at. Great load of good that did me.But it was big. It was the usual sixty-some-odd-thousand people that had been in previous years, as opposed to the hard cap of 30 or so that they had last year so it felt smaller and more accessible. Nope. Right back to bizarre numbers of people. But fewer sponsors than most years, so it felt like their budget was severely constrained. And they were trying to have not as many sponsors, but still an overwhelming crush of attendees. It felt odd, but definitely very large scale.Pete: Yeah, I can echo that a hundred percent. I'm sure we've talked about this in previous ones, but I've had the pleasure—well, I don't know, some might call it not a pleasure, but it's been a pleasure to watch re:Invent grow over so many years. I went to the first re:Invent. A company I was at actually sponsored it. And remembering back to that first re:Invent, it was kind of quaint by comparison.There were 4000 people at the first re:Invent, which again, it's a big conference, especially when a lot of the conferences that I think I was really attending at the time were like, you know, 600, 1000, maybe tops. To go to a 4000-person event in Vegas especially, it's again, in the same Expo Hall it's been since that first one, it still felt big. But every person stayed in the Venetian. Pretty much everyone was in the same hotel, all of the attendees that year. All the talks were there.There was, you know, a lot [laugh]—I mean, a lot less of everything that was there. And so, watching it grow over time, not only as a sponsor because I've actually been—kind of worked re:Invent as a, like, a booth person for many of these years for multiple different sponsors and had to coordinate that aspect of it, but then also a couple of times just being more, like, attendee, right, just someone who could go and kind of consume the content. This year was more on the side of being more of an attendee where I got to just kind of experience the Expo Hall. You know, I actually spent a lot of time in the Expo Hall because a big part of why I was there was—Corey: To get t-shirts.Pete: Yeah, we'll get to—I was running low on not only t-shirts but socks. My socks were really worse for wear the last few years. I had to, like, re-up that, right [laugh]?Corey: Yeah, you look around. It's like, “Well, none of you people have, like, logoed pants? What's the deal here? Like, I have to actually buy those myself. I don't—I'm not here to spend money.”Pete: Yeah, I know. So. And so yeah, this year, it felt—it was like Covid wasn't a thing. It wasn't in anyone's mind. Just walking around—Vegas in general, obviously, it's kind of in its own little bubble, but, you know, I've been to other events this year that were much more controlled and had a lot more cautious attendees and this was definitely not like that at all. It felt very much, like the last one I was at. The last one I was at was 2019 and it was a big huge event with probably 50,000-plus people. And this one felt like to me at least, attendee-wise, it definitely felt bigger than that one in a lot of ways.Corey: I think that when all is said and done, it was a good event, but it wasn't necessarily what a lot of folks were expecting. What was your take on the content and how the week played out?Pete: Yeah, so I do, in many ways, kind of miss [laugh] the event of yore that was a little bit more of a targeted, focused event. And I understand that it will never be that kind of event anymore. Maybe they start splitting it off to be, you know, there's—just felt much more like a builder event in previous years. The content in the keynotes, you know, the big keynotes and things like that would be far more, these big, iterative improvements to the cloud. That's something that always felt kind of amazing to see. I mean, for years and years, it was like, “Who's ready for another re:Invent price drop?” Right? It was all about, like, what's the next big price drop going to be?Corey: Was it though because I never was approaching with an eye toward, “Oh, great. What are they going to cut prices on now?” That feels like the least interesting things that ever came out of re:Invent, at least for me. It's, what are they doing architecturally that lets me save money, yes. Or at least do something interesting architecturally, great. I didn't see Lambda when it first came out, for example, as a cost opportunity, although, of course, it became one. I saw it as this is a neat capability that I'm looking forward to exploring.Pete: Yeah, and I think that's what was really cool about some of those early ones is these, like, big things would get released. Like, Lambda was a big thing that got released. There was just these larger types of services coming out. And I think it's one of your quotes, right? Like, there's a service for everyone, but every service isn't for everyone.Corey: Yeah.Pete: And I feel like, you know, again, years ago, looking back, it felt like more of the services were more geared towards the operational, the early adopters of Amazon, a lot of those services was for those people. And it makes sense. They got to spread out further, they've got to have kind of a wider reach to grow into all of these different areas. And so, when they come out with things that, yeah, to me, I'm like, “This is ridiculous feature. Who would ever use this?” Like, there's probably a dozen other people at different companies that are obscenely excited because they're at some enterprise that has been ignored for years and now finally they're getting the exact tooling that they need, right?Corey: That made sense for a long time. I think that now, the idea that we're going to go and see an Andy Jassy-era style feature drop of, “Here's five new databases and a whole new compute platform and 17,000 more ways to run containers,” is not necessarily what is good for the platform, certainly not good for customers. I think that we're seeing an era of consolidation where, okay, you have all these services to do a thing. How do I pick which one to use? How do I get onto a golden path that I can also deviate from without having to rebuild everything? That's where customers seem to be. And it feels like AWS has been relatively slow to acknowledge or embrace that to me.Pete: Yeah, a lot of the services, you know, are services they're probably building for just their own internal purposes, as well. You know, I know, they are for a while very motivated to get off anything Oracle-related, so they started building these services that would help migrate, you know, away from Oracle because they were trying to do it themselves. But also, it's like, there's still—I mean, I talk with friends of mine who have worked at Amazon for many years and I'm always fascinated by how excited they are still to be there because they're operating at a scale that just doesn't exist anywhere else, right? It's like, they're off on their lone island that go into work somewhere else is almost going backwards because you've already solved problems at this lower level of scale. That's obviously not what you want to be doing anymore.And at the scale that they're at for some of these services, even like the core services, the small improvements they're making, they seem so simple and basic, like a tiny EBS improvement, you're like, “Ugh, that's so boring.” But at their level of scale for, like, something like an EBS, like one of those top five services, the impact of that tiny little change is probably even so amazingly impactful. Like it's just so huge [laugh], you know, inside that scope of the business that is just—that's what—if you really start pulling the thread, you're like, “Wow, actually, that is a massive improvement.” It just doesn't feel that way because it's just oh, it's just this tiny little thing [laugh]. It's like, just almost—it's too simple. It's too simple to be complex, except at massive scale.Corey: Exactly. The big problem I ran into is, I should have noticed it last year, but it was Adam Selipsky's first re:Invent and I didn't want to draw too many conclusions on it, but now we have enough dots to make a line—specifically two—where he is not going to do the Andy Jassy thing of getting on stage and reading off of a giant 200 item list of new feature and service announcements, which in AWS parlance, are invariably the same thing, and they wind up rolling all of that out. And me planning my content schedule for re:Quinnvent around that was a mistake. I had to cancel a last-minute rebuttal to his keynote because there was so little there of any substance that all would have been a half-hour-long personal attack and I try not to do that.Pete: Mmm. Yeah, the online discussion, I feel like, around the keynote was really, like, lackluster. It was yeah, like you said, very devoid of… not value; it's not really the right word, but just substance and heft to it. And maybe, look, we were just blessed with many, many years of these dense, really, really dense, full keynotes that were yeah, just massive feature drops, where here's a thing and here's a thing, and it was almost that, like, Apple-esque style kind of keynote where it was like, we're just going to bombard you with so many amazing things that kind of is in a cohesive storyline. I think that's the thing that they were always very good about in the past was having a cohesive story to tell about all of these crazy features.All of these features that they were just coming out with at this incredible velocity, they could weave the story around it. And you felt like, yeah, keynote was whatever hour, two hours long, but it would go by—it always felt like it would go by quickly because they were just they had down kind of really tight messaging and kept your attention the whole way through because you were kind of like, “Well, what's next? There's always—there's more. There's got to be more.” And there would be, right? There would be that payoff.Corey: I'm glad that they recognized that what got them here won't get them there, but I do wish that they had done a better job of signaling that to us in more effective ways. Does that make any sense?Pete: Yeah, that's an interesting… it's kind of an interesting thought exercise. I mean, you kind of mentioned before earlier, before we started recording, the CMO job is still available, it's still open [laugh] at AWS. So, if this was a good way to attract a top-tier CMO, I'd almost feel like if you were that person to come in and be like, “Hey, this did not work. Here are the following reasons and here's what you need to do to improve it.” Like, you might have a pretty solid shot of landing that role [laugh].Corey: Yeah, I'm not trying to make people feel intentionally bad over it. This stuff is very hard, particularly at scale. The problem I had with his keynote was not in fact that he was a bad speaker, far from it. He was good a year ago, he's clearly put work and energy into becoming better over the past year. From a technical analysis of how is Adam Selipsky as a public speaker, straight A's as far as I'm concerned, and I spent a lot of time focusing on this stuff myself as a professional speaker myself. I have no problems with how he wound up delivering any of the content. My problem was with the content itself. It feels like he was let down by the content team.Pete: Yeah, it definitely felt not as dense or as rich as we had come to expect in previous years. I don't think it was that the content didn't exist. It's not like they didn't build just as much, if not way, way more than they have in previous years. It just seemed to just not be part of the talk.I don't know. I always kind of wonder, too, is this just an audience thing? Which is, like, maybe I'm just not the audience for his talk, right? Was there someone else in that Expo Hall, someone else watching the stream, that was just kept on the edge of their seat hearing these stories? I don't know. I'm really kind of curious. Like, you know, are we only representing this one slice of the pie, basically?Corey: I think part of the problem is that re:Invent has grown so big, that it doesn't know what it wants to be anymore. Is it a sales event? By the size of the Expo Hall, yeah, it kind of is. Is it a partner expo where they talk about how they're going to milk various companies? Possibly. There's certainly one of those going on.There was an analyst summit that I attended for a number of days during re:Invent this year. They have a whole separate thing for press. The community has always been a thriving component of re:Invent, at least for me, and seeing those folks is always terrific. Is it supposed to be where they dump a whole bunch of features and roadmap information? Is it time for them to wind up having executive meetings with their customers? It tries to be all of those things and as a result, at this scale it feels like it is failing to be any of them, at least in an effective, cohesive way.Pete: Yeah, and you really nailed each of the personas, like, of a re:Invent attendee. I've talked to many people who are considering going to re:Invent, and they're, “I don't really know if I want to go, but I really want to go to some sessions, and I really want to do this.” And I always have to kind of push back and say, “Look, if you're only going there to attend talks,” like, just don't bother, right? As everyone knows, the talks are all recorded, you can watch them later. I did have conversations with some engineer, principal engineer level software folks that were there and the prevailing consensus from chatting with those folks, kind of anecdotally, is that, like, they had actually a lot of struggles even getting into some of these sessions, which for anyone who has been to re:Invent in the last, I don't know, four or five years, like, it's still a challenge, right?There's—you got to register for a lot of these talks way far in advance, there'll be a standby list, there'll be a standby line. It's a lot of a lot. And so, there's not usually a ton of value there. And so, I always try to say, like, “If you're going to re:Invent your, kind of, main purpose to go would be more for networking,” or just you're going because of the human interaction that you hope to get out of it, right, the high bandwidth conversations that are really hard to do in other areas. And I think you've nailed a bunch of those, right? Like, an analyst briefing is really efficient if you can get all the analysts in a room versus doing one-off analyst meetings.Meeting with big enterprises and hearing their thoughts and feelings and needs and requirements, you can get a lot of those conversations. And especially, too, if, like, talking to an enterprise and they got a dozen people all spread over the world, well you can get them all in one room, like, that's pretty amazing in this world. And then on the sales side, I feel like granted, I spent most of my time in the Expo Hall, but that was probably the area that I think you said earlier which I really picked up on, which was the balance between sponsors and attendees felt out of whack. Like it felt like there were way more attendees than the sponsors that I would have kind of expect to see.Now, there were a lot of sponsors on that Expo Hall and it took days. I mean, I was on the Expo Hall for days walking around and chatting with different companies and people. But one of the things that I saw that I have never seen before was a number of sponsorship booths, right—and these are booths that are, like, prebuilt, ten by ten-foot size or the smaller ones—that were blanks. They were like, you know, like, in a low-quality car where you have blank buttons that, like, if you paid more you get that feature. Walking around, there was a nonzero number of just straight-up empty booth, blank booths around which, I don't know, like, that felt kind of telling. Like, did they not sell all their sponsorships? Has that ever happened? I don't even know. But this was I felt like the first I've had—Corey: Or did companies buy the sponsorship and then realize that it was so expensive to go on top of it, throwing bad money after good might not have made sense. Because again, when people—Pete: Right.Corey: —brought out these sponsorships, in many cases, it was in the very early days of the growing recession we've seen now. And they may have been more optimistic, their circumstances may certainly have changed. I do know that pricing for re:Invent sponsorships was lower this past year than it had been in previous years. In 2019, for example, they had two Expo Halls, one at the ARIA Quad and the other at the Venetian. They had just one this year, which made less running around on my part, but still.Pete: Yeah, I do remember that, that they had so many sponsors. What I would say about the sponsors that there's two parts of this that were actually interesting. One, you're definitely right. As someone who has sponsored re:Invent before and has had to navigate that world, you are likely going to commit to the sponsorship as early as June, you know, could even be earlier than June depending on how big of a thing that you're doing. But it's early. It's usually in the summertime that you're—if you haven't made a decision by the summertime, like, you could actually not get a booth, right?And this was, I remember, the last one that I had sponsored was maybe 2018, 2019. And, like, you don't want those last few booths. Like, they put you in the back and not a good way. But going there, were a lot of—I did notice a lot of booths that had pretty massive layoffs who still had the booths, you know, and again, large booths, large companies, which again, same thing. I kind of am like, “Wow, like, how many employees did that booth cost you, right?”Because like [laugh], some of these booths are hundreds of thousands of dollars to sponsor. And then the other thing that I actually noticed, too, which I was honestly a little surprised by, with the exception of the Datadog booth; I love my friends at Datadog, they have the most amazingly aggressive booth BDRs who are always just, they'll get you if you're, like, hovering near them. And there's always someone to talk to over there. Like, they staff it really, really well. But there were some other booths that I was actually really interested in talking to some of the people to learn about their technology, that I actually waited to talk to someone. Like, I waited for someone to talk to, and then finally I'm like, “You know what? I'm going to come back.”And then I came back and waited again. So, it's like, how many of these sponsors obviously spent a lot of money to go there, then months later, they start looking at the people that they have to support this, they've already had some layoffs and probably sent a much smaller audience there to actually, like, operate the booth.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: One bright light that I did enjoy and I always enjoy, though I'm not certain how actionable it is in the direct sense, was Peter DeSantis' Monday Night Live keynote. It was great. I mean, the one criticism I had of it—on Twitter at the time, before that thing melted and exploded—was that it was a bit mislabeled because it really should have been what it turned into midway through of surprise computer science lecture with Professor DeSantis. And I was totally there for it. But it was fun just watching some of the deep magic that makes this all work be presented in a way that most of us normally don't get to see.Pete: Well, this was the first year they did not do their Midnight Madness over-the-top kind of thing. And I also I don't recall that I saw them doing one of the other things I feel like is at night is they're, like, giant wing-eating competition. Am I wrong? Did they do that this year and I just missed hearing about it?Corey: They did not. Turns out that competitive Gluttony is not as compelling as it once was. But they also canceled their Midnight Madness event a month or two before re:Invent itself. What was super weird to me was that there was no event—community or otherwise—that sprung forth to seize that mantle. So, you had a whole bunch of people who were used to going for several hours that night to a big event with nothing to do.And at 9 p.m. they started just dumping a whole bunch of service releases in their blog and RSS feeds and the rest, and it just felt very weird and discordant. Like, do they think that we have nothing better to do than sit here and read through this on a Sunday night where we would have otherwise been at a party? Well yeah, in my case, I'm super sad and of course, I had nothing better to do that night. But most people had things going on.Pete: Yeah. Yeah, exactly. I think also, if you—maybe it's a little bit better now but I don't know when you have to buy that many chicken wings in advance, but with supply chains being what they are and the cost of chicken wings, I mean, not that I track the cost of chicken wings, except I absolutely do every time I go to Costco, they're up substantially. So, that was probably a contributing factor to the wing-eating contest: supply chain pain and suffering. But yeah, it's really interesting that just even in what some of the sponsors kind of were doing this year over previous years, I doubt they did this in 2021—but maybe, I don't know—but definitely not in 2019, something that I don't recall to this level was the sponsors essentially booking out entire restaurants near the venue every single day of the conference.And so again, if you were at this event like we were, and you at the end of the thing, were just like, I just want to sit and I've got a handful of friends, I want to sit and, like, have a drink, and just, like, chat and catch up and hear how the day went and everything else, finding a place to actually go to do that was very, very hard to do. And the thing that I noticed was—again, seemed like it was new this year; I don't recall it in 2019 to this level is, there were a lot of the big sponsors that had just booked a whole restaurant, breakfast, lunch, and dinner, like, from open to close, fully booked it, which was honestly, brilliant.Corey: Oh, yeah. If you bring 200, 300 people to an event, you've got to feed him somehow. And, “Hey, can we just rent out your restaurant for the entirety of this week?” Is not out of the question compared to what you'd even spend just reimbursing that sea of people to go and eat somewhere else.Pete: Exactly. The reason—I'm approaching this from, like, a business perspective—if I had a large group of enterprise salespeople and they need a place to book meetings, well, it's super compelling if I'm being courted by one of these salespeople and they're like, “Hey, come and have breakfast. Come and grab a coffee.” You know, and there's a place where you can sit down and quietly enjoy that meal or coffee while having a sale. Like, I'll have that sales conversation and I'm going to be way more motivated to show up to it because you're telling me it's like, this is where we're going to meet.Versus some of my friends were trying to, like, coordinate a lunch or a coffee and it's like, do we want to go to the Starbucks that has 500 people in line or do we want to walk four hotels, you know, down the street to find a bar that has video poker that no one will be sitting at and that we can just sit down and talk, right? It kind of felt like those were your two options.Corey: One thing that MongoDB did is rented out the Sugarcane restaurant. And they did this a couple of years in a row and they wound up effectively making it available to community leaders, heroes, and whatnot, for meetings or just a place to sit down and catch your breath. And I think that was a brilliant approach. You've gone to the trouble of setting this thing up for meetings for your execs and whatnot. Why not broaden it out?You can't necessarily do it for everyone, for obvious reasons, but it was nice to just reach out to folks in your orbit and say, “Yeah, this is something available to you.” I thought that was genius. And I—Pete: Oh yeah.Corey: —wish I thought of doing something like that. Let's be clear. I also wish I had rent-out-Sugarcane-for-a-week budget. But you know—Pete: [laugh].Corey: —we take what we can get.Pete: Yeah. That'll be a slight increase to the Spite Budget to support that move.Corey: Just a skosh, yeah.Pete: Yeah, the MongoDB, they were one that I do remember had done it similarly. I don't know if they had done it, kind of, full-time before, but a friend of mine work there, had invited me over and said, “Hey, like, come by, let's grab a drink. You know, we've got this hotel, you know, this restaurant kind of booked out.” And that was back in 2019. Really enjoyed it.And yeah, I noticed it was like, you know, basically, they had this area available, again, a place to sit down, to open your laptop, to respond to some emails, making it available to community people should have been a no-brainer to, really, all of these other sponsors that may have times of less kind of attendance, right? So obviously, at any of the big meals, maybe that's when you can't make it available for all the people you want to, but there's going to be off hours in between times that making that available and offering that up generates a supreme amount of goodwill, you know, in the community because you know, you're just looking for a place to sit out and drink some water [laugh].Corey: Yeah, that was one challenge that I saw across the board. There were very few places to just sit and work on something. And I'm not talking a lounge everywhere around every corner was needed necessarily or even advisable. No, the problem I've got was that I just wanted to sit down for two or three minutes and just type up an email quickly, or a tweet or something, and nope, you're walking and moving the whole time.Pete: Yeah. Now honestly, this would be a—this was a big missed opportunity for the Amazon event planning folks. There was a lot of unnecessary space usage that I understand why they had it. Here's an area you could play Foursquare, here's an area that had seesaws that you could sit on. Like, just, I don't know, kitschy stuff like that, and it was kind of off to the side or whatever.Those areas honestly, like, we're kind of off to the side, they were a little bit quieter. Would have been a great spot to just, like, load up some chairs and couches and little coffee tables and just having places that people could sit down because what ended up happening—and I'm sure you saw it just like I did—is that any hallway that had somewhere that you could lean your back against had a line of people just sitting there on their laptops because again, a lot of us are at this event, but we're also have jobs that we're working at, too, and at some point during the day, you need to check in, you need to check some stuff out. It felt like a lack of that kind of casual space that you can just relax in. And when you add on top of all the restaurants nearby being essentially fully booked, it really, really leaves you hanging for any sort of area to sit and relax and just check a thing or talk to a person or anything like that.Corey: Yeah, I can't wait to see what lessons get learned from this and how it was a mapping to next year, across the board. Like, I have a laundry list of things that I'm going to do differently at re:Invent next year. I do every year. And sometimes it works out; sometimes it really doesn't. And it's a constant process of improvement.I mean, one of the key breakthroughs for me was when I finally internalized the idea that, yeah, this isn't going to be like most jobs where I get fired in the next six months, where when I'm planning to go to re:Invent this is not the last re:Invent I will be at in my current capacity, doing what I do professionally. And that was no small thing. Where oh, yeah. So, I'm already making plans, not just for next re:Invent, but laying the groundwork for the re:Invent after that.Pete: Yeah, I mean, that's smart way to do it. And especially, too, when you don't consider yourself an analyst, even though you obviously are an analyst. Maybe you do consider yourself an analyst, but you're [laugh] more, you know, you're also the analyst who will go and actually use the product and start being like, “Why does this work the way it does?” But you're kind of a little bit the re:Invent target audience in a lot of ways, right? You're kind of equal parts on the analyst expert and user as well. It's like you kind of touch in a bunch of those areas.But yeah, I mean, I would say the one part that I definitely enjoyed was the nature walk that you did. And just seeing the amount of people that also enjoyed that and came by, it was kind of surreal to watch you in, like, full safari garb, basically meandering through the Expo Hall with this, like, trail of, like, backpacks [laugh] following you around. It was a lot of fun. And, you know, it's like stuff like that, where people are looking for interesting takes on, kind of, the state of something that is its own organism. Like, the Expo Hall is kind of its own thing that is outside of the re:Invent control. It's kind of whatever is made up by the people who are actually sponsoring it.Corey: Yeah, it was neat to see it play out. I'm curious to see how it winds up continuing to evolve in future years. Like right now, the Nature Walk is a blast, but it was basically at the top and I had something like 50 people following me around at one point. And that is too big for the Expo Hall. And I'm not there to cause a problem for AWS. Truly, I'm not. So, I need to find ways to embrace that in ways that don't kill the mojo or the energy but also don't create problems for, you know, the company whose backup I am perched upon, yelling more or less ridiculous things.Pete: [laugh]. I think it was particularly interested in how many people I'd be walking by and every once a while I would see, like, a friend of mine, someone actually working one of the booths and just be like, “What's going on here?” Like, I know one of my friends even said, “Yeah, like, nothing draws a crowd like a crowd.” And you can almost see more people [laugh] just, like, connecting themselves onto this safari train moving their way through. Yeah, it's a sight to see, that's for sure [laugh].Corey: Yeah, I'll miss aspects of this. Again, nothing can ever stay the same, on some level. You've got to wind up continuing to evolve and grow or you wind up more or less just frozen in place[ and nothing great ever happens for you.Pete: Yeah, I mean, again, Expo Hall has gone through these different iterations, and I—you know, when it does come to the event, as I kind of think back, I probably have spent most of my time actually in the Expo Hall, usually just related to the fact that, like, when you're a sponsor, like, you're just—that's where you're at. For better or worse, you're going to be in there. And especially if you're a sponsor, you want to check out what other sponsors are doing because you want to get ideas around things that you might want to try in later years. I mentioned Datadog before because Datadog to this day continues to have the best-designed booth ever, right? Like when it comes to a product that is highly demoable, I've been myself as a sponsor, it has always been a struggle to have a very effective demo setup.And I actually remember, kind of, recommending to a startup that I was at years ago, I'm doing a demo setup that was very, very similar to how Datadog did it because it was brilliant, where you have this, like, octagon around a main area of tables, and having double-sided demo stations. A lot more people are doing this now, but again, as I walked by I was again reminded just how effective that setup is because not only do you have people that just they don't want to talk, they just want to look, and they can kind of safely stand there and look, but you also have enough people staffing the booth for conversations that for, like me who actually might want to ask for questions, I don't have to wait and I can get an answer and be taken care of right away, versus some other booths. This year, one of the areas that I actually really enjoyed—and I don't even know the details of, like, how it all came about—but it looked like some sort of like Builder's Expo. I don't know if you remember walking by there, but there was a whole area of different people who had these little IoT or various powered things. One of them was, like, a marble sorting thing that was set up with a bunch of AWS services. I think there was like the Simple Beer Service V… four or five at this point. I had one of those iterations.It was some sort of mixture between Amazon software services that were powering these, like, physical things that you can interact with. But what was interesting is like, I have no idea, like, how it was set up, and who—I'm assuming it was Amazon specific—but each of these little booths were like chocked up with information about who they were and what they built, which gave it a feel of, like, this was like a last-minute builder event thing. It didn't feel like it was a highly produced thing. It had a much more casual feel to it, which honestly got me more interested to spend time there and check out the different booths.Corey: It was really nice to be able to go and I feel like you got to see all of the booths and whatnot. I know in previous years, it feels like you go looking for specific companies and you never find them. And you thought, “Oh”—Pete: [laugh].Corey: —“They must not have been here.” You find out after the fact, oh no, just you were looking in the wrong direction because there was so much to see.Pete: There were definitely still a couple of those. I had a list of a handful of booths I wanted to stop by, either to say hi to someone I knew who was going to be there or just to chat with them in general, there was a couple that I had to do a couple loops to really track them down. But yeah, I mean, it didn't feel as overly huge as a previous one, or as previous ones. I don't know, maybe it was like the way they designed it, the layout was maybe a little bit more efficient so that you could do loops through, like, an outer loop and an inner loop and actually see everything, or if it just was they just didn't have enough sponsors to truly fill it out and maybe that's why it felt like it was a little bit more approachable.I mean, it was still massive. I mean, it was still completely over the top, and loud and shiny lights and flashing things and millions of people. But it is kind of funny that, like, if you do enough of these, you can start to say, “Oh well, I don't know, it's still felt… a little bit less [laugh] for some reason.”Corey: Yeah, just a smidgen. Yeah. Pete, it is always a pleasure to get your take on re:Invent and see what you saw that I didn't and vice versa. And same time next year, same place?Pete: Yeah. I mean, like I said, one of my favorite parts of re:Invent is, you know, we always try to schedule, like, an end-of-event breakfast when we're both just supremely exhausted. Most of us don't even have a voice by the end. But just being able to, like, catch up and do our quick little recap and then obviously to be able to get on a podcast and talk about it is always a lot of fun. And yeah, thanks again for having me. This is—it's always, it's always a blast.Corey: It really is. Pete Cheslock, Head of Growth and Community at AppMap. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and then put something insulting about me in the next keynote because you probably work on that content team.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About TimTim Gonda is a Cloud Security professional who has spent the last eight years securing and building Cloud workloads for commercial, non-profit, government, and national defense organizations. Tim currently serves as the Technical Director of Cloud at Praetorian, influencing the direction of its offensive-security-focused Cloud Security practice and the Cloud features of Praetorian's flagship product, Chariot. He considers himself lucky to have the privilege of working with the talented cyber operators at Praetorian and considers it the highlight of his career.Tim is highly passionate about helping organizations fix Cloud Security problems, as they are found, the first time, and most importantly, the People/Process/Technology challenges that cause them in the first place. In his spare time, he embarks on adventures with his wife and ensures that their two feline bundles of joy have the best playtime and dining experiences possible.Links Referenced: Praetorian: https://www.praetorian.com/ LinkedIn: https://www.linkedin.com/in/timgondajr/ Praetorian Blog: https://www.praetorian.com/blog/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Thinkst Canary. Most Companies find out way too late that they've been breached. Thinkst Canary changes this. Deploy Canaries and Canarytokens in minutes and then forget about them. Attackers tip their hand by touching 'em giving you the one alert, when it matters. With 0 admin overhead and almost no false-positives, Canaries are deployed (and loved) on all 7 continents. Check out what people are saying at canary.love today!Corey: Kentik provides Cloud and NetOps teams with complete visibility into hybrid and multi-cloud networks. Ensure an amazing customer experience, reduce cloud and network costs, and optimize performance at scale — from internet to data center to container to cloud. Learn how you can get control of complex cloud networks at www.kentik.com, and see why companies like Zoom, Twitch, New Relic, Box, Ebay, Viasat, GoDaddy, booking.com, and many, many more choose Kentik as their network observability platform. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while, I like to branch out into new and exciting territory that I've never visited before. But today, no, I'd much rather go back to complaining about cloud security, something that I tend to do an awful lot about. Here to do it with me is Tim Gonda, Technical Director of Cloud at Praetorian. Tim, thank you for joining me on this sojourn down what feels like an increasingly well-worn path.Tim: Thank you, Corey, for having me today.Corey: So, you are the Technical Director of Cloud, which I'm sort of short-handing to okay, everything that happens on the computer is henceforth going to be your fault. How accurate is that in the grand scheme of things?Tim: It's not too far off. But we like to call it Praetorian for nebula. The nebula meaning that it's Schrödinger's problem: it both is and is not the problem. Here's why. We have a couple key focuses at Praetorian, some of them focusing on more traditional pen testing, where we're looking at hardware, hit System A, hit System B, branch out, get to goal.On the other side, we have hitting web applications and [unintelligible 00:01:40]. This insecure app leads to this XYZ vulnerability, or this medical appliance is insecure and therefore we're able to do XYZ item. One of the things that frequently comes up is that more and more organizations are no longer putting their applications or infrastructure on-prem anymore, so therefore, some part of the assessment ends up being in the cloud. And that is the unique rub that I'm in. And that I'm responsible for leading the direction of the cloud security focus group, who may not dive into a specific specialty that some of these other teams might dig into, but may have similar responsibilities or similar engagement style.And in this case, if we discover something in the cloud as an issue, or even in your own organization where you have a cloud security team, you'll have a web application security team, you'll have your core information security team that defends your environment in many different methods, many different means, you'll frequently find that the cloud security team is the hot button for hey, the server was misconfigured at one certain level, however the cloud security team didn't quite know that this web application was vulnerable. We did know that it was exposed to the internet but we can't necessarily turn off all web applications from the internet because that would no longer serve the purpose of a web application. And we also may not know that a particular underlying host's patch is out of date. Because technically, that would be siloed off into another problem.So, what ends up happening is that on almost every single incident that involves a cloud infrastructure item, you might find that cloud security will be right there alongside the incident responders. And yep, this [unintelligible 00:03:20] is here, it's exposed to the internet via here, and it might have the following application on it. And they get cross-exposure with other teams that say, “Hey, your web application is vulnerable. We didn't quite inform the cloud security team about it, otherwise this wouldn't be allowed to go to the public internet,” or on the infrastructure side, “Yeah, we didn't know that there was a patch underneath it, we figured that we would let the team handle it at a later date, and therefore this is also vulnerable.” And what ends up happening sometimes, is that the cloud security team might be the onus or might be the hot button in the room of saying, “Hey, it's broken. This is now your problem. Please fix it with changing cloud configurations or directing a team to make this change on our behalf.”So, in essence, sometimes cloud becomes—it both is and is not your problem when a system is either vulnerable or exposed or at some point, worst case scenario, ends up being breached and you're performing incident response. That's one of the cases why it's important to know—or important to involve others in the cloud security problem, or to be very specific about what the role of a cloud security team is, or where cloud security has to have certain boundaries or has to involve certain extra parties have to be involved in the process. Or when it does its own threat modeling process, say that, okay, we have to take a look at certain cloud findings or findings that's within our security realm and say that these misconfigurations or these items, we have to treat the underlying components as if they are vulnerable, whether or not they are and we have to report on them as if they are vulnerable, even if it means that a certain component of the infrastructure has to already be assumed to either have a vulnerability, have some sort of misconfiguration that allows an outside attacker to execute attacks against whatever the [unintelligible 00:05:06] is. And we have to treat and respond our security posture accordingly.Corey: One of the problems that I keep running into, and I swear it's not intentional, but people would be forgiven for understanding or believing otherwise, is that I will periodically inadvertently point out security problems via Twitter. And that was never my intention because, “Huh, that's funny, this thing isn't working the way that I would expect that it would,” or, “I'm seeing something weird in the logs in my test account. What is that?” And, “Oh, you found a security vulnerability or something akin to one in our environment. Oops. Next time, just reach out to us directly at the security contact form.” That's great. If I'd known I was stumbling blindly into a security approach, but it feels like the discovery of these things is not heralded by an, “Aha, I found it.” But, “Huh, that's funny.”Tim: Of course. Absolutely. And that's where some of the best vulnerabilities come where you accidentally stumble on something that says, “Wait, does this work how—what I think it is?” Click click. Like, “Oh, boy, it does.”Now, I will admit that certain cloud providers are really great about with proactive security reach outs. If you either just file a ticket or file some other form of notification, just even flag your account rep and say, “Hey, when I was working on this particular cloud environment, the following occurred. Does this work the way I think it is? Is this is a problem?” And they usually get back to you with reporting it to their internal team, so on and so forth. But let's say applications are open-source frameworks or even just organizations at large where you might have stumbled upon something, the best thing to do was either look up, do they have a public bug bounty program, do they have a security contact or form reach out that you can email them, or do you know, someone that the organization that you just send a quick email saying, “Hey, I found this.”And through some combination of those is usually the best way to go. And to be able to provide context of the organization being, “Hey, the following exists.” And the most important things to consider when you're sending this sort of information is that they get these sorts of emails almost daily.Corey: One of my favorite genre of tweet is when Tavis Ormandy and Google's Project Zero winds up doing a tweet like, “Hey, do I know anyone over at the security apparatus at insert company here?” It's like, “All right. I'm sure people are shorting stocks now [laugh], based upon whatever he winds up doing that.”Tim: Of course.Corey: It's kind of fun to watch. But there's no cohesive way of getting in touch with companies on these things because as soon as you'd have something like that, it feels like it's subject to abuse, where Comcast hasn't fixed my internet for three days, now I'm going to email their security contact, instead of going through the normal preferred process of wait in the customer queue so they can ignore you.Tim: Of course. And that's something else you want to consider. If you broadcast that a security vulnerability exists without letting the entity or company know, you're also almost causing a green light, where other security researchers are going to go dive in on this and see, like, one, does this work how you described. But that actually is a positive thing at some point, where either you're unable to get the company's attention, or maybe it's an open-source organization, or maybe you're not being fully sure that something is the case. However, when you do submit something to the customer and you want it to take it seriously, here's a couple of key things that you should consider.One, provide evidence that whatever you're talking about has actually occurred, two, provide repeatable steps that the layman's term, even IT support person can attempt to follow in your process, that they can repeat the same vulnerability or repeat the same security condition, and three, most importantly, detail why this matters. Is this something where I can adjust a user's password? Is this something where I can extract data? Is this something where I'm able to extract content from your website I otherwise shouldn't be able to? And that's important for the following reason.You need to inform the business what is the financial value of why leaving this unpatched becomes an issue for them. And if you do that, that's how those security vulnerabilities get prioritized. It's not necessarily because the coolest vulnerability exists, it's because it costs the company money, and therefore the security team is going to immediately jump on it and try to contain it before it costs them any more.Corey: One of my least favorite genres of security report are the ones that I get where I found a vulnerability. It's like, that's interesting. I wasn't aware that I read any public-facing services, but all right, I'm game; what have you got? And it's usually something along the lines of, “You haven't enabled SPF to hard fail an email that doesn't wind up originating explicitly from this list of IP addresses. Bug bounty, please.” And it's, “No genius. That is very much an intentional choice. Thank you for playing.”It comes down to also an idea of whenever I have reported security vulnerabilities in the past, the pattern I always take is, “I'm seeing something that I don't fully understand. I suspect this might have security implications, but I'm also more than willing to be proven wrong.” Because showing up with, “You folks are idiots and have a security problem,” is a terrific invitation to be proven wrong and look like an idiot. Because the first time you get that wrong, no one will take you seriously again.Tim: Of course. And as you'll find that most bug bounty programs are, if you participate in those, the first couple that you might have submitted, the customer might even tell you, “Yeah, we're aware that that vulnerability exists, however, we don't view it as a core issue and it cannot affect the functionality of our site in any meaningful way, therefore we're electing to ignore it.” Fair.Corey: Very fair. But then when people write up about those things, well, they've they decided this is not an issue, so I'm going to do a write-up on it. Like, “You can't do that. The NDA doesn't let you expose that.” “Really? Because you just said it's a non-issue. Which is it?”Tim: And the key to that, I guess, would also be that is there an underlying technology that doesn't necessarily have to be attributed to said organization? Can you also say that, if I provide a write-up or if I put up my own personal blog post—let's say, we go back to some of the OpenSSL vulnerabilities including OpenSSL 3.0, that came out not too long ago, but since that's an open-source project, it's fair game—let's just say that if there was a technology such as that, or maybe there's a wrapper around it that another organization could be using or could be implementing a certain way, you don't necessarily have to call the company up by name, or rather just say, here's the core technology reason, and here's the core technology risk, and here's the way I've demoed exploiting this. And if you publish an open-source blog like that and then you tweet about that, you can actually gain security support around such issue and then fight for the research.An example would be that I know a couple of pen testers who have reported things in the past, and while the first time they reported it, the company was like, “Yeah, we'll fix it eventually.” But later, when another researcher report this exact same finding, the company is like, “We should probably take this seriously and jump on it.” It sometimes it's just getting in front of that and providing frequency or providing enough people around to say that, “Hey, this really is an issue in the security community and we should probably fix this item,” and keep pushing others organizations on it. A lot of times, they just need additional feedback. Because as you said, somebody runs an automated scanner against your email and says that, “Oh, you're not checking SPF as strictly as the scanner would have liked because it's a benchmarking tool.” It's not necessarily a security vulnerability rather than it's just how you've chosen to configure something and if it works for you, it works for you.Corey: How does cloud change this? Because a lot of what we talked about so far could apply to anything. Go back in time to 1995 and a lot of what we're talking about mostly holds true. It feels like cloud acts as a significant level of complexity on top of all of this. How do you view the differentiation there?Tim: So, I think it differentiated two things. One, certain services or certain vulnerability classes that are handled by the shared service model—for the most part—are probably secure better than you might be able to do yourself. Just because there's a lot of research, the team is [experimented 00:13:03] a lot of time on this. An example of if there's a particular, like, spoofing or network interception vulnerability that you might see on a local LAN network, you probably are not going to have the same level access to be able to execute that on a virtual private cloud or VNet, or some other virtual network within cloud environment. Now, something that does change with the paradigm of cloud is the fact that if you accidentally publicly expose something or something that you've created expo—or don't set a setting to be private or only specific to your resources, there is a couple of things that could happen. The vulnerabilities exploitability based on where increases to something that used to be just, “Hey, I left a port open on my own network. Somebody from HR or somebody from it could possibly interact with it.”However, in the cloud, you've now set this up to the entire world with people that might have resources or motivations to go after this product, and using services like Shodan—which are continually mapping the internet for open resources—and they can quickly grab that, say, “Okay, I'm going to attack these targets today,” might continue to poke a little bit further, maybe an internal person that might be bored at work or a pen tester just on one specific engagement. Especially in the case of let's say, what you're working on has sparked the interest of a nation-state and they want to dig into a little bit further, they have the resources to be able to dedicate time, people, and maybe tools and tactics against whatever this vulnerability that you've given previously the example of—maybe there's a specific ID and a URL that just needs to be guessed right to give them access to something—they might spend the time trying to brute force that URL, brute force that value, and eventually try to go after what you have.The main paradigm shift here is that there are certain things that we might consider less of a priority because the cloud has already taken care of them with the shared service model, and rightfully so, and there's other times that we have to take heightened awareness on is, one, we either dispose something to the entire internet or all cloud accounts within creations. And that's actually something that we see commonly. In fact, one thing I would like to say we see very common is, all AWS users, regardless if it's in your account or somewhere else, might have access to your SNS topic or SQS Queue. Which doesn't seem like that big of vulnerability, but I changed the messages, I delete messages, I viewed your messages, but rather what's connected to those? Let's talk database Lambda functions where I've got source code that a developer has written to handle that source code and may not have built in logic to handle—maybe there was a piece of code that could be abused as part of this message that might allow an attacker to send something to your Lambda function and then execute something on that attacker's behalf.You weren't aware of it, you weren't thinking about it, and now you've exposed it to almost the entire internet. And since anyone can go sign up for an AWS account—or Azure or GCP account—and then they're able to start poking at that same piece of code that you might have developed thinking, “Well, this is just for internal use. It's not a big deal. That one static code analysis tool isn't probably too relevant.” Now, it becomes hyper-relevant and something you have to consider with a little more attention and dedicated time to making sure that these things that you've written or deploying, are in fact, safe because misconfigured or mis-exposed, and suddenly the entire world is starts knocking at it, and increases the risk of, it may really well be a problem. The severity of that issue could increase dramatically.Corey: As you take a look across, let's call it the hyperscale clouds, the big three—which presumably I don't need to define out—how do you wind up ranking them in terms of security from top to bottom? I have my own rankings that I like to dole out and basically, this is the, let's offend someone at every one of these companies, no matter how we wind up playing it. Because I will argue with you just on principle on them. How do you view them stacking up against each other?Tim: So, an interesting view on that is based on who's been around longest and who is encountered of the most technical debt. A lot of these security vulnerabilities or security concerns may have had to deal with a decision made long ago that might have made sense at the time and now the company has kind of stuck with that particular technology or decision or framework, and are now having to build or apply security Band-Aids to that process until it gets resolved. I would say, ironically, AWS is actually at the top of having that technical debt, and actually has so many different types of access policies that are very complex to configure and not very user intuitive unless you speak intuitively JSON or YAML or some other markdown language, to be able to tell you whether or not something was actually set up correctly. Now, there are a lot of security experts who make their money based on knowing how to configure or be able to assess whether or not these are actually the issue. I would actually bring them as, by default, by design, between the big three, they're actually on the lower end of certain—based on complexity and easy-to-configure-wise.The next one that would also go into that pile, I would say is probably Microsoft Azure, who [sigh] admittedly, decided to say that, “Okay, let's take something that was very complicated and everyone really loved to use as an identity provider, Active Directory, and try to use that as a model for.” Even though they made it extensively different. It is not the same as on-prem directory, but use that as the framework for how people wanted to configure their identity provider for a new cloud provider. The one that actually I would say, comes out on top, just based on use and based on complexity might be Google Cloud. They came to a lot of these security features first.They're acquiring new companies on a regular basis with the acquisition of Mandiant, the creation of their own security tooling, their own unique security approaches. In fact, they probably wrote the book on Kubernetes Security. Would be on top, I guess, from usability, such as saying that I don't want to have to manage all these different types of policies. Here are some buttons I would like to flip and I'd like my resources, for the most part by default, to be configured correctly. And Google does a pretty good job of that.Also, one of the things they do really well is entity-based role assumption, which inside of AWS, you can provide access keys by default or I have to provide a role ID after—or in Azure, I'm going to say, “Here's a [unintelligible 00:19:34] policy for something specific that I want to grant access to a specific resource.” Google does a pretty good job of saying that okay, everything is treated as an email address. This email address can be associated in a couple of different ways. It can be given the following permissions, it can have access to the following things, but for example, if I want to remove access to something, I just take that email address off of whatever access policy I had somewhere, and then it's taken care of. But they do have some other items such as their design of least privilege is something to be expected when you consider their hierarchy.I'm not going to say that they're not without fault in that area—in case—until they had something more recently, as far as finding certain key pieces of, like say, tags or something within a specific sub-project or in our hierarchy, there were cases where you might have granted access at a higher level and that same level of access came all the way down. And where at least privilege is required to be enforced, otherwise, you break their security model. So, I like them for how simple it is to set up security at times, however, they've also made it unnecessarily complex at other times so they don't have the flexibility that the other cloud service providers have. On the flip side of that, the level of flexibility also leads to complexity at times, which I also view as a problem where customers think they've done something correctly based on their best knowledge, the best of documentation, the best and Medium articles they've been researching, and what they have done is they've inadvertently made assumptions that led to core anti-patterns, like, [unintelligible 00:21:06] what they've deployed.Corey: This episode is sponsored in part by our friends at Uptycs, because they believe that many of you are looking to bolster your security posture with CNAPP and XDR solutions. They offer both cloud and endpoint security in a single UI and data model. Listeners can get Uptycs for up to 1,000 assets through the end of 2023 (that is next year) for $1. But this offer is only available for a limited time on UptycsSecretMenu.com. That's U-P-T-Y-C-S Secret Menu dot com.Corey: I think you're onto something here, specifically in—well, when I've been asked historically and personally to rank security, I have viewed Google Cloud as number one, and AWS is number two. And my reasoning behind that has been from an absolute security of their platform and a pure, let's call it math perspective, it really comes down to which of the two of them had what for breakfast on any given day there, they're so close on there. But in a project that I spin up in Google Cloud, everything inside of it can talk to each other by default and I can scope that down relatively easily, whereas over an AWS land, by default, nothing can talk to anything. And that means that every permission needs to be explicitly granted, which in an absolutist sense and in a vacuum, yeah, that makes sense, but here in reality, people don't do that. We've seen a number of AWS blog posts over the last 15 years—they don't do this anymore—but it started off with, “Oh, yeah, we're just going to grant [* on * 00:22:04] for the purposes of this demo.”“Well, that's horrible. Why would you do that?” “Well, if we wanted to specify the IAM policy, it would take up the first third of the blog post.” How about that? Because customers go through that exact same thing. I'm trying to build something and ship.I mean, the biggest lie in any environment or any codebase ever, is the comment that starts with, “To do.” Yeah, that is load-bearing. You will retire with that to do still exactly where it is. You have to make doing things the right way at least the least frictionful path because no one is ever going to come back and fix this after the fact. It's never going to happen, as much as we wish that it did.Tim: At least until after the week of the breach when it was highlighted by the security team to say that, “Hey, this was the core issue.” Then it will be fixed in short order. Usually. Or a Band-Aid is applied to say that this can no longer be exploited in this specific way again.Corey: My personal favorite thing that, like, I wouldn't say it's a lie. But the favorite thing that I see in all of these announcements right after the, “Your security is very important to us,” right after it very clearly has not been sufficiently important to them, and they say, “We show no signs of this data being accessed.” Well, that can mean a couple different things. It can mean, “We have looked through the audit logs for a service going back to its launch and have verified that nothing has ever done this except the security researcher who found it.” Great. Or it can mean, “What even are logs, exactly? We're just going to close our eyes and assume things are great.” No, no.Tim: So, one thing to consider there is in that communication, that entire communication has probably been vetted by the legal department to make sure that the company is not opening itself up for liability. I can say from personal experience, when that usually has occurred, unless it can be proven that breach was attributable to your user specifically, the default response is, “We have determined that the security response of XYZ item or XYZ organization has determined that your data was not at risk at any point during this incident.” Which might be true—and we're quoting Star Wars on this one—from a certain point of view. And unfortunately, in the case of a post-breach, their security, at least from a regulation standpoint where they might be facing a really large fine, is absolutely probably their top priority at this very moment, but has not come to surface because, for most organizations, until this becomes something that is a financial reason to where they have to act, where their reputation is on the line, they're not necessarily incentivized to fix it. They're incentivized to push more products, push more features, keep the clients happy.And a lot of the time going back and saying, “Hey, we have this piece of technical debt,” it doesn't really excite our user base or doesn't really help us gain a competitive edge in the market is considered an afterthought until the crisis occurs and the information security team rejoices because this is the time they actually get to see their stuff fixed, even though it might be a super painful time for them in the short run because they get to see these things fixed, they get to see it put to bed. And if there's ever a happy medium, where, hey, maybe there was a legacy feature that wasn't being very well taken care of, or maybe this feature was also causing the security team a lot of pain, we get to see both that feature, that item, that service, get better, as well as security teams not have to be woken up on a regular basis because XYZ incident happened, XYZ item keeps coming up in a vulnerability scan. If it finally is put to bed, we consider that a win for all. And one thing to consider in security as well as kind of, like, we talk about the relationship between the developers and security and/or product managers and security is if we can make it a win, win, win situation for all, that's the happy path that we really want to be getting to. If there's a way that we can make sure that experience is better for customers, the security team doesn't have to be broken up on a regular basis because an incident happened, and the developers receive less friction when they want to go implement something, you find that that secure feature, function, whatever tends to be the happy path forward and the path of least resistance for everyone around it. And those are sometimes the happiest stories that can come out of some of these incidents.Corey: It's weird to think of there being any happy stories coming out of these things, but it's definitely one of those areas that there are learnings there to be had if we're willing to examine them. The biggest problem I see so often is that so many companies just try and hide these things. They give the minimum possible amount of information so the rest of us can't learn by it. Honestly, some of the moments where I've gained the most respect for the technical prowess of some of these cloud providers has been after there's been a security issue and they have disclosed either their response or why it was a non-issue because they took a defense-in-depth approach. It's really one of those transformative moments that I think is an opportunity if companies are bold enough to chase them down.Tim: Absolutely. And in a similar vein, when we think of certain cloud providers outages and we're exposed, like, the major core flaw of their design, and if it kept happening—and again, these outages could be similar and analogous to an incident or a security flaw, meaning that it affected us. It was something that actually happened. In the case of let's say, the S3 outage of, I don't know, it was like 2017, 2018, where it turns out that there was a core DNS system that inside of us-east-1, which is actually very close to where I live, apparently was the core crux of, for whatever reason, the system malfunctioned and caused a major outage. Outside of that, in this specific example, they had to look at ways of how do we not have a single point of failure, even if it is a very robust system, to make sure this doesn't happen again.And there was a lot of learnings to be had, a lot of in-depth investigation that happened, probably a lot of development, a lot of research, and sometimes on the outside of an incident, you really get to understand why a system was built a certain way or why a condition exists in the first place. And it sometimes can be fascinating to kind of dig into that very deeper and really understand what the core problem is. And now that we know what's an issue, we can actually really work to address it. And sometimes that's actually one of the best parts about working at Praetorian in some cases is that a lot of the items we find, we get to find them early before it becomes one of these issues, but the most important thing is we get to learn so much about, like, why a particular issue is such a big problem. And you have to really solve the core business problem, or maybe even help inform, “Hey, this is an issue for it like this.”However, this isn't necessarily all bad in that if you make these adjustments of these items, you get to retain this really cool feature, this really cool thing that you built, but also, you have to say like, here's some extra, added benefits to the customers that you weren't really there. And—such as the old adage of, “It's not a bug, it's a feature,” sometimes it's exactly what you pointed out. It's not necessarily all bad in an incident. It's also a learning experience.Corey: Ideally, we can all learn from these things. I want to thank you for being so generous with your time and talking about how you view this increasingly complicated emerging space. If people want to learn more, where's the best place to find you?Tim: You can find me on LinkedIn which will be included in this podcast description. You can also go look at articles that the team is putting together at praetorian.com. Unfortunately, I'm not very big on Twitter.Corey: Oh, well, you must be so happy. My God, what a better decision you're making than the rest of us.Tim: Well, I like to, like, run a little bit under the radar, except on opportunities like this where I can talk about something I'm truly passionate about. But I try not to pollute the airwaves too much, but LinkedIn is a great place to find me. Praetorian blog for stuff the team is building. And if anyone wants to reach out, feel free to hit the contact page up in praetorian.com. That's one of the best places to get my attention.Corey: And we will, of course, put links to that in the [show notes 00:30:19]. Thank you so much for your time. I appreciate it. Tim Gonda, Technical Director of Cloud at Praetorian. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment talking about how no one disagrees with you based upon a careful examination of your logs.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.