Podcasts about corey is

Dave Mangot, CEO and founder of Mangoteque, joins Coreyon Screaming in the Cloud to explain how leveraging DevOps improves the lives of engineers and results in stronger businesses. Dave talks about the importance of exclusively working for private equity firms that act ethically, the key difference between venture capital and private equity, and how conveying issues and ideas to your CEO using language he understands leads to faster results. Corey and Dave discuss why successful business are built on two things: infrastructure as code and monitoring.About DaveDave Mangot, author of DevOps Patterns for Private Equity, helps portfolio companies get good at delivering software.  He is a leading consultant, author, and speaker as the principal at Mangoteque.  A DevOps veteran, Dave has successfully led digital, SRE, and DevOps transformations at companies such as Salesforce, SolarWinds, and Cable & Wireless. He has a proven track record of working with companies to quickly mature their existing culture to improve the speed, frequency, and resilience of their software service delivery.Links Referenced: Mangoteque: https://www.mangoteque.com DevOps Patterns for Private Equity: https://www.amazon.com/DevOps-Patterns-Private-Equity-organization/dp/B0CHXVDX1K “How to Talk Business: A Short Guide for Tech Leaders”: https://itrevolution.com/articles/how-to-talk-business-a-short-guide-for-tech-leaders/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is someone that I have known for, well, longer than I've been doing this show. Dave Mangot is the founder and CEO at Mangoteque. Dave, thank you for joining me.Dave: Hey, Corey, it's great to be here. Nice to see you again.Corey: I have to say, your last name is Mangot and the name of your company is Mangoteque, spelled M-A-N-G-O-T-E-Q-U-E, if I got that correctly, which apparently I did. What an amazing name for a company. How on earth did you name a company so well?Dave: Yeah, I don't know. I have to think back, a few years ago, I was just getting started in consulting, and I was talking to some friends of mine who were giving me a bunch of advice—because they had been doing consulting for quite some time—about what my rates should be, about all kinds of—you know, which vendors I should work with for my legal advice. And I said, “I'm having a lot of trouble coming up with a name for the company.” And this guy, Corey Quinn, was like, “Hey, I got a name for you.” [laugh].Corey: I like that story, just because it really goes to show the fine friends of mine over at all of the large cloud services companies—but mostly AWS—that it's not that hard to name something well. The trick, I think, is just not to do it in committee.Dave: Yeah. And you know, it was a very small committee obviously of, like, three. But yeah, it's been great. I have a lot of compliments on the name of my company. And I was like, oh, “You know that guy, the QuinnyPig dude?” And they're like, “Yeah?” “Oh, yeah, it was—that was his idea.” And I liked it. And it works really well for the things that I do.Corey: It seems to. So, talk to you about what it is that you do because back when we first met and many, many years ago, you were an SRE manager at a now defunct observability company. This was so long ago, I don't think that they used the term observability. It was Librato, which, “What do you do?” “We do monitoring,” back when that didn't sound like some old-timey thing. Like, “Oh, yeah. Right, between the blacksmith and the cobbler.” But you've evolved significantly since you were doing the mundane, pedestrian tasks of keeping the service up and running. What do you do these days?Dave: Yeah, that was before the observability wars [laugh] [whatever you like 00:02:55] to call it. But over time, that company was owned by SolarWinds and I wound up being responsible for all the SolarWinds cloud company SRE organizations. So, started—ran a global organization there. And they were owned by a couple of private equity firms. And I got to know one of the firms rather well, and then when I left SolarWinds, I started working with private equity firm portfolio companies, especially software investments. And what I like to say is I teach people how to get good at delivering software.Corey: So, you recently wrote a book, and I know this because I make it a point to get a copy of the book—usually by buying it, but you beat me to it by gifting me one—of every guest I have on the show who's written a book. Sometimes that means I wind up with the eclectic collections of poetry, other times, I wind up with a number of different books around the DevOps and cloud space. And one of these days, I'm going to wind up talking to someone who wound up writing an encyclopedia or something, to where I have to back the truck around. But what I wanted to ask is about your title, of all things. It's called DevOps Patterns for Private Equity. And I have to ask, what makes private equity special?Dave: I think as a cloud economist, what you also just told me, is you owe me $17.99 for the book because it was gifted.Corey: Is that how expensive books are these days? My God, I was under the impression once you put the word ‘DevOps' in the title, that meant you're above 40 bucks, just as, you know, entrance starting fees here.Dave: I think I need to talk to my local cloud economist on how to price things. Yeah, the book is about things that I've basically seen at portfolio companies over the years. The thing about, you know, why private equity, I think it would be one question, just because I've been involved in the DevOps movement since pretty much the start, when John Willis calls me a DevOps OG, which I think is a compliment. But the thing that I like about working with private equity, and more specifically, private equity portfolio companies is, like I wrote in the book, they're serious. And serious means that they're not afraid to make a big investment, they're not afraid to change things quickly, they're not afraid to reorganize, or rethink, or whatever because a lot of these private equity firms have, how they describe it as a three to five year investment thesis. So, in three to five years, they want to have some kind of an exit event, which means that they can't just sit around and talk about things and try it and see what happens—Corey: In the fullness of time, 20 years from now. Yeah, it doesn't work that well. But let's back up a little bit here because something that I have noticed over the years is that, especially when it comes to financial institutions, the general level of knowledge is not terrific. For a time, a lot of people were very angry at Goldman Sachs, for example. But okay, fair enough. What does Goldman Sachs do? And the answer was generally incoherent.And again, I am in no way, shape or form, different from people who form angry opinions without having all of the facts. I do that myself three times before breakfast. My last startup was acquired by BlackRock, and I was the one that raised our hand internally, at the 40-person company when that was announced, as everyone was sort of sitting there stunned: “What's a BlackRock?” Because I had no idea. Well, for the next nine months, I assure you, I found out what a BlackRock is. But what is private equity? Because I see a lot of them getting beaten up for destroying companies. Everyone wants to bring up the Toys-R-Us story as a for instance. But I don't get the sense that that is the full picture. Tell me more.Dave: Yes. So, I'm probably not the best spokesperson for private equity. But—Corey: Because you don't work for a private equity firm, you only work with them, that makes you a terrific spokesperson because you're not [in 00:06:53] this position of, “Well, justify what your company does here,” situation, there's something to be said for objectivity.Dave: So, you know, like I wrote in the book, there are approximately 10,000 private equity firms in the United States. They are not all going to be ethical. That is just not a thing. I choose to work with a specific segment of private equity companies, and these private equity companies want to make a good business. That's what they're going for.And you and I, having had worked at many companies in our careers, know that there's a lot of companies out there that aren't a good business. You're like, “Why are we doing this? This doesn't make any sense. This isn't a good investment. This”—there's a lot of things and what I would call the professional level private equity firms, the ones at the top—and not all of them at the top are ethical, don't get me wrong; I have a blacklist here of companies I won't work for. I will not say who those companies are.Corey: I am in the same boat. I think that anyone who works in an industry at all and doesn't have a list of companies that they would not do business with, is, on some level, either haven't thought it through, hasn't been in business long enough, or frankly, as long as you're paying them, everything you can do is a-okay. And you know, I'm not going to sit here and say that those are terrible people, but I never wanted to do that soul-searching. I always thought the only way to really figure out where you stand is to figure it out in advance before there's money on the table. Like, do you want to go do contracting for a defense company? Well no, objectively, I don't, but that's a lot harder to say when they're sitting on the table with $20 million in front of you of, “Do you want to work with a defense company?” Because you can rationalize your way into anything when the stakes are high enough. That's where I've always stood on it. But please, continue.Dave: I'd love to be in that situation to turn down $20 million [laugh].Corey: Yeah, that's a hard situation to find yourself in, right?Dave: But regardless, there's a lot of different kinds of private equity firms. Generally the firms that I work with, they all want—not generally; the ones I work with want to make better companies. I have had operating partners at these companies tell me—because this always comes up with private equity—there's no way to cut your way to a good company. So, the private equity firms that I work with invest in these companies. Do they sell off unprofitable things? Of course they do. Do they try to streamline some things sometimes so that the company is only focused on X or Y, and then they tuck other companies into it—that's called a buy and build strategy or a platform strategy—yes. But the purpose of that is to make a better company.The thing that I see a lot of people in our industry—meaning, like, us tech kind of folks—get confused about is what the difference is between venture capital and private equity. And private equity, in general, is the thing that is the kind of financing that follows on after venture capital. So, in venture capital, you are trying to find product-market fit. The venture capitalists are putting all their bets down like they're in Vegas at re:Invent, and trying to figure out which bet is going to pay off, but they have no expectation that all of the bets are going to pay off. With private equity, the companies have product-market fit, they're profitable. If they're not profitable, they have a very clear line to profitability.And so, what these private equity firms are trying to do, no matter what the size of the company is, whether it's a 50-person company or a 5000-person company, they're trying to get these companies up to another level so that they're more profitable and more valuable, so that either a larger fish will gobble them up or they'll go out on the public markets, like onto the stock market, those kinds of things, but they're trying to make a company that's more valuable. And so, not everything looks so good [laugh] when you're looking at it from the outside, not understanding what these people are trying to do. That's not to say they're not complete jerks who are in private equity because there are.Corey: Because some parts are missing. Kidding. Kidding. Kidding.Dave: [laugh].Corey: It's a nuanced area, and it's complicated, just from the perspective of… finance is deceptively complicated. It looks simple, on some level, because on some level, you can always participate in finance. I have $10. I want to buy a thing that costs $7. How does that work? But it gets geometrically more complex the further you go. Financial engineering is very much a thing.And it is not at all obvious how those things interplay with different dynamics. One of the private equity outcomes, as you alluded to a few minutes ago, is the idea that they need to be able to rapidly effect change. It becomes a fast turnaround situation, and then have an exit event of some kind. So, the DevOps patterns that you write about are aligned with an idea of being effective, presumably, rather than, well, here's how you slowly introduce a sweeping cultural mindset shift across the organization. Like, that's great, but some of us don't have that kind of runway for what we're trying to achieve to be able to pull that off. So, I'm assuming that a lot of the patterns you talk about are emphasizing rapid results.Dave: Well, I think the best way to describe this, right, is what we've talked about is they want to make a better company. And for those of us who have worked in the DevOps movement for all these years, what's one great way of making a better company? Adopting DevOps principles, right? And so, for me, one of the things I love about my job is I get to go in and make engineers' lives better. No more working on weekends, no more we're only going to do deployments at 11 o'clock at night, no more we're going to batch things up and ship them three or four times a year, which all of us who've done DevOps stuff for years know, like, fastest way to have a catastrophe is batch up as many things as possible and release them all at once.So like, for me, I'm going in making engineers' lives better. When their lives are better, they produce better results because they're not stressed out, they're not burned out, they get to spend time with their families, all those kinds of things. When they start producing better results, the executives are happier. The executives can go to the investors and show all the great results they're getting, so the investors are happier. So, for me, I always say, like, I'm super lucky because I have a job that's win, win, win.And like, I'm helping them to make a better company, I'm helping them to ship faster, I'm helping them do things in the cloud, I'm helping them get more reliability, which helps them retain customers, all these things. Because we know from the—you know, remember the 2019 State of DevOps Report: highest performers are twice as likely to meet or exceed their organization's performance goals, and those can be customer retention, revenue, whatever those goals are. And so, I get to go in and help make a better company because I'm making people's lives better and, kind of, everybody wins. And so, for me, it's super rewarding.Corey: That's a good way of framing it. I have to ask, since the goal for private equity, as you said, is to create better companies, to effectively fix a bunch of things that, for better or worse, had not been working optimally. Let me ask the big, dumb, naive question here. Isn't that ostensibly the goal of every company? Now, everyone says it's their goal, but whether that is their goal or not, I think, is a somewhat separate question.Dave: Yeah. I—that should be the goal of every company, I agree. There are people who read my book and said, “Hey, this stuff applies far beyond private equity.” And I say, “Yeah, it absolutely does.” But there are constraints—[gold rat 00:15:10]—within private equity, about the timing, about the funding, about whatever, to get the thing to another level. And that's an interesting thing that I've seen is I've seen private equity companies take a company up to another level, have some kind of exit event, and then buy that company again years later. Which, like, what? Like, how could that be?Corey: I've seen that myself. It feels, on some level, like that company goes public, and then goes private, then goes public, then goes private to the same PE firm, and it's like, are you really a PE company or are you just secretly a giant cat, perpetually on the wrong side of a door somewhere?Dave: But that's because they will take it to a level, the company does things, things happen out in the market, and then they see another opportunity to grow them again. Where in a regular company—in theory—you're going to want to just get better all the time, forever. This is the Toyota thesis about continual improvement.Corey: I am curious as far as what you are seeing changing in the market with the current macroeconomic conditions, which is a polite way to say the industry going wonky after ten years of being relatively up and to the right.Dave: Yeah, well, I guess the fun thing is, we have interest rates, we had a pandemic, we had [laugh], like, all this exciting stuff. There's, you know, massive layoffs, [unintelligible 00:16:34] and then all this, kind of like, super churn-y things. I think the fun thing for me is, I went to a private equity conference in San Francisco, I don't know, a month ago or something like that, and they had all these panelists on stage pontificating about this and that and the other thing, and one of the women said something that I thought was really great, especially for someone like me. She said, “The next five to ten years in private equity are going to be about growth and operational efficiency.” And I was like, “That's DevOps. That's awesome.” [laugh].That really works well for me because, like, we want to have people twice as likely to meet or exceed their organization's performance goals. That's growth. And we want operational efficiency, right? Like, stop manually copying files around, start putting stuff in containers, do all these things that enable us to go fast speed and also do that with high quality. So, if the next five to ten years are going to be about growth and operational efficiency, I think it's a great opportunity for people to take in a lot of these DevOps principles.And so, the being on the Screaming in the Cloud podcast, like, I think cloud is a huge part of that. I think that's a big way to get growth and operational efficiency. Like, how better to be able to scale? How better to be able to Deming's PDSA cycle, right—Plan, Do, Study, Act—how better to run all these experiments to find out, like, how to get better, how to be more efficient, how to meet our customers' demands. I think that's a huge part of it.Corey: That is, I think, a very common sentiment as far as how folks are looking at things from a bigger picture these days. I want to go back as well to something you said earlier that I was joking around at the start of the episode about, “Wow, what an amazing name for the company. How did you come up with it?” And you mentioned that you had been asking a bunch of people for advice—or rather, you mentioned you had gotten advice from people. I want to clarify, you were in fact asking. I wasn't basically the human form of Clippy popping up, “It looks like you're starting a business. Let me give you unsolicited advice on what you should be doing.”What you've done, I think, is a terrific example of the do what I say not what I do type of problem, where you have focused on your positioning on a specific segment of the market: private equity firms and their portfolio companies. If I had been a little bit smarter, I would have done something similar in my own business. I would fix AWS bills for insurance companies in the Pacific Northwest or something like that, where people can hear the type of company they are reflected in the name of what it is that you do. I was just fortunate enough or foolish enough to be noisy enough in order to talk about what I do in a way that I was able to overcome that. But targeting the way that you have, I think is just so spot on. And it's clearly working out for you.Dave: I think a Corey Quinn Clippy would be very distracting in [laugh] my Microsoft Word, first of all [laugh]. Second of all—Corey: They're calling it Copilot now.Dave: [laugh]—there's this guy Corey and his partner Mike who turned me on to this guy, Jonathan Stark, who has his theory about your business. He calls it, like, elucidating, like, a Rolodex moment. So, if somebody's talking about X or Y, and they say, “Oh, yeah. You want to talk to Corey about that.” Or, “You want to talk to Mike about that.”And so, for me, working with private equity portfolio companies, that's a Rolodex moment. When people are like, “I'm at a portfolio company. We just got bought. They're coming in, and they want to understand what our spend is on the cloud, and this and that. Like, I don't know what I'm supposed to do here.” A lot of times people think of me because I tend to work on those kinds of problems. And so, it doesn't mean I can't work on other things, and I definitely do work on other things, I've definitely worked with companies that are not owned by private equity, but for me, that's really a place that I enjoy working, and thankfully, I get Rolodex moments from those things.Corey: That's the real value that I've found. The line I've heard is always it's not just someone at a party popping up and saying, “Oh, yeah, I have that problem.” But, “Oh, my God, you need to talk to this person I know who has that problem.” It's the introduction moment. In my case at least, it became very hard for me to find people self-identifying as having large AWS bills, just because, yeah, individual learners or small startup founders, for example, might talk about it here and there, but large companies do not tend to complain about that in Twitter because that tends to, you know, get them removed from their roles when they start going down that path. Do you find that it is easier for you to target what you do to people because it's easier to identify them in public? Because I assure you, someone with a big AWS bill is hard to spot out of a crowd.Dave: Well, I think you need to meet people where they are, I think is probably the best way of saying that. So, if you are—and this isn't something I need to explain to you, obviously, so this is more for your listeners, but like, if you're going to talk about, “Hey, I'm looking for companies with large AWS bills,” [pthhh] like that's, maybe kind of whatever. But if you say, “Hey, I want to improve your margins and your operational efficiencies,” all of a sudden, you're starting to speak their language, right? And that language is where people start to understand that, “Hey, Corey's talking about me.”Corey: A large part of how I talk about this was shaped by some of the early conversations I had. The way that I think about this stuff and the way that I talk is not necessarily what terms my customers use. Something that I found that absolutely changed my approach was having an investigative journalist—or a former investigative journalist, in this case—interview people I'd worked with to get case studies and testimonials from them. But what she would also do was get the exact phrasing that they use to describe the value that I did, and how they talked about what we'd done. Because that became something that was oh, you're effectively writing the rough draft of my marketing copy when you do that. Speaking in the language of your customer is so important, and I meet a lot of early-stage startups that haven't quite unlocked that bit of insight yet.Dave: And I think looking at that from a slightly different perspective is also super important. So, not only speaking the language of your customer, but let's say you're not a consultant like me or you. Let's say you work inside of a company. You need to learn to speak the language of business, right? And this is, like, something I wrote about in the beginning of the book about the guy in San Francisco who got locked up for not giving away the Cisco passwords, and Gavin Newsom had to go to his jail cell and all this other crazy stuff that happened is, technologists often think that the reason that they go to work is to play with technology. The reason we go to work is to enable the business.And—so shameless plug here I—wrote a paper that came out, like, two months ago with IT Revolution—so the people who do The Phoenix Project, and Accelerate, and The DevOps Handbook, and all that other stuff, I wrote this paper with, like, Courtney Kissler, and Paul Gaffney, and Scott Nasello, and a whole bunch of amazing technologists, but it's about speaking the language of business. And as technologists, if we want to really contribute and feel like the work that we're doing is contributing and valuable, you need to start understanding how those other people are talking. So, you and I were just talking about, like, operational efficiencies, and margins, and whatever. What is all that stuff? And figuring that out and being able to have that conversation with your CEO or whoever, those are the things that get people to understand exactly what you're trying to do, and what you're doing, and why this thing is so important.I talk to so many engineers that are like, “Ah, I talked to management and they just don't understand, and [da-dah].” Yeah, they don't understand because you're speaking technology language. They don't want to hear about, like, CNCF compliant this, that, and the—that doesn't mean anything to them. You need to understand in their lang—talk to them and their language and say like, “Hey, this is why this is good for the business.” And I think that's a really important thing for people to start to learn.Corey: So, a question that I have, given that you have been doing this stuff, I think, longer than I have, back when cloud wasn't really a thing, and then it was a thing, but it seemed really irresponsible to do. And then it went through several more iterations to the point where now it's everywhere. What's your philosophy of cloud?Dave: So, I'll go back to something that just came out, the 2023 State of DevOps Report just came out. I follow those things pretty closely. One of the things they talked about in the paper is one of the key differentiators to get your business to have what they call high organizational performance—again, this [laugh] is going back to business talk again—is what they call infrastructure flexibility. And I just don't think you can get infrastructure flexibility if you're not in the cloud. Can you do it? Absolutely.You know, back over a decade ago, I built out a bunch of stuff in a data center on what I called cloud principles. We could shoot things in the head, get new ones back, we did all kinds of things, we identified SKUs of, like, what kind of classes of machines we had. All that looks like a lot of stuff that you would just do in AWS, right? Like, I know, my C instances are compute. I know my M instances are memory. Like, they're all just SKUs, right?Corey: Yeah, that changed a little bit now to the point where they have so many different instance families that some of their names look like dumps of their firmware.Dave: [laugh]. That is probably true. But like, this idea that, like, I want to have this infrastructure flexibility isn't just my idea that it's going to turn out well. Like, the State of DevOps Report kind of proves it. And so, for me, like, I go back to some of the principles of the DevOps movement, and like, if you look at the DORA metrics, let's say you've got deployment frequency and lead time for changes. That's speed: how fast can I do something? And you've got time-to-recover, and you've got change failure rate. That's quality: how much can I ship without having problems, and how fast can I recover when I do?And I think this is one of the things I teach to a lot of my clients about moving into the cloud. If you want to be successful, you have to deliver with speed and quality. Speed: Infrastructure as Code, full stop. If I want to be able to go fast, I need to be able to destroy an environment, bring a new environment up, I need to be able to do that in minutes. That's speed.And then the second requirement, and the only other requirement, is build monitoring in from the start. Everything gets monitored. And that's quality. Like, if I monitor stuff, I know when I've deployed something that's spiking CPU. If that's monitored, I know that this thing is costing me a hell of a lot more than other things. I know all this stuff. And I can do capacity planning, I can do whatever the heck I want. But those are the two fundamental things: Infrastructure as Code and monitoring.And yes, like you said, I worked at a monitoring or observability company, so perhaps I'm slightly biased, but what I've seen is, like, companies that adopt those two principles, and everything else comes from that—so all my Kubernetes stuff and all those other things are not at odds with those principles—those are the people who actually wind up doing really well. And I think those are the people that have—State of DevOps Report—infrastructure flexibility, and that enables them to have higher organizational performance.Corey: I think you're onto something. Like, I still remember the days of having to figure out the number of people who you had in your ops team versus how many servers they could safely and reasonably run. And now that question has little, if any, meaning. If someone asked me, “Okay, so we're running right now 10,000 instances in our cloud environment. How many admins should it take us to run those?” The correct response is, “How the heck are you running those things?” Like, tell me more because the answer is probably terrifying. Because right now, if you do that correctly, it's you want to make a change to all of them or some subset of them? You change a parameter somewhere and computers do the heavy lifting.Dave: Yeah, I ran a content delivery network for cable and wireless. We had three types of machines. You know, it was like Windows Media Server and some squid-cache thing or whatever. And it didn't matter how many we had. It's all the same. Like, if I had 10,000 and I had 50,000, it's irrelevant. Like, they're all the same kind of crap. It's not that hard to manage a bunch of stuff that's all the same.If I have 10,000 servers and each one is a unique, special snowflake because I'm running in what I call a hosted configuration, I have 10,000 customers, therefore I have 10,000 servers, and each of them is completely different than the other, then that's going to be a hell of a lot harder to manage than 10,000 things that the load balancer is like [bbbrrrp bbbrrrp] [laugh] like, just lay it out. So, it's sort of a… kind of a nonsense question at this point. Like you're saying, like, it doesn't really matter how many. It's complexity. How much complexity do I have? And as we all say, in the DevOps movement, complexity isn't free. Which I'll bet is a large component of how you save companies money with The Duckbill Group.Corey: It goes even beyond that because cloud infrastructure is always less expensive than the people working on it, unless you do something terrifying. Otherwise, everything should be running an EC2 instances. Nothing higher-level built on top of it because if people's time is free, the cheapest thing you're going to get is a bunch of instances. The end. That is not really how you should be thinking about this.Dave: [laugh]. I know a lot of private equity firms that would love to find a place where time was free [laugh]. They could make a lot of money.Corey: Yeah. Pretty sure that the biggest—like, “What's your biggest competitive headwind?” You know [laugh], “Wage laws.” Like it doesn't work that way. I'm sorry, but it doesn't [laugh].I really want to thank you for taking the time to talk to me about what you're up to, how things are going over in your part of the universe. If people want to learn more, where's the best place for them to go to find you?Dave: They can go to mangoteque.com. I've got all the links to my blog, my mailing list. Definitely, if you're interested in this intersection of DevOps and private equity, sign up for the mailing list. For people who didn't get Corey's funky spelling of my last name, it is a play on the fact that it is French and I also work with technology companies. So, it's M-A-N-G-O-T-E-Q-U-E dot com.If you type that in—Mangoteque—to any search engine, obviously, you will find me. I am not difficult to find on the internet because I've been doing this for quite some time. But thank you for having me on the show. It's always great to catch up with you. I love hearing about what you're doing. I super appreciate you're asking me about the things that I'm working on, and you know, been a big help.Corey: No, it's deeply fascinating. It's neat to watch you continue to meet your market in a variety of different ways. Dave Mangot, CEO and founder of Mangoteque, which is excellently named. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this episode, please leave a five-star review on your podcast platform of choice, along with an angry comment almost certainly filled with incoherent screaming because you tuned out just as soon as you heard the words ‘private equity.'Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business, and we get to the point. Visit duckbillgroup.com to get started.

Seif Lotfy, Co-Founder and CTO at Axiom, joins Corey on Screaming in the Cloud to discuss how and why Axiom has taken a low-cost approach to event data. Seif describes the events that led to him helping co-found a company, and explains why the team wrote all their code from scratch. Corey and Seif discuss their views on AWS pricing, and Seif shares his views on why AWS doesn't have to compete on price. Seif also reveals some of the exciting new products and features that Axiom is currently working on. About SeifSeif is the bubbly Co-founder and CTO of Axiom where he has helped build the next generation of logging, tracing, and metrics. His background is at Xamarin, and Deutche Telekom and he is the kind of deep technical nerd that geeks out on white papers about emerging technology and then goes to see what he can build.Links Referenced: Axiom: https://axiom.co/ Twitter: https://twitter.com/seiflotfy TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted guest episode is brought to us by my friends, and soon to be yours, over at Axiom. Today I'm talking with Seif Lotfy, who's the co-founder and CTO of Axiom. Seif, how are you?Seif: Hey, Corey, I am very good, thank you. It's pretty late here, but it's worth it. I'm excited to be on this interview. How are you today?Corey: I'm not dead yet. It's weird, I see you at a bunch of different conferences, and I keep forgetting that you do in fact live half a world away. Is the entire company based in Europe? And where are you folks? Where do you start and where do you stop geographically? Let's start there. We over—everyone dives right into product. No, no, no. I want to know where in the world people sit because apparently, that's the most important thing about a company in 2023.Seif: Unless you ask Zoom because they're undoing whatever they did. We're from New Zealand, all the way to San Francisco, and everything in between. So, we have people in Egypt and Nigeria, all around Europe, all around the US… and UK, if you don't consider it Europe anymore.Corey: Yeah, it really depends. There's a lot of unfortunate naming that needs to get changed in the wake of that.Seif: [laugh].Corey: But enough about geopolitics. Let's talk about industry politics. I've been a fan of Axiom for a while and I was somewhat surprised to realize how long it had been around because I only heard about you folks a couple of years back. What is it you folks do? Because I know how I think about what you're up to, but you've also gone through some messaging iteration, and it is a near certainty that I am behind the times.Seif: Well, at this point, we just define ourselves as the best home for event data. So, Axiom is the best home for event data. We try to deal with everything that is event-based, so time-series. So, we can talk metrics, logs, traces, et cetera. And right now predominantly serving engineering and security.And we're trying to be—or we are—the first cloud-native time-series platform to provide streaming search, reporting, and monitoring capabilities. And we're built from the ground up, by the way. Like, we didn't actually—we're not using Parquet [unintelligible 00:02:36] thing. We're completely everything from the ground up.Corey: When I first started talking to you folks a few years back, there were two points to me that really stood out, and I know at least one of them still holds true. The first is that at the time, you were primarily talking about log data. Just send all your logs over to Axiom. The end. And that was a simple message that was simple enough that I could understand it, frankly.Because back when I was slinging servers around and you know breaking half of them, logs were effectively how we kept track of what was going on, where. These days, it feels like everything has been repainted with a very broad brush called observability, and the takeaway from most company pitches has been, you must be smarter than you are to understand what it is that we're up to. And in some cases, you scratch below the surface and realize it no, they have no idea what they're talking about either and they're really hoping you don't call them on that.Seif: It's packaging.Corey: Yeah. It is packaging and that's important.Seif: It's literally packaging. If you look at it, traces and logs, these are events. There's a timestamp and just data with it. It's a timestamp and data with it, right? Even metrics is all the way to that point.And a good example, now everybody's jumping on [OTel 00:03:46]. For me, OTel is nothing else, but a different structure for time series, for different types of time series, and that can be used differently, right? Or at least not used differently but you can leverage it differently.Corey: And the other thing that you did that was interesting and is a lot, I think, more sustainable as far as [moats 00:04:04] go, rather than things that can be changed on a billboard or whatnot, is your economic position. And your pricing has changed around somewhat, but I ran a number of analyses on your cost that you were passing on to customers and my takeaway was that it was a little bit more expensive to store data for logs in Axiom than it was to store it in S3, but not by much. And it just blew away the price point of everything else focused around logs, including AWS; you're paying 50 cents a gigabyte to ingest CloudWatch logs data over there. Other companies are charging multiples of that and Cisco recently bought Splunk for $28 billion because it was cheaper than paying their annual Splunk bill. How did you get to that price point? Is it just a matter of everyone else being greedy or have you done something different?Seif: We looked at it from the perspective of… so there's the three L's of logging. I forgot the name of the person at Netflix who talked about that, but basically, it's low costs, low latency, large scale, right? And you will never be able to fulfill all three of them. And we decided to work on low costs and large scale. And in terms of low latency, we won't be low as others like ClickHouse, but we are low enough. Like, we're fast enough.The idea is to be fast enough because in most cases, I don't want to compete on milliseconds. I think if the user can see his data in two seconds, he's happy. Or three seconds, he's happy. I'm not going to be, like, one to two seconds and make the cost exponentially higher because I'm one second faster than the other. And that's, I think, that the way we approached this from day one.And from day one, we also started utilizing the idea of existence of Open—Object Storage, we have our own compressions, our own encodings, et cetera, from day one, too, so and we still stick to that. That's why we never converted to other existing things like Parquet. Also because we are a Schema-On-Read, which Parquet doesn't allow you really to do. But other than that, it's… from day one, we wanted to save costs by also making coordination free. So, ingest has to be coordination free, right, because then we don't run a shitty Kafka, like, honestly a lot—a lot of the [logs 00:06:19] companies who running a Kafka in front of it, the Kafka tax reflects in what they—the bill that you're paying for them.Corey: What I found fun about your pricing model is it gets to a point that for any reasonable workload, how much to log or what to log or sample or keep everything is no longer an investment decision; it's just go ahead and handle it. And that was originally what you wound up building out. Increasingly, it seems like you're not just the place to send all the logs to, which to be honest, I was excited enough about that. That was replacing one of the projects I did a couple of times myself, which is building highly available, fault-tolerant, rsyslog clusters in data centers. Okay, great, you've gotten that unlocked, the economics are great, I don't have to worry about that anymore.And then you started adding interesting things on top of it, analyzing things, replaying events that happen to other players, et cetera, et cetera, it almost feels like you're not just a storage depot, but you also can forward certain things on under a variety of different rules or guises and format them as whatever on the other side is expecting them to be. So, there's a story about integrating with other observability vendors, for example, and only sending the stuff that's germane and relevant to them since everyone loves to charge by ingest.Seif: Yeah. So, we did this one thing called endpoints, the number one. Endpoints was a beginning where we said, “Let's let people send us data using whatever API they like using, let's say Elasticsearch, Datadog, Honeycomb, Loki, whatever, and we will just take that data and multiplex it back to them.” So, that's how part of it started. This allows us to see, like, how—allows customers to see how we compared to others, but then we took it a bit further and now, it's still in closed invite-only, but we have Pipelines—codenamed Pipelines—which allows you to send data to us and we will keep it as a source of truth, then we will, given specific rules, we can then ship it anywhere to a different destination, right, and this allows you just to, on the fly, send specific filter things out to, I don't know, a different vendor or even to S3 or you could send it to Splunk. But at the same time, you can—because we have all your data, you can go back in the past, if the incident happens and replay that completely into a different product.Corey: I would say that there's a definite approach to observability, from the perspective of every company tends to visualize stuff a little bit differently. And one of the promises of OTel that I'm seeing that as it grows is the idea of oh, I can send different parts of what I'm seeing off to different providers. But the instrumentation story for OTel is still very much emerging. Logs are kind of eternal and the only real change we've seen to logs over the past decade or so has been instead of just being plain text and their positional parameters would define what was what—if it's in this column, it's an IP address and if it's in this column, it's a return code, and that just wound up being ridiculous—now you see them having schemas; they are structured in a variety of different ways. Which, okay, it's a little harder to wind up just cat'ing a file together and piping it to grep, but there are trade-offs that make it worth it, in my experience.This is one of those transitional products that not only is great once you get to where you're going, from my playing with it, but also it meets you where you already are to get started because everything you've got is emitting logs somewhere, whether you know it or not.Seif: Yes. And that's why we picked up on OTel, right? Like, one of the first things, we now support… we have an OTel endpoint natively bec—or as a first-class citizen because we wanted to build this experience around OTel in general. Whether we like it or not, and there's more reasons to like it, OTel is a standard that's going to stay and it's going to move us forward. I think of OTel as will have the same effect if not bigger as [unintelligible 00:10:11] back of the day, but now it just went away from metrics, just went to metrics, logs, and traces.Traces is, for me, very interesting because I think OTel is the first one to push it in a standard way. There were several attempts to make standardized [logs 00:10:25], but I think traces was something that OTel really pushed into a proper standard that we can follow. It annoys me that everybody uses a different bits and pieces of it and adds something to it, but I think it's also because it's not that mature yet, so people are trying to figure out how to deliver the best experience and package it in a way that it's actually interesting for a user.Corey: What I have found is that there's a lot that's in this space that is just simply noise. Whenever I spend a protracted time period working on basically anything and I'm still confused by the way people talk about that thing, months or years later, I'm starting to get the realization that maybe I'm not the problem here. And I'm not—I don't mean this to be insulting, but one of the things I've loved about you folks is I've always understood what you're saying. Now, you can hear that as, “Oh, you mean we talk like simpletons?” No, it means what you're talking about resonates with at least a subset of the people who have the problem you solve. That's not nothing.Seif: Yes. We've tried really hard because one of the things we've tried to do is actually bring observability to people who are not always busy or it's not part of their day to day. So, we try to bring into [Versal 00:11:37] developers, right, with doing a Versal integration. And all of a sudden, now they have their logs, and they have a metrics, and they have some traces. So, all of a sudden, they're doing the observability work. Or they have actual observability, for their Versal based, [unintelligible 00:11:54]-based product.And we try to meet the people where they are, so we try to—instead of actually telling people, “You should send us data.”—I mean, that's what they do now—we try to find, okay, what product are you using and how can we grab data from there and send it to us to make your life easier? You see that we did that with Versal, we did that with Cloudflare. AWS, we have extensions, Lambda extensions, et cetera, but we're doing it for more things. For Netlify, it's a one-click integration, too, and that's what we're trying to do to actually make the experience and the journey easier.Corey: I want to change gears a little bit because something that we spent a fair bit of time talking about—it's why we became friends, I would think anyway—is that we have a shared appreciation for several things. One of which, at most notable to anyone around us is whenever we hang out, we greet each other effusively and then immediately begin complaining about costs of cloud services. What is your take on the way that clouds charge for things? And I know it's a bit of a leading question, but it's core and foundational to how you think about Axiom, as well as how you serve customers.Seif: They're ripping us off. I'm sorry [laugh]. They just—the amount of money they make, like, it's crazy. I would love to know what margins they have. That's a big question I've always had. I'm like, what are the margins they have at AWS right now?Corey: Across the board, it's something around 30 to 40%, last time I looked at it.Seif: That's a lot, too.Corey: Well, that's also across the board of everything, to be clear. It is very clear that some services are subsidized by other services. As it should be. If you start charging me per IAM call, we're done.Seif: And also, I mean, the machine learning stuff. Like, they won't be doing that much on top of it right now, right, [else nobody 00:13:32] will be using it.Corey: But data transfer? Yeah, there's a significant upcharge on that. But I hear you. I would moderate it a bit. I don't think that I would say that it's necessarily an intentional ripoff. My problem with most cloud services that they offer is not usually that they're too expensive—though there are exceptions to that—but rather that the dimensions are unpredictable in advance. So, you run something for a while and see what it costs. From where I sit, if a customer uses your service and then at the end of usage is surprised by how much it cost them, you've kind of screwed up.Seif: Look, if they can make egress free—like, you saw how Cloudflare just did the egress of R2 free? Because I am still stuck with AWS because let's face it, for me, it is still my favorite cloud, right? Cloudflare is my next favorite because of all the features that are trying to develop and the pace they're picking, the pace they're trying to catch up with. But again, one of the biggest things I liked is R2, and R2 egress is free. Now, that's interesting, right?But I never saw anything coming back from S3 from AWS on S3 for that, like you know. I think Amazon is so comfortable because from a product perspective, they're simple, they have the tools, et cetera. And the UI is not the flashiest one, but you know what you're doing, right? The CLI is not the flashiest one, but you know what you're doing. It is so cool that they don't really need to compete with others yet.And I think they're still dominantly the biggest cloud out there. I think you know more than me about that, but [unintelligible 00:14:57], like, I think they are the biggest one right now in terms of data volume. Like, how many customers are using them, and even in terms of profiles of people using them, it's very, so much. I know, like, a lot of the Microsoft Azure people who are using it, are using it because they come from enterprise that have been always Microsoft… very Microsoft friendly. And eventually, Microsoft also came in Europe in these all these different weird ways. But I feel sometimes ripped off by AWS because I see Cloudflare trying to reduce the prices and AWS just looking, like, “Yeah, you're not a threat to us so we'll keep our prices as they are.”Corey: I have it on good authority from folks who know that there are reasons behind the economic structures of both of those companies based—in terms of the primary direction the traffic flows and the rest. But across the board, they've done such a poor job of articulating this that, frankly, I think the confusion is on them to clear up, not us.Seif: True. True. And the reason I picked R2 and S3 to compare there and not look at Workers and Lambdas because I look at it as R2 is S3 compatible from an API perspective, right? So, they're giving me something that I already use. Everything else I'm using, I'm using inside Amazon, so it's in a VPC, but just the idea. Let me dream. Let me dream that S3 egress will be free at some point.Corey: I can dream.Seif: That's like Christmas. It's better than Christmas.Corey: What I'm surprised about is how reasonable your pricing is in turn. You wind up charging on the basis of ingest, which is basically the only thing that really makes sense for how your company is structured. But it's predictable in advance, the free tier is, what, 500 gigs a month of ingestion, and before people think, “Oh, that doesn't sound like a lot,” I encourage you to just go back and think how much data that really is in the context of logs for any toy project. Like, “Well, our production environment spits out way more than that.” Yes, and by the word production that you just used, you probably shouldn't be using a free trial of anything as your critical path observability tooling. Become a customer, not a user. I'm a big believer in that philosophy, personally. For all of my toy projects that are ridiculous, this is ample.Seif: People always tend to overestimate how much logs they're going to be sending. Like so, there's one thing. What you said it right: people who already have something going on, they already know how much logs they'll be sending around. But then eventually they're sending too much, and that's why we're back here and they're talking to us. Like, “We want to ttry your tool, but you know, we'll be sending more than that.” So, if you don't like our pricing, go find something else because I think we are the cheapest out there right now. We're the competitive the cheapest out there right now.Corey: If there is one that is less expensive, I'm unaware of it.Seif: [laugh].Corey: And I've been looking, let's be clear. That's not just me saying, “Well, nothing has skittered across my desk.” No, no, no, I pay attention to this space.Seif: Hey, where's—Corey, we're friends. Loyalty.Corey: Exactly.Seif: If you find something, you tell me.Corey: Oh, if I find something, I'll tell everyone.Seif: Nononon, you tell me first and you tell me in a nice way so I can reduce the prices on my site [laugh].Corey: This is how we start a price was, industry-wide, and I would love to see it.Seif: [laugh]. But there's enough channels that we share at this point across different Slacks and messaging apps that you should be able to ping me if you find one. Also, get me the name of the CEO and the CTO while you're at it.Corey: And where they live. Yes, yes, of course. The dire implications will be awesome.Seif: That was you, not me. That was your suggestion.Corey: Exactly.Seif: I will not—[laugh].Corey: Before we turn into a bit of an old thud and blunder, let's talk about something else that I'm curious about here. You've been working on Axiom for something like seven years now. You come from a world of databases and events and the like. Why start a company in the model of Axiom? Even back then, when I looked around, my big problem with the entire observability space could never have been described as, “You know what we need? More companies that do exactly this.” What was it that you saw that made you say, “Yeah, we're going to start a company because that sounds easy.”Seif: So, I'll be very clear. Like, I'm not going to, like, sugarcoat this. We kind of got in a position where it [forced counterweighted 00:19:10]. And [laugh] by that I mean, we came from a company where we were dealing with logs. Like, we actually wrote an event crash analytics tool for a company, but then we ended up wanting to use stuff like Datadog, but we didn't have the budget for that because Datadog was killing us.So, we ended up hosting our own Elasticsearch. And Elasticsearch, it costs us more to maintain our Elasticsearch cluster for the logs than to actually maintain our own little infrastructure for the crash events when we were getting, like, 1 billion crashes a month at this point. So eventually, we just—that was the first burn. And then you had alert fatigue and then you had consolidating events and timestamps and whatnot. The whole thing just seemed very messy.So, we started off after some company got sold, we started off by saying, “Okay, let's go work on a new self-hosted version of the [unintelligible 00:20:05] where we do metrics and logs.” And then that didn't go as well as we thought it would, but we ended up—because from day one, we were working on cloud na—because we d—we cloud ho—we were self-hosted, so we wanted to keep costs low, we were working on and making it stateless and work against object store. And this is kind of how we started. We realized, oh, our cost, we can host this and make it scale, and won't cost us that much.So, we did that. And that started gaining more attention. But the reason we started this was we wanted to start a self-hosted version of Datadog that is not costly, and we ended up doing a Software as a Service. I mean, you can still come and self-hosted, but you'll have to pay money for it, like, proper money for that. But we do as a SaaS version of this and instead of trying to be a self-hosted Datadog, we are now trying to compete—or we are competing with Datadog.Corey: Is the technology that you've built this on top of actually that different from everything else out there, or is this effectively what you see in a lot of places: “Oh, yeah, we're just going to manage Elasticsearch for you because that's annoying.” Do you have anything that distinguishes you from, I guess, the rest of the field?Seif: Yeah. So, very just bluntly, like, I think Scuba was the first thing that started standing out, and then Honeycomb came into the scene and they start building something based on Scuba, the [unintelligible 00:21:23] principles of Scuba. Then one of the authors of actual Scuba reached out to me when I told him I'm trying to build something, and he's gave me some ideas, and I start building that. And from day one, I said, “Okay, everything in S3. All queries have to be serverless.”So, all the queries run on functions. There's no real disks. It's just all on S3 right now. And the biggest issue—achievement we got to lower our cost was to get rid of Kafka, and have—let's say, in behind the scenes we have our own coordination-free mechanism, but the idea is not to actually have to use Kafka at all and thus reduce the costs incredibly. In terms of technology, no, we don't use Elasticsearch.We wrote everything from the ground up, from scratch, even the query language. Like, we have our own query language that's based—modeled after Kusto—KQL by Microsoft—so everything we have is built from absolutely from the ground up. And no Elastic. I'm not using Elastic anymore. Elastic is a horror for me. Absolutely horror.Corey: People love the API, but no, I've never met anyone who likes managing Elasticsearch or OpenSearch, or whatever we're calling your particular flavor of it. It is a colossal pain, it is subject to significant trade-offs, regardless of how you work with it, and Amazon's managed offering doesn't make it better; it makes it worse in a bunch of ways.Seif: And the green status of Elasticsearch is a myth. You'll only see it once: the first time you start that cluster, that's what the Elasticsearch cluster is green. After that, it's just orange, or red. And you know what? I'm happy when it's orange. Elasticsearch kept me up for so long. And we had actually a very interesting situation where we had Elasticsearch running on Azure, on Windows machines, and I would have server [unintelligible 00:23:10]. And I'd have to log in and every day—you remember, what's it called—RP… RP Something. What was it called?Corey: RDP? Remote Desktop Protocol, or something else?Seif: Yeah, yeah. Where you have to log in, like, you actually have visual thing, and you have to go in and—Corey: Yep.Seif: And visually go in and say, “Please don't restart.” Every day, I'd have to do that. Please don't restart, please don't restart. And also a lot of weird issues, and also at that point, Azure would decide to disconnect the pod, wanted to try to bring in a new pod, and all these weird things were happening back then. So, eventually, end up with a [unintelligible 00:23:39] decision. I'm talking 2013, '14, so it was back in the day when Elasticsearch was very young. And so, that was just a bad start for me.Corey: I will say that Azure is the most cost-effective cloud because their security is so clown shoes, you can just run whatever you want in someone else's account and it's free to you. Problem solved.Seif: Don't tell people how we save costs, okay?Corey: [laugh]. I love that.Seif: [laugh]. Don't tell people how we do this. Like, Corey, come on [laugh], you're exposing me here. Let me tell you one thing, though. Elasticsearch is the reason I literally use a shock collar or a shock bracelet on myself every time it went down—which was almost every day, instead of having PagerDuty, like, ring my phone.And, you know, I'd wake up and my partner back then would wake up. I bought a Bluetooth collar off of Alibaba that would tase me every time I'd get a notification, regardless of the notification. So, some things are false alarm, but I got tased for at least two, three weeks before I gave up. Every night I'd wake up, like, to a full discharge.Corey: I would never hook myself up to a shocker tied to outages, even if I owned a company. There are pleasant ways to wake up, unpleasant ways to wake up, and even worse. So, you're getting shocked for some—so someone else can wind up effectively driving the future of the business. You're, more or less, the monkey that gets shocked awake to go ahead and fix the thing that just broke.Seif: [laugh]. Well, the fix to that was moving from Azure to AWS without telling anybody. That got us in a lot of trouble. Again, that wasn't my company.Corey: They didn't notice that you did this, or it caused a lot of trouble because suddenly nothing worked where they thought it would work?Seif: They—no, no, everything worked fine on AWS. That's how my love story began. But they didn't notice for, like, six months.Corey: That's kind of amazing.Seif: [laugh]. That was specta—we rewrote everything from C# to Node.js and moved everything away from Elasticsearch, started using Redshift, Redis and a—you name it. We went AWS all the way and they didn't even notice. We took the budget from another department to start filling that in.But we cut the costs from $100,000 down to, like, 40, and then eventually down to $30,000 a month.Corey: More than a little wild.Seif: Oh, God, yeah. Good times, good times. Next time, just ask me to tell you the full story about this. I can't go into details on this podcast. I'll get in a lot—I think I'll get in trouble. I didn't sign anything though.Corey: Those are the best stories. But no, I hear you. I absolutely hear you. Seif, I really want to thank you for taking the time to speak with me. If people want to learn more, where should they go?Seif: So, axiom.co—not dot com. Dot C-O. That's where they learn more about Axiom. And other than that, I think I have a Twitter somewhere. And if you know how to write my name, you'll—it's just one word and find me on Twitter.Corey: We will put that all in the [show notes 00:26:33]. Thank you so much for taking the time to speak with me. I really appreciate it.Seif: Dude, that was awesome. Thank you, man.Corey: Seif Lotfy, co-founder and CTO of Axiom, who has brought this promoted guest episode our way. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that one of these days, I will get around to aggregating in some horrifying custom homebrew logging system, probably built on top of rsyslog.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Forrest Brazeal, Head of Developer Media at Google Cloud, joins Corey on Screaming in the Cloud to discuss how AI, current job markets, and more are impacting software engineers. Forrest and Corey explore whether AI helps or hurts developers, and what impact it has on the role of a junior developer and the rest of the development team. Forrest also shares his viewpoints on how he feels AI affects people in creative roles. Corey and Forrest discuss the pitfalls of a long career as a software developer, and how people can break into a career in cloud as well as the necessary pivots you may need to make along the way. Forrest then describes why he feels workers are currently staying put where they work, and how he predicts a major shift will happen when the markets shift.About ForrestForrest is a cloud educator, cartoonist, author, and Pwnie Award-winning songwriter. He currently leads the content marketing team at Google Cloud. You can buy his book, The Read Aloud Cloud, from Wiley Publishing or attend his talks at public and private events around the world.Links Referenced: Personal Website: https://goodtechthings.com Newsletter signup: https://cloud.google.com/innovators TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn and I am thrilled to have a returning guest on, who has been some would almost say suspiciously quiet over the past year or so. Forrest Brazeal is the Head of Developer Media over at Google Cloud, and everyone sort of sits there and cocks their head, like, “What does that mean?” And then he says, “Oh, I'm the cloud bard.” And everyone's, “Oh, right. Get it: the song guy.” Forrest, welcome back.Forrest: Thanks, Corey. As always, it's great to be here.Corey: So, what have you been up to over the past, oh let's call it, I don't know, a year, since I think, is probably the last time you're on the show.Forrest: Well, gosh, I mean, for one thing, it seems like I can't call myself the cloud bard anymore because Google rolled out this thing called Bard and I've started to get some DMs from people asking for, you know, tech support on Bard. So, I need to make that a little bit clearer that I do not work on Bard. I am a lowercase bard, but I was here first, so if anything, you know, Google has deprecated me.Corey: Honestly, this feels on some level like it's more cloudy if we define cloudy as what, you know, Amazon does because they launched a quantum computing service about six months after they launched some unrelated nonsense that they called [QuantumDB 00:01:44], which you'd think if you're launching quantum stuff, you'd reserve the word quantum for that. But no, they're going to launch things that stomp all over other service names as well internally, so customers just wind up remarkably confused. So, if you find a good name, just we're going to slap it on everything, seems to be the way of cloud.Forrest: Yeah, naming things has proven to be harder than either quantum computing or generative AI at this point, I think.Corey: And in fairness, I will point out that naming things is super hard; making fun of names is not. So, that is—everyone's like, “Wow, you're so good at making fun of names. Can you name something well?” [laugh]. Absolutely not.Forrest: Yeah, well, one of the things you know, that I have been up to over the past year or so is just, you know, getting to learn more about what it's like to have an impact in a very, very large organizational context, right? I mean, I've worked in large companies before, but Google is a different size and scale of things and it takes some time honestly, to, you know, figure out how you can do the best for the community in an environment like that. And sometimes that comes down to the level of, like, what are things called? How do we express things in a way that makes sense to everyone and takes into account people's different communication styles and different preferences, different geographies, regions? And that's something that I'm still learning.But you know, hopefully, we're getting to a point where you're going to start hearing some things come out of Google Cloud that answer your questions and makes sense to you. That's supposed to be part of my job, anyway.Corey: So, I want to talk a bit about the idea of generative AI because there has been an awful lot of hype in the space, but you have never given me a bum steer. You have always been a level-headed, reasonable voice. You are not—to my understanding—a VC trying desperately to prop up an industry that you may or may not believe in, but you are financially invested into. What is your take on the last, let's call it, year of generative AI enhancements?Forrest: So, to be clear, while I do have a master's degree in interactive intelligence, which is kind of AI adjacent, this is not something that I build with day-to-day professionally. But I have spent a lot of time over the last year working with the people who do that and trying to understand what is the value that gen AI can bring to the domains that I do care about and have a lot of interest in, which of course, are cloud developers and folks trying to build meaningful enterprise applications, take established workloads and make them better, and as well work with folks who are new to their careers and trying to figure out, you know, what's the most appropriate technology for me to bet on? What's going to help me versus what's going to hurt me?And I think one of the things that I have been telling people most frequently—because I talk to a lot of, like, new cloud learners, and they're saying, “Should I just drop what I'm doing? Should I stop building the projects I'm working on and should I instead just go and get really good at generating code through something like a Bard or a ChatGPT or what have you?” And I went down a rabbit hole with this, Corey, for a long time and spent time building with these tools. And I see the value there. I don't think there's any question.But what has come very, very clearly to the forefront is, the better you already are at writing code, the more help a generative AI coding assistant is going to give you, like a Bard or a ChatGPT, what have you. So, that means the way to get better at using these tools is to get better at not using these tools, right? The more time you spend learning to code without AI input, the better you'll be at coding with AI input.Corey: I'm not sure I entirely agree because for me, the wake-up call that I had was a singular moment using I want to say it was either Chat-Gippity—yes, that's how it's pronounced—or else it was Gif-Ub Copilot—yes, also how it's pronounced—and the problem that I was having was, I wanted to query probably the worst API in the known universe—which is, of course, the AWS pricing API: it returns JSON, that kind of isn't, it returns really weird structures where you have to correlate between a bunch of different random strings to get actual data out of it, and it was nightmarish and of course, it's not consistent. So, I asked it to write me a Python script that would contrast the hourly cost of a Managed NAT gateway in all AWS regions and return a table sorted by the most to least expensive. And it worked.Now, this is something that I could have done myself in probably half a day because my two programming languages of choice remain brute force and enthusiasm, but it wound up taking away so much of the iterative stuff that doesn't work of oh, that's not quite how you'd handle that data structure. Oh, you think it's a dict, but no, it just looks like one. It's a string first; now you have to convert it, or all kinds of other weird stuff like that. Like, this is not senior engineering work, but it really wound up as a massive accelerator to get the answer I was after. It was almost an interface to a bad API. Or rather, an interface to a program—to a small script that became an interface itself to a bad API.Forrest: Well, that's right. But think for a minute, Corey, about what's implicit in that statement though. Think about all the things you had to know to get that value out of ChatGPT, right? You had to know, A, what you were looking for: how these prices worked, what the right price [style 00:06:52] was to look for, right, why NAT gateway is something you needed to be caring about in the first place. There's a pretty deep stack of things—actually, it's what we call a context window, right, that you needed to know to make this query take a half-day of work away from you.And all that stuff that you've built up through years and years of being very hands-on with this technology, you put that same sentence-level task in the hands of someone who doesn't have that background and they're not going to have the same results. So, I think there's still tremendous value in expanding your personal mental context window. The more of that you have, the better and faster results you're going to get.Corey: Oh, absolutely. I do want to steer away from this idea that there needs to be this massive level of subject matter expertise because I don't disagree with it, but you're right, the question I asked was highly contextual to the area of expertise that I have. But everyone tends to have something like that. If you're a marketer for example, and you wind up with an enormous pile of entrants on a feedback form, great. Can you just dump it all in and say, can you give me a sentiment analysis on this?I don't know how to run a sentiment analysis myself, but I'm betting that a lot of these generative AI models do, or being able to direct me in the right area on this. The question I have is—it can even be distilled down into simple language of, “Here's a bunch of comments. Do people love the thing or hate the thing?” There are ways to get there that apply, even if you don't have familiarity with the computer science aspects of it, you definitely have aspect to the problem in which you are trying to solve.Forrest: Oh, yeah, I don't think we're disagreeing at all. Domain expertise seems to produce great results when you apply it to something that's tangential to your domain expertise. But you know, I was at an event a month or two ago, and I was talking to a bunch of IT executives about ChatGPT and these other services, and it was interesting. I heard two responses when we were talking about this. The first thing that was very common was I did not hear any one of these extremely, let's say, a little bit skeptical—I don't want to say jaded—technical leaders—like, they've been around a long time; they've seen a lot of technologies come and go—I didn't hear a single person say, “This is something that's not useful to me.”Every single one of them immediately was grasping the value of having a service that can connect some of those dots, can in-between a little bit, if you will. But the second thing that all of them said was, “I can't use this inside my company right now because I don't have legal approval.” Right? And then that's the second round of challenges is, what does it look like to actually take these services and make them safe and effective to use in a business context where they're load-bearing?Corey: Depending upon what is being done with them, I am either sympathetic or dismissive of that concern. For example, yesterday, I wound up having fun with it, and—because I saw a query, a prompt that someone had put in of, “Create a table of the US presidents ranked by years that they were in office.” And it's like, “Okay, that's great.” Like, I understand the value here. But if you have a magic robot voice from the future in a box that you can ask it any question and as basically a person, why not have more fun with it?So, I put to it the question of, “Rank the US presidents by absorbency.” And it's like, “Well, that's not a valid way of rating presidential performance.” I said, “It is if I have a spill and I'm attempting to select the US president with which to mop up the spill.” Like, “Oh, in that case, here you go.” And it spat out a bunch of stuff.That was fun and exciting. But one example he gave was it ranked Theodore Roosevelt very highly. Teddy Roosevelt was famous for having a mustache. That might be useful to mop up a spill. Now, I never would have come up in isolation with the idea of using a president's mustache to mop something up explicitly, but that's a perfect writer's room style Yes, And approach that I could then springboard off of to continue iterating on if I'm using that as part of something larger. That is a far cry from copying and pasting whatever it is to say into an email, whacking send before realizing it makes no sense.Forrest: Yeah, that's right. And of course, you can play with what we call the temperatures on these models, right, to get those very creative, off-the-wall kind of answers, or to make them very, kind of, dry and factual on the other end. And Google Cloud has been doing some interesting things there with Generative AI Studio and some of the new features that have come to Vertex AI. But it's just—it's going to be a delicate dance, honestly, to figure out how you tune those things to work in the enterprise.Corey: Oh, absolutely. I feel like the temperature dial should instead be instead relabeled as ‘corporate voice.' Like, do you want a lot of it or a little of it? And of course, they have to invert it. But yeah, the idea is that, for some things, yeah, you definitely just want a just-the-facts style of approach.Another demo that I saw, for example, that I thought showed a lack of imagination was, “Here's a transcript of a meeting. Extract all the to-do items.” Okay. Yeah, I suppose that works, but what about, here's a transcript of the meeting. Identify who the most unpleasant, passive-aggressive person in this meeting is to work with.And to its credit—because of course this came from something corporate, none of the systems that I wound up running that particular query through could identify anyone because of course the transcript was very bland and dry and not actually how human beings talk, other than in imagined corporate training videos.Forrest: Yes, well again, I think that gets us into the realm of just because you can doesn't mean you should use it for this.Corey: Oh, I honestly, most of what I use this stuff for—or use anything for—should be considered a cautionary tale as opposed to guidance for the future. You write parody songs a fair bit. So do I, and I've had an attempt to write versions of, like, write parody lyrics for some random song about this theme. And it's not bad, but for a lot of that stuff, it's not great, either. It is a starting point.Forrest: Now, hang on, Corey. You know, as well as I do that I don't write parody songs. We've had this conversation before. A parody is using existing music and adding new lyrics to it. I write my own music and my own lyrics and I'll have you know, that's an important distinction. But—Corey: True.Forrest: I think you're right on that, you know, having these services give you creative output. What you're getting is an average of a lot of other creative output, right, which is—could give you a perfectly average result, but it's difficult to get a first pass that gives you something that really stands out. I do also find, as a creative, that starting with something that's very average oftentimes locks me into a place where I don't really want to be. In other words, I'm not going to potentially come up with something as interesting if I'm starting with a baseline like that. It's almost a little bit polluting to the creative process.I know there's a lot of other creatives that feel that way as well, but you've also got people that have found ways to use generative AI to stimulate some really interesting creative things. And I think maybe the example you gave of the president's rank by absorbency is a great way to do that. Now, in that case, the initial creativity, a lot of it resided in the prompt, Corey. I mean, you're giving it a fantastically creative, unusual, off-the-wall place to start from. And just about any average of five presidents that come out of that is going to be pretty funny and weird because of just how funny and weird the idea was to begin with. That's where I think AI can give you that great writer's room feel.Corey: It really does. It's a Yes, And approach where there's a significant way that it can build on top of stuff. I've been looking for a, I guess, a writer's room style of approach for a while, but it's hard to find the right people who don't already have their own platform and voice to do this. And again, it's not a matter of payment. I'm thrilled to basically pay any reasonable out of money to build a writer's room here of people who get the cloud industry to work with me and workshops on some of the bigger jokes.The challenge is that those people are very hard to find and/or are conflicted out. Having just a robot who, with infinite patience for tomfoolery—because the writing process can look kind of dull and crappy until you find the right thing—has been awesome. There's also a sense of psychological safety in not poisoning people. Like, “I thought you were supposed to be funny, but this stuff is all terrible. What's the deal here?” I've already poisoned that well with my business partner, for example.Forrest: Yeah, there's only so many chances you get to make that first impression, so why not go with AI that never remembers you or any of your past mistakes?Corey: Exactly. Although the weird thing is that I found out that when they first launched Chat-Gippity, it already knew who I was. So, it is in fact familiar, so at least my early work of my entire—I guess my entire life. So that's—Forrest: Yes.Corey: —kind of worrisome.Forrest: Well, I know it credited to me books I hadn't written and universities I hadn't attended and all kinds of good stuff, so it made me look better than I was.Corey: So, what have you been up to lately in the context of, well I said generative AI is a good way to start, but I guess we can also call it at Google Cloud. Because I have it on good authority that, marketing to the contrary, all of the cloud providers do other things in addition to AI and ML work. It's just that's what's getting in the headline these days. But I have noticed a disturbing number of virtual machines living in a bunch of customer environments relative to the amount of AI workloads that are actually running. So, there might be one or two other things afoot.Forrest: That's right. And when you go and talk to folks that are actively building on cloud services right now, and you ask them, “Hey, what is the business telling you right now? What is the thing that you have to fix? What's the thing that you have to improve?” AI isn't always in the conversation.Sometimes it is, but very often, those modernization conversations are about, “Hey, we've got to port some of these services to a language that the people that work here now actually know how to write code in. We've got to find a way to make this thing a little faster. Or maybe more specifically, we've got to figure out how to make it run at the same speed while using less or less expensive resources.” Which is a big conversation right now. And those are things that they are conversations as old as time. They're not going away, and so it's up to the cloud providers to continue to provide services and features that help make that possible.And so, you're seeing that, like, with Cloud Run, where they've just announced this CPU Boost feature, right, that gives you kind of an additional—it's like a boost going downhill or a push on the swing as you're getting started to help you get over that cold-start penalty. Where you're seeing the session affinity features for Cloud Run now where you have the sticky session ability that might allow you to use something like, you know, a container-backed service like that, instead of a more traditional load balancer service that you'd be using in the past. So, you know, just, you take your eye off the ball for a minute, as you know, and 10 or 20, more of these feature releases come out, but they're all kind of in service of making that experience better, broadening the surface area of applications and workloads that are able to be moved to cloud and able to be run more effectively on cloud than anywhere else.Corey: There's been a lot of talk lately about how the idea of generative AI might wind up causing problems for people, taking jobs away, et cetera, et cetera. You almost certainly have a borderline unique perspective on this because of your work with, honestly, one of the most laudable things I've ever seen come out of anywhere which is The Cloud Resume Challenge, which is a build a portfolio site, then go ahead and roll that out into how you interview. And it teaches people how to use cloud, step-by-step, you have multi-cloud versions, you have them for specific clouds. It's nothing short of astonishing. So, you find yourself talking to an awful lot of very early career folks, folks who are transitioning into tech from other places, and you're seeing an awful lot of these different perspectives and AI plays come to the forefront. How do you wind up, I guess, making sense of all this? What guidance are you giving people who are worried about that?Forrest: Yeah, I mean, I, you know—look, for years now, when I get questions from these, let's call them career changers, non-traditional learners who tend to be a large percentage, if not a plurality, of the people that are working on The Cloud Resume Challenge, for years now, the questions that they've come to me with are always, like, you know, “What is the one thing I need to know that will be the magic technology, the magic thing that will unlock the doors and give me the inside track to a junior position?” And what I've always told them—and it continues to be true—is, there is no magic thing to know other than magically going and getting two years of experience, right? The way we hire juniors in this industry is broken, it's been broken for a long time, it's broken not because of any one person's choice, but because of this sort of tragedy of the commons situation where everybody's competing over a dwindling pool of senior staff level talent and hopes that the next person will, you know, train the next generation for them so they don't have to expend their energy and interview cycles and everything else on it. And as long as that remains true, it's just going to be a challenge to stand out.Now, you'll hear a lot of people saying that, “Well, I mean, if I have generative AI, I'm not going to need to hire a junior developer.” But if you're saying that as a hiring manager, as a team member, then I think you always had the wrong expectation for what a junior developer should be doing. A junior developer is not your mini me who sits there and takes the little challenges, you know, the little scripts and things like that are beneath you to write. And if that's how you treat your junior engineers, then you're not creating an environment for them to thrive, right? A junior engineer is someone who comes in who, in a perfect world, is someone who should be able to come in almost in more of an apprentice context, and somebody should be able to sit alongside you learning what you know, right, and having education integrated into their actual job experience so that at the end of that time, they're able to step back and actually be a full-fledged member of your team rather than just someone that you kind of throw tasks over the wall to, and they don't have any career advancement potential out of that.So, if anything, I think the advancement of generative AI, in a just world, ought to give people a wake-up call that, hey, training the next generation of engineers is something that we're actually going to have to actively create programs around, now. It's not something that we can just, you know, give them the scraps that fall off of our desks. Unfortunately, I do think that in some cases, the gen AI narrative more than the reality is being used to help people put off the idea of trying to do that. And I don't believe that that's going to be true long-term. I think that if anything, generative AI is going to open up more need for developers.I mean, it's generating a lot of code, right, and as we know, Jevons paradox says that when you make it easier to use something and there's elastic demand for that thing, the amount of creation of that thing goes up. And that's going to be true for code just like it was for electricity and for code and for GPUs and who knows what all else. So, you're going to have all this code that has a much lower barrier of entry to creating it, right, and you're going to need people to harden that stuff and operate it in production, be on call for it at three in the morning, debug it. Someone's going to have to do all that, you know? And what I tell these junior developers is, “It could be you, and probably the best thing for you to do right now is to, like I said before, get good at coding on your own. Build as much of that personal strength around development as you can so that when you do have the opportunity to use generative AI tools on the job, that you have the maximum amount of mental context to put around them to be successful.”Corey: I want to further point out that there are a number of folks whose initial reaction to a lot of this is defensiveness. I showed that script that wound up spitting out the Managed NAT gateway ranked-by-region table to one of our contract engineers, who's very senior. And the initial response I got from them was almost defensive, were, “Okay, yeah. That'll wind up taking over, like, a $20 an hour Upwork coder, but it's not going to replace a senior engineer.” And I felt like that was an interesting response psychologically because it felt defensive for one, and two, not for nothing, but senior developers don't generally spring fully formed from the forehead of some ancient God. They start off as—dare I say it—junior developers who learn and improve as they go.So, I wonder what this means. If we want to get into a point where generative AI takes care of all the quote-unquote, “Easy programming problems,” and getting the easy scripts out, what does that mean for the evolution and development of future developers?Forrest: Well, keep in mind—Corey: And that might be a far future question.Forrest: Right. That's an argument as old as time, right, or a concern is old as time and we hear it anew with each new level of automation. So, people were saying this a few years ago about the cloud or about virtual machines, right? Well, how are people going to, you know, learn how to do the things that sit on top of that if they haven't taken the time to configure what's below the surface? And I'm sympathetic to that argument to some extent, but at the same time, I think it's more important to deal with the reality we have now than try to create an artificial version of realities' past.So, here's the reality right now: a lot of these simple programming tasks can be done by AI. Okay, that's not likely to change anytime soon. That's the new reality. So now, what does it look like to bring on juniors in that context? And again, I think that comes down to don't look at them as someone who's there just to, you know, be a pair of hands on a keyboard, spitting out tiny bits of low-level code.You need to look at them as someone who needs to be, you know, an effective user of general AI services, but also someone who is being trained and given access to the things they'll need to do on top of that, so the architectural decisions, the operational decisions that they'll need to make in order to be effective as a senior. And again, that takes buy-in from a team, right, to make that happen. That is not going to happen automatically. So, we'll see. That's one of those things that's very hard to automate the interactions between people and the growth of people. It takes people that are willing to be mentors.Corey: I'm also curious as to how you see the guidance shifting as computers get better. Because right now, one of my biggest problems that I see is that if I have an idea for a company I want to start or a product I want to build that involves software, step one is, learn to write a bunch of code. And I feel like there's a massive opportunity for skipping aspects of that, whereas effectively have the robot build me the MVP that I describe. Think drag-and-drop to build a web app style of approach.And the obvious response to that is, well, that's not going to go to hyperscale. That's going to break in a bunch of different ways. Well, sure, but I can get an MVP out the door to show someone without having to spend a year building it myself by learning the programming languages first, just to throw away as soon as I hire someone who can actually write code. It cuts down that cycle time massively, and I can't shake the feeling that needs to happen.Forrest: I think it does. And I think, you know, you were talking about your senior engineer that had this kind of default defensive reaction to the idea that something like that could meaningfully intrude on their responsibilities. And I think if you're listening to this and you are that senior engineer, you're five or more years into the industry and you've built your employability on the fact that you're the only person who can rough out these stacks, I would take a very, very hard look at yourself and the value that you're providing. And you say, you know—let's say that I joined a startup and the POC was built out by this technical—or possibly the not-that-technical co-founder, right—they made it work and that thing went from, you know, not existing to having users in the span of a week, which we're seeing more now and we're going to see more and more of. Okay, what does my job look like in that world? What am I actually coming on to help with?Am I—I'm coming on probably to figure out how to scale that thing and make it maintainable, right, operate it in a way that is not going to cause significant legal and financial problems for the company down the road. So, your role becomes less about being the person that comes in and does this totally greenfield thing from scratch and becomes more about being the person who comes in as the adult in the room, technically speaking. And I think that role is not going away. Like I said, there's going to be more of those opportunities rather than less. But it might change your conception of yourself a little bit, how you think about yourself, the value that you provide, now's the time to get ahead of that.Corey: I think that it is myopic and dangerous to view what you do as an engineer purely through the lens of writing code because it is a near certainty that if you are learning to write code today and build systems involving technology today, that you will have multiple careers between now and retirement. And in fact, if you're entering the workforce now, the job that you have today will not exist in anything remotely approaching the same way by the time you leave the field. And the job you have then looks borderline unrecognizable, if it even exists at all today. That is the overwhelming theme that I've got on this ar—the tech industry moves quickly and is not solidified like a number of other industries have. Like, accountants: they existed a generation ago and will exist in largely the same form a generation from now.But software engineering in particular—and cloud, of course, as well, tied to that—have been iterating so rapidly, with such sweepingly vast changes, that that is something that I think we're going to have a lot of challenge with, just wrestling with. If you want a job that doesn't involve change, this is the wrong field.Forrest: Is it the wrong field. And honestly, software engineering is, has been, and will continue to be a difficult business to make a 40-year career in. And this came home to me really strongly. I was talking to somebody a couple of months ago who, if I were to say the name—which I won't—you and I would both know it, and a lot of people listening to this would know as well. This is someone who's very senior, very well respected is, by name, identified in large part with the creation of a significant movement in technology. So, someone who you would never think of would be having a problem getting a job.Corey: Is it me? And is it Route 53 as a database, as the movement?Forrest: No, but good guess.Corey: Excellent.Forrest: This is someone I was talking to because I had just given a talk where I was pleading with IT leaders to take more responsibility for building on-ramps for non-traditional learners, career changers, people that are doing something a little different with their career. And I was mainly thinking of it as people that had come from a completely non-technical background or maybe people that were you know, like, I don't know, IT service managers with skills 20 years out of date, something like that. But this is a person who you and I would think of as someone at the forefront, the cutting edge, an incredibly employable person. And this person was a little bit farther on in their career and they came up to me and said, “Thank you so much for giving that talk because this is the problem I have. Every interview that I go into, I get told, ‘Oh, we probably can't afford you,' or, ‘Oh well, you say you want to do AI stuff now, but we see that all your experience is doing this other thing, and we're just not interested in taking a chance on someone like that at the salary you need to be at.'” and this person's, like, “What am I going to do? I don't see the roadmap in front of me anymore like I did 10, 15, or 20 years ago.”And I was so sobered to hear that coming from, again, someone who you and I would consider to be a luminary, a leading light at the top of the, let's just broadly say IT field. And I had to go back and sit with that. And all I could come up with was, if you're looking ahead and you say I want to be in this industry for 30 years, you may reach a point where you have to take a tremendous amount of personal control over where you end up. You may reach a point where there is not going to be a job out there for you, right, that has the salary and the options that you need. You may need to look at building your own path at some point. It's just it gets really rough out there unless you want to continue to stagnate and stay in the same place. And I don't have a good piece of advice for that other than just you're going to have to find a path that's unique to you. There is not a blueprint once you get beyond that stage.Corey: I get asked questions around this periodically. The problem that I have with it is that I can't take my own advice anymore. I wish I could. But what I used to love doing was, every quarter or so, I'd make it a point to go on at least one job interview somewhere else. This wound up having a few great features.One, interviewing is a skill that atrophies if you don't use it. Two, it gives me a finger on the pulse of what the market is doing, what the industry cares about. I dismissed Docker the first time I heard about it, but after the fourth interview where people were asking about Docker, okay, this is clearly a thing. And it forced me to keep my resume current because I've known too many people who spend seven years at a company and then wind up forgetting what they did years three, four, and five, where okay, then what was the value of being there? It also forces you to keep an eye on how you're evolving and growing or whether you're getting stagnant.I don't ever want to find myself in the position of the person who's been at a company for 20 years and gets laid off and discovers to their chagrin that they don't have 20 years of experience; they have one year of experience repeated 20 times. Because that is a horrifying and scary position to be in.Forrest: It is horrifying and scary. And I think people broadly understand that that's not a position they want to be in, hence why we do see people that are seeking out this continuing education, they're trying to find—you know, trying to reinvent themselves. I see a lot of great initiative from people that are doing that. But it tends to be more on the company side where, you know, they get pigeonholed into a position and the company that they're at says, “Yeah, no. We're not going to give you this opportunity to do something else.”So, we say, “Okay. Well, I'm going to go and interview other places.” And then other companies say, “No, I'm not going to take a chance on someone that's mid-career to learn something brand new. I'm going to go get someone that's fresh out of school.” And so again, that comes back to, you know, where are we as an industry on making space for non-traditional learners and career changers to take the maturity that they have, right, even if it's not specific familiarity with this technology right now, and let them do their thing, let them get untracked.You know, there's tremendous potential being untapped there and wasted, I would say. So, if you're listening to this and you have the opportunity to hire people, I would just strongly encourage you to think outside the box and consider people that are farther on in their careers, even if their technical skill set doesn't exactly line up with the five pieces of technology that are on your job req, look for people that have demonstrated success and ability to learn at whatever [laugh] the things are that they've done in the past, people that are tremendously highly motivated to succeed, and let them go win on your behalf. There's—you have no idea the amount of talent that you're leaving on the table if you don't do that.Corey: I'd also encourage people to remember that job descriptions are inherently aspirational. If you take a job where you know how to do every single item on the list because you've done it before, how is that not going to be boring? I love being given problems. And maybe I'm weird like this, but I love being given a problem where people say, “Okay, so how are you going to solve this?” And the answer is, “I have no idea yet, but I can't wait to find out.” Because at some level, being able to figure out what the right answer is, pick up the skill sets I don't need, the best way to learn something that I've ever found, at least for me.Forrest: Oh, I hear that. And what I found, you know, working with a lot of new learners that I've given that advice to is, typically the ones that advice works best for, unfortunately, are the ones who have a little bit of baked-in privilege, people that tend to skate by more on the benefit of the doubt. That is a tough piece of advice to fulfill if you're, you know, someone who's historically underrepresented or doesn't often get the chance to prove that you can do things that you don't already have a testament to doing successfully. So again, takes it back to the hiring side. Be willing to bet on people, right, and not just to kind of look at their resume and go from there.Corey: So, I'm curious to see what you've noticed in the community because I have a certain perspective on these things, and a year ago, everyone was constantly grousing about dissatisfaction with their employers in a bunch of ways. And that seems to have largely vanished. I know, there have been a bunch of layoffs and those are tragic on both sides, let's be very clear. No one is happy when a layoff hits. But I'm also seeing a lot more of people keeping their concerns to either private channels or to themselves, and I'm seeing what seems to be less mobility between companies than I saw previously. Is that because people are just now grateful to have a job and don't want to rock the boat, or is it still happening and I'm just not seeing it in the same way?Forrest: No, I think the vibe has shifted, for sure. You've got, you know, less opportunities that are available, you know that if you do lose your job that you're potentially going to have fewer places to go to. I liken it to like if you bought a house with a sub-3% mortgage and 2021, let's say, and now you want to move. Even though the housing market may have gone down a little bit, those interest rates are so high that you're going to be paying more, so you kind of are stuck where you are until the market stabilizes a little bit. And I think there's a lot of people in that situation with their jobs, too.They locked in salaries at '21, '22 prices and now here we are in 2023 and those [laugh] those opportunities are just not open. So, I think you're seeing a lot of people staying put—rationally, I would say—and waiting for the market to shift. But I think that at the point that you do see that shift, then yes, you're going to see an exodus; you're going to see a wave and there will be a whole bunch of new think pieces about the great resignation or something, but all it is just that pent up demand as people that are unhappy in their roles finally feel like they have the mobility to shift.Corey: I really want to thank you for taking the time to speak with me. If people want to learn more, where's the best place for them to find you?Forrest: You can always find me at goodtechthings.com. I have a newsletter there, and I like to post cartoons and videos and other fun things there as well. If you want to hear my weekly take on Google Cloud, go to cloud.google.com/innovators and sign up there. You will get my weekly newsletter The Overwhelmed Person's Guide to Google Cloud where I try to share just the Google Cloud news and community links that are most interesting and relevant in a given week. So, I would love to connect with you there.Corey: I have known you for years, Forrest, and both of those links are new to me. So, this is the problem with being active in a bunch of different places. It's always difficult to—“Where should I find you?” “Here's a list of 15 places,” and some slipped through the cracks. I'll be signing up for both of those, so thank you.Forrest: Yeah. I used to say just follow my Twitter, but now there's, like, five Twitters, so I don't even know what to tell you.Corey: Yes. The balkanization of this is becoming very interesting. Thanks [laugh] again for taking the time to chat with me and I look forward to the next time.Forrest: All right. As always, Corey, thanks.Corey: Forrest Brazeal, Head of Developer Media at Google Cloud, and of course the Cloud Bard. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment that you undoubtedly had a generative AI model write for you and then failed to proofread it.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Turn your investment plans into reality with private money lending expert Corey Dutton. From finding the right lending company to navigating loan applications and effectively managing deals, this episode provides all the assistance you need to get started. Don't miss out!Key Takeaways to Listen forHow to make sure your hard money loan application gets approvedWhat you should look for in lending companies before making a loanIs it possible for lenders to change your loan's interest rate?Top attributes to consider when determining the right lender for youPrecautionary measures you should know to avoid loan scamsResources Mentioned in This EpisodeGoogle Search ConsoleApartment Syndication Due Diligence Checklist for Passive Investor If you want to know more about loan requests and summaries, visit: https://privatemoneyutah.com/intake and accelerate the growth of your business via hard money!About Corey DuttonCorey Dutton is a real estate lender and the founder of Private Money Utah. Corey specializes in the placement of private debt funds, and she lends on both residential and commercial real estate in the Western United States. Mrs. Dutton has funded $550 MM in loans for real estate investors since 2008. Corey obtained her MBA in 2005 from Thunderbird Graduate School of Global Management. Corey is originally from Austin, Texas, and enjoys spending time in the great outdoors. Corey Is the President of a trails foundation that builds recreational and transportation trails in rural areas of Utah.Connect with CoreyWebsite: Private Money UtahInstagram: @private_money_utah | @coreydutton.lender YouTube: Private Money UtahTo Connect With UsPlease visit our website: www.bonavestcapital.com, and please click here, to leave a rating and review!SponsorsGrow Your Show, LLCThinking About Creating and Growing Your Own Podcast But Not Sure Where To Start?Visit GrowYourShow.com and Schedule a call with Adam A. Adams

Brandon Sherman, Cloud Security Engineer at Temporal Technologies Inc., joins Corey on Screaming in the Cloud to discuss his experiences at recent cloud conferences and the ongoing changes in cloud computing. Brandon shares why he enjoyed fwd:cloudsec more than this year's re:Inforce, and how he's seen AWS events evolve over the years. Brandon and Corey also discuss how the cloud has matured and why Brandon feels ongoing change can be expected to be the continuing state of cloud. Brandon also shares insights on how his perspective on Google Cloud has changed, and why he's excited about the future of Temporal.io.About BrandonBrandon is currently a Cloud Security Engineer at Temporal Technologies Inc. One of Temporal's goals is to make our software as reliable as running water, but to stretch the metaphor it must also be *clean* water. He has stared into the abyss and it stared back, then bought it a beer before things got too awkward. When not at work, he can be found playing with his kids, working on his truck, or teaching his kids to work on his truck.Links Referenced: Temporal: https://temporal.io/ Personal website: https://brandonsherman.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: In the cloud, ideas turn into innovation at virtually limitless speed and scale. To secure innovation in the cloud, you need Runtime Insights to prioritize critical risks and stay ahead of unknown threats. What's Runtime Insights, you ask? Visit sysdig.com/screaming to learn more. That's S-Y-S-D-I-G.com/screaming.My thanks as well to Sysdig for sponsoring this ridiculous podcast.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined today by my friend who I am disappointed to say I have not dragged on to this show before. Brandon Sherman is a cloud security engineer over at Temporal. Brandon, thank you for finally giving in.Brandon: Thanks, Corey, for finally pestering me enough to convince me to join. Happy to be here.Corey: So, a few weeks ago as of this recording—I know that time is a flexible construct when it comes to the podcast production process—you gave a talk at fwd:cloudsec, the best cloud security conference named after an email subject line. Yes, I know re:Inforce also qualifies; this one's better. Tell me about what you talked about.Brandon: Yeah, definitely agree on this being the better the two conferences. I gave a talk about how the ground shifts underneath us, kind of touching on how these cloud services that we operate—and I'm mostly experienced in AWS and that's kind of the references that I can give—but these services work as a contract basis, right? We use their APIs and we don't care how they're implemented behind the scenes. At this point, S3 has been rewritten I don't know how many times. I'm sure that other AWS services, especially the longer-lived ones have gone through that same sort of rejuvenation cycle.But as a security practitioner, these implementation details that get created are sort of byproducts of, you know, releasing an API or releasing a managed service can have big implications to how you can either secure that service or respond to actions or activities that happen in that service. And when I say actions and activity, I'm kind of focused on, like, security incidents, breaches, your ability to do incident response from that.Corey: One of the reasons I've always felt that cloud providers have been cagey around how the services work under the hood is not because they don't want to talk about it so much as they don't want to find themselves committed to certain patterns that are not guaranteed as a part of the definition of the service. So if, “Yeah, this is how it works under the hood,” and you start making plans and architecting in accordance with that and they rebuild the service out from under you like they do with S3, then very often, those things that you depend upon being true could very easily no longer be true. And there's no announcement around those things.Brandon: No. It's very much Amazon is… you know, they're building a service to meet the needs of their customers. And they're trying to grow these services as the customers grow along with them. And it's absolutely within their right to act that way, to not have to tell us when they make a change because in some contexts, right, Amazon's feature update might be me as a customer a breaking change. And Amazon wants to try and keep that, what they need to tell me, as small as possible, probably not out of malice, but just because there's a lot of people out there using their services and trying to figure out what they've promised to each individual entity through either literal contracts or their API contracts is hard work. And that's not the job I would want.Corey: No. It seems like it's one of those thankless jobs where you don't get praise for basically anything. Instead, all you get to do is deal with the grim reality that people either view as invisible or a problem.Brandon: Yeah. It sort of feels like documentation. Everyone wants more and better documentation, but it's always an auxiliary part of the service creation process. The best documentation always starts out when you write the documentation first and then kind of build backwards from that, but that's rarely how I've seen software get made.Corey: No. I feel like I left them off the hook, on some level, when we say this, but I also believe in being fair. I think there's a lot of things that cloud providers get right and by and large, with any of the large cloud providers, they are going to do a better job of securing the fundamentals than you are yourself. I know that that is a controversial statement to some folks who spent way too much time in the data centers, but I stand by it.Brandon: Yeah, I agree. I've had to work in both environments and some of the easiest, best wins in security is just what do I have, so that way I know what I have to protect, what that is there. But even just that asset inventory, that's the sort of thing that back in the days of data centers—and still today; it was data centers all over the place—to do an inventory you might need to go and send an actual human with an actual clipboard or iPad or whatever, to the actual physical location and hope that they read the labels on hundreds of thousands of servers correctly and get their serial numbers and know what you have. And that doesn't even tell you what's running on them, what ports are open, what stuff you have to care about. In AWS, I can run a couple of describe calls or list calls and that forms the backbone of my inventory.There's no server that, you know, got built into a wall or lost behind and some long-forgotten migration. A lot of those basic stuff that really, really helps. Not to mention then the user-managed service like S3, you never have to care about patch notes or what an update might do. Plenty of times I've, like, hesitated upgrading a software package because I didn't know what was going to happen. Control Tower, I guess, is kind of an exception to that where you do have to care about the version of your cloud service, but stuff like, yeah, these other services is absolutely right. The undifferentiated heavy lifting it's taken care of. And hopefully, we always kind of hope that the undifferentiated heavy lifting doesn't become differentiated and heavy and lands on us.Corey: So, now that we've done the obligatory be nice to cloud providers thing, let's potentially be a little bit harsher. While you were speaking at fwd:cloudsec, did you take advantage of the fact that you were in town to also attend re:Inforce?Brandon: I did because I was given a ticket, and I wanted to go see some people who didn't have tickets to fwd:cloudsec. Yeah, we've been nice to cloud providers, but as—I haven't found I've learned a lot from the re:Inforce sessions. They're all recorded anyway. There's not even an open call for papers, right, for talking about at a re:Inforce session, “Hey, like, this would be important and fresh or things that I would be wanting to share.” And that's not the sort of thing that Amazon does with their conferences.And that's something that I think would be really interesting to change if there was a more community-minded track that let people submit, not just handpicked—although I suppose any kind of Amazon selection committee is going to be involved, but to pick out, from the community, stories or projects that are interesting that can be, not just have to get filtered through your TAM but something you can actually talk to and say, “Hey, this is something I'd like to talk about. Maybe other people would find it useful.”Corey: One of the things that I found super weird about re:Inforce this year has been that, in a normal year, it would have been a lot more notable, I think. I know for a fact that if I had missed re:Invent, for example, I would have had to be living in a cave not to see all of the various things coming out of that conference on social media, in my email, in all the filters I put out there. But unless you're looking for it, you've would not know that they had a conference that costs almost as much.Brandon: Yeah. The re:Invent-driven development cycle is absolutely a real thing. You can always tell in the lead up to re:Invent when there's releases that get pushed out beforehand and you think, “Oh, that's cool. I wonder why this doesn't get a spot at re:Invent, right, some kind of announcement or whatever.” And I was looking for that this year for re:Inforce and didn't see any kind of announcement or that kind of pre-release trickle of things that are like, oh, there's a bunch of really cool stuff. And that's not to say that cool stuff didn't happen; it just there was a very different marketing feel to it. Hard to say, it's just the vibes around felt different [laugh].Corey: Would you recommend that people attend next year—well let me back up. I've heard that they had not even announced a date for next year. Do you think there will be a re:Inforce next year?Brandon: Making me guess, predict the future, something that I'm—Corey: Yeah, do a prediction. Why not?Brandon: [laugh]. Let's engage in some idle speculation, right? I think that not announcing it was kind of a clue that there's a decent chance it won't happen because in prior years, it had been pre-announced at the—I think it was either at closing or opening ceremonies. Or at some point. There's always the, “Here's what you can look forward to next year.”And that didn't happen, so I think that's there's a decent chance this may have been the last re:Inforce, especially once all the data is crunched and people look at the numbers. It might just be… I don't know, I'm not a marketing-savvy kind of person, but it might just be that a day at re:Invent next year is dedicated to security. But then again, security is always job zero at Amazon so maybe re:Invent just becomes re:Inforce all the time, right? Do security, everybody.Corey: It just feels like a different type of conference. Whenever re:Invent there's something for everyone. At re:Inforce, there's something for everyone as long as they work in InfoSec. Because other than that, you wind up just having these really unfortunate spiels of them speaking to people that are not actually present, and it winds up missing the entire forest for the trees, really.Brandon: I don't know if I'd characterize it as that. I feel like some of the re:Inforce content was people who were maybe curious about the cloud or making progress in their companies and moving to the cloud—and in Amazon's case when they say the cloud, they mean themselves. They don't mean any other cloud. And re:Inforce tries to dispel the notion there are any other clouds.But at the same time, it feels like an attempt to try and make people feel better. There's a change underway in the industry and it still is going to continue for a while. There's still all kinds of non-cloud environments people are going to operate for probably until the end of time. But at the same time, a lot of these are moving to the cloud and they want the people who are thinking about this or engaged in it, to be comforted by that Amazon that either has these services, or there's a pattern you can follow to do something in a secure manner. I think that's that was kind of the primary audience of re:Inforce was people who were charged with doing cloud security or were exploring moving their corporate systems to AWS and they wanted some assurance that they're going to actually be doing things the right way, or someone else hadn't made those mistakes first. And if that audience has been sort of saturated, then maybe there isn't a need for that style of conference anymore.Corey: It feels like it's not intended to be the same thing at re:Invent, which is probably I guess, a bigger problem. Re:Invent for a long time has attempted to be all things to all people, and it has grown to a scale where that is no longer possible. So, they've also done a poor job of signaling that, so you wind up attending Adam Selipsky's keynote, and in many cases, find yourself bored absolutely to tears. Or you go in expecting it to be an Andy Jassy style of, “Here are 200 releases, four of them good,” and instead, you wind up just having what feels like a relatively paltry number doled out over a period of days. And I don't know that their wrong to do it; I just think it doesn't align with pre-existing expectations. I also think people expecting to go to re:Inforce to see a whole bunch of feature releases are bound to be disappointed.Brandon: Like, both of those are absolutely correct. The number of releases on the slide must always increase up and the right; away we go; we're pushing more code and making more changes to services. I mean, if you look at the history, there's always new instance types. Do they count each instance type as a new release, or they not do that?Corey: Yeah, it honestly feels like that sometimes. They also love to do price cuts where they—you wind up digging into them and something like 90% of them are services you've never heard of in regions you couldn't find on a map if your life depended on it. It's not quite the, “Yeah, the bill gets lower all the time,” that they'd love to present it as being.Brandon: Yeah. And you may even find that there's services that had updates that you didn't know about until you go and check the final bill, the Cost and Usage Report, and you look and go, “Oh, hey. Look at all the services that we were using, that our engineers started using after they heard announcements at re:Invent.” And then you find out how much you're actually paying for them. [pause]. Or that they were in use in the first place. There's no better way to find what is actually happening in your environment than, look at the bill.Corey: It's depressing that that's true. At least they finally stopped doing the slides where they talk about year-over-year, they have a histogram of number of feature and service releases. It's, no one feels good about that, even the people building the services and features because they look at that and think, “Oh, whatever I do is going to get lost in the noise.” And they're not wrong. Customers see it and freak out because how am I ever going to keep current with all this stuff? I take a week off and I spend a month getting caught back up again.Brandon: Yeah. And are you going to—you know, what's your strategy for dealing with all these new releases and features? Do you want to have a strategy of saying, “No, you can't touch any of those until we've vetted and understand them?” I mean, you don't even have to talk about security in that context; just the cost alone, understanding it's someone, someone going to run an experiment that bankrupts your company by forgetting about it or by growing into some monster in the bill. Which I suspect helps [laugh] helps you out when those sorts of things happen, right, for companies don't have that strategy.But at the same time, all these things are getting released. There's not really a good way of understanding which of these do I need to care about. Which of these is going to really impact my operational flow, my security impacts? What does this mean to me as a user of the service when there's, I don't know, an uncountable number really, or at least a number that's so big, it stops mattering that it got any bigger?Corey: One thing that I will say was great about re:Invent, I want to say 2021, was how small it felt. It felt like really a harkening back to the old re:Invents. And then you know, 2022 hit, and we go there and half of us wound up getting Covid because of course we did. But it was also this just this massive rush of, we're talking with basically the population of a midsize city just showing up inside of this entire enormous conference. And you couldn't see the people you wanted to see, it was difficult to pay attention to all there was to pay attention to, and it really feels like we've lost something somewhere.Brandon: Yeah, but at the same time is that just because there are more people in this ecosystem now? You know, 2021 may have been a callback to that a decade ago. And these things were smaller when it was still niche, but growing in kind of the whole ecosystem. And parts of—let's say, the ecosystem there, I'm talking about like, how—when I say that ecosystem there, I'm kind of talking about how in general, I want to run something in technology, right? I need a server, I need an object store, I need compute, whatever it is that you need, there is more attractive services that Amazon offers to all kinds of customers now.So, is that just because, right, we've been in this for a while and we've seen the cloud grow up and like, oh, wow, you're now in your awkward teenage phase of cloud computing [laugh]? Have we not yet—you know, we're watching the maturity to adulthood, as these things go? I really don't know. But it definitely feels a little, uh… feels a little like we've watched this cloud thing grow from a half dozen services to now, a dozen-thousand services all operating different ways.Corey: Part of me really thinks that we could have done things differently, had we known, once upon a time, what the future was going to hold. So, much of the pain I see in Cloud is functionally people trying to shove things into the cloud that weren't designed with Cloud principles in mind. Yeah, if I was going to build a lot of this stuff from scratch myself, then yeah, I would have absolutely made a whole universe of different choices. But I can't predict the future. And yet, here we are.Brandon: Yep. If I could predict the future, I would have definitely won the lottery a lot more times, avoided doing that one thing I regretted that once back in my history [laugh]. Like, knowing the future change a lot of things. But at least unless you're not letting on with something, then that's something that no one's got the ability to, do not even at Amazon.Corey: So, one of the problems I've always had when I come back from a conference, especially re:Invent, it takes me a few… well, I'll be charitable and say days, but it's more like weeks, to get back into the flow of my day-to-day work life. Was there any of that with you and re:Inforce? I mean, what is your day job these days anyway? What are you up to?Brandon: What is my day job? There's a lot. So, Temporal is a small, but quickly growing company. A lot of really cool customers that are doing really cool things with our technology and we need to build a lot of basics, essentially, making sure that when we grow, that we're going to kind of grow into our security posture. There's not anything talking about predicting the future. My prediction is that the company I work for is going to do well. You can hold your analysis on that [laugh].So, while I'm predicting what the company that I'm working at is going to do well, part of it is also what are the things that I'm going to regret not having in two or three years' time. So, some baseline cloud monitoring, right? I want that asset inventory across all of our accounts; I want to know what's going on there. There's other things that are sort of security adjacent. So, things like DNS records, domain names, a lot of those things where if we can capture this and centralize it early and build it in a way—especially that users are less unhappy about, like, not everyone, for example, is hosting their own—buying their own domains on personal cards and filing for reimbursement, that DNS records aren't scattered across a dozen different software projects and manipulated in different ways, then that sets us up.It may not be perfect today, but in a year, year-and-a-half, two years, we have the ability to then say, “Okay, we know what we're pointing at. What are the dangling subdomains? What are the things that are potential avenues of being taken over? What do we have? What are people doing?” And trying to understand how we can better help users with their needs day-to-day.Also as a side part of my day job is advising a startup Common Fate. Does just-in-time access management. And that's been a lot of fun to do as well because fundamentally—this is maybe a hot take—that, in a lot of cases, you really only need admin access and read-only access when you're doing really intensive work. In Temporal day job, we've got infrastructure teams that are building stuff, they need lots of permissions and it'd be very silly to say you can't do your job just because you could potentially use IAM and privilege escalate yourself to administrator. Let's cut that out. Let's pretend that you are a responsible adult. We can monitor you in other ways, we're not going to put restrictions between you and doing your job. Have admin access, just only have it for a short period of time, when you say you're going to need it and not all the time, every account, every service, all the time, all day.Corey: I do want to throw a shout-in for that startup you advise, Common Fate. I've been a big fan of their Granted offering for a while now. granted.dev for those who are unfamiliar. I use that to automatically generate console logins, do all kinds of other things. When you're moving between a bunch of different AWS accounts, which it kind of feels like people building the services don't have to do somehow because of their Isengard system handling it for them. Well, as a customer, can I just say that experience absolutely sucks and Granted goes a long way toward making it tolerable, if not great.Brandon: Mm-hm. Yeah, I remember years ago, the way that I would have to handle this is I would have probably a half-dozen different browsers at the same time, Safari, Chrome, the Safari web developer preview, just so I could have enough browsers to log into with, to see all the accounts I needed to access. And that was an extremely painful experience. And it still feels so odd that the AWS console today still acts like you have one account. You can switch roles, you can type in a [role 00:21:23] on a different account, but it's very clunky to use, and having software out there that makes this easier is definitely, definitely fills a major pain point I have with using these services.Corey: Tired of Apache Kafka's complexity making your AWS bill look like a phone number? Enter Redpanda. You get 10x your streaming data performance without having to rob a bank. And migration? Smoother than a fresh jar of peanut butter. Imagine cutting as much as 50% off your AWS bills. With Redpanda, it's not a dream, it's reality. Visit go.redpanda.com/duckbill. Redpanda: Because Kafka shouldn't cause you nightmares.Corey: Do you believe that there's hope? Because we have seen some changes where originally AWS just had the AWS account you'd log into, it's the root user. Great. Then they had IAM. Now, they're using what used to be known as AWS SSO, which they wound up calling IAM Access Identity Center, or—I forget the exact words they put in order, but it's confusing and annoying. But it does feel like the trend is overall towards something that's a little bit more coherent.Brandon: Mm-hm.Corey: Is the future five years from now better than it looks like today?Brandon: That's certainly the hope. I mean, we've talked about how we both can't predict the future, but I would like to hope that the future gets better. I really like GCP's project model. There's complaints I have with how Google Cloud works, and it's going to be here next year, and if the permission model is exactly how I'd like to use it, but I do like the mental organization that feels like Google was able to come in and solve a lot of those problems with running projects and having a lot of these different things. And part of that is, there's still services in AWS that don't really respect resource-based permissions or tag-based permissions, or I think the new one is attribute-based access control.Corey: One of the challenges I see, too, is that I don't think that there's been a lot of thought put into how a lot of these things are going to work between different AWS accounts. One of my bits of guidance whenever I'm talking to someone who's building anything, be it at AWS or external is, imagine an architecture diagram and now imagine that between any two resources in that diagram is now an account boundary. Because someone somewhere is going to have one there, so it sounds ridiculous, but you can imagine a microservices scenario where every component is in its own isolated account. What are you going to do now as a result? Because if you're going to build something that scales, you've got to respect those boundaries. And usually, that just means the person starts drinking.Brandon: Not a bad place to start, the organizational structure—lowercase organizations, not the Amazon service, Organizations—it's still a little tricky to get it in a way that sort of… I guess, I always kind of feel that these things are going to change and that the—right, the only constant is change. That's true. The services we use are going to change. The way that we're going to want to organize them is going to change. Our researcher is going to come out with something and say, “Hey, I found a really cool way to do something really terrible to the stuff in your cloud environment.”And that's going to happen eventually, in the fullness of time. So, how do we be able to react quickly to those kinds of changes? And how can we make sure that if you know, suddenly, we do need to separate out these services to go, you know, to decompose the monolith even more, or whatever the cool, current catchphrase is, and we have those account boundaries, which are phenomenal boundaries, they make it so much easier to do—if you can do multi-account then you've solved multi-regional on the way, you've sold failover, you've solve security issues. You have not solved the fact that your life is considerably more challenging at the moment, but I would really hope that in you know, even next year, but by the time five years comes around, that that's really been taken to heart within Amazon and it's a lot easier to be working creating services in different accounts that can talk to each other, especially in the current environment where it's kind of a mess to wire these things all together. ClickOps has its place, but some console applications just don't want to believe that you have a KMS key in another account because well, why would you put that over there? It's not like if your current account has a problem, you want to lose all your data that's encrypted.Corey: It's one of those weird things, too, where the clouds almost seem to be arguing against each other. Like, I would be hard-pressed to advise someone not to put a ‘rehydrate the entire business' level of backups into a different cloud provider entirely, but there's so steeped in the orthodoxy of no other clouds ever, that that message is not something that they can effectively communicate. And I think they're doing their customers a giant disservice by that, just because it is so much easier to explain to your auditor that you've done it than to explain why it's not necessary. And it's never true; you always have the single point of failure of the payment instrument, or the contract with that provider that could put things at risk.Is it a likely issue? No. But if you're running a publicly traded company on top of it, you'd be negligent not to think about it that way. So, why pretend otherwise?Brandon: Is that a question for me because [laugh]—Corey: Oh, that was—no, absolutely. That was a rant ending in a rhetorical question. So, don't feel you have to answer it. But getting the statement out there because hopefully, someone at Amazon is listening to this.Brandon: That's, uh, hopefully, if you find out who's the one that listens to this and can affect it, then yeah, I'd like to send them a couple of emails because absolutely. There's room out there, there will always be room for at least two providers.Corey: Yeah, I'd say a third, but I don't know that Google is going to have the attention span to still have a cloud offering by lunchtime today.Brandon: Yeah. I really wish that I had more faith in the services and that they weren't going—you know, speaking of services changing underneath you, that's definitely a—speaking of services changing underneath, you definitely a major disservice if you don't know—if you're going to put into work into architecting and really using cloud providers as they're meant to be used. Not in a, sort of, least common denominator sense, in which case, you're not in good shape.Corey: Right. You should not be building something with an idea toward what if this gets deprecated. You shouldn't have to think about that on a consistent basis.Brandon: Mm-hm. Absolutely. You should expect those things to change because they will, right, the performance impact. I mean, the performance of these services is going to change, the underlying technology that the providers use is going to change, but you should still be able to mostly expect that at least the API calls you make are going to still be there and still be consistent come this time next year.Corey: The thing that really broke me was the recent selling off of Google domains to Squarespace. Nothing against Squarespace, but they have a different target market in many respects. And oh, I'm a Google customer, you're now going to give all of my information to a third party I never asked to deal with. Great. And more to the point, if I recommend Google to folks because as has happened in years past, then they canceled the thing that I recommended, then I looked like a buffoon. So, we've gotten to a point now where it has become so steady and so consistent, that I fear I cannot, in good conscience, recommend a Google product without massive caveats. Otherwise, I look like a clown or worse, a paid shill.Brandon: Yeah. And when you want to start incorporating these things into the core of your business, to take that point about, you know, total failover scenarios, you should, you know, from you want it to have a domain registered in a Google service that was provisioned to Google Cloud services, that whole sort of ecosystem involved there, that's now gone, right? If I want to use Google Cloud with a Google Cloud native domain name hosting services, I can't. How am—I just—now I can't [laugh]. There's, like, not workarounds available.I've got to go to some other third-party and it just feels odd that an organization would sort of take those core building blocks and outsource them. [I know 00:29:05] that Google's core offering isn't Google Cloud; it's not their primary focus, and it kind of reflects that, which was a shame. There's things that I'd love to see grow out of Google Cloud and get better. And, you know, competition is good for the whole cloud computing industry.Corey: I think that it's a sad thing, but it's real, that there are people who were passionate defenders of Google over the years. I used to be one. We saw a bunch of them with Stadia fans coming out of the woodwork, and then all those people who have defended Google and said, “No, no, you can trust Google on this service because it's different,” for some reason or other, then wind up looking ridiculous. And some of the staunchest Google defenders that I've seen are starting to come around to my point of view. Eventually, you've run out of people who are willing to get burned if you burn them all.Brandon: Yeah. I've always been a little, uh… maybe this is the security Privacy part of me; I've always been a little leery of the services that really want to capture and gather your data. But I always respected the Google engineering that went into building these things at massive scale. It's something beyond my ability to understand as I haven't worked in something that big before. And Google made it look… maybe not effortless, but they made it look like they knew what they were doing, they could build something really solid.And I don't know if that's still true because it feels like they might know how to build something, and then they'll just dismantle it and turn it over to somebody else, or just dismantle it completely. And I think humans, we do a lot of things because we don't want to look foolish and… now recommending Google Cloud starts to make you wonder, “Am I going to look foolish?” Is this going to be a reflection on me in a year or two years, when you got to come in to say, “Hey, I guess that whole thing we architected around, it's being sold to someone else. It's being closed down. We got to transfer and rearchitect our whole whatever we built because of factors out of our control.” I want to be rearchitecting things because I screwed it up. I want to be rearchitecting things because I made an interesting novel mistake, not something that's kind of mundane, like, oh, I guess the thing we were going to use got shut down. Like, that makes it look like not only can I not predict the future, but I can't even pretend to read the tea leaves.Corey: And that's what's hard is because, on some level, our job, when we work in operations and cloud and try and make these decisions, is to convince the business we know what we're talking about. And when we look foolish, we don't make that same mistake again.Brandon: Mm-hm. Billing and security are oftentimes frequently aligned with each other. We're trying to convince the business that we need to build things a certain way to get a certain outcome, right? Either lower costs or more performance for the dollar, so that way, we don't wind up in the front page of newspapers, any kinds of [laugh] any kind of those things.Corey: Oh, yes. I really want to thank you for taking the time to speak with me. If people want to learn more, where's the best place for them to find you?Brandon: The best place to find me, I have a website about me, [brandonsherman.com 00:32:13]. That's where I post stuff. There's some links to—I have a [Mastodon 00:32:18] profile. I'm not much of a social, sort of post your information out there kind of person, but if you want to get a hold of me, then that's probably the best way to find me and contact me. Either that or head out to the desert somewhere, look for a silver truck out in the dunes and without technology around. It's another good spot if you can find me there.Corey: And I will include a link to that, of course, in the [show notes 00:32:45]. Thank you so much for taking the time to speak with me today. As always, I appreciate it.Brandon: Thank you very much for having me, Corey. Good to chat with you.Corey: Brandon Sherman, cloud security engineer at Temporal. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that will somehow devolve into you inviting me to your new uninspiring cloud security conference that your vendor is putting on, and is of course named after an email subject line.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Jonathan (Koz) Kozolchyk, General Manager for Certificate Services at AWS, joins Corey on Screaming in the Cloud to discuss the best practices he recommends around certificates. Jonathan walks through when and why he recommends private certs, and the use cases where he'd recommend longer or unusual expirations. Jonathan also highlights the importance of knowing who's using what cert and why he believes in separating expiration from rotation. Corey and Jonathan also discuss their love of smart home devices as well as their security concerns around them and how they hope these concerns are addressed moving forward. About JonathanJonathan is General Manager of Certificate Services for AWS, leading the engineering, operations, and product management of AWS certificate offerings including AWS Certificate Manager (ACM) AWS Private CA, Code Signing, and Encryption in transit. Jonathan is an experienced leader of software organizations, with a focus on high availability distributed systems and PKI. Starting as an intern, he has built his career at Amazon, and has led development teams within our Consumer and AWS businesses, spanning from Fulfillment Center Software, Identity Services, Customer Protection Systems and Cryptography. Jonathan is passionate about building high performing teams, and working together to create solutions for our customers. He holds a BS in Computer Science from University of Illinois, and multiple patents for his work inventing for customers. When not at work you'll find him with his wife and two kids or playing with hobbies that are hard to do well with limited upside, like roasting coffee.Links Referenced: AWS website: https://www.aws.com Email: mailto:koz@amazon.com Twitter: https://twitter.com/seakoz TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: In the cloud, ideas turn into innovation at virtually limitless speed and scale. To secure innovation in the cloud, you need Runtime Insights to prioritize critical risks and stay ahead of unknown threats. What's Runtime Insights, you ask? Visit sysdig.com/screaming to learn more. That's S-Y-S-D-I-G.com/screaming.My thanks as well to Sysdig for sponsoring this ridiculous podcast.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. As I record this, we are about a week and a half from re:Inforce in Anaheim, California. I am not attending, not out of any moral reason not to because I don't believe in cloud security or conferences that Amazon has that are named after subject lines, but rather because I am going to be officiating a wedding on the other side of the world because I am an ordained minister of the Church of There Is A Problem With This Website's Security Certificate. So today, my guest is going to be someone who's a contributor, in many ways, to that religion, Jonathan Kozolchyk—but, you know, we all call him Koz—is the general manager for Certificate Services at AWS. Koz, thank you for joining me.Koz: Happy to be here, Corey.Corey: So, one of the nice things about ACM historically—the managed service that handles certificates from AWS—is that for anything public-facing, it's free—which is always nice, you should not be doing upcharges for security—but you also don't let people have the private portion of the cert. You control all of the endpoints that terminate SSL. Whereas when I terminate SSL myself, it terminates on the floor because I've dropped things here and there, which means that suddenly the world of people exposing things they shouldn't or expiry concerns just largely seemed to melt away. What was the reason that Amazon looked around at the landscape and said, “Ah, we're going to launch our own certificate service, but bear with me here, we're not going to charge people money for it.” It seems a little bit out of character.Koz: Well, Amazon itself has been battling with certificates for years, long before even AWS was a thing, and we learned that you have to automate. And even that's not enough; you have to inspect and you have to audit, you need a controlled loop. And we learned that you need a closed loop to truly manage it and make sure that you don't have outages. And so, when we built ACM, we built it saying, we need to provide that same functionality to our customers, that certificates should not be the thing that makes them go out. Is that we need to keep them available and we need to minimize the sharp edges customers have to deal with.Corey: I somewhat recently caught some flack on one of the Twitter replacement social media sites for complaining about the user experience of expired SSL certs. Because on the one hand, if I go to my bank's website, and the response is that instead, the server is sneakyhackerman.com, it has the exact same alert and failure mode as, holy crap, this certificate reached its expiry period 20 minutes ago. And from my perspective, one of those is a lot more serious than the other. What also I wind up encountering is not just when I'm doing banking, but when I'm trying to read some random blog on how to solve a technical problem. I'm not exactly putting personal information into the thing. It feels like that was a missed opportunity, agree or disagree?Koz: Well, I wouldn't categorize it as a missed opportunity. I think one of the things you have to think about with security is you have to keep it simple so that everyone, whether they're a technologist or not, can abide by the rules and be safe. And so, it's much easier to say to somebody, “There's something wrong. Period. Stop.” versus saying there are degrees of wrongness. Now, that said, boy, do I wish we had originally built PKI and TLS such that you could submit multiple certificates to somebody, in a connection for example, so that you could always say, you know, my certificates can expire, but I've got two, and they're off by six months, for example. Or do something so that you don't have to close failed because the certificate expired.Corey: It feels like people don't tend to think about what failure modes are going to look like. Because, pfhh, as an expired certificate? What kind of irresponsible buffoon would do such a thing? But I've worked in enough companies where you have historically, the wildcard cert because individual certs cost money, once upon a time. So, you wound up getting the one certificate that could work on all of the stuff that ends in the same domain.And that was great, but then whenever it expired, you had to go through and find all the places that you put it and you always miss some, so things would break for a while and the corporate response was, “Ugh, that was awful. Instead of a one-year certificate, let's get a five-year or a ten-year certificate this time.” And that doesn't make the problem better; it makes it absolutely worse because now it proliferates forever. Everyone who knows where that thing lives is now long gone by the time it hits again. Counterintuitively, it seems the industry has largely been moving toward short-lived certs. Let's Encrypt, for example, winds up rotating every 90 days, by my estimation. ACM is a year, if memory serves.Koz: So, ACM certs are 13 months, and we start rotating them around the 11th month. And Let's Encrypt offers you 90-day certs, but they don't necessarily require you to rotate every 90 days; they expire in 90 days. My tip for everybody is divorce expiration from rotation. So, if your cert is a 90-day cert, rotate it at 45 days. If your cert is a year cert, give yourself a couple of months before expiration to start the rotation. And then you can alarm on it on your own timeline when something fails, and you still have time to fix it.Corey: This makes a lot of sense in—you know, the second time because then you start remembering, okay, everywhere I use this cert, I need to start having alarms and alerts. And people are bad at these things. What ACM has done super well is that it removes that entire human from the loop because you control all of the endpoints. You folks have the ability to rotate it however often you'd like. You could have picked arbitrary timelines of huge amounts of time or small amounts of time and it would have been just fine.I mean, you log into an EC2 instance role and I believe the credentials get passed out of either a 6 or a 12-hour validity window, and they're consistently rotating on the back end and it's completely invisible to the customer. Was there ever thought given to what that timeline should be,j what that experience should be? Or did you just, like, throw a dart at a wall? Like, “Yeah, 13 months feels about right. We're going to go with that.” And never revisited it. I have a guess which—Koz: [laugh].Corey: Side of that it was. Did you think at all about what you were doing at the time, or—yeah.Koz: So, I will admit, this happened just before I got there. I got to ACM after—Corey: Ah, blame the predecessor. Always a good call.Koz: —the launch. It's a God-given right to blame your predecessor.Corey: Oh, absolutely. It's their entire job.Koz: I think they did a smart job here. What they did was they took the longest lifetime cert that was then allowed, at 13 months, knowing that we were going to automate the rotation and basically giving us as much time as possible to do it, right, without having to worry about scaling issues or having to rotate overly frequently. You know, there are customers who while I don't—I strongly disagree with [pinning 00:07:35], for example, but there are customers out there who don't like certs to change very often. I don't recommend pinning at all, but I understand these cases are out there, and changing it once every year can be easier on customers than changing it every 20 minutes, for example. If I were to pick an ideal rotation time, it'd probably be under ten days because an OCSP response is good for ten days and if you rotate before, then I never have to update an OCSP response, for example. But changing that often would play havoc with many systems because of just the sheer frequency you're rotating what is otherwise a perfectly valid certificate.Corey: It is computationally expensive to generate certificates at scale, I would imagine.Koz: It starts to be a problem. You're definitely putting a lot of load on the HSMs at that point, [laugh] when you're generating. You know, when you have millions of certs out in deployment, you're generating quite a few at a time.Corey: There is an aspect of your service that used to be part of ACM and now it's its own service—which I think is probably the right move because it was confusing for a lot of customers—Amazon looks around and sees who can we compete with next, it feels like sometimes. And it seemed like you were squarely focused on competing against your most desperate of all enemies, my crappy USB key where I used to keep the private CA I used at any given job—at the time; I did not keep it after I left, to be very clear—for whatever I'm signing things for certificates for internal use. You're, like, “Ah, we can have your crappy USB key as a service.” And sure enough, you wound up rolling that out. It seems like adoption has been relatively brisk on that, just because I see it in almost every client account I work with.Koz: Yeah. So, you're talking about the private CA offering which is—Corey: I—that's right. Private CA was the new service name. Yes, it used to be a private certificate authority was an aspect of ACM, and now you're—mmm, we're just going to move that off.Koz: And we split it out because like you said customers got confused. They thought they had to only use it with ACM. They didn't understand it was a full standalone service. And it was built as a standalone service; it was not built as part of ACM. You know, before we built it, we talked to customers, and I remember meeting with people running fairly large startups, saying, “Yes, please run this for me. I don't know why, but I've got this piece of paper in my sock drawer that one of my security engineers gave me and said, ‘if something goes wrong with our CA, you and two other people have to give me this piece of paper.'” And others were like, “Oh, you have a piece of paper? I have a USB stick in my sock drawer.” And like, this is what, you know, the startup world was running their CAs from sock drawers as far as I can tell.Corey: Yeah. A piece of paper? Someone wrote out the key by hand? That sounds like hell on earth.Koz: [sigh]. It was a sharding technique where you needed, you know, three of five or something like that to—Corey: Oh, they, uh, Shamir's Secret Sharing Service.Koz: Yes.Corey: The SSSS. Yeah.Koz: Yes. You know, and we looked at it. And the other alternative was people would use open-source or free certificate authorities, but without any of the security, you'd want, like, HSM backing, for example, because that gets really expensive. And so yeah, we did what our customers wanted: we built this service. We've been very happy with the growth it's taken and, like you said, we love the places we've seen it. It's gone into all kinds of different things, from the traditional enterprise use cases to IoT use cases. At one point, there's a company that tracks sheep and every collar has one of our certs in it. And so, I am active in the sheep-tracking industry.Corey: I am certain that some wit is going to comment on this. “Oh, there's a company out there that tracks sheep. Yeah, it's called Apple,” or Facebook, or whatever crappy… whatever axe someone has to grind against any particular big company. But you're talking actual sheep as in baa, smell bad, count them when going to sleep?Koz: Yes. Actual sheep.Corey: Excellent, excellent.Koz: The certs are in drones, they're in smart homes, so they're everywhere now.Corey: That is something I want to ask you about because I found that as a competition going on between your service, ACM because you won't give me the private keys for reasons that we already talked about, and Let's Encrypt. It feels like you two are both competing to not take my money, which is, you know, an odd sort of competition. You're not actually competing, you're both working for a secure internet in different ways, but I wind up getting certificates made automatically for me for all of my internal stuff using Let's Encrypt, and with publicly resolvable domain names. Why would someone want a private CA instead of an option that, okay, yeah, we're only using it internally, but there is public validity to the certificate?Koz: Sure. And just because I have to nitpick, I wouldn't say we're competing with them. I personally love Let's Encrypt; I use them at home, too. Amazon supports them financially; we give them resources. I think they're great. I think—you know, as long as you're getting certs I'm happy. The world is encrypted and I—people use private CA because fundamentally, before you get to the encryption, you need secure identity. And a certificate provides identity. And so, Let's Encrypt is great if you have a publicly accessible DNS endpoint that you can prove you own and get a certificate for and you're willing to update it within their 90-day windows. Let's use the sheep example. The sheep don't have publicly valid DNS endpoints and so—Corey: Or to be very direct with you, they also tend to not have terrific operational practices around updating their own certificates.Koz: Right. Same with drones, same with internal corporate. You may not want your DNS exposed to the internet, your internal sites. And so, you use a private certificate where you own both sides of the connection, right, where you can say—because you can put the CA in the trust store and then that gets you out of having to be compliant with the CA browser form and the web trust rules. A lot of the CA browser form dictates what a public certificate can and can't do and the rules around that, and those are built very much around the idea of a browser connecting to a client and protecting that user.Corey: And most people are not banking on a sheep.Koz: Most people are not banking on a sheep, yes. But if you have, for example, a database that requires a restart to pick up a new cert, you're not going to want to redo that every 90 days. You're probably going to be fine with a five-year certificate on that because you want to minimize your downtime. Same goes with a lot of these IoT devices, right? You may want a thousand-year cert or a hundred-year cert or cert that doesn't expire because this is a cert that happens at—that is generated at creation for the device. And it's at birth, the machine is manufactured and it gets a certificate and you want it to live for the life of that device.Or you have super-secret-project.internal.mycompany.com and you don't want a publicly visible cert for that because you're not ready to launch it, and so you'll start with a private cert. Really, my advice to customers is, if you own both pieces of the connection, you know, if you have an API that gets called by a client you own, you're almost always better off with a private certificate and managing that trust store yourself because then you are subject not to other people's rules, but the rules that fit the security model and the threat assessment you've done.Corey: For the publication system for my newsletter, when I was building it out, I wanted to use client certificates as a way of authenticating that it was me. Because I only have a small number of devices that need to talk to this thing; other people don't, so how do I submit things into my queue and manage it? And back in those ancient days, the API Gateways didn't support TLS authentication. Now, they do. I would redo it a bunch of different ways. They did support API key as an authentication mechanism, but the documentation back then was so terrible, or I was so new to this stuff, I didn't realize what it was and introduced it myself from first principles where there's a hard-coded UUID, and as long as there's the right header with that UUID, I accept it, otherwise drop it on the floor. Which… there are probably better ways to do that.Koz: Sure. Certificates are, you know, a very popular way to handle that situation because they provide that secure identity, right? You can be assured that the thing connecting to you can prove it is who they say they are. And that's a great use of a private CA.Corey: Changing gears slightly. As we record this, we are about two weeks before re:Inforce, but I will be off doing my own thing on that day. Anything interesting and exciting coming out of your group that's going to be announced, with the proviso, of course, that this will not air until after re:Inforce.Koz: Yes. So, we are going to be pre-announcing the launch of a connector for Active Directory. So, you will be able to tie your private CA instance to your Active Directory tree and use private CA to issue certificates for use by Active Directory for all of your Windows hosts for the users in that Active Directory tree.Corey: It has been many years since I touched Windows in anger, but in 2003 or so, I was a mediocre Small Business Windows Server Admin. Doesn't Active Directory have a private CA built into it by default for whenever you're creating a new directory?Koz: It does.Corey: Is that one of the FSMO roles? I'm trying to remember offhand.Koz: What's a Fimal?Corey: FSMO. F-S-M-O. There are—I forget, it's some trivia question that people love to haze each other with in Microsoft interviews. “What are the seven FSMO roles?” At least back then. And have to be moved before you decommission a domain controller or you're going to have tears before bedtime.Koz: Ah. Yeah, so Microsoft provides a certificate authority for use with Active Directory. They've had it for years and they had to provide it because back then nobody had a certificate authority, but AD needed one. The difference here is we manage it for you. And it's backed by HSMs. We ensure that the keys are kept secure. It's a serverless connection to your Active Directory tree, you don't have to run any software of ours on your hosts. We take care of all of it.And it's been the top requests from customers for years now. It's been quite [laugh] a bit of effort to build it, but we think customers are going to love it because they're going to get all the security and best practices from private CA that they're used to and they can decommission their on-prem certificate authority and not have to go through the hassle of running it.Corey: A big area where I see a lot of private CA work has been in the realm of desktops for corporate environments because when you can pass out your custom trusted root or trusted CA to all of the various nodes you have and can control them, it becomes a lot easier. I always tended to shy away from it, just because in small businesses like the one that I own, I don't want to play corporate IT guy more than I absolutely have to.Koz: Yeah. Trust or management is always a painful part of PKI. As if there weren't enough painful things in PKI. Trust store management is yet another one. Thankfully, in the large enterprises, there are good tooling out there to help you manage it for the corporate desktops and things like that.And with private CA, you can also, if you already have an offline root that is in all of your trust stores in your enterprise, you can cross-sign the route that we give you from private CA into that hierarchy. And so, then you don't have to distribute a new trust store out if you don't want to.Corey: This is a tricky release and I'm very glad I'm taking the week off it's getting announced because there are two reactions that are going to happen to any snarking I can do about this. The first is no one knows what the hell this is and doesn't have any context for the rest, and the other folks are going to be, “Yes, shut up clown. This is going to change my workflow in amazing ways. I'll deal with your nonsense later. I want to do this.” And I feel like one of those constituencies is very much your target market and the other isn't. Which is fine. No service that AWS offers—except the bill—is for every customer, but every service is for someone.Koz: That's right. We've heard from a lot of our customers, especially as they—you know, the large international ones, right, they find themselves running separate Active Directory CAs in different countries because they have different regulatory requirements and separations that they want to do. They are chomping at the bit to get this functionality because we make it so easy to run a private CA in these different regions. There's certainly going to be that segment at re:Inforce, that's just happy certificates happen in the background and they don't think anything about where they come from and this won't resonate with them, but I assure you, for every one of them, they have a colleague somewhere else in the building that is going to do a happy dance when this launches because there's a great deal of customer heavy-lifting and just sharp edges that we're taking away from them. And we'll manage it for them, and they're going to love it.[midroll 0:21:08]Corey: One thing that I have seen the industry shift to that I love is the Let's Encrypt model, where the certificate expires after 90 days. And I love that window because it is a quarter, which means yes, you can do the crappy thing and have a calendar reminder to renew the thing. It's not something you have to do every week, so you will still do it, but you're also not going to love it. It's just enough friction to inspire people to automate these things. And that I think is the real win.There's a bunch of things like Certbot, I believe the protocol is called ACME A-C-M-E, always in caps, which usually means an acronym or someone has their caps lock key pressed—which is of course cruise control for cool. But that entire idea of being able to have a back-and-forth authentication pass and renew certificates on a schedule, it's transformative.Koz: I agree. ACM, even Amazon before ACM, we've always believed that automation is the way out of a lot of this pain. As you said earlier, moving from a one-year cert to a five-year cert doesn't buy you anything other than you lose even more institutional knowledge when your cert expires. You know, I think that the move to further automation is great. I think ACME is a great first step.One of the things we've learned is that we really do need a closed loop of monitoring to go with certificate issuance. So, at Amazon, for example, every cert that we issue, we also track and the endpoints emit metrics that tell us what cert they're using. And it's not what's on disk, it's what's actually in the endpoint and what they're serving from memory. And we know because we control every cert issued within the company, every cert that's in use, and if we see a cert in use that, for example, isn't the latest one we issued, we can send an alert to the team that's running it. Or if we've issued a cert and we don't see it in use, we see the old ones still in use, we can send them an alert, they can alarm and they can see that, oh, we need to do something because our automation failed in this case.And so, I think ACME is great. I think the push Let's Encrypt did to say, “We're going to give you a free certificate, but it's going to be short-lived so you have to automate,” that's a powerful carrot and stick combination they have going, and I think for many customers Certbot's enough. But you'll see even with ACM where we manage it for our customers, we have that closed loop internally as well to make sure that the cert when we issue a new cert to our client, you know, to the partner team, that it does get picked up and it does get loaded. Because issuing you a cert isn't enough; we have to make sure that you're actually using the new certificate.Corey: I also have learned as a result of this, for example, that AWS certificate manager—Amazon Certificate Manager, the ACM, the certificate thingy that you run, that so many names, so many acronyms. It's great—but it has a limit—by default—of 2500 certificates. And I know this because I smacked into it. Why? I wasn't sitting there clicking and adding that many certificates, but I had a delightful step function pattern called ‘The Lambda invokes itself.' And you can exhaust an awful lot of resources that way because I am bad at programming. That is why for safety, I always recommend that you iterate development-wise in an account that is not production, and preferably one that belongs to someone else.Koz: [laugh]. We do have limits on cert issuance.Corey: You have limits on everything in AWS. As it should because it turns out that whatever there's not a limit, A, free database just dropped, and B, things get hammered to death. You have to harden these things. And it's one of those things that's obvious once you've operated at a certain point of scale, but until you do, it just feels arbitrary and capricious. It's one of those things where I think Amazon is still—and all the cloud companies who do this—are misunderstood.Koz: Yeah. So, in the case of the ACM limits, we look at them fairly regularly. Right now, they're high enough that most of our customers, vast majority, never come close to hitting it. And the ones that do tend to go way over.Corey: And it's been a mistake, as in my case as well. This was not a complaint, incidentally. It was like, well, I want to wind up having more waste and more ridiculous nonsense. It was not my concern.Koz: No no no, but we do, for those customers who have not mistake use cases but actual use cases where they need more, we're happy to work with their account teams and with the customer and we can up those limits.Corey: I've always found that limit increases, with remarkably few exceptions, the process is, “Explain to you what your use case is here.” And I feel like that is a screen for, first, are you doing something horrifying for which there's a better solution? And two, it almost feels like it's a bit of a customer research approach where this is fine for most customers. What are you folks doing over there and is there a use case we haven't accounted for in how we use the service?Koz: I always find we learned something when we look at the [P100 00:26:05] accounts that they use the most certificates, and how they're operating.Corey: Every time I think I've seen it all on AWS, I just talk to one more customer, and it's back to school I go.Koz: Yep. And I thank them for that education.Corey: Oh, yeah. That is the best part of working with customers and honestly being privileged enough to work with some of these things and talk to the people who are building really neat stuff. I'm just kibitzing from the sideline most of the time.Koz: Yeah.Corey: So, one last topic I want to get into before we call it a show. You and I have been talking a fair bit, out of school, for lack of a better term, around a couple of shared interests. The one more germane to this is home automation, which is always great because especially in a married situation, at least as I am and I know you are as well, there's one partner who is really into home automation and the other partner finds himself living in a haunted house.Koz: [laugh]. I knew I had won that battle when my wife was on a work trip and she was in a hotel and she was talking to me on the phone and she realized she had to get out of bed to turn the lights off because she didn't have our Alexa Good Night routine available to her to turn all the lights off and let her go to bed. And so, she is my core customer when I do the home automation stuff. And definitely make sure my use cases and my automations work for her. But yeah, I'm… I love that space.Coincidentally, it overlaps with my work life quite a bit because identity in smart home is a challenge. We're really excited about the Matter standard. For those listening who aren't sure what that is, it's a new end-all be-all smart home standard for defining devices in a protocol-independent way that lets your hubs talk to devices without needing drivers from each company to interact with them. And one of the things I love about it is every device needs a certificate to identify it. And so, private CA has been a great partner with Matter, you know, it goes well with it.In fact, we're one of the leading certificate authorities for Matter devices. Customers love the pricing and the way they can get started without talking to anybody. So yeah, I'm excited to see, you know, as a smart home junkie and as a PKI guy, I'm excited to see Matter take off. Right now I have a huge amalgamation of smart home devices at home and seeing them all go to Matter will be wonderful.Corey: Oh, it's fantastic. I am a little worried about aspects of this, though, where you have things that get access to the internet and then act as a bridge. So suddenly, like, I have a IoT subnet with some controls on it for obvious reasons and honestly, one of the things I despise the most in this world has been the rise of smart TVs because I just want you to be a big dumb screen. “Well, how are you going to watch your movies?” “With the Apple TV I've plugged into the thing. I just want you to be a screen. That's it.” So, I live a bit in fear of the day where these things find alternate ways to talk to the internet and, you know, report on what I'm watching.Koz: Yeah, I think Matter is going to help a lot with this because it's focused on local control. And so, you'll have to trust your hub, whether that's your TV or your Echo device or what have you, but they all communicate securely amongst themselves. They use certificates for identification, and they're building into Matter a robust revocation mechanism. You know, in my case at home, my TV's not connected to the internet because I use my Fire TV to talk to it, similar to your Apple TV situation. I want a device I control not my TV, doing it. I'm happy with the big dumb screen.And I think, you know, what you're going to end up doing is saying there's a device out there you'll trust maybe more than others and say, “That's what I'm going to use as my hub for my Matter devices and that's what will speak to the internet,” and otherwise my Matter devices will talk directly to my hub.Corey: Yeah, there's very much a spectrum of trust. There's the, this is a Linux distribution on a computer that I installed myself and vetted and wound up contributing to at one point on the one end of the spectrum, and the other end of the spectrum of things you trust the absolute least in this world, which are, of course, printers. And most things fall somewhere in between.Koz: Yes, right, now, it is a Wild West of rebranded white-label applications, right? You have all kinds of companies spitting out reference designs as products and white labeling the control app for it. And so, your phone starts collecting these smart home applications to control each one of these things because you buy different switches from different people. I'm looking forward to Matter collapsing that all down to having one application and one control model for all of the smart home devices.Corey: Wemo explicitly stated that they're not going to be pursuing this because it doesn't let them differentiate the experience. Read as, cash grab. I also found out that Wemo—which is, of course, a Belkin subsidiary—had a critical vulnerability in some of the light switches it offered, including the one built into the wall in this room—until a week ago—where they're not going to be releasing a patch for it because those are end-of-life. Really? Because I log into the Wemo app and the only way I would have known this has been the fact that it's been a suspiciously long time since there was a firmware update available for it. But that's it. Like, the only way I found this out was via a security advisory, at which point that got ripped out of the wall and replaced with something that isn't, you know, horrifying. But man did that bother me.Koz: Yeah. I think this is still an open issue for the smart home world.Corey: Every company wants a moat of some sort, but I don't want 15 different apps to manage this stuff. You turned me on to Home Assistant, which is an open-source, home control automation system and, on some level, the interface is very clearly built by a bunch of open-source people—good for them; they could benefit from a graphic designer or three to—or user experience person to tie it all together, but once you wrap your head around it, it works really well, where I have automations let me do different things. They even have an Apple Watch app [without its 00:32:14] complications on it. So, I can tap the thing and turn on the lights in my office to different levels if I don't want to talk to the robot that runs my house. And because my daughter has started getting very deeply absorbed into some YouTube videos from time to time, after the third time I asked her what—I call her name, I tap a different one and the internet dies to her iPad specifically, and I wait about 30 to 45 seconds, and she'll find me immediately.Koz: That's an amazing automation. I love Home Assistant. It's certainly more technical than I could give to my parents, for example, right now. I think things like Matter are going to bring a lot of that functionality to the easier-to-use hubs. And I think Home Assistant will get better over time as well.I think the only way to deal with these devices that are going to end-of-life and stop getting support is have them be local control only and so then it's your hub that keeps getting support and that's what talks to the internet. And so, you don't—you know, if there's a vulnerability in the TCP stack, for example, in your light switch, but your light switch only talks to the hub and isn't allowed to talk to anything else, how severe is that? I don't think it's so bad. Certainly, I wall off all of my IoT devices so that they don't talk to the rest of my network, but now you're getting a fairly complicated networking… mojo that listeners to your podcast I'm sure capable of, but many people aren't.Corey: I had something that did something very similar and then I had to remove a lot of those restrictions, try to diagnose a phantom issue that it appears was an unreported bug in the wireless AP when you use its second ethernet port as a bridge, where things would intermittently not be able to cross VLANs when passing through that. As in, the initial host key exchange for SSH would work and then it would stall and resets on both sides and it was a disaster. It was, what is going on here? And the answer was it was haunted. So, a small architecture change later, and the problem has not recurred. I need to reapply those restrictions.Koz: I mean, these are the kinds of things that just make me want to live in a shack in the woods, right? Like, I don't know how you manage something like that. Like, these are just pain points all over. I think over time, they'll get better, but until then, that shack in the woods with not even running water sounds pretty appealing.Corey: Yeah, at some level, having smart lights, for example, one of the best approaches that all the manufacturers I've seen have taken, it still works exactly as you would expect when you hit the light switch on the wall because that's something that you really need to make work or it turns out for those of us who don't live alone, we will not be allowed to smart home things anymore.Koz: Exactly. I don't have any smart bulbs in my house. They're all smart switches because I don't want to have to put tape over something and say, “Don't hit that switch.” And then watch one of my family members pull the tape off and hit the switch anyways.Corey: I have floor lamps with smart bulbs in them, but I wind up treating them all as one device. And I mean, I've taken the switch out from the root because it's, like, too many things to wind up slicing and dicing. But yeah, there's a scaling problem because right now a lot of this stuff—because Matter is not quite there all winds up using either Zigbee—which is fine; I have no problem with that it feels like it's becoming Matter quickly—or WiFi. And there is an upper bound to how many devices you want or can have on some fairly limited frequency.Koz: Yeah. I think this is still something that needs to be resolved. You know, I've got hundreds of devices in my house. Thankfully, most of them are not WiFi or Zigbee. But I think we're going to see this evolve over time and I'm excited for it.Corey: I was talking to someone where I was explaining that, well, how this stuff works. Like, “Well, how many devices could you possibly have on your home network?” And at the time it was about 70 or 80. And they just stared at me for the longest time. I mean, it used to be that I could name all the computers in my house. I can no longer do that.Koz: Sure. Well, I mean, every light switch ends up being a computer.Corey: And that's the weirdest thing is that it's, I'm used to computers, being a thing that requires maintenance and care and feeding and security patches and—yes, relevant to your work—an SSL certificate. It's like, so what does all of that fancy wizardry do? Well, when it receives a signal, it completes a circuit. The end. And it's, are really better off for some of these things? There are days we wonder.Koz: Well, my light bill, my electric bill, is definitely better off having these smart switches because nobody in my house seems to know how to turn a light switch off. And so, having the house do it itself helps quite a bit.Corey: To be very clear, I would skewer you if you worked on an AWS service that actually charged money for anything for what you just said about the complaining about light bills and optimizing light bills and the rest—Koz: [laugh].Corey: —but I've never had to optimize your service's certificate bill beca—after you've spun off the one thing that charges—because you can't cost optimize free, as it turns out, and I've yet to find a way to the one optimization possible where now you start paying customers money. I'm sure there's a way to do that somewhere but damned if I can find it.Koz: Well, if you find a way to optimize free, please let me know and I'll share it with all of our customers.Corey: [laugh]. Isn't that the truth? I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place for them to find you?Koz: I can give you the standard AWS answer.Corey: Yeah, www.aws.com. Yeah.Koz: Well, I would have said koz@amazon.com. I'm always happy to talk about certs and PKI. I find myself less active on social media lately. You can find me, I guess, on Twitter as @seakoz and on Bluesky as [kozolchyk.com 00:38:03].Corey: And we will put links to all of that in the [show notes 00:38:06]. Thank you so much for being so generous with your time. I appreciate it.Koz: Always happy, Corey.Corey: Jonathan Kozolchyk, or Koz as we all call him, general manager for Certificate Services at AWS. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment that then will fail to post because your podcast platform of choice has an expired security certificate.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Jack Roehrig, Technology Evangelist at Uptycs, joins Corey on Screaming in the Cloud for a conversation about security awareness, ChatGPT, and more. Jack describes some of the recent developments at Uptycs, which leads to fascinating insights about the paradox of scaling engineering teams large and small. Jack also shares how his prior experience working with AskJeeves.com has informed his perspective on ChatGPT and its potential threat to Google. Jack and Corey also discuss the evolution of Reddit, and the nuances of developing security awareness trainings that are approachable and effective.About JackJack has been passionate about (obsessed with) information security and privacy since he was a child. Attending 2600 meetings before reaching his teenage years, and DEF CON conferences shortly after, he quickly turned an obsession into a career. He began his first professional, full-time information-security role at the world's first internet privacy company; focusing on direct-to-consumer privacy. After working the startup scene in the 90's, Jack realized that true growth required a renaissance education. He enrolled in college, completing almost six years of coursework in a two-year period. Studying a variety of disciplines, before focusing on obtaining his two computer science degrees. University taught humility, and empathy. These were key to pursuing and achieving a career as a CSO lasting over ten years. Jack primarily focuses his efforts on mentoring his peers (as well as them mentoring him), advising young companies (especially in the information security and privacy space), and investing in businesses that he believes are both innovative, and ethical.Links Referenced: Uptycs: https://www.uptycs.com/ jack@jackroehrig.com: mailto:jack@jackroehrig.com jroehrig@uptycs.com: mailto:jroehrig@uptycs.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey:  LANs of the late 90's and early 2000's were a magical place to learn about computers, hang out with your friends, and do cool stuff like share files, run websites & game servers, and occasionally bring the whole thing down with some ill-conceived software or network configuration. That's not how things are done anymore, but what if we could have a 90's style LAN experience along with the best parts of the 21st century internet? (Most of which are very hard to find these days.) Tailscale thinks we can, and I'm inclined to agree. With Tailscale I can use trusted identity providers like Google, or Okta, or GitHub to authenticate users, and automatically generate & rotate keys to authenticate devices I've added to my network. I can also share access to those devices with friends and teammates, or tag devices to give my team broader access. And that's the magic of it, your data is protected by the simple yet powerful social dynamics of small groups that you trust. Try now - it's free forever for personal use. I've been using it for almost two years personally, and am moderately annoyed that they haven't attempted to charge me for what's become an absolutely-essential-to-my-workflow service.Corey: Kentik provides Cloud and NetOps teams with complete visibility into hybrid and multi-cloud networks. Ensure an amazing customer experience, reduce cloud and network costs, and optimize performance at scale — from internet to data center to container to cloud. Learn how you can get control of complex cloud networks at www.kentik.com, and see why companies like Zoom, Twitch, New Relic, Box, Ebay, Viasat, GoDaddy, booking.com, and many, many more choose Kentik as their network observability platform. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted episode is brought to us by our friends at Uptycs and they have once again subjected Jack Roehrig, Technology Evangelist, to the slings, arrows, and other various implements of misfortune that I like to hurl at people. Jack, thanks for coming back. Brave of you.Jack: I am brave [laugh]. Thanks for having me. Honestly, it was a blast last time and I'm looking forward to having fun this time, too.Corey: It's been a month or two, ish. Basically, the passing of time is one of those things that is challenging for me to wrap my head around in this era. What have you folks been up to? What's changed since the last time we've spoken? What's coming out of Uptycs? What's new? What's exciting? Or what's old with a new and exciting description?Jack: Well, we've GA'ed our agentless architecture scanning system. So, this is one of the reasons why I joined Uptycs that was so fascinating to me is they had kind of nailed XDR. And I love the acronyms: XDR and CNAPP is what we're going with right now. You know, and we have to use these acronyms so that people can understand what we do without me speaking for hours about it. But in short, our agentless system looks at the current resting risk state of production environment without the need to deploy agents, you know, as we talked about last time.And then the XDR piece, that's the thing that you get to justify the extra money on once you go to your CTO or whoever your boss is and show them all that risk that you've uncovered with our agentless piece. It's something I've done in the past with technologies that were similar, but Uptycs is continuously improving, our anomaly detection is getting better, our threat intel team is getting better. I looked at our engineering team the other day. I think we have over 300 engineers or over 250 at least. That's a lot.Corey: It's always wild for folks who work in small shops to imagine what that number of engineers could possibly be working on. Then you go and look at some of the bigger shops and you talk to them and you hear about all the different ways their stuff is built and how they all integrate together and you come away, on some level, surprised that they're able to work with that few engineers. So, it feels like there's a different perspective on scale. And no one has it right, but it is easy, I think, in the layperson's mindset to hear that a company like Twitter, for example, before it got destroyed, had 5000 engineers. And, “What are they all doing?” And, “Well, I can see where that question comes from and the answer is complicated and nuanced, which means that no one is going to want to hear it if it doesn't fit into a tweet itself.” But once you get into the space, you start realizing that everything is way more complicated than it looks.Jack: It is. Yeah. You know, it's interesting that you mention that about Twitter. I used to work for a company called Interactive Corporation. And Interactive Corporation is an internet conglomerate that owns a lot of those things that are at the corners of the internet that not many people know about. And also, like, the entire online dating space. So, I mean, it was a blast working there, but at one point in my career, I got heavily involved in M&A. And I was given the nickname Jack the RIFer. RIF standing for Reduction In Force.Corey: Oof.Jack: So, Jack the RIFer was—yeah [laugh] I know, right?Corey: It's like Buzzsaw Ted. Like, when you bring in the CEO with the nickname of Buzzsaw in there, it's like, “Hmm, I wonder who's going to hire a lot of extra people?” Not so much.Jack: [laugh]. Right? It's like, hey, they said they were sending, “Jack out to hang out with us,” you know, in whatever country we're based out of. And I go out there and I would drink them under the table. And I'd find out the dirty secrets, you know.We would be buying these companies because they would need optimized. But it would be amazing to me to see some of these companies that were massive and they produced what I thought was so little, and then to go on to analyze everybody's job and see that they were also intimately necessary.Corey: Yeah. And the question then becomes, if you were to redesign what that company did from scratch. Which again, is sort of an architectural canard; it was the easiest thing in the world to do is to design an architecture from scratch on a whiteboard with almost an arbitrary number of constraints. The problem is that most companies grow organically and in order to get to that idealized architecture, you've got to turn everything off and rebuild it from scratch. The problem is getting to something that's better without taking 18 months of downtime while you rebuild everything. Most companies cannot and will not sustain that.Jack: Right. And there's another way of looking at it, too, which is something that's been kind of a thought experiment for me for a long time. One of the companies that I worked with back at IC was Ask Jeeves. Remember Ask Jeeves?Corey: Oh, yes. That was sort of the closest thing we had at the time to natural language search.Jack: Right. That was the whole selling point. But I don't believe we actually did any natural language processing back then [laugh]. So, back in those days, it was just a search index. And if you wanted to redefine search right now and you wanted to find something that was like truly a great search engine, what would you do differently?If you look at the space right now with ChatGPT and with Google, and there's all this talk about, well, ChatGPT is the next Google killer. And then people, like, “Well, Google has Lambda.” What are they worried about ChatGPT for? And then you've got the folks at Google who are saying, “ChatGPT is going to destroy us,” and the folks in Google who are saying, “ChatGPT's got nothing on us.” So, if I had to go and do it all over from scratch for search, it wouldn't have anything to do with ChatGPT. I would go back and make a directed, cyclical graph and I would use node weight assignments based on outbound links. Which is exactly what Google was with the original PageRank algorithm, right [laugh]?Corey: I've heard this described as almost a vector database in various terms depending upon what it is that—how it is you're structuring this and what it looks like. It's beyond my ken personally, but I do see that there's an awful lot of hype around ChatGPT these days, and I am finding myself getting professionally—how do I put it—annoyed by most of it. I think that's probably the best way to frame it.Jack: Isn't it annoying?Corey: It is because it's—people ask, “Oh, are you worried that it's going to take over what you do?” And my answer is, “No. I'm worried it's going to make my job harder more than anything else.” Because back when I was a terrible student, great, write an essay on this thing, or write a paper on this. It needs to be five pages long.And I would write what I thought was a decent coverage of it and it turned out to be a page-and-a-half. And oh, great. What I need now is a whole bunch of filler fluff that winds up taking up space and word count but doesn't actually get us to anywhere—Jack: [laugh].Corey: —that is meaningful or useful. And it feels like that is what GPT excels at. If I worked in corporate PR for a lot of these companies, I would worry because it takes an announcement that fits in a tweet—again, another reference to that ailing social network—and then it turns it into an arbitrary length number of pages. And it's frustrating for me just because that's a lot more nonsense I have to sift through in order to get the actual, viable answer to whatever it is I'm going for here.Jack: Well, look at that viable answer. That's a really interesting point you're making. That fluff, right, when you're writing that essay. Yeah, that one-and-a-half pages out. That's gold. That one-and-a-half pages, that's the shit. That's the stuff you want, right? That's the good shit [laugh]. Excuse my French. But ChatGPT is what's going to give you that filler, right? The GPT-3 dataset, I believe, was [laugh] I think it was—there's a lot of Reddit question-and-answers that were used to train it. And it was trained, I believe—the data that it was trained with ceased to be recent in 2021, right? It's already over a year old. So, if your teacher asked you to write a very contemporary essay, ChatGPT might not be able to help you out much. But I don't think that that kind of gets the whole thing because you just said filler, right? You can get it to write that extra three-and-a-half pages from that five pages you're required to write. Well, hey, teachers shouldn't be demanding that you write five pages anyways. I once heard, a friend of mine arguing about one presidential candidate saying, “This presidential candidate speaks at a third-grade level.” And the other person said, “Well, your presidential candidate speaks at a fourth-grade level.” And I said, “I wish I could convey presidential ideas at a level that a third or a fourth grader could understand” You know? Right?Corey: On some level, it's actually not a terrible thing because if you can only convey a concept at an extremely advanced reading level, then how well do you understand—it felt for a long time like that was the problem with AI itself and machine-learning and the rest. The only value I saw was when certain large companies would trot out someone who was themselves deep into the space and their first language was obviously math and they spoke with a heavy math accent through everything that they had to say. And at the end of it, I didn't feel like I understood what they were talking about any better than I had at the start. And in time, it took things like ChatGPT to say, “Oh, this is awesome.” People made fun of the Hot Dog/Not A Hot Dog App, but that made it understandable and accessible to people. And I really think that step is not given nearly enough credit.Jack: Yeah. That's a good point. And it's funny, you mentioned that because I started off talking about search and redefining search, and I think I use the word digraph for—you know, directed gra—that's like a stupid math concept; nobody understands what that is. I learned that in discrete mathematics a million years ago in college, right? I mean, I'm one of the few people that remembers it because I worked in search for so long.Corey: Is that the same thing is a directed acyclic graph, or am I thinking of something else?Jack: Ah you're—that's, you know, close. A directed acyclic graph has no cycles. So, that means you'll never go around in a loop. But of course, if you're just mapping links from one website to another website, A can link from B, which can then link back to A, so that creates a cycle, right? So, an acyclic graph is something that doesn't have that cycle capability in it.Corey: Got it. Yeah. Obviously, my higher math is somewhat limited. It turns out that cloud economics doesn't generally tend to go too far past basic arithmetic. But don't tell them. That's the secret of cloud economics.Jack: I think that's most everything, I mean, even in search nowadays. People aren't familiar with graph theory. I'll tell you what people are familiar with. They're familiar with Google. And they're familiar with going to Google and Googling for something, and when you Google for something, you typically want results that are recent.And if you're going to write an essay, you typically don't care because only the best teachers out there who might not be tricked by ChatGPT—honestly, they probably would be, but the best teachers are the ones that are going to be writing the syllabi that require the recency. Almost nobody's going to be writing syllabi that requires essay recency. They're going to reuse the same syllabus they've been using for ten years.Corey: And even that is an interesting question there because if we talk about the results people want from search, you're right, I have to imagine the majority of cases absolutely care about recency. But I can think of a tremendous number of counterexamples where I have been looking for things explicitly and I do not want recent results, sometimes explicitly. Other times because no, I'm looking for something that was talked about heavily in the 1960s and not a lot since. I don't want to basically turn up a bunch of SEO garbage that trawled it from who knows where. I want to turn up some of the stuff that was digitized and then put forward. And that can be a deceptively challenging problem in its own right.Jack: Well, if you're looking for stuff has been digitized, you could use archive.org or one of the web archive projects. But if you look into the web archive community, you will notice that they're very secretive about their data set. I think one of the best archive internet search indices that I know of is in Portugal. It's a Portuguese project.I can't recall the name of it. But yeah, there's a Portuguese project that is probably like the axiomatic standard or like the ultimate prototype of how internet archiving should be done. Search nowadays, though, when you say things like, “I want explicitly to get this result,” search does not want to show you explicitly what you want. Search wants to show you whatever is going to generate them the most advertising revenue. And I remember back in the early search engine marketing days, back in the algorithmic trading days of search engine marketing keywords, you could spend $4 on an ad for flowers and if you typed the word flowers into Google, you just—I mean, it was just ad city.You typed the word rehabilitation clinic into Google, advertisements everywhere, right? And then you could type certain other things into Google and you would receive a curated list. These things are obvious things that are identified as flaws in the secrecy of the PageRank algorithm, but I always thought it was interesting because ChatGPT takes care of a lot of the stuff that you don't want to be recent, right? It provides this whole other end to this idea that we've been trained not to use search for, right?So, I was reviewing a contract the other day. I had this virtual assistant and English is not her first language. And she and I red-lined this contract for four hours. It was brutal because I kept on having to Google—for lack of a better word—I had to Google all these different terms to try and make sense of it. Two days later, I'm playing around with ChatGPT and I start typing some very abstract commands to it and I swear to you, it generated that same contract I was red-lining. Verbatim. I was able to get into generating multiple [laugh] clauses in the contract. And by changing the wording in ChatGPT to save, “Create it, you know, more plaintiff-friendly,” [laugh] that contract all of a sudden, was red-lined in a way that I wanted it to be [laugh].Corey: This is a fascinating example of this because I'm married to a corporate attorney who does this for a living, and talking to her and other folks in her orbit, the problem they have with it is that it works to a point, on a limited basis, but it then veers very quickly into terms that are nonsensical, terms that would absolutely not pass muster, but sound like something a lawyer would write. And realistically, it feels like what we've built is basically the distillation of a loud, overconfident white guy in tech because—Jack: Yes.Corey: —they don't know exactly what they're talking about, but by God is it confident when it says it.Jack: [laugh]. Yes. You hit the nail on that. Ah, thank you. Thank you.Corey: And there's as an easy way to prove this is pick any topic in the world in which you are either an expert or damn close to it or know more than the average bear about and ask ChatGPT to explain that to you. And then notice all the things that glosses over or what it gets subtly wrong or is outright wrong about, but it doesn't ever call that out. It just says it with the same confident air of a failing interview candidate who gets nine out of ten questions absolutely right, but the one they don't know they bluff on, and at that point, you realize you can't trust them because you never know if they're bluffing or they genuinely know the answer.Jack: Wow, that is a great analogy. I love that. You know, I mentioned earlier that the—I believe the part of the big portion of the GPT-3 training data was based on Reddit questions and answers. And now you can't categorize Reddit into a single community, of course; that would be just as bad as the way Reddit categories [laugh] our community, but Reddit did have a problem a wh—I remember, there was the Ellen Pao debacle for Reddit. And I don't know if it was so much of a debacle if it was more of a scapegoat situation, but—Corey: I'm very much left with a sense that it's the scapegoat. But still, continue.Jack: Yeah, we're adults. We know what happened here, right? Ellen Pao is somebody who is going through some very difficult times in her career. She's hired to be a martyr. They had a community called fatpeoplehate, right?I mean, like, Reddit had become a bizarre place. I used Reddit when I was younger and it didn't have subreddits. It was mostly about programming. It was more like Hacker News. And then I remember all these people went to Hacker News, and a bunch of them stayed at Reddit and there was this weird limbo of, like, the super pretentious people over at Hacker News.And then Reddit started to just get weirder and weirder. And then you just described ChatGPT in a way that just struck me as so Reddit, you know? It's like some guy mansplaining some answer. It starts off good and then it's overconfidently continues to state nonsensical things.Corey: Oh yeah, I was a moderator of the legal advice and personal finance subreddits for years, and—Jack: No way. Were you really?Corey: Oh, absolutely. Those corners were relatively reasonable. And like, “Well, wait a minute, you're not a lawyer. You're correct and I'm also not a financial advisor.” However, in both of those scenarios, what people were really asking for was, “How do I be a functional adult in society?”In high school curricula in the United States, we insist that people go through four years of English literature class, but we don't ever sit down and tell them how to file their taxes or how to navigate large transactions that are going to be the sort of thing that you encounter in adulthood: buying a car, signing a lease. And it's more or less yeah, at some point, you wind up seeing someone with a circumstance that yeah, talk to a lawyer. Don't take advice on the internet for this. But other times, it's no, “You cannot sue a dog. You have to learn to interact with people as a grown-up. Here's how to approach that.” And that manifests as legal questions or finance questions, but it all comes down to I have been left on prepared for the world I live in by the school system. How do I wind up addressing these things? And that is what I really enjoyed.Jack: That's just prolifically, prolifically sound. I'm almost speechless. You're a hundred percent correct. I remember those two subreddits. It always amazes me when I talk to my friends about finances.I'm not a financial person. I mean, I'm an investor, right, I'm a private equity investor. And I was on a call with a young CEO that I've been advising for while. He runs a security awareness training company, and he's like, you know, you've made 39% off of your investment three months. And I said, “I haven't made anything off of my investment.”I bought a safe and, you know—it's like, this is conversion equity. And I'm sitting here thinking, like, I don't know any of the stuff. And I'm like, I talk to my buddies in the—you know, that are financial planners and I ask them about finances, and it's—that's also interesting to me because financial planning is really just about when are you going to buy a car? When are you going to buy a house? When are you going to retire? And what are the things, the securities, the companies, what should you do with your money rather than store it under your mattress?And I didn't really think about money being stored under a mattress until the first time I went to Eastern Europe where I am now. I'm in Hungary right now. And first time I went to Eastern Europe, I think I was in Belgrade in Serbia. And my uncle at the time, he was talking about how he kept all of his money in cash in a bank account. In Serbian Dinar.And Serbian Dinar had already gone through hyperinflation, like, ten years prior. Or no, it went through hyperinflation in 1996. So, it was not—it hadn't been that long [laugh]. And he was asking me for financial advice. And here I am, I'm like, you know, in my early-20s.And I'm like, I don't know what you should do with your money, but don't put it under your mattress. And that's the kind of data that Reddit—that ChatGPT seems to have been trained on, this GPT-3 data, it seems like a lot of [laugh] Redditors, specifically Redditors sub-2001. I haven't used Reddit very much in the last half a decade or so.Corey: Yeah, I mean, I still use it in a variety of different ways, but I got out of both of those cases, primarily due to both time constraints, as well as my circumstances changed to a point where the things I spent my time thinking about in a personal finance sense, no longer applied to an awful lot of folk because the common wisdom is aimed at folks who are generally on a something that resembles a recurring salary where they can calculate in a certain percentage raises, in most cases, for the rest of their life, plan for other things. But when I started the company, a lot of the financial best practices changed significantly. And what makes sense for me to do becomes actively harmful for folks who are not in similar situations. And I just became further and further attenuated from the way that you generally want to give common case advice. So, it wasn't particularly useful at that point anymore.Jack: Very. Yeah, that's very well put. I went through a similar thing. I watched Reddit quite a bit through the Ellen Pao thing because I thought it was a very interesting lesson in business and in social engineering in general, right? And we saw this huge community, this huge community of people, and some of these people were ridiculously toxic.And you saw a lot of groupthink, you saw a lot of manipulation. There was a lot of heavy-handed moderation, there was a lot of too-late moderation. And then Ellen Pao comes in and I'm, like, who the heck is Ellen Pao? Oh, Ellen Pao is this person who has some corporate scandal going on. Oh, Ellen Pao is a scapegoat.And here we are, watching a community being socially engineered, right, into hating the CEO who's just going to be let go or step down anyways. And now they ha—their conversations have been used to train intelligence, which is being used to socially engineer people [laugh] into [crosstalk 00:22:13].Corey: I mean you just listed something else that's been top-of-mind for me lately, where it is time once again here at The Duckbill Group for us to go through our annual security awareness training. And our previous vendor has not been terrific, so I start looking to see what else is available in that space. And I see that the world basically divides into two factions when it comes to this. The first is something that is designed to check the compliance boxes at big companies. And some of the advice that those things give is actively harmful as in, when I've used things like that in the past, I would have an addenda that I would send out to the team. “Yeah, ignore this part and this part and this part because it does not work for us.”And there are other things that start trying to surface it all the time as it becomes a constant awareness thing, which makes sense, but it also doesn't necessarily check any contractual boxes. So it's, isn't there something in between that makes sense? I found one company that offered a Slackbot that did this, which sounded interesting. The problem is it was the most condescendingly rude and infuriatingly slow experience that I've had. It demanded itself a whole bunch of permissions to the Slack workspace just to try it out, so I had to spin up a false Slack workspace for testing just to see what happens, and it was, start to finish, the sort of thing that I would not inflict upon my team. So, the hell with it and I moved over to other stuff now. And I'm still looking, but it's the sort of thing where I almost feel like, this is something ChatGPT could have built and cool, give me something that sounds confident, but it's often wrong. Go.Jack: [laugh]. Yeah, Uptycs actually is—we have something called a Otto M8—spelled O-T-T-O space M and then the number eight—and I personally think that's the cutest name ever for Slackbot. I don't have a picture of him to show you, but I would personally give him a bit of a makeover. He's a little nerdy for my likes. But he's got—it's one of those Slackbots.And I'm a huge compliance geek. I was a CISO for over a decade and I know exactly what you mean with that security awareness training and ticking those boxes because I was the guy who wrote the boxes that needed to be ticked because I wrote those control frameworks. And I'm not a CISO anymore because I've already subjected myself to an absolute living hell for long enough, at least for now [laugh]. So, I quit the CISO world.Corey: Oh yeah.Jack: Yeah.Corey: And so, much of it also assumes certain things like I've had people reach out to me trying to shill whatever it is they've built in this space. And okay, great. The problem is that they've built something that is aligned at engineers and developers. Go, here you go. And that's awesome, but we are really an engineering-first company.Yes, most people here have an engineering background and we build some internal tooling, but we don't need an entire curriculum on how to secure the tools that we're building as web interfaces and public-facing SaaS because that's not what we do. Not to mention, what am I supposed to do with the accountants in the sales folks and the marketing staff that wind up working on a lot of these things that need to also go through training? Do I want to sit here and teach them about SQL injection attacks? No, Jack. I do not want to teach them that.Jack: No you don't.Corey: I want them to not plug random USB things into the work laptop and to use a password manager. I'm not here trying to turn them into security engineers.Jack: I used to give a presentation and I onboarded every single employee personally for security. And in the presentation, I would talk about password security. And I would have all these complex passwords up. But, like, “You know what? Let me just show you what a hacker does.”And I'd go and load up dhash and I'd type in my old email address. And oh, there's my password, right? And then I would—I copied the cryptographic hash from dhash and I'd paste that into Google. And I'd be like, “And that's how you crack passwords.” Is you Google the cryptographic hash, the insecure cryptographic hash and hope somebody else has already cracked it.But yeah, it's interesting. The security awareness training is absolutely something that's supposed to be guided for the very fundamental everyman employee. It should not be something entirely technical. I worked at a company where—and I love this, by the way; this is one of the best things I've ever read on Slack—and it was not a message that I was privy to. I had to have the IT team pull the Slack logs so that I could read these direct communications. But it was from one—I think it was the controller to the Vice President of accounting, and the VP of accounting says how could I have done this after all of those phishing emails that Jack sent [laugh]?Corey: Oh God, the phishing emails drives me up a wall, too. It's you're basically training your staff not to trust you and waste their time and playing gotcha. It really creates an adversarial culture. I refuse to do that stuff, too.Jack: My phishing emails are fun, all right? I did one where I pretended that I installed a camera in the break room refrigerator, and I said, we've had a problem with food theft out of the Oakland refrigerator and so I've we've installed this webcam. Log into the sketchy website with your username and password. And I got, like, a 14% phish rate. I've used this campaign at multinational companies.I used to travel around the world and I'd grab a mic at the offices that wanted me to speak there and I'd put the mic real close to my head and I say, “Why did you guys click on the link to the Oakland refrigerator?” [laugh]. I said, “You're in Stockholm for God's sake.” Like, it works. Phishing campaigns work.They just don't work if they're dumb, honestly. There's a lot of things that do work in the security awareness space. One of the biggest problems with security awareness is that people seem to think that there's some minimum amount of time an employee should have to spend on security awareness training, which is just—Corey: Right. Like, for example, here in California, we're required to spend two hours on harassment training every so often—I think it's every two years—and—Jack: Every two years. Yes.Corey: —at least for managerial staff. And it's great, but that leads to things such as, “Oh, we're not going to give you a transcript if you can read the video more effectively. You have to listen to it and make sure it takes enough time.” And it's maddening to me just because that is how the law is written. And yes, it's important to obey the law, don't get me wrong, but at the same time, it just feels like it's an intentional time suck.Jack: It is. It is an intentional time suck. I think what happens is a lot of people find ways to game the system. Look, when I did security awareness training, my controls, the way I worded them, didn't require people to take any training whatsoever. The phishing emails themselves satisfied it completely.I worded that into my control framework. I still held the trainings, they still made people take them seriously. And then if we have a—you know, if somebody got phished horrifically, and let's say wired $2 million to Hong Kong—you know who I'm talking about, all right, person who might is probably not listening to this, thankfully—but [laugh] she did. And I know she didn't complete my awareness training. I know she never took any of it.She also wired $2 million to Hong Kong. Well, we never got that money back. But we sure did spend a lot of executive time trying to. I spent a lot of time on the phone, getting passed around from department to department at the FBI. Obviously, the FBI couldn't help us.It was wired from Mexico to Hong Kong. Like the FBI doesn't have anything to do with it. You know, bless them for taking their time to humor me because I needed to humor my CEO. But, you know, I use those awareness training things as a way to enforce the Code of Conduct. The Code of Conduct requiring disciplinary action for people who didn't follow the security awareness training.If you had taken the 15 minutes of awareness training that I had asked people to do—I mean, I told them to do it; it was the Code of Conduct; they had to—then there would be no disciplinary action for accidentally wiring that money. But people are pretty darn diligent on not doing things like that. It's just a select few that seems to be the ones that get repeatedly—Corey: And then you have the group conversations. One person screws something up and then you wind up with the emails to everyone. And then you have the people who are basically doing the right thing thinking they're being singled out. And—ugh, management is hard, people is hard, but it feels like a lot of these things could be a lot less hard.Jack: You know, I don't think management is hard. I think management is about empathy. And management is really about just positive reinforce—you know what management is? This is going to sound real pretentious. Management's kind of like raising a kid, you know? You want to have a really well-adjusted kid? Every time that kid says, “Hey, Dad,” answer. [crosstalk 00:30:28]—Corey: Yeah, that's a good—that's a good approach.Jack: I mean, just be there. Be clear, consistent, let them know what to expect. People loved my security program at the places that I've implemented it because it was very clear, it was concise, it was easy to understand, and I was very approachable. If anybody had a security concern and they came to me about it, they would [laugh] not get any shame. They certainly wouldn't get ignored.I don't care if they were reporting the same email I had had reported to me 50 times that day. I would personally thank them. And, you know what I learned? I learned that from raising a kid, you know? It was interesting because it was like, the kid I was raising, when he would ask me a question, I would give him the same answer every time in the same tone. He'd be like, “Hey, Jack, can I have a piece of candy?” Like, “No, your mom says you can't have any candy today.” They'd be like, “Oh, okay.” “Can I have a piece of candy?” And I would be like, “No, your mom says you can't have any candy today.” “Can I have a piece of candy, Jack?” I said, “No. Your mom says he can't have any candy.” And I'd just be like a broken record.And he immediately wouldn't ask me for a piece of candy six different times. And I realized the reason why he was asking me for a piece of candy six different times is because he would get a different response the sixth time or the third time or the second time. It was the inconsistency. Providing consistency and predictability in the workforce is key to management and it's key to keeping things safe and secure.Corey: I think there's a lot of truth to that. I really want to thank you for taking so much time out of your day to talk to me about think topics ranging from GPT and ethics to parenting. If people want to learn more, where's the best place to find you?Jack: I'm jack@jackroehrig.com, and I'm also jroehrig@uptycs.com. My last name is spelled—heh, no, I'm kidding. It's a J-A-C-K-R-O-E-H-R-I-G dot com. So yeah, hit me up. You will get a response from me.Corey: Excellent. And I will of course include links to that in the show notes. Thank you so much for your time. I appreciate it.Jack: Likewise.Corey: This promoted guest episode has been brought to us by our friends at Uptycs, featuring Jack Roehrig, Technology Evangelist at same. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment ghostwritten for you by ChatGPT so it has absolutely no content worth reading.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

About AerinAerin is a Cloud Sustainability Advocate and neurodiverse founder in tech on a mission to help developers understand the real impact that cloud computing has on the world and reduce their carbon emissions in the cloud. Did you know that internet and cloud computing contribute over 4% of annual carbon emissions? Twice that of the airline industry!Aerin also hosts "Public Cloud for Public Good," a podcast targeted towards developers and senior leaders in tech. Every episode, they also donate £500 to charities and highlight organisations that are working towards a better future. Listen and learn how you can contribute towards making the world a better place through the use of public cloud services.Links Referenced: Twitter: https://twitter.com/aerincloud LinkedIn: https://www.linkedin.com/in/aerinb/ Public Cloud for Public Good: https://publicgood.cloud/ duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Uptycs, because they believe that many of you are looking to bolster your security posture with CNAPP and XDR solutions. They offer both cloud and endpoint security in a single UI and data model. Listeners can get Uptycs for up to 1,000 assets through the end of 2023 (that is next year) for $1. But this offer is only available for a limited time on UptycsSecretMenu.com. That's U-P-T-Y-C-S Secret Menu dot com.Corey: Cloud native just means you've got more components or microservices than anyone (even a mythical 10x engineer) can keep track of. With OpsLevel, you can build a catalog in minutes and forget needing that mythical 10x engineer. Now, you'll have a 10x service catalog to accompany your 10x service count. Visit OpsLevel.com to learn how easy it is to build and manage your service catalog. Connect to your git provider and you're off to the races with service import, repo ownership, tech docs, and more. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn and I am joined what feels like roughly a year later by a returning guest, Aerin Booth. How long have you been?Aerin: I've been really great. You know, it's been a journey of a year, I think, since we sort of did this podcast even, like, you know, a year and a bit since we met, and, like, I'm doing so much and I think it's doing, like, a big difference. And yeah, I can't wait for everything else. It's just yeah, a lot of work right now, but I'm really enjoying it. So, I'm really well, thank you.Corey: Normally, I like to introduce people by giving their job title and the company in which they work because again, that's a big deal for an awful lot of people. But a year ago, you were independent. And now you still are. And back when I was doing my own consulting independently, it felt very weird to do that, so I'm just going to call you the Ted Lasso of cloud at this point.Aerin: [laugh].Corey: You've got the mustache, you've got the, I would say, obnoxiously sunny disposition. It's really, there's a certain affinity right there. So, there we go. I feel like that is the best descriptor for what you have become.Aerin: I—do know what, I only just watched Ted Lasso over Christmas and I really found it so motivational in some ways because wow, like, it's not just who we'd want to be in a lot of ways? And I think, you know, for the work that I do, which is focused on sustainability, like, I want to present a positive future, I want to encourage people to achieve more and collaborate, and yeah, basically work on all these problems that we need to be worked on. And yeah, I think that's [laugh] [crosstalk 00:02:02]—Corey: One of the challenges of talking to you sometimes is you talk about these depressing things, but there's such a—you take such an upbeat, positive approach to it that I, by comparison, invariably come away from our conversations during, like, I'm Surly McBastard over here.Aerin: [laugh]. Yeah, you can be the bad cop of cloud computing and I'll try and be the good cop. Do you know, you say that the stuff I talk about is depressing, and it is true and people do worry about climate change. Like I did an online conference recently, it's focused on FinOps, and we had a survey, “Do you worry about climate change?” 70% of the people that responded said they worry about it.So, we all know, it's something we worry about and we care about. And, you know, I guess what I'm really trying to do is encourage people to care a bit more and start taking action and look after yourself. Because you know, when you do start taking action towards it, when you join those communities that are also working on it, it is good, it is helpful. And, you know, I've gone through some ups and downs and some of this, like, just do I throw in the towel because no one cares about it? Like, we spoke last year; I had attended re:Invent for the first time.This year, I was able to speak at re:Invent. So, I did a talk on being ethical in tech. And it was fun, it was good. I enjoyed what I delivered, but I had about 35 people sign up to that. I'm pretty sure if I talked about serverless or the next Web3 blockchain product, I would have got hundreds more. But what I'm starting to realize is that I think people just aren't ready to, sort of, want to do this yet. And yeah, I'm hoping that'll change.Corey: Let's first talk about, I guess, something that is more temporally pressing than some other things. Not that it is more important than climate change, mind you, but it feels like it's on a shorter timeline which is, relatively soon after this recording, there is a conference that you are kicking off called The State of Open. Ajar, Aerin. The State of Open is ajar. What is this conference? Is it in person? Is it virtual? Is it something where you and three friends are going to show up and basically talk to each other? How big? How small? What is it? What's it about? Tell me more, please. I'm riveted.Aerin: So, State of Open conference is a conference that's been in the works now for maybe about two weeks, a little bit longer in the planning, but the work we've been putting in over the last two weeks. It'll be on the seventh and eighth of February in London as a physical event in the QEII Conference Centre, but it will also be available online. And you know, when we talk about the State of Open, it's that question: what is the State of Open? The state of open-source, the state of open hardware, and the state of open data. And it is going to be probably the first and hopefully the biggest open-source conference in the UK.We already have over 100 confirmed guest speakers from Jimmy Wales, the co-founder of Wikipedia, to many of our great guests and headliners who haven't even announced yet for the plenary. So, I'm really excited. And the reason why I wanted to get involved with this is because one of the coolest things about this conference—compared to some others like re:Invent, for example—is that sustainability and diversity run through every single thing that we do. So, as the content director, I reviewed every single CFP for both of these things. I mean, you couldn't get a better person than someone like me, who's the queer person who won't shut up about sustainability to sort of do this thing.So, you know, I looked after those scorings for the CFPs in support of the CFP chairs. And now, as I'm working with those individual speakers on their content and making sure that diversity is included in the content. It's not just the diversity of the speaker, for example it's, who were the other people whose voice you're raising? What other people if you worked on this? Are there anyone that you've mentored, like, you know, actually, you know, let's have this as a wider conversation?Corey: Thank God. I thought you were about to say diversity of thought, and I was about to reach through the screen to strangle you.Aerin: [laugh]. No, no. I mean, we're doing really well, so of the announced speakers online, we are 40% non-male and about 18% non-white, which to be honest, for a fair sheer conference, when we didn't really do that much to specifically call this out, but I would probably raise this to Amanda Brock, who is the CEO of OpenUK, you know, she has built a community in the UK and around the world over the last few years which has been putting women forward and building these links. And that's why we've had such a great response for our first-year conferences, the work she's put in. It's hard.Like, this isn't easy. You know, we've had to do a lot of work to make sure that it is representative, at least better than other conferences, at least. So, I'm really excited. And like, there's so much, like, open-source is probably going to be the thing that saves the world. If we're going to end up looking at two different futures with monopolies and closed systems and all the money going towards cloud providers versus a fair and equitable society, open-source is the thing that's going to get us closer to that. So yeah, this conference will be a great event.Corey: Is it all in person? Is it being live-streamed as well? What is the deal here?Aerin: So, in person, we have loads of different things going on, but what will be streamed online if you sign up for virtual ticket is five different tracks. So, our platform engineering track, our security track, government law and policy, open data, and open hardware. And of course, the keynote and plenaries. But one of the things I'm also really proud about this conference is that we're really focusing on the developer experience, like, you know, what is your experience at the conference? So, we also have an unconference, we have a sub-conference run by Sustain OSS focused on workshops related to climate change and sustainability.We have loads of developer experience halls in the event itself. And throughout the day, over the two days, we have two one-hour blocks with no speaking content at all so that we can really make sure that people have that hardware track and are out there meeting each other and having a good time. And obviously, of course, like any good conference, the all-hands party on the first night. So, it really is a conference that's doing things differently from diversity to sustainability to that experience. So, it's awesome.Corey: One of the challenges that I've seen historically around things aiming at the idea of open conferences—and when we talk open-source, et cetera, et cetera—open' seems like it is a direction parallel to, we haven't any money, where it's, “Yes, we're a free software foundation,” and it turns out conferences themselves are not free. And you wind up with a whole bunch of folks showing up to it who are, in many cases, around the fringes of things. There are individual hobbyists who are very passionate about a thing but do not have the position in the corporate world. I'm looking through the lengthy list of speakers you have here and that is very much not this. These are serious people at serious companies. Not that there are not folks who are individual practitioners and passionate advocates and hobbyists than the rest. This is, by virtually any way you look at it, a remarkably diverse conference.Aerin: Mmm. You know, you are right about, like, that problem in open-source. It's like, you know, we look at open and whether we want to do open and we just go, “Well, it won't make me any money. I can't do that. I don't have the time. I need to bring in some money.”And one of the really unique things, again, about this conference is—I have not even mentioned it yet—we have an entrepreneurship room. So, we have 20 tables filled with entrepreneurs and CEOs and founders of open-source companies throughout the two days where you can book in time to sit at that table and have conversations with them. Ask them the questions that you want to ask about, whether it's something that you want to work on, or a company you want to found, and you'll be able to get that time. I had a very similar experience in some ways. It was re:Invent.I was a peer talk expert and you know, I had 15 or so conversations with some really interesting people just because they were able put that time in and they were able to find me on the website. So, that's something we are replicating to get those 20 also entrepreneurs and co-founders out to everyone else. They want to be able to help you and support you.Corey: That is an excellent segue if I do say so myself. Let's talk about re:Invent. It's the one time of the year you and I get to spend time in the same room. One thing that I got wrong is that I overbooked myself as I often do, and I didn't have time to do anything on their peer talk expert program, which is, you more or less a way that any rando can book time to sit down and chat with you. Now, in my case, I have assassination concerns because it turns out Amazon employees can read that thing too and some of them might work on billing. One wonders.So yeah, I have to be a little careful for personal reasons but for most people, it's a non-issue. I didn't get as much time as I wanted to talk to folks in the community. That is not going to repeat itself at the end of this year. But what was your take on re:Invent, because I was in meetings for most of them?Aerin: So, comparing this re:Invent to the re:Invent I went to, my first re:Invent when we met in 2021, you know, that was the re:Invent that inspired me to get into sustainability. They'd announced stuff to do with the shared responsibility model. A few months later, they released their carbon calculator, and I was like, “Yeah, this is the problem. This is the thing I want to work on and it will make me happy.” And a lot of that goes into, you know, finding a passion that keeps me motivated when things aren't that great.When maybe not a lot of money is coming in, at least I know, I'm doing everything I can to help save the world. So, re:Invent 2021 really inspired me to get involved with sustainability. When I look at re:Invent 2022, you might have Adam Selipsky on the main stage saying that sustainability is the problem of our generation, but that is just talk and bluster compared to what they were putting out in terms of content and their experience of, like, let's say the sustainability—I don't know what to call it—tiny little square in the back of the MGM Grand compared to the paid hall in the expo. Like, you know, that's the sort of thing where you can already see the prioritization of money. Let's put the biggest sponsors and all the money that we can bring it in the big hall where everyone is, and then put the thing we care about the most, apparently—sustainability—in the back of the MGM.And that in itself was annoying, but then you get there in the content, and it was like a massive Rivian van, like, an advert for, “Oh, Amazon has done all this to electrify Rivian and deliver you Prime.” But where was the people working on sustainability in the cloud? You know, we had a couple of teams who were talking about the customer carbon footprint tool, but there was just not much. And I spoke to a lot of people and they were saying similar things, like, “Where are the announcements? Where are the actual interesting things?” Rather than just—which is kind of what I'm starting to realize is that a lot of the conversations about sustainability is about selling yourself as sustainable.Use me rather than my competitors because we're 88% more, kind of, carbon neutral when it comes to traditional data centers, not because we are really going to solve these problems. And not to say that Amazon isn't doing innovative, amazing things that no one else can't do, because that is true, and cloud as part of the solution, but you know, sustainability shouldn't be about making more sales and growing your business, it should be about making the world a better place, not just in terms of carbon emissions, but you know, our life, the tech that we can access. Three billion people on this planet have never accessed the internet. And as we continue to grow all of our services like AI and machine learning and new Web3, bloody managed services come online, that's going to be more carbon, more compute power going towards the already rich and the already westernized people, rather than solving the problems we need to solve in the face of climate change.So, I was a little bit disappointed. And I did put a tweet thread out about it afterwards. And I just hope it can be different next year and I hope more people will start to ask for this. And that also what I'm starting to realize is that until more Amazon customers put this as their number one priority and say, “I'm not going to do business with you because of this issue,” or, you know, “This is what we really care about,” they're not going to make a change. Unless it starts to impact their bottom lines and people start to choose other cloud providers, they're not going to prioritize it.And I think up until this point, we're not seeing that from customers. We're kind of getting some people like me shouting about it, but across the board, sustainability isn't the number one priority right now. It's, like what Amazon says, security or resiliency or something else.Corey: And I think that, at least from where I set, the challenge is that if you asked me what I got out of re:Invent, and what the conversations I had—going into it, what are my expectations, and what do I hope to get and how's it going to end up, and then you ask you that same question—though maybe you are a poor example of this—and then you ask someone who works out as an engineer at a company that uses AWS and their two or three years into their career, why don't you talk to a manager or director or someone else? And the problem is if you start polling the entire audience, you'll find that this becomes—you're going to wind up with 20 different answers, at least. The conference doesn't seem like it has any idea of what it wants to be and to whom and in that vacuum, it tries to be all things to all people. And surprise, just like the shooting multifunction printer some of us have in our homes, it doesn't do well with any of those things because it's trying to stand in too many worlds at the same time.Aerin: You know, let's not, like, look at this from a way that you know, re:Invent is crap and, like, do all the work that everyone puts it is wasted because it is a really great event for a lot of different things for a lot of different people. And to be honest, the work that the Amazon staff put into it is pretty out of this world. I feel sorry though because you know, the rush for AWS sell more and do this massive event, they put people through the grinder. And I feel like, I don't know, we could see the cracks in some of that, the way that works. But, you know, there's so many people that I speak to who were like, “Yeah, I'm definitely not going again. I'm not even going to go anywhere near submitting a talk.”And, sort of, the thing is, like, I can imagine if the conference was something different; it was focused at sustainability at number one, it was about making the world a better place from everything that they do, it was about bringing diverse communities together. Like, you know, bringing these things up the list would make the whole thing a lot better. And to be honest, it would probably make it a lot more enjoyable [laugh] for the Amazon staff who end up talking at it. Because, you know, I guess it can feel a bit soulless over time is all you're doing is making money for someone else and selling more things. And, yeah, I think there's a lot more… different things we can do and a lot more things we can talk about if people just start to talk about, like you know, if you care about this as well and you work at Amazon, then start saying that as well.It'll really make a difference if you say we want re:Invent to look different. I mean, even Amazon staff, [laugh] and we've not even mentioned this one because I got Covid straight after re:Invent, nine days and staring at a wall in hotel room in Vegas was not my idea of a good time post-conference. So, that was a horrible, horrible experience. But, you know, I've had people call it re:Infect. Like, where are the Covid support?Like, there was hardly any conversation about that. It was sort of like, “Don't mention it because oh, s”—whatever else. But imagine if you just did something a little bit differently to look like you care about your customers. Just say, “We recommend people mask or take a test,” or even provide tests and masks. Like, even if it's not mandatory, they could have done a lot more to make it safer for everyone. Because, yeah, imagine having the reputation of re:Infect rather than re:Invent?Corey: I can only imagine how that would play out.Aerin: Only imagine.Corey: Yeah, it's it feels like we're all collectively decided to pretend that the pandemic is over. Because yeah, that's a bummer. I don't want to think about it. You know, kind of like we approach climate change.Aerin: Yeah. At the end of the day, like, and I keep coming across this more and more, you know, my thinking has changed over the last year because, like, you know, initially it was like a hyperactive puppy. Why are we caring about this? Like, yeah, if I say it, people will come, but the reality is, we have to blinker ourselves in order to deal with a lot of this stuff. We can't always worry about all of this stuff all of the time. And that's fine. That's acceptable. We do that in so many different parts of our life.But there comes to a point when you kind of think, “How much do I care about this?” And for a lot of people, it's because they have kids. Like, anyone who has kids right now must have to think, “Wow, what's the future going to look like?” And if you worry about what the future is going to look like, make sure you're taking steps to make the world a better place and make it the future you want it to look like. You know, I made the decision a long time ago not to have kids because I don't think I'd want to bring anyone into the world on what it might actually end up being, but you know, when I speak to people who are older in the 60s and they're like, “Oh, you've got 100 years. You don't need to worry about it.” Like, “Maybe you can say that because you're closer to dying than I am.” But yeah, I have to worry about this now because I'll still be eighty when all this shit is kicking off [laugh].Corey: This episode is sponsored in part by our friends at Strata. Are you struggling to keep up with the demands of managing and securing identity in your distributed enterprise IT environment? You're not alone, but you shouldn't let that hold you back. With Strata's Identity Orchestration Platform, you can secure all your apps on any cloud with any IDP, so your IT teams will never have to refactor for identity again. Imagine modernizing app identity in minutes instead of months, deploying passwordless on any tricky old app, and achieving business resilience with always-on identity, all from one lightweight and flexible platform.Want to see it in action? Share your identity challenge with them on a discovery call and they'll hook you up with a complimentary pair of AirPods Pro. Don't miss out, visit Strata.io/ScreamingCloud. That's Strata dot io slash ScreamingCloud.Corey: That I guess is one of the big fears I have—and I think it's somewhat unfounded—is that every year starts to look too much like the year before it. Because it's one of those ideas where we start to see the pace of innovation is slowing at AWS—and I'm not saying that to piss people at Amazon off and have them come after me with pitchforks and torches again—but they're not launching new services at the rate they once did, which is good for customers, but it starts to feel like oh, have we hit peak cloud this is what it's going to look like? Absolutely not. I don't get the sense that the world is like, “Well, everything's been invented. Time to shut down the patent office,” anytime soon.And in the short term, it feels like oh, there's not a lot exciting going on, but you look back the last five years even and look at how far we've come even in that period of time and—what is it? “The days are long, but the years are short.” It becomes a very macro thing of as things ebb and flow, you start to see the differences but the micro basis on a year-to-year perspective, it seems harder to detect. So longer term, I think we're going to see what the story looks like. And it's going to be satisfying one. Just right now, it's like, well, this wasn't as entertaining as I would have hoped, so I'm annoyed. Which I am because it wasn't, but that's not the biggest problem in the world.Aerin: It's not. And, you know, you look at okay, cool, there wasn't all these new flashy services. There was a few things are announced, I mean, hopefully that are going to contribute towards climate change. One of them is called AWS Supply Chain. And the irony of seeing sort of like AWS Supply Chain where a company that already has issues with data and conversations around competition, saying to everyone, “Hey, trust us and give all of your supply chain information and put it into one of our AWS products,” while at the same time their customer carbon footprint tool won't even show the full scope for their emissions of their own supply chain is not lost on me.And you do say, “Maybe we should start seeing things at a macro level,” but unless Amazon and other cloud hyperscalers start pulling the finger out and showing us how they have got a vision between now and 2040, and now in 2050, of how they're going to get there, it kind of just feels like they're saying, “It'll all be fine as long as we continue to grow, as long as we keep sucking up the market.” And, you know, an interesting thing that just kicked off in the UK back in November was the Competition and Markets Authority have started an investigation into the cloud providers on how they are basically sucking up all these markets, and how the growth of things that are not hyperscale is going. So, in the UK, the percentage of cloud has obviously gone up—more and more cloud spending has gone up—but kind of usage across non-hyperscalers has gone down over that same period. And they really are at risk of sucking up the world. Like, I have got involved in a lot of different things.I'm an AWS community builder; like, I do promote AWS. And, you know, the reason why I promote cloud, for example is serverless. We need serverless as the way we run our IT because that's the only way we'll do things like time shifting or demand shifting. So, when we look at renewable energy on the grid if that really high, the wind is blowing and the sun is shining, we want more workloads to be running then and when they're tiny, and they're [unintelligible 00:21:03], and what's the call it serverless generally, uh—Corey: Hype?Aerin: Function as a Code?Corey: Function—yeah, Function as a Service and all kinds of other nonsense. But I have to ask, when you're talking about serverless, in this context, is a necessary prerequisite of serverless that scale to zero when it's [unintelligible 00:21:19].Aerin: [laugh]. I kind of go back to marketing. What Amazon releasing these days when it relates to serverless that isn't just marketing and saying, “Oh, it's serverless.” Because yeah, there was a few products this year that is not scaled to zero is it? It's a 100-pound minimum. And when you're looking at number of accounts that you have, that can add up really quickly and it excludes people from using it.Corey: It's worse than that because it's not number of accounts. I consider DynamoDB to be serverless, by any definition of the term. Because it is. And what I like about it is I can have a separate table for every developer, for every service or microservice or project that they have, and in fact, each branch can have its own stuff like that. I look at some of the stuff that I build with multi-branch testing and whatnot, and, “Oh, wow. That would cost more than the engineer if they were to do that with some of the serverless offerings that AWS has put out.”Which makes that entire philosophy a complete non-starter, which means that invariably as soon as you start developing down that path, you are making significant trade-offs. That's just from a economics slash developer ergonomics slash best practices point of view. But there's a sustainability story to it as well.Aerin: Yeah. I mean, this sustainability thing is like, if you're not going to encourage this new way of working, like, if you're not going to move everyone to this point of view and this is how we need to do things, then you kind of just propagating the old world, putting it into your data center. For every managed service that VMware migrated piece of crap, just that land in the cloud, it's not making a real difference in the world because that's still going to exist. And we mentioned this just before the podcast and, you know, a lot of focus these days and for a lot of people is, “Okay, green energy is the problem. We need to solve green energy.”And Amazon is the biggest purchaser of power purchase agreements in renewable energy around the world, more than most governments. Or I think that the biggest corporate purchaser of it anyway. And that all might sound great, like, “Oh, the cloud is going to solve this problem for me and Amazon is going to solve it for me even better because they're bigger.” But at the end of the day, when we think about a data center, it exists in the real world.It's made of concrete. You know, when you pour concrete and when you make concrete, it releases CO2. It's got racks of servers that all are running. So, those individual servers had to be made by whoever it is in Asia or mined from rare earth metals and end up in the supply chain and then transported into the data centers in us-east-1. And then things go wrong. You have to repair you have to replace and you have to maintain them.Unless we get these circular economies going in a closed system, we can't just continue to grow like this. Because carbon emissions related to Scope 3, all those things I've just been talking about, basically anything that isn't the energy, is about 80 to 90% of all the carbon emissions. So, when Amazon says, “Oh, we're going to go green and get energy done by 2030”—which is seven years away—they've then got ten years to solve 90% of the problem. And we cannot all just continue to grow and think of tech as neutral and better for the world if we still got that 90% problem, which we do right now. And it really frustrates me when you look at the world and the way we've jumped on technology just go on, “Oh, it must be good.”Like Bitcoin, for example. Bitcoin has released 200 million metric tons of CO2 since its inception. And for something that is basically a glorified Ponzi scheme, I can't see how that is making the world a better place. So, when cloud providers are making managed services for Web3 and for blockchain, and they're selling more and more AI and machine learning, basically so they can keep on selling GPU access, I do worry about whether our path to infinite growth with all of these hyperscalers is probably the wrong way of looking at things. So, linking back to, you know, the conference, open-source and, you know, thinking about things differently is really important in tech right now.And not just for your own well-being and being able to sleep at night, but this is how we're going to solve our problems. When all companies on the planet want people to be sustainable and we have to start tackling this because there's a financial cost related to it, then you're going to be in the vogue. If you're really good developer, thinking about things differently can be efficient, then yeah, you're the developer that's going to win in the future. You might be assisted by ChatGPT three or whatever else, but yeah, sustainability and efficiency can really be the number one priority because it's a win, win, win. We save the world, we make ourselves better, we sleep better at night, and you just become a better developer.I keep monologuing at this point, but you know, when it comes to stuff like games design, we look at things like Quake and Pokemon and all these things when there's like, “How did they get these amazing games and these amazing experiences in such small sizes,” they had boundaries. They had boundaries to innovate within because they had to. They couldn't release the game if they couldn't fit into the cartridge, therefore, they made it work. When the cloud is sold as infinitely scalable and horizontally scalable and no one needs to worry about this stuff because you can get your credit card out, people stop caring about being innovative and being more efficient. So yeah, let's get some more boundaries in the cloud.Corey: What I find that is super helpful, has been, like, if I can, like, descri—like, Instagram is down. Describe your lunch to me style meme description, like, the epic handshake where you have two people clasping hands, and one side is labeled in this case, ‘sustainability advocates,' and the other side should be labeled ‘cloud economists,' and in the middle, it's, “Turn that shit off.” Because it's not burning carbon if it's not running, and it's not costing you anything—ideally—if it's not running, so it's one of those ideas where we meet in the middle. And that's important, not just because it makes both of us independently happy because it's both good for the world and you'll get companies on board with this because, “Wait. We can do this thing and it saves us money?” Suddenly, you're getting them aligned because that is their religion.If companies could be said to have a religion, it is money. That's the way it works. So, you have to make it worth money for them to do the right thing or you're always going to be swimming upstream like a depressed salmon.Aerin: I mean, look at why [unintelligible 00:27:11] security is near the top: because there's so many big fines related to security breaches. It will cost them money not to be secure. Right now, it doesn't cost companies money to be inefficient or to release all this carbon, so they get away with it or they choose to do it. And I think that's going to change. We see in regulations across you're coming out.So, you know, if you work for a big multinational that operates in Europe, by next year, you'll have to report on all of your Scope 3 carbon emissions. If you're a customer of AWS right now, you have no ability to do that. So, you know, this is going to be crunch time over the next 18 months to two years for a lot of big businesses, for Amazon and the other hyperscalers, to really start demonstrating that they can do this. And I guess that's my big push. And, you know, I want to work with anyone, and it's funny because I have been running this business for about, you know, a couple of years now, it's been going really well, I did my podcast, I'm on this path.But I did, last year, take some time, and I applied into AWS. And you know, I was like, “Okay, maybe I'll apply for this big tech company and help Amazon out.” And because I'll take that salary and I'll do something really good with it afterwards, I'll do my time for three years and attend re:Invent and deliver 12 talks and never sleep, but you know, at the end of it, I'll say, “Okay, I've done that and now I can do something really good.” Unfortunately, I didn't get the role—or fortunately—but you know, when I applied for that role, what I said to them is, “I really care about sustainability. I want to make the world a better place. I want to help your customers be more sustainable.”And they didn't want me to join. So, I'm just going to continue doing that but from the outside. And whether that means working with politicians or developers or anyone else to try and make the world better and to kind of help fight against climate change, then, yeah, that's definitely what I'm doing.Corey: So, one last question before we wind up calling it an episode. How do we get there? What is the best next step that folks can take? Because it's easy to look at this as a grand problem and realize it's too big to solve. Well, great. You don't need to solve the entire problem. You need take the first step. What is that first step?Aerin: Individuals, I would say it's just realizing that you do care about it and you want to take action. And you're going to say to yourself, “Even if I do little things, I'm going to move forward towards that point.” So, if that is being a more sustainable engineer or getting more conversations about climate change or even just doing other things in your community to make the world a better place than it is, taking that action. But one thing that I can definitely help about and talk a bit more of is that at the conference itself, I'll be running a panel with some great experts called the, “Next Generation of Cloud Education.” So, I really think we need to—like I said earlier in the podcast—to think differently about the cloud and IT.So, I am doing this panel and I'm bringing together someone like Simon Wardley to help people do Wardley Mapping. Like, that is a tool that allows you to see the landscape that you're operating in. You know, if you use that sort of tool to understand the real-world impact of what you're doing, then you can start caring about it a bit more. I'm bringing in somebody called Anne Currie, who is a tech ethicist and speaker and lecturer, and she's actually written some [laugh] really great nonfiction books, which I'd recommend everyone reads. It starts with Utopia Five.And that's about asking, “Well, is this ethical? Can we continue to do these things?” Can't—talks about things about sustainability. If it's not sustainable for everyone, it's not ethical. So, when I mentioned 3 billion people currently don't use the internet, it's like, can we continue to just keep on doing things the same way?And then John Booth, who is a data center expert, to help us really understand what the reality is on the ground. What are these data centers really look like? And then Amanda Brock, from OpenUK in the conference will joining as well to talk about, kind of, open-source and how we can make the world kind of a better place by getting involved in these communities. So, that'll be a really great panel.But what I'm also doing is releasing this as an online course. So, for people who want to get involved, it will be very intimate, about 15 seats on each core, so three weeks for you to actually work and talk directly with some of these experts and me to figure out what you want to do in the world of climate change and how you can take those first steps. So, it'll be a journey that even starts with an ecotherapist to help us deal with climate grief and wonder about the things we can do as individuals to feel better ourselves and be happier. So, I think that'd be a really great thing for a lot of people. And, yeah, not only that, but… it'll be great for you, but it also goes towards making the world a better place.So, 50% of the course fees will be donated, 25%, to charity, and 25% supporting open-source projects. So, I think it kind of just win, win, win. And that's the story of sustainability in general. It's a win, win, win for everyone. If you start seeing the world through a lens of sustainability, you'll save money, you'll sleep better at night, you'll get involved with some really great communities, and meet some really great people who care about this as well. And yeah, it'll be a brighter future.Corey: If people want to learn more, where can they find you?Aerin: So, if you want to learn more about what I'm up to, I'm on Twitter under @aerincloud, that A-E-R-I-N cloud. And then you can also find me on LinkedIn. But I also run my own podcast that was inspired by Corey, called Public Cloud for Public Good talking about cloud sustainability and how to make the world a better place for the use of public cloud services.Corey: And we will, of course, put a link to that in the [show notes 00:32:32]. Thank you so much for your time. I appreciate it, as always.Aerin: Thank you.Corey: Aerin Booth, the Ted Lasso of cloud. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this episode, please leave a five-star review on your podcast platform of choice, along with an angry and insulting comment that I will immediately scale to zero in true serverless fashion.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About ChristinaChristina Maslach, PhD, is a Professor of Psychology (Emerita) and a researcher at the Healthy Workplaces Center at the University of California, Berkeley.  She received her A.B. from Harvard, and her Ph.D. from Stanford.  She is best known as the pioneering researcher on job burnout, producing the standard assessment tool (the Maslach Burnout Inventory, MBI), books, and award-winning articles.  The impact of her work is reflected by the official recognition of burnout, as an occupational phenomenon with health consequences, by the World Health Organization in 2019.  In 2020, she received the award for Scientific Reviewing, for her writing on burnout, from the National Academy of Sciences.  Among her other honors are: Fellow of the American Association for the Advancement of Science (1991 -- "For groundbreaking work on the application of social psychology to contemporary problems"), Professor of the Year (1997), and the 2017 Application of Personality and Social Psychology Award (for her research career on job burnout).  Links: The Truth About Burnout: https://www.amazon.com/Truth-About-Burnout-Organizations-Personal/dp/1118692136 TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: If you asked me to rank which cloud provider has the best developer experience, I'd be hard-pressed to choose a platform that isn't Google Cloud. Their developer experience is unparalleled and, in the early stages of building something great, that translates directly into velocity. Try it yourself with the Google for Startups Cloud Program over at cloud.google.com/startup. It'll give you up to $100k a year for each of the first two years in Google Cloud credits for companies that range from bootstrapped all the way on up to Series A. Go build something, and then tell me about it. My thanks to Google Cloud for sponsoring this ridiculous podcast.Corey: This episode is brought to us by our friends at Pinecone. They believe that all anyone really wants is to be understood, and that includes your users. AI models combined with the Pinecone vector database let your applications understand and act on what your users want… without making them spell it out. Make your search application find results by meaning instead of just keywords, your personalization system make picks based on relevance instead of just tags, and your security applications match threats by resemblance instead of just regular expressions. Pinecone provides the cloud infrastructure that makes this easy, fast, and scalable. Thanks to my friends at Pinecone for sponsoring this episode. Visit Pinecone.io to understand more.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One subject that I haven't covered in much depth on this show has been a repeated request from the audience, and that is to talk a bit about burnout. So, when I asked the audience who I should talk to about burnout, there were really two categories of responses. The first was, “Pick me. I hate my job, and I'd love to talk about that.” And the other was, “You should speak to Professor Maslach.” Christina Maslach is a Professor of Psychology at Berkeley. She's a teacher and a researcher, particularly in the area of burnout. Professor, welcome to the show.Dr. Maslach: Well, thank you for inviting me.Corey: So, I'm going to assume from the outset that the reason that people suggest that I speak to you about burnout is because you've devoted a significant portion of your career to studying the phenomenon, and not just because you hate your job and are ready to go do something else. Is that directionally correct?Dr. Maslach: That is directionally correct, yes. I first stumbled upon the phenomenon back in the 1970s—which is, you know, 45, almost 50 years ago now—and have been fascinated with trying to understand what is going on.Corey: So, let's start at the very beginning because I'm not sure in, I guess, the layperson context that I use the term that I fully understand it. What is burnout?Dr. Maslach: Well, burnout as we have been studying it over many years, it's a stress phenomenon, okay, it's a response to stressors, but it's not just the exhaustion of stress. That's one component of it, but it actually has two other components that go along with it. One is this very negative, cynical, hostile attitude toward the job and the other people in it, you know, “Take this job and shove it,” kind of feeling. And usually, people don't begin their job like that, but that's where they go if they become more burned out.Corey: I believe you may have just inadvertently called out a decent proportion of the tech sector.Dr. Maslach: [laugh].Corey: Or at least, that might just be my internal cynicism rising to the foreground.Dr. Maslach: No, it's not. Actually, I have heard from a number of tech people over the past decades about just this kind of issue. And so I think it's particularly relevant. The third component that we see going along with this, it usually comes in a little bit later, but I've heard a lot about this from tech people as well, and that is that you begin to develop a very negative sense of your own self, and competence, and where you're going, and what you're able to do. So, the stress response of exhaustion, the negative cynicism towards the job, the negative evaluation of yourself, that's the trifecta of burnout.Corey: You've spent a lot of your early research at least focusing on, I guess, occupations that you could almost refer to as industrial, in some respects: working with heavy equipment, working with a variety of different professionals in very stressful situations. It feels weird, on some level, to say, “Oh, yeah, my job is very stressful. In that vein, I have to sit in front of a computer all day, and sometimes I have to hop on a meeting with people.” And it feels, on some level, like that even saying, “I'm experiencing burnout,” in my role is a bit of an overreach.Dr. Maslach: Yeah, that's an interesting point because, in fact, yes, when we think about OSHA, you know, and occupational risks and hazards, we do think about the chemicals, and the big equipment, and the hazards, so having more psychological and social risk factors, is something that probably a lot of people don't resonate to immediately and think, well, if you're strong, and if you're resilient, and whatever, you can—anybody can handle that, and that's really a test almost of your ability to do your work. But what we're finding is that it has its own hazards, psychological and social as well. And so, burnout is something that we've seen in a lot of more people-oriented professions, from the beginning. Healthcare has had this for a long time. Various kinds of social services, teaching, all of these other things. So, it's actually not a sign of weakness as some people might think.Corey: Right. And that's part of the challenge and, honestly, one of the reasons that I've stayed away from having in-depth discussions about the topic of burnout on the show previously is it feels that—rightly or wrongly, and I appreciate your feedback on this one either way—it feels like it's approaching the limits of what could be classified as mental health. And I can give terrible advice on how computers work—in fact, I do on a regular basis; it's kind of my thing—and that's usually not going to have any lasting impact on people who don't see through the humor part of that. But when we start talking about mental health, I'm cautious because it feels like an inadvertent story or advice that works for some but not all, has the potential to do a tremendous bit of damage, and I'm very cautious about that. Is burnout a mental health issue? Is it a medical issue that is recognized? Where does it start, okay does it stop on that spectrum?Dr. Maslach: It is not a medical issue—and the World Health Organization, which just came out with a statement about this in 2019 on burnout, they're recognizing it as an occupational risk factor—made it very clear that this is not a medical thing. It is not a medical disease, it doesn't have a certain set of medical diagnoses, although people tend to sometimes go there. Can it have physical health outcomes? In other words, if you're burning out and you're not sleeping well, and you're not eating well, and not taking care of yourself, do you begin to impair your physical health down the road? Yes.Could it also have mental health outcomes, that you begin to feel depressed, and anxious, and not knowing what to do, and afraid of the future? Yes, it could have those outcomes as well. So, it certainly is kind of like—I can put it this way, like a stepping stone in a path to potential negative health: physical health, or mental health issues. And I think that's one of the reasons why it is so important. But unfortunately, a lot of people still view it as somebody who's burned out isn't tough enough, strong enough, they're wimpy, they're not good enough, they're not a hundred percent.And so the stigma that is often attached to burnout, people not only indulge it, but they feel it directed towards them, and often they will try to hide the kinds of experiences they're having because they worry that they are going to be judged negatively, thrown under the bus, you know, let go from the job, whatever, if they talk about what's actually happening with them.Corey: What do you see, as you look around, I guess, the wide varieties of careers that are susceptible to burnout—which I have a sneaking suspicion based upon what you've said rounds to all of them—what do you think is the most misunderstood, or misunderstood aspects of burnout?Dr. Maslach: I think what's most misunderstood is that people assume that it is a problem of the individual person. And if somebody is burned out, then they've got to just take care of themselves, or take a break, or eat better, or get more sleep, all of those kinds of things which cope with stressors. What's not as well understood or focused on is the fact that this is a response to other stressors, and these stressors are often in the workplace—this is where I've been studying it—but in essentially in the larger social, physical environment that people are functioning in. They're not burning out all by themselves.There's a reason why they are feeling the kind of exhaustion, developing that cynicism, beginning to doubt themselves, that we see with burnout. So there, if you ever want to talk about preventing burnout, you really have to be focusing on what are the various kinds of things that seem to be causing the problem, and how do we modify those? Coping with stressors is a good thing, but it doesn't change the stressors. And so we really have to look at that, as well as what people can bring about, you know, taking care of themselves or trying to do the job better or differently.Corey: I feel like it's impossible to have a conversation like this without acknowledging the background of the past year that many of us have spent basically isolated, working from home. And for some folks, okay, they were working from home before, but it feels different now. At least that's the position I find myself in. Other folks are used to going into an office and now they're either isolated—and research shows that it has been worse, statistically, for single people versus married people, but married people are also trapped at home with their spouse, which sounds half-joking but it is very real. At some point, distance is useful.And it feels like everyone is sort of a bit at their wit's end. It feels like things are closer to being frayed, there's a constant sense that there's this, I guess, pervasive dread for the past year. Are you seeing that that has a potential to affect how burnout is being expressed or perceived?Dr. Maslach: I think it has, and one of the things that we clearly see is that people are using the word burnout, more and more and more and more. It's almost becoming the word du jour, and using it to describe, things are going wrong and it's not good. And it may be overstretching the use of burnout, but I think the reason of the popularity of the term is that it has this kind of very vivid imagery of things going up in smoke, and can't handle it, and flames licking at your heels, and all this sort of stuff so that they can do that. I even got a comment from a colleague in France just a few days ago, where they're talking about, “Is burnout the malady of the century?” you know, kind of thing. And it's being used a lot; it's sometimes maybe overused, but I think it's also striking a chord with people as a sign that things are going badly, and I don't know how to deal with it in some way.Corey: It also feels, on some level, for those of us who are trapped inside, it kind of almost feels like it's a tremendous expression of privilege because who am I to have a problem with this? Oh, I have to go inside and order a lot of takeout and spend time with my family. And I look at how folks who are nowhere near as privileged have to go and be essential workers and show up in increasingly dangerous positions. And it almost feels like burnout isn't something that I'm entitled to, if that makes sense.Dr. Maslach: [laugh]. Yeah. It's an interesting description of that because I think there are ways in which people are looking at their experience and dealing with it, and like many things in life, I find that all of these things are a bit of a double-edged sword; there's positive and there's negative aspects to them. And so when I've talked with some people about now having to work from home rather than working in their office, they're also bringing up, “Well, hey, I've noticed that the interviews I'm doing with potential clients are actually going a little better”—you know, this is from a law office—“And trying to figure out how—are we doing it differently so that people can actually relate to each other as human beings instead of the suit and tie in the big office? What's going on in terms of how we're doing the work that there may be actually a benefit here?”For others. It's been, “Oh, my gosh. I don't have to commute, but endless meetings and people are thinking I'm not doing my job, and I don't know how to get in touch, and how do we work together effectively?” And so there's other things that are much more difficult, in some sense. I think another thing that you have to keep in mind that it's not just about how you're doing your work, perhaps differently, or you're under different circumstances, but people, so many people have lost their jobs, and are worried that they may lose their jobs.That we're actually finding that people are going into overdrive and working harder and more hours as a way of trying to protect from being the next one who won't have any income at all. So, there's a lot of other dynamics that are going on as a result of the pandemic, I think, that we need to be aware of.Corey: One thing that I'd like to point out is that you are a Professor Emerita of Psychology at Berkeley, which means you presumably wound up formulating this based upon significant bodies of peer-reviewed research, as opposed to just coming up with a thesis, stating it as if it were fact, and then writing an entire series of books on it. I mean, that path, I believe, is called being a venture capitalist, but I may be mistaken on that front. How do you effectively study something like burnout? It feels like it is so subjective and situation-specific, but it has to have a normalization aspect to it.Dr. Maslach: Uh, yeah, that's a good point. I think, in fact, the first time I ever wrote about some of the stuff that I was learning about burnout back in the mid '70s—I think it was '75, '76 maybe—and it was in a magazine, it wasn't in a journal. It wasn't peer-reviewed because not even peer-reviewed journals would review this; they thought it was pop psychology, and eh. So, I would get, in those days, snail mail by the sackfuls from people saying, “Oh, my God. I didn't know anybody else felt like this. Let me tell you my story.”You know, kind of thing. And so that was really, after doing a lot of interviews with people, following them on the job when possible to, sort of, see how things were going, and then writing about the basic themes that were coming out of this, it turned out that there were a lot of people who responded and said, “I know that. I've been there. I'm experiencing it.” Even though each of them were sort of thinking, “I'm the only one. What's wrong with me? Everybody else seems fine.”And so part of the research in trying to get it out in whatever form you can is trying to share it because that gives you feedback from a wide variety of people, not only the peers reviewing the quality of the research, but the people who are actually trying to figure out how to deal effectively with this problem. So it's, how do I and my colleagues actually have a bigger, broader conversation with people from which we learn a lot, and then try and say, okay, and here's everything we've heard, and let's throw it back out and share it and see what people think.Corey: You have written several books on the topic, if I'm not mistaken. And one thing that surprises me is how much what you talk about in those books seems to almost transcend time. I believe your first was published in 1982—Dr. Maslach: Right.Corey: —if I'm not mistaken—Dr. Maslach: Yes.Corey: —and it's an awful lot of what it talks about still feels very much like it could be written today. Is this just part of the quintessential human experience? Or has nothing new changed in the last 200 years since the Industrial Revolution? How is it progressing, if at all, and what does the future look like?Dr. Maslach: Great questions and I don't have a good answer for you. But we have sort of struggled with this because if you look at older literature, if you even go back centuries, if you even go back in parts of the Bible or something, you're seeing phrases and descriptions sometime that says sounds a lot like burnout, although we're not using that term. So, it's not something that I think just somehow got invented; it wasn't invented in the '70s or anything like that. But trying to trace back those roots and get a better sense of what are we capturing here is fascinating, and I think we're still working on it.People have asked, well, where did the term ‘burnout' as opposed to other kinds of terms come from? And it's been around for a while, again, before the '70s or something. I mean, we have Graham Greene writing the novel A Burnt-Out Case, back in the early '60s. My dad was an engineer, rarefied gas dynamics, so he was involved with the space program and engineers talk about burnout all the time: ball bearings burn out, rocket boosters burn out. And when they started developing Silicon Valley, all those little startups and enterprises, they advertised as burnout shops. And that was, you know, '60s, into the '70s, et cetera, et cetera. So, the more modern roots, I think probably have some ties to that use of the term before I and other researchers even got started with it.Corey: This episode is sponsored in part by our friends at Uptycs, because they believe that many of you are looking to bolster your security posture with CNAPP and XDR solutions. They offer both cloud and endpoint security in a single UI and data model. Listeners can get Uptycs for up to 1,000 assets through the end of 2023 (that is next year) for $1. But this offer is only available for a limited time on UptycsSecretMenu.com. That's U-P-T-Y-C-S Secret Menu dot com.Corey: This is one of those questions that is incredibly self-serving, and I refuse to apologize for it. How can I tell whether I'm suffering from burnout, versus I'm just a jerk with an absolutely terrible attitude? And that is not as facetious a question as it probably sounds like.Dr. Maslach: [laugh]. Yeah. Well, part of the problem for me—or the challenge for me—is to understand what it is people need to know about themselves. Can I take a diagnostic test which tells me if I am burned out or if I'm something else?Sort of the more important question is, what is feeling right and what is not feeling so good—or even wrong—about my experience? And usually, you can't figure that all out by yourself and you need to get other input from other people. And it could be a counselor or therapist, or it could be friends or colleagues who you have to be able to get to a point where we can talk about it, and hear each other, and get some feedback without putdowns, just sort of say, “Yeah, have you ever thought about the fact that when you get this kind of a task, you usually just go crazy for a while and not really settle down and figure out what you really need to do as opposed to what you think you have to do?” Part of this, are you bringing yourself in terms of the stress response, but what is it that you're not doing—or that you're doing not well—to figure out solutions, to get help or advice or better input from others. So, it takes time, but it really does take a lot of that kind of social feedback.So, when I said—if I can stay with it a little bit more—when I first was writing and publishing about and all these people were writing back saying, “I thought I was the only one,” that phenomenon of putting on a happy face and not letting anybody else see that you're going through some difficult challenges, or feeling bad, or depressed, or whatever is something we call pluralistic ignorance; means we don't have good knowledge about what is normal, or what is being shared, or how other people are because we're all pretending to put on the happy face, to pretend and make sure that everybody thinks we're okay and is not going to come after us. But if we all do that, then we all, together, are creating a different social reality that people perceive rather than actually what is happening behind that mask.Corey: It feels, on some level, like this is an aspect of the social media problem, where we're comparing our actual lives and all the bloopers that we see to other people's highlight reels because few people wind up talking very publicly about their failures.Dr. Maslach: Oh, yeah. Yeah. And often for good reason because they know they will be attacked and dumped. And there could be some serious consequences, and you just say, “I'm going to figure out what I'm going to do on my own.”But one of the things that when I work with people, and I'm asking them, “What do you think would help? What sort of things that don't happen could happen?” And so forth, one of the things that goes to the top of the list is having somebody else; a safe relationship, a safe place where we can talk, where we can unburden, where you're not going to spill the beans to everybody else, and you're getting advice, or you're getting a pat on the back, or a shoulder to cry on, and that you're there for them for the same kind of reason. So, it's a different form of what we think of as social network. It used to be that a network like that meant that you had other people, whether family, friends, neighbors, colleagues, whoever, that you knew, you could go to; a mentor, an advisor, a trusted ally, and that you would perform that role for them and other people, as well.And what has happened, I think, to add to the emphasis on burnout these days, is that those social connections, those trusts, between people has really been shredding, and, you know—or cut off or broken apart. And so people are feeling isolated, even if they're surrounded by a lot of other people, don't want to raise their hand, don't want to say, “Can we talk over coffee? I'm really having a bad day. I need some help to figure out this problem.” And so one of those most valuable resources that human beings need—which is other people—is, if we're working in environments where that gets pulled apart, and shredded, and it's lacking, that's a real risk factor for burnout.Corey: What are the things that contribute to burnout? It doesn't feel, based upon what you've said so far, that it's one particular thing. There has to be points of commonality between all of this, I have to imagine.Dr. Maslach: Yeah.Corey: Is it possible to predict that, oh, this is a scenario in which either I or people who are in this role are likely to become burned out faster?Dr. Maslach: Mm-hm. Yeah. Good question and I don't know if we have a final answer, but at this point, in terms of all the research that's been done, not just on burnout, but on much larger issues of health, and wellbeing, and stress, and coping, and all the rest of it, there are clearly six areas in which the fit between people and their job environment are critical. And if the fit is—or the match, or the balance—is better, they are going to be at less risk for burnout, they're more likely to be engaged with work.But if some real bad fits, or mismatches, occur in one or more of these areas, that raises the risk factor for burnout. So, if I can just mention those six quickly. And these are not in any particular order because I find that people assume the first one is the worst or the best, and it's not. Any rate, one of them has to do with that social environment I was just talking about; think of it as the workplace community. All the people whose paths you cross at various points—you know, coworkers, the people you supervise, your bosses, et cetera—so those social relationships, that culture, do you have a supportive environment which really helps people thrive? Can you trust people, there's respect, and all that kind of thing going on? Or is it really what people are now describing as a socially toxic work environment?A second area has to do with reward. And it turns out not so much salary and benefits, it's more about social recognition and the intrinsic reward you get from doing a good job. So, if you work hard, do some special things, and nothing positive happens—nobody even pats you on the back, nobody says, “Gee, why don't you try this new project? I think you're really good at it,” anything that acknowledges what you've done—it's a very difficult environment to work in. People who are more at risk of burnout, when I asked them, “What is a good day for you? A good day. A really good day.” And the answer is often, “Nothing bad happens.” But it's not the presence of good stuff happening, like people glad that you did such good work or something like that.Third area has to do with values—and this is one that also often gets ignored, but sometimes this is the critical bottom line—that you're doing work that you think is meaningful, where you're working has integrity, and you're in line with that as opposed to value conflicts where you're doing things that you think are wrong: “I want to help people, I want to help cure patients, and here, I'm actually only supposed to be trying to help the hospital get more money.” When they have that kind of value conflict, this is often where they have to say, “I don't want to sell my soul and I'm leaving.”The fourth area is one of fairness. And this is really about that whatever the policies, the principles, et cetera, they're administered fairly. So, when things are going badly here—the mismatch—this is where discrimination lives, this is where glass ceilings are going on, that people are not being treated fairly in terms of the work they do, how they're promoted, or all of those kinds of things. So, that interpersonal respect, and, sort of, social justice is missing.The next two areas—the fifth and six—are probably the two that had been the most well-known for a long time. One has to do with workload and how manageable it is. Given the demands that you have, do you have sufficient resources, like time, and tools, and whatever other kind of teams support you need to get the job done. And control is about the amount of autonomy and the opportunities you have to perhaps improvise, or innovate, or correct, or figure out how to do the job better in some way. So, when people are having mismatches in work overload; a lack of control; you cannot improvise; where you have unfairness; where there is values that are just incompatible with what you believe is right, a sort of moral issue; where you're not getting any kind of positive feedback, even when it's deserved, for the kind of work you're doing; and when you're working in a socially toxic relationship where you can't trust people, you don't know who to turn to, people are having unresolved conflicts all the time. Those six areas are, those are the markers really of risk factors for burnout.Corey: I know that I'm looking back through my own career history listening to you recount those and thinking, “Oh, maybe I wasn't just a terrible employee in every one of those situations.”Dr. Maslach: Exactly.Corey: I'm sure a lot of it did come from me, I want to be very clear here. But there's also that aspect of this that might not just be a ‘me' problem.Dr. Maslach: Yeah. That's a good way of putting it. It's really in some sense, it's more of a ‘we' problem than a ‘me' problem. Because again, you're not working in isolation, and the reciprocal relationship you have with other people, and other policies, and other things that are happening in whatever workplace that is, is creating a kind of larger environment in which you and many others are functioning.And we've seen instances where people begin to make changes in that environment—how do we do this differently? How can we do this better, let's try it out for a while and see if this can work—and using those six areas, the value is not just, “Oh, it's really in bad shape. We have huge unfairness issues.” But then it says, “It would be better if we could figure out a way to get rid of that fairness problem, or to make a modification so that we have a more fair process on that.” So, they're like guideposts as well.As people start thinking through these six areas, you can sort of say, “What's working well, in terms of workload, what's working badly? Where do we run into problems on control? How do we improve the social relationships between colleagues who have to work together on a team?” They're not just markers of what's gone wrong, but they can—if you flip it around and look at it, let's look at the other end—okay is a path that we could get better? Make it right?Corey: If people want to learn more about burnout in general, and you're working in it specifically, where can they go to find your work and learn more about what you have to say?Dr. Maslach: Obviously, there's been a lot of articles, and now lots of things on the web, and in past books that I've written. And as you said, in many ways, they are still pretty relevant. The Truth About Burnout came out, oh gosh, '97. So, that's 25 years ago and it's still work.But my colleague, Michael Leiter from Canada, and I have just written up a new manuscript for a new book in which we really are trying to focus on sharing everything we have learned about, you know, what burnout has taught us, and put that into a format of a book that will allow people to really take what we've learned and figure out how does this apply? How can this be customized to our situation? So, I'm hoping that that will be coming out within the next year.Corey: And you are, of course, welcome back to discuss your book when it releases.Dr. Maslach: I would be honored if you would have me back. That would be a wonderful treat.Corey: Absolutely. But in return, I do expect a pre-release copy of the manuscript, so I have something intelligent to talk about.Dr. Maslach: [laugh]. Of course, of course.Corey: Thank you so much for your time. I really appreciate it.Dr. Maslach: Well, thank you for having me. I appreciate the opportunity to share this, especially during these times.Corey: Indeed. Professor Christina Maslach, Professor Emeritus of Psychology at Berkeley, I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an insulting comment telling me why you're burned out on this show.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About RamDr. Ram Sriharsha held engineering, product management, and VP roles at the likes of Yahoo, Databricks, and Splunk. At Yahoo, he was both a principal software engineer and then research scientist; at Databricks, he was the product and engineering lead for the unified analytics platform for genomics; and, in his three years at Splunk, he played multiple roles including Sr Principal Scientist, VP Engineering and Distinguished Engineer.Links Referenced: Pinecone: https://www.pinecone.io/ XKCD comic: https://www.explainxkcd.com/wiki/index.php/1425:_Tasks TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Chronosphere. Tired of observability costs going up every year without getting additional value? Or being locked into a vendor due to proprietary data collection, querying, and visualization? Modern-day, containerized environments require a new kind of observability technology that accounts for the massive increase in scale and attendant cost of data. With Chronosphere, choose where and how your data is routed and stored, query it easily, and get better context and control. 100% open-source compatibility means that no matter what your setup is, they can help. Learn how Chronosphere provides complete and real-time insight into ECS, EKS, and your microservices, wherever they may be at snark.cloud/chronosphere that's snark.cloud/chronosphere.Corey: This episode is brought to you in part by our friends at Veeam. Do you care about backups? Of course you don't. Nobody cares about backups. Stop lying to yourselves! You care about restores, usually right after you didn't care enough about backups. If you're tired of the vulnerabilities, costs, and slow recoveries when using snapshots to restore your data, assuming you even have them at all living in AWS-land, there is an alternative for you. Check out Veeam, that's V-E-E-A-M for secure, zero-fuss AWS backup that won't leave you high and dry when it's time to restore. Stop taking chances with your data. Talk to Veeam. My thanks to them for sponsoring this ridiculous podcast.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today's promoted guest episode is brought to us by our friends at Pinecone and they have given their VP of Engineering and R&D over to suffer my various sling and arrows, Ram Sriharsha. Ram, thank you for joining me.Ram: Corey, great to be here. Thanks for having me.Corey: So, I was immediately intrigued when I wound up seeing your website, pinecone.io because it says right at the top—at least as of this recording—in bold text, “The Vector Database.” And if there's one thing that I love, it is using things that are not designed to be databases as databases, or inappropriately referring to things—be they JSON files or senior engineers—as databases as well. What is a vector database?Ram: That's a great question. And we do use this term correctly, I think. You can think of customers of Pinecone as having all the data management problems that they have with traditional databases; the main difference is twofold. One is there is a new data type, which is vectors. Vectors, you can think of them as arrays of floats, floating point numbers, and there is a new pattern of use cases, which is search.And what you're trying to do in vector search is you're looking for the nearest, the closest vectors to a given query. So, these two things fundamentally put a lot of stress on traditional databases. So, it's not like you can take a traditional database and make it into a vector database. That is why we coined this term vector database and we are building a new type of vector database. But fundamentally, it has all the database challenges on a new type of data and a new query pattern.Corey: Can you give me an example of what, I guess, an idealized use case would be of what the data set might look like and what sort of problem you would have in a vector database would solve?Ram: A very great question. So, one interesting thing is there's many, many use cases. I'll just pick the most natural one which is text search. So, if you're familiar with the Elastic or any other traditional text search engines, you have pieces of text, you index them, and the indexing that you do is traditionally an inverted index, and then you search over this text. And what this sort of search engine does is it matches for keywords.So, if it finds a keyword match between your query and your corpus, it's going to retrieve the relevant documents. And this is what we call text search, right, or keyword search. You can do something similar with technologies like Pinecone, but what you do here is instead of searching our text, you're searching our vectors. Now, where do these vectors come from? They come from taking deep-learning models, running your text through them, and these generate these things called vector embeddings.And now, you're taking a query as well, running them to deep-learning models, generating these query embeddings, and looking for the closest record embeddings in your corpus that are similar to the query embeddings. This notion of proximity in this space of vectors tells you something about semantic similarity between the query and the text. So suddenly, you're going beyond keyword search into semantic similarity. An example is if you had a whole lot of text data, and maybe you were looking for ‘soda,' and you were doing keyword search. Keyword search will only match on variations of soda. It will never match ‘Coca-Cola' because Coca-Cola and soda have nothing to do with each other.Corey: Or Pepsi, or pop, as they say in the American Midwest.Ram: Exactly.Corey: Yeah.Ram: Exactly. However, semantic search engines can actually match the two because they're matching for intent, right? If they find in this piece of text, enough intent to suggest that soda and Coca-Cola or Pepsi or pop are related to each other, they will actually match those and score them higher. And you're very likely to retrieve those sort of candidates that traditional search engines simply cannot. So, this is a canonical example, what's called semantic search, and it's known to be done better by these other vector search engines. There are also other examples in say, image search. Just if you're looking for near duplicate images, you can't even do this today without a technology like vector search.Corey: What is the, I guess, translation or conversion process of existing dataset into something that a vector database could use? Because you mentioned it was an array of floats was the natural vector datatype. I don't think I've ever seen even the most arcane markdown implementation that expected people to wind up writing in arrays of floats. What does that look like? How do you wind up, I guess, internalizing or ingesting existing bodies of text for your example use case?Ram: Yeah, this is a very great question. This used to be a very hard problem and what has happened over the last several years in deep-learning literature, as well as in deep-learning as a field itself, is that there have been these large, publicly trained models, examples will be OpenAI, examples will be the models that are available in Hugging Face like Cohere, and a large number of these companies have come forward with very well trained models through which you can pass pieces of text and get these vectors. So, you no longer have to actually train these sort of models, you don't have to really have the expertise to deeply figured out how to take pieces of text and build these embedding models. What you can do is just take a stock model, if you're familiar with OpenAI, you can just go to OpenAIs homepage and pick a model that works for you, Hugging Face models, and so on. There's a lot of literature to help you do this.Sophisticated customers can also do something called fine-tuning, which is built on top of these models to fine-tune for their use cases. The technology is out there already, there's a lot of documentation available. Even Pinecone's website has plenty of documentation to do this. Customers of Pinecone do this [unintelligible 00:07:45], which is they take piece of text, run them through either these pre-trained models or through fine-tuned models, get the series of floats which represent them, vector embeddings, and then send it to us. So, that's the workflow. The workflow is basically a machine-learning pipeline that either takes a pre-trained model, passes them through these pieces of text or images or what have you, or actually has a fine-tuning step in it.Corey: Is that ingest process something that not only benefits from but also requires the use of a GPU or something similar to that to wind up doing the in-depth, very specific type of expensive math for data ingestion?Ram: Yes, very often these run on GPUs. Sometimes, depending on budget, you may have compressed models or smaller models that run on CPUs, but most often they do run on GPUs, most often, we actually find people make just API calls to services that do this for them. So, very often, people are actually not deploying these GPU models themselves, they are maybe making a call to Hugging Face's service, or to OpenAI's service, and so on. And by the way, these companies also democratized this quite a bit. It was much, much harder to do this before they came around.Corey: Oh, yeah. I mean, I'm reminded of the old XKCD comic from years ago, which was, “Okay, I want to give you a picture. And I want you to tell me it was taken within the boundaries of a national park.” Like, “Sure. Easy enough. Geolocation information is attached. It'll take me two hours.” “Cool. And I also want you to tell me if it's a picture of a bird.” “Okay, that'll take five years and a research team.”And sure enough, now we can basically do that. The future is now and it's kind of wild to see that unfolding in a human perceivable timespan on these things. But I guess my question now is, so that is what a vector database does? What does Pinecone specifically do? It turns out that as much as I wish it were otherwise, not a lot of companies are founded on, “Well, we have this really neat technology, so we're just going to be here, well, in a foundational sense to wind up ensuring the uptake of that technology.” No, no, there's usually a monetization model in there somewhere. Where does Pinecone start, where does it stop, and how does it differentiate itself from typical vector databases? If such a thing could be said to exist yet.Ram: Such a thing doesn't exist yet. We were the first vector database, so in a sense, building this infrastructure, scaling it, and making it easy for people to operate it in a SaaS fashion is our primary core product offering. On top of that, this very recently started also enabling people who have who actually have raw text to not just be able to get value from these vector search engines and so on, but also be able to take advantage of traditional what we call keyword search or sparse retrieval and do a combined search better, in Pinecone. So, there's value-add on top of this that we do, but I would say the core of it is building a SaaS managed platform that allows people to actually easily store as data, scale it, query it in a way that's very hands off and doesn't require a lot of tuning or operational burden on their side. This is, like, our core value proposition.Corey: Got it. There's something to be said for making something accessible when previously it had only really been available to people who completed the Hello World tutorial—which generally resembled a doctorate at Berkeley or Waterloo or somewhere else—and turn it into something that's fundamentally, click the button. Where on that, I guess, a spectrum of evolution do you find that Pinecone is today?Ram: Yeah. So, you know, prior to Pinecone, we didn't really have this notion of a vector database. For several years, we've had libraries that are really good that you can pre-train on your embeddings, generate this thing called an index, and then you can search over that index. There is still a lot of work to be done even to deploy that and scale it and operate it in production and so on. Even that was not being, kind of, offered as a managed service before.What Pinecone does which is novel, is you no longer have to have this pre-training be done by somebody, you no longer have to worry about when to retrain your indexes, what to do when you have new data, what to do when there is deletions, updates, and the usual data management operations. You can just think of this is, like, a database that you just throw your data in. It does all the right things for you, you just worry about querying. This has never existed before, right? This is—it's not even like we are trying to make the operational part of something easier. It is that we are offering something that hasn't existed before, at the same time, making it operationally simple.So, we're solving two problems, which is we building a better database that hasn't existed before. So, if you really had this sort of data management problems and you wanted to build an index that was fresh that you didn't have to super manually tune for your own use cases, that simply couldn't have been done before. But at the same time, we are doing all of this in a cloud-native fashion; it's easy for you to just operate and not worry about.Corey: You've said that this hasn't really been done before, but this does sound like it is more than passingly familiar specifically to the idea of nearest neighbor search, which has been around since the '70s in a bunch of different ways. So, how is it different? And let me of course, ask my follow-up to that right now: why is this even an interesting problem to start exploring?Ram: This is a great question. First of all, nearest neighbor search is one of the oldest forms of machine learning. It's been known for decades. There's a lot of literature out there, there are a lot of great libraries as I mentioned in the passing before. All of these problems have primarily focused on static corpuses. So basically, you have a set of some amount of data, you want to create an index out of it, and you want to query it.A lot of literature has focused on this problem. Even there, once you go from small number of dimensions to large number of dimensions, things become computationally far more challenging. So, traditional nearest neighbor search actually doesn't scale very well. What do I mean by large number of dimensions? Today, deep-learning models that produce image representations typically operate in 2048 dimensions of photos [unintelligible 00:13:38] dimensions. Some of the OpenAI models are even 10,000 dimensional and above. So, these are very, very large dimensions.Most of the literature prior to maybe even less than ten years back has focused on less than ten dimensions. So, it's like a scale apart in dealing with small dimensional data versus large dimensional data. But even as of a couple of years back, there hasn't been enough, if any, focus on what happens when your data rapidly evolves. For example, what happens when people add new data? What happens if people delete some data? What happens if your vectors get updated? These aren't just theoretical problems; they happen all the time. Customers of ours face this all the time.In fact, the classic example is in recommendation systems where user preferences change all the time, right, and you want to adapt to that, which means your user vectors change constantly. When even these sort of things change constantly, you want your index to reflect it because you want your queries to catch on to the most recent data. [unintelligible 00:14:33] have to reflect the recency of your data. This is a solved problem for traditional databases. Relational databases are great at solving this problem. A lot of work has been done for decades to solve this problem really well.This is a fundamentally hard problem for vector databases and that's one of the core focus areas [unintelligible 00:14:48] painful. Another problem that is hard for these sort of databases is simple things like filtering. For example, you have a corpus of say product images and you want to only look at images that maybe are for the Fall shopping line, right? Seems like a very natural query. Again, databases have known and solved this problem for many, many years.The moment you do nearest neighbor search with these sort of constraints, it's a hard problem. So, it's just the fact that nearest neighbor search and lots of research in this area has simply not focused on what happens to that, so those are of techniques when combined with data management challenges, filtering, and all the traditional challenges of a database. So, when you start doing that you enter a very novel area to begin with.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open-source database. If you're tired of managing open-source Redis on your own, or if you are looking to go beyond just caching and unlocking your data's full potential, these folks have you covered. Redis Enterprise is the go-to managed Redis service that allows you to reimagine how your geo-distributed applications process, deliver and store data. To learn more from the experts in Redis how to be real-time, right now, from anywhere, visit snark.cloud/redis. That's snark dot cloud slash R-E-D-I-S.Corey: So, where's this space going, I guess is sort of the dangerous but inevitable question I have to ask. Because whenever you talk to someone who is involved in a very early stage of what is potentially a transformative idea, it's almost indistinguishable from someone who is whatever the polite term for being wrapped around their own axle is, in a technological sense. It's almost a form of reverse Schneier's Law of anyone can create an encryption algorithm that they themselves cannot break. So, the possibility that this may come back to bite us in the future if it turns out that this is not potentially the revelation that you see it as, where do you see the future of this going?Ram: Really great question. The way I think about it is, and the reason why I keep going back to databases and these sort of ideas is, we have a really great way to deal with structured data and structured queries, right? This is the evolution of the last maybe 40, 50 years is to come up with relational databases, come up with SQL engines, come up with scalable ways of running structured queries on large amounts of data. What I feel like this sort of technology does is it takes it to the next level, which is you can actually ask unstructured questions on unstructured data, right? So, even the couple of examples we just talked about, doing near duplicate detection of images, that's a very unstructured question. What does it even mean to say that two images are nearly duplicate of each other? I couldn't even phrase it as kind of a concrete thing. I certainly cannot write a SQL statement for it, but I cannot even phrase it properly.With these sort of technologies, with the vector embeddings, with deep learning and so on, you can actually mathematically phrase it, right? The mathematical phrasing is very simple once you have the right representation that understands your image as a vector. Two images are nearly duplicate if they are close enough in the space of vectors. Suddenly you've taken a problem that was even hard to express, let alone compute, made it precise to express, precise to compute. This is going to happen not just for images, not just for semantic search, it's going to happen for all sorts of unstructured data, whether it's time series, where it's anomaly detection, whether it's security analytics, and so on.I actually think that fundamentally, a lot of fields are going to get disrupted by this sort of way of thinking about things. We are just scratching the surface here with semantic search, in my opinion.Corey: What is I guess your barometer for success? I mean, if I could take a very cynical point of view on this, it's, “Oh, well, whenever there's a managed vector database offering from AWS.” They'll probably call it Amazon Basics Vector or something like that. Well, that is a—it used to be a snarky observation that, “Oh, we're not competing, we're just validating their market.” Lately, with some of their competitive database offerings, there's a lot more truth to that than I suspect AWS would like.Their offerings are nowhere near as robust as what they pretend to be competing against. How far away do you think we are from the larger cloud providers starting to say, “Ah, we got the sense there was money in here, so we're launching an entire service around this?”Ram: Yeah. I mean, this is a—first of all, this is a great question. There's always something that's constantly, things that any innovator or disrupter has to be thinking about, especially these days. I would say that having a multi-year head, start in the use cases, in thinking about how this system should even look, what sort of use cases should it [unintelligible 00:19:34], what the operating points for the [unintelligible 00:19:37] database even look like, and how to build something that's cloud-native and scalable, is very hard to replicate. Meaning if you look at what we have already done and kind of tried to base the architecture of that, you're probably already a couple of years behind us in terms of just where we are at, right, not just in the architecture, but also in the use cases in where this is evolving forward.That said, I think it is, for all of these companies—and I would put—for example, Snowflake is a great example of this, which is Snowflake needn't have existed if Redshift had done a phenomenal job of being cloud-native, right, and kind of done that before Snowflake did it. In hindsight, it seems like it's obvious, but when Snowflake did this, it wasn't obvious that that's where everything was headed. And Snowflake built something that's very technologically innovative, in a sense that it's even now hard to replicate. Plus, it takes a long time to replicate something like that. I think that's where we are at.If Pinecone does its job really well and if we simply execute efficiently, it's very hard to replicate that. So, I'm not super worried about cloud providers, to be honest, in this space, I'm more worried about our execution.Corey: If it helps anything, I'm not very deep into your specific area of the world, obviously, but I am optimistic when I hear people say things like that. Whenever I find folks who are relatively early along in their technological journey being very concerned about oh, the large cloud provider is going to come crashing in, it feels on some level like their perspective is that they have one weird trick, and they were able to crack that, but they have no defensive mode because once someone else figures out the trick, well, okay, now we're done. The idea of sustained and lasting innovation in a space, I think, is the more defensible position to take, with the counterargument, of course, that that's a lot harder to find.Ram: Absolutely. And I think for technologies like this, that's the only solution, which is, if you really want to avoid being disrupted by cloud providers, I think that's the way to go.Corey: I want to talk a little bit about your own background. Before you wound up as the VP of R&D over at Pinecone, you were in a bunch of similar… I guess, similar styled roles—if we'll call it that—at Yahoo, Databricks, and Splunk. I'm curious as to what your experience in those companies wound up impressing on you that made you say, “Ah, that's great and all, but you know what's next? That's right, vector databases.” And off, you went to Pinecone. What did you see?Ram: So, first of all, in was some way or the other, I have been involved in machine learning and systems and the intersection of these two for maybe the last decade-and-a-half. So, it's always been something, like, in the in between the two and that's been personally exciting to me. So, I'm kind of very excited by trying to think about new type of databases, new type of data platforms that really leverages machine learning and data. This has been personally exciting to me. I obviously learned very different things from different companies.I would say that Yahoo was just the learning in cloud to begin with because prior to joining Yahoo, I wasn't familiar with Silicon Valley cloud companies at that scale and Yahoo is a big company and there's a lot to learn from there. It was also my first introduction to Hadoop, Spark, and even machine learning where I really got into machine learning at scale, in online advertising and areas like that, which was a massive scale. And I got into that in Yahoo, and it was personally exciting to me because there's very few opportunities where you can work on machine learning at that scale, right?Databricks was very exciting to me because it was an earlier-stage company than I had been at before. Extremely well run and I learned a lot from Databricks, just the team, the culture, the focus on innovation, and the focus on product thinking. I joined Databricks as a product manager. I hadn't played the product manager hat before that, so it was very much a learning experience for me and I think I learned from some of the best in that area. And even at Pinecone, I carry that forward, which is think about how my learnings at Databricks informs how we should be thinking about products at Pinecone, and so on. So, I think I learned—if I had to pick one company I learned a lot from, I would say, it's Databricks. The most [unintelligible 00:23:50].Corey: I would also like to point out, normally when people say, “Oh, the one company I've learned the most from,” and they pick one of them out of their history, it's invariably the most recent one, but you left there in 2018—Ram: Yeah.Corey: —then went to go spend the next three years over at Splunk, where you were a Senior Principal, Scientist, a Senior Director and Head of Machine-Learning, and then you decided, okay, that's enough hard work. You're going to do something easier and be the VP of Engineering, which is just wild at a company of that scale.Ram: Yeah. At Splunk, I learned a lot about management. I think managing large teams, managing multiple different teams, while working on very different areas is something I learned at Splunk. You know, I was at this point in my career when I was right around trying to start my own company. Basically, I was at a point where I'd taken enough learnings and I really wanted to do something myself.That's when Edo and I—you know, the CEO of Pinecone—and I started talking. And we had worked together for many years, and we started working together at Yahoo. We kept in touch with each other. And we started talking about the sort of problems that I was excited about working on and then I came to realize what he was working on and what Pinecone was doing. And we thought it was a very good fit for the two of us to work together.So, that is kind of how it happened. It sort of happened by chance, as many things do in Silicon Valley, where a lot of things just happen by network and chance. That's what happened in my case. I was just thinking of starting my own company at the time when just a chance encounter with Edo led me to Pinecone.Corey: It feels from my admittedly uninformed perspective, that a lot of what you're doing right now in the vector database area, it feels on some level, like it follows the trajectory of machine learning, in that for a long time, the only people really excited about it were either sci-fi authors or folks who had trouble explaining it to someone without a degree in higher math. And then it turned into—a couple of big stories from the mid-2010s stick out at me when we've been people were trying to sell this to me in a variety of different ways. One of them was, “Oh, yeah, if you're a giant credit card processing company and trying to detect fraud with this kind of transaction volume—” it's, yeah, there are maybe three companies in the world that fall into that exact category. The other was WeWork where they did a lot of computer vision work. And they used this to determine that at certain times of day there was congestion in certain parts of the buildings and that this was best addressed by hiring a second barista. Which distilled down to, “Wait a minute, you're telling me that you spent how much money on machine-learning and advanced analyses and data scientists and the rest have figured out that people like to drink coffee in the morning?” Like, that is a little on the ridiculous side.Now, I think that it is past the time for skepticism around machine learning when you can go to a website and type in a description of something and it paints a picture of the thing you just described. Or you can show it a picture and it describes what is in that picture fairly accurately. At this point, the only people who are skeptics, from my position on this, seem to be holding out for some sort of either next-generation miracle or are just being bloody-minded. Do you think that there's a tipping point for vector search where it's going to become blindingly obvious to, if not the mass market, at least more run-of-the-mill, more prosaic level of engineer that haven't specialized in this?Ram: Yeah. It's already, frankly, started happening. So, two years back, I wouldn't have suspected this fast of an adoption for this new of technology from this varied number of use cases. I just wouldn't have suspected it because I, you know, I still thought, it's going to take some time for this field to mature and, kind of, everybody to really start taking advantage of this. This has happened much faster than even I assumed.So, to some extent, it's already happening. A lot of it is because the barrier to entry is quite low right now, right? So, it's very easy and cost-effective for people to create these embeddings. There is a lot of documentation out there, things are getting easier and easier, day by day. Some of it is by Pinecone itself, by a lot of work we do. Some of it is by, like, companies that I mentioned before who are building better and better models, making it easier and easier for people to take these machine-learning models and use them without having to even fine-tune anything.And as technologies like Pinecone really mature and dramatically become cost-effective, the barrier to entry is very low. So, what we tend to see people do, it's not so much about confidence in this new technology; it is connecting something simple that I need this sort of value out of, and find the least critical path or the simplest way to get going on this sort of technology. And as long as it can make that barrier to entry very small and make this cost-effective and easy for people to explore, this is going to start exploding. And that's what we are seeing. And a lot of Pinecone's focus has been on ease-of-use, in simplicity in connecting the zero-to-one journey for precisely this reason. Because not only do we strongly believe in the value of this technology, it's becoming more and more obvious to the broader community as well. The remaining work to be done is just the ease of use and making things cost-effective. And cost-effectiveness is also what the focus on a lot. Like, this technology can be even more cost-effective than it is today.Corey: I think that it is one of those never-mistaken ideas to wind up making something more accessible to folks than keeping it in a relatively rarefied environment. We take a look throughout the history of computing in general and cloud in particular, were formerly very hard things have largely been reduced down to click the button. Yes, yes, and then get yelled at because you haven't done infrastructure-as-code, but click the button is still possible. I feel like this is on that trendline based upon what you're saying.Ram: Absolutely. And the more we can do here, both Pinecone and the broader community, I think the better, the faster the adoption of this sort of technology is going to be.Corey: I really want to thank you for spending so much time talking me through what it is you folks are working on. If people want to learn more, where's the best place for them to go to find you?Ram: Pinecone.io. Our website has a ton of information about Pinecone, as well as a lot of standard documentation. We have a free tier as well where you can play around with small data sets, really get a feel for vector search. It's completely free. And you can reach me at Ram at Pinecone. I'm always happy to answer any questions. Once again, thanks so much for having me.Corey: Of course. I will put links to all of that in the show notes. This promoted guest episode is brought to us by our friends at Pinecone. Ram Sriharsha is their VP of Engineering and R&D. And I'm Cloud Economist Corey Quinn. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment that I will never read because the search on your podcast platform is broken because it's not using a vector database.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About RichardRichard "RichiH" Hartmann is the Director of Community at Grafana Labs, Prometheus team member, OpenMetrics founder, OpenTelemetry member, CNCF Technical Advisory Group Observability chair, CNCF Technical Oversight Committee member, CNCF Governing Board member, and more. He also leads, organizes, or helps run various conferences from hundreds to 18,000 attendess, including KubeCon, PromCon, FOSDEM, DENOG, DebConf, and Chaos Communication Congress. In the past, he made mainframe databases work, ISP backbones run, kept the largest IRC network on Earth running, and designed and built a datacenter from scratch. Go through his talks, podcasts, interviews, and articles at https://github.com/RichiH/talks or follow him on Twitter at https://twitter.com/TwitchiH for musings on the intersection of technology and society.Links Referenced: Grafana Labs: https://grafana.com/ Twitter: https://twitter.com/TwitchiH Richard Hartmann list of talks: https://github.com/richih/talks TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at AWS AppConfig. Engineers love to solve, and occasionally create, problems. But not when it's an on-call fire-drill at 4 in the morning. Software problems should drive innovation and collaboration, NOT stress, and sleeplessness, and threats of violence. That's why so many developers are realizing the value of AWS AppConfig Feature Flags. Feature Flags let developers push code to production, but hide that that feature from customers so that the developers can release their feature when it's ready. This practice allows for safe, fast, and convenient software development. You can seamlessly incorporate AppConfig Feature Flags into your AWS or cloud environment and ship your Features with excitement, not trepidation and fear. To get started, go to snark.cloud/appconfig. That's snark.cloud/appconfig.Corey: This episode is brought to us in part by our friends at Datadog. Datadog's SaaS monitoring and security platform that enables full stack observability for developers, IT operations, security, and business teams in the cloud age. Datadog's platform, along with 500 plus vendor integrations, allows you to correlate metrics, traces, logs, and security signals across your applications, infrastructure, and third party services in a single pane of glass.Combine these with drag and drop dashboards and machine learning based alerts to help teams troubleshoot and collaborate more effectively, prevent downtime, and enhance performance and reliability. Try Datadog in your environment today with a free 14 day trial and get a complimentary T-shirt when you install the agent.To learn more, visit datadoghq/screaminginthecloud to get. That's www.datadoghq/screaminginthecloudCorey: Welcome to Screaming in the Cloud, I'm Corey Quinn. There are an awful lot of people who are incredibly good at understanding the ins and outs and the intricacies of the observability world. But they didn't have time to come on the show today. Instead, I am talking to my dear friend of two decades now, Richard Hartmann, better known on the internet as RichiH, who is the Director of Community at Grafana Labs, here to suffer—in a somewhat atypical departure for the theme of this show—personal attacks for once. Richie, thank you for joining me.Richard: And thank you for agreeing on personal attacks.Corey: Exactly. It was one of your riders. Like, there have to be the personal attacks back and forth or you refuse to appear on the show. You've been on before. In fact, the last time we did a recording, I believe you were here in person, which was a long time ago. What have you been up to?You're still at Grafana Labs. And in many cases, I would point out that, wow, you've been there for many years; that seems to be an atypical thing, which is an American tech industry perspective because every time you and I talk about this, you look at folks who—wow, you were only at that company for five years. What's wrong with you—you tend to take the longer view and I tend to have the fast twitch, time to go ahead and leave jobs because it's been more than 20 minutes approach. I see that you're continuing to live what you preach, though. How's it been?Richard: Yeah, so there's a little bit of Covid brains, I think. When we talked in 2018, I was still working at SpaceNet, building a data center. But the last two-and-a-half years didn't really happen for many people, myself included. So, I guess [laugh] that includes you.Corey: No, no you're right. You've only been at Grafana Labs a couple of years. One would think I would check the notes for shooting my mouth off. But then, one wouldn't know me.Richard: What notes? Anyway, I've been around Prometheus and Grafana Since 2015. But it's like, real, full-time everything is 2020. There was something in between. Since 2018, I contracted to do vulnerability handling and everything for Grafana Labs because they had something and they didn't know how to deal with it.But no, full time is 2020. But as to the space in the [unintelligible 00:02:45] of itself, it's maybe a little bit German of me, but trying to understand the real world and trying to get an overview of systems and how they actually work, and if they are working correctly and as intended, and if not, how they're not working as intended, and how to fix this is something which has always been super important to me, in part because I just want to understand the world. And this is a really, really good way to automate understanding of the world. So, it's basically a work-saving mechanism. And that's why I've been sticking to it for so long, I guess.Corey: Back in the early days of monitoring systems—so we called it monitoring back then because, you know, are using simple words that lack nuance was sort of de rigueur back then—we wound up effectively having tools. Nagios is the one that springs to mind, and it was terrible in all the ways you would expect a tool written in janky Perl in the early-2000s to be. But it told you what was going on. It tried to do a thing, generally reach a server or query it about things, and when things fell out of certain specs, it screamed its head off, which meant that when you had things like the core switch melting down—thinking of one very particular incident—you didn't get a Nagios alert; you got 4000 Nagios alerts. But start to finish, you could wrap your head rather fully around what Nagios did and why it did the sometimes strange things that it did.These days, when you take a look at Prometheus, which we hear a lot about, particularly in the Kubernetes space and Grafana, which is often mentioned in the same breath, it's never been quite clear to me exactly where those start and stop. It always feels like it's a component in a larger system to tell you what's going on rather than a one-stop shop that's going to, you know, shriek its head off when something breaks in the middle of the night. Is that the right way to think about it? The wrong way to think about it?Richard: It's a way to think about it. So personally, I use the terms monitoring and observability pretty much interchangeably. Observability is a relatively well-defined term, even though most people won't agree. But if you look back into the '70s into control theory where the term is coming from, it is the measure of how much you're able to determine the internal state of a system by looking at its inputs and its outputs. Depending on the definition, some people don't include the inputs, but that is the OG definition as far as I'm aware.And from this, there flow a lot of things. This question of—or this interpretation of the difference between telling that, yes, something's broken versus why something's broken. Or if you can't ask new questions on the fly, it's not observability. Like all of those things are fundamentally mapped to this definition of, I need enough data to determine the internal state of whatever system I have just by looking at what is coming in, what is going out. And that is at the core the thing. Now, obviously, it's become a buzzword, which is oftentimes the fate of successful things. So, it's become a buzzword, and you end up with cargo culting.Corey: I would argue periodically, that observability is hipster monitoring. If you call it monitoring, you get yelled at by Charity Majors. Which is tongue and cheek, but she has opinions, made, nonetheless shall I say, frustrating by the fact that she is invariably correct in those opinions, which just somehow makes it so much worse. It would be easy to dismiss things she says if she weren't always right. And the world is changing, especially as we get into the world of distributed systems.Is the server that runs the app working or not working loses meaning when we're talking about distributed systems, when we're talking about containers running on top of Kubernetes, which turns every outage into a murder mystery. We start having distributed applications composed of microservices, so you have no idea necessarily where an issue is. Okay, is this one microservice having an issue related to the request coming into a completely separate microservice? And it seems that for those types of applications, the answer has been tracing for a long time now, where originally that was something that felt like it was sprung, fully-formed from the forehead of some God known as one of the hyperscalers, but now is available to basically everyone, in theory.In practice, it seems that instrumenting applications still one of the hardest parts of all of this. I tried hooking up one of my own applications to be observed via OTEL, the open telemetry project, and it turns out that right now, OTEL and AWS Lambda have an intersection point that makes everything extremely difficult to work with. It's not there yet; it's not baked yet. And someday, I hope that changes because I would love to interchangeably just throw metrics and traces and logs to all the different observability tools and see which ones work, which ones don't, but that still feels very far away from current state of the art.Richard: Before we go there, maybe one thing which I don't fully agree with. You said that previously, you were told if a service up or down, that's the thing which you cared about, and I don't think that's what people actually cared about. At that time, also, what they fundamentally cared about: is the user-facing service up, or down, or impacted? Is it slow? Does it return errors every X percent for requests, something like this?Corey: Is the site up? And—you're right, I was hand-waving over a whole bunch of things. It was, “Okay. First, the web server is returning a page, yes or no? Great. Can I ping the server?” Okay, well, there are ways of server can crash and still leave enough of the TCP/IP stack up or it can respond to pings and do little else.And then you start adding things to it. But the Nagios thing that I always wanted to add—and had to—was, is the disk full? And that was annoying. And, on some level, like, why should I care in the modern era how much stuff is on the disk because storage is cheap and free and plentiful? The problem is, after the third outage in a month because the disk filled up, you start to not have a good answer for well, why aren't you monitoring whether the disk is full?And that was the contributors to taking down the server. When the website broke, there were what felt like a relatively small number of reasonably well-understood contributors to that at small to midsize applications, which is what I'm talking about, the only things that people would let me touch. I wasn't running hyperscale stuff where you have a fleet of 10,000 web servers and, “Is the server up?” Yeah, in that scenario, no one cares. But when we're talking about the database server and the two application servers and the four web servers talking to them, you think about it more in terms of pets than you do cattle.Richard: Yes, absolutely. Yet, I think that was a mistake back then, and I tried to do it differently, as a specific example with the disk. And I'm absolutely agreeing that previous generation tools limit you in how you can actually work with your data. In particular, once you're with metrics where you can do actual math on the data, it doesn't matter if the disk is almost full. It matters if that disk is going to be full within X amount of time.If that disk is 98% full and it sits there at 98% for ten years and provides the service, no one cares. The thing is, will it actually run out in the next two hours, in the next five hours, what have you. Depending on this, is this currently or imminently a customer-impacting or user-impacting then yes, alert on it, raise hell, wake people, make them fix it, as opposed to this thing can be dealt with during business hours on the next workday. And you don't have to wake anyone up.Corey: Yeah. The big filer with massive amounts of storage has crossed the 70% line. Okay, now it's time to start thinking about that, what do you want to do? Maybe it's time to order another shelf of discs for it, which is going to take some time. That's a radically different scenario than the 20 gigabyte root volume on your server just started filling up dramatically; the rate of change is such that'll be full in 20 minutes.Yeah, one of those is something you want to wake people up for. Generally speaking, you don't want to wake people up for what is fundamentally a longer-term strategic business problem. That can be sorted out in the light of day versus, “[laugh] we're not going to be making money in two hours, so if I don't wake up and fix this now.” That's the kind of thing you generally want to be woken up for. Well, let's be honest, you don't want that to happen at all, but if it does happen, you kind of want to know in advance rather than after the fact.Richard: You're literally describing linear predict from Prometheus, which is precisely for this, where I can look back over X amount of time and make a linear prediction because everything else breaks down at scale, blah, blah, blah, to detail. But the thing is, I can draw a line with my pencil by hand on my data and I can predict when is this thing going to it. Which is obviously precisely correct if I have a TLS certificate. It's a little bit more hand-wavy when it's a disk. But still, you can look into the future and you say, “What will be happening if current trends for the last X amount of time continue in Y amount of time.” And that's precisely a thing where you get this more powerful ability of doing math with your data.Corey: See, when you say it like that, it sounds like it actually is a whole term of art, where you're focusing on an in-depth field, where salaries are astronomical. Whereas the tools that I had to talk about this stuff back in the day made me sound like, effectively, the sysadmin that I was grunting and pointing: “This is gonna fill up.” And that is how I thought about it. And this is the challenge where it's easy to think about these things in narrow, defined contexts like that, but at scale, things break.Like the idea of anomaly detection. Well, okay, great if normally, the CPU and these things are super bored and suddenly it gets really busy, that's atypical. Maybe we should look into it, assuming that it has a challenge. The problem is, that is a lot harder than it sounds because there are so many factors that factor into it. And as soon as you have something, quote-unquote, “Intelligent,” making decisions on this, it doesn't take too many false positives before you start ignoring everything it has to say, and missing legitimate things. It's this weird and obnoxious conflation of both hard technical problems and human psychology.Richard: And the breaking up of old service boundaries. Of course, when you say microservices, and such, fundamentally, functionally a microservice or nanoservice, picoservice—but the pendulum is already swinging back to larger units of complexity—but it fundamentally does not make any difference if I have a monolith on some mainframe or if I have a bunch of microservices. Yes, I can scale differently, I can scale horizontally a lot more easily, vertically, it's a little bit harder, blah, blah, blah, but fundamentally, the logic and the complexity, which is being packaged is fundamentally the same. More users, everything, but it is fundamentally the same. What's happening again, and again, is I'm breaking up those old boundaries, which means the old tools which have assumptions built in about certain aspects of how I can actually get an overview of a system just start breaking down, when my complexity unit or my service or what have I, is usually congruent with a physical piece, of hardware or several services are congruent with that piece of hardware, it absolutely makes sense to think about things in terms of this one physical server. The fact that you have different considerations in cloud, and microservices, and blah, blah, blah, is not inherently that it is more complex.On the contrary, it is fundamentally the same thing. It scales with users' everything, but it is fundamentally the same thing, but I have different boundaries of where I put interfaces onto my complexity, which basically allow me to hide all of this complexity from the downstream users.Corey: That's part of the challenge that I think we're grappling with across this entire industry from start to finish. Where we originally looked at these things and could reason about it because it's the computer and I know how those things work. Well, kind of, but okay, sure. But then we start layering levels of complexity on top of layers of complexity on top of layers of complexity, and suddenly, when things stop working the way that we expect, it can be very challenging to unpack and understand why. One of the ways I got into this whole space was understanding, to some degree, of how system calls work, of how the kernel wound up interacting with userspace, about how Linux systems worked from start to finish. And these days, that isn't particularly necessary most of the time for the care and feeding of applications.The challenge is when things start breaking, suddenly having that in my back pocket to pull out could be extremely handy. But I don't think it's nearly as central as it once was and I don't know that I would necessarily advise someone new to this space to spend a few years as a systems person, digging into a lot of those aspects. And this is why you need to know what inodes are and how they work. Not really, not anymore. It's not front and center the way that it once was, in most environments, at least in the world that I live in. Agree? Disagree?Richard: Agreed. But it's very much unsurprising. You probably can't tell me how to precisely grow sugar cane or corn, you can't tell me how to refine the sugar out of it, but you can absolutely bake a cake. But you will not be able to tell me even a third of—and I'm—for the record, I'm also not able to tell you even a third about the supply chain which just goes from I have a field and some seeds and I need to have a package of refined sugar—you're absolutely enabled to do any of this. The thing is, you've been part of the previous generation of infrastructure where you know how this underlying infrastructure works, so you have more ability to reason about this, but it's not needed for cloud services nearly as much.You need different types of skill sets, but that doesn't mean the old skill set is completely useless, at least not as of right now. It's much more a case of you need fewer of those people and you need them in different places because those things have become infrastructure. Which is basically the cloud play, where a lot of this is just becoming infrastructure more and more.Corey: Oh, yeah. Back then I distinctly remember my elders looking down their noses at me because I didn't know assembly, and how could I possibly consider myself a competent systems admin if I didn't at least have a working knowledge of assembly? Or at least C, which I, over time, learned enough about to know that I didn't want to be a C programmer. And you're right, this is the value of cloud and going back to those days getting a web server up and running just to compile Apache's httpd took a week and an in-depth knowledge of GCC flags.And then in time, oh, great. We're going to have rpm or debs. Great, okay, then in time, you have apt, if you're in the dev land because I know you are a Debian developer, but over in Red Hat land, we had yum and other tools. And then in time, it became oh, we can just use something like Puppet or Chef to wind up ensuring that thing is installed. And then oh, just docker run. And now it's a checkbox in a web console for S3.These things get easier with time and step by step by step we're standing on the shoulders of giants. Even in the last ten years of my career, I used to have a great challenge question that I would interview people with of, “Do you know what TinyURL is? It takes a short URL and then expands it to a longer one. Great, on the whiteboard, tell me how you would implement that.” And you could go up one side and down the other, and then you could add constraints, multiple data centers, now one goes offline, how do you not lose data? Et cetera, et cetera.But these days, there are so many ways to do that using cloud services that it almost becomes trivial. It's okay, multiple data centers, API Gateway, a Lambda, and a global DynamoDB table. Now, what? “Well, now it gets slow. Why is it getting slow?”“Well, in that scenario, probably because of something underlying the cloud provider.” “And so now, you lose an entire AWS region. How do you handle that?” “Seems to me when that happens, the entire internet's kind of broken. Do people really need longer URLs?”And that is a valid answer, in many cases. The question doesn't really work without a whole bunch of additional constraints that make it sound fake. And that's not a weakness. That is the fact that computers and cloud services have never been as accessible as they are now. And that's a win for everyone.Richard: There's one aspect of accessibility which is actually decreasing—or two. A, you need to pay for them on an ongoing basis. And B, you need an internet connection which is suitably fast, low latency, what have you. And those are things which actually do make things harder for a variety of reasons. If I look at our back-end systems—as in Grafana—all of them have single binary modes where you literally compile everything into a single binary and you can run it on your laptop because if you're stuck on a plane, you can't do any work on it. That kind of is not the best of situations.And if you have a huge CI/CD pipeline, everything in this cloud and fine and dandy, but your internet breaks. Yeah, so I do agree that it is becoming generally more accessible. I disagree that it is becoming more accessible along all possible axes.Corey: I would agree. There is a silver lining to that as well, where yes, they are fraught and dangerous and I would preface this with a whole bunch of warnings, but from a cost perspective, all of the cloud providers do have a free tier offering where you can kick the tires on a lot of these things in return for no money. Surprisingly, the best one of those is Oracle Cloud where they have an unlimited free tier, use whatever you want in this subset of services, and you will never be charged a dime. As opposed to the AWS model of free tier where well, okay, it suddenly got very popular or you misconfigured something, and surprise, you now owe us enough money to buy Belize. That doesn't usually lead to a great customer experience.But you're right, you can't get away from needing an internet connection of at least some level of stability and throughput in order for a lot of these things to work. The stuff you would do locally on a Raspberry Pi, for example, if your budget constrained and want to get something out here, or your laptop. Great, that's not going to work in the same way as a full-on cloud service will.Richard: It's not free unless you have hard guarantees that you're not going to ever pay anything. It's fine to send warning, it's fine to switch the thing off, it's fine to have you hit random hard and soft quotas. It is not a free service if you can't guarantee that it is free.Corey: I agree with you. I think that there needs to be a free offering where, “Well, okay, you want us to suddenly stop serving traffic to the world?” “Yes. When the alternative is you have to start charging me through the nose, yes I want you to stop serving traffic.” That is definitionally what it says on the tin.And as an independent learner, that is what I want. Conversely, if I'm an enterprise, yeah, I don't care about money; we're running our Superbowl ad right now, so whatever you do, don't stop serving traffic. Charge us all the money. And there's been a lot of hand wringing about, well, how do we figure out which direction to go in? And it's, have you considered asking the customer?So, on a scale of one to bank, how serious is this account going to be [laugh]? Like, what are your big concerns: never charge me or never go down? Because we can build for either of those. Just let's make sure that all of those expectations are aligned. Because if you guess you're going to get it wrong and then no one's going to like you.Richard: I would argue this. All those services from all cloud providers actually build to address both of those. It's a deliberate choice not to offer certain aspects.Corey: Absolutely. When I talk to AWS, like, “Yeah, but there is an eventual consistency challenge in the billing system where it takes”—as anyone who's looked at the billing system can see—“Multiple days, sometimes for usage data to show up. So, how would we be able to stop things if the usage starts climbing?” To which my relatively direct responses, that sounds like a huge problem. I don't know how you'd fix that, but I do know that if suddenly you decide, as a matter of policy, to okay, if you're in the free tier, we will not charge you, or even we will not charge you more than $20 a month.So, you build yourself some headroom, great. And anything that people are able to spin up, well, you're just going to have to eat the cost as a provider. I somehow suspect that would get fixed super quickly if that were the constraint. The fact that it isn't is a conscious choice.Richard: Absolutely.Corey: And the reason I'm so passionate about this, about the free space, is not because I want to get a bunch of things for free. I assure you I do not. I mean, I spend my life fixing AWS bills and looking at AWS pricing, and my argument is very rarely, “It's too expensive.” It's that the billing dimension is hard to predict or doesn't align with a customer's experience or prices a service out of a bunch of use cases where it'll be great. But very rarely do I just sit here shaking my fist and saying, “It costs too much.”The problem is when you scare the living crap out of a student with a surprise bill that's more than their entire college tuition, even if you waive it a week or so later, do you think they're ever going to be as excited as they once were to go and use cloud services and build things for themselves and see what's possible? I mean, you and I met on IRC 20 years ago because back in those days, the failure mode and the risk financially was extremely low. It's yeah, the biggest concern that I had back then when I was doing some of my Linux experimentation is if I typed the wrong thing, I'm going to break my laptop. And yeah, that happened once or twice, and I've learned not to make those same kinds of mistakes, or put guardrails in so the blast radius was smaller, or use a remote system instead. Yeah, someone else's computer that I can destroy. Wonderful. But that was on we live and we learn as we were coming up. There was never an opportunity for us, to my understanding, to wind up accidentally running up an $8 million charge.Richard: Absolutely. And psychological safety is one of the most important things in what most people do. We are social animals. Without this psychological safety, you're not going to have long-term, self-sustaining groups. You will not make someone really excited about it. There's two basic ways to sell: trust or force. Those are the two ones. There's none else.Corey: Managing shards. Maintenance windows. Overprovisioning. ElastiCache bills. I know, I know. It's a spooky season and you're already shaking. It's time for caching to be simpler. Momento Serverless Cache lets you forget the backend to focus on good code and great user experiences. With true autoscaling and a pay-per-use pricing model, it makes caching easy. No matter your cloud provider, get going for free at gomemento.co/screaming That's GO M-O-M-E-N-T-O dot co slash screamingCorey: Yeah. And it also looks ridiculous. I was talking to someone somewhat recently who's used to spending four bucks a month on their AWS bill for some S3 stuff. Great. Good for them. That's awesome. Their credentials got compromised. Yes, that is on them to some extent. Okay, great.But now after six days, they were told that they owed $360,000 to AWS. And I don't know how, as a cloud company, you can sit there and ask a student to do that. That is not a realistic thing. They are what is known, in the United States at least, in the world of civil litigation as quote-unquote, “Judgment proof,” which means, great, you could wind up finding that someone owes you $20 billion. Most of the time, they don't have that, so you're not able to recoup it. Yeah, the judgment feels good, but you're never going to see it.That's the problem with something like that. It's yeah, I would declare bankruptcy long before, as a student, I wound up paying that kind of money. And I don't hear any stories about them releasing the collection agency hounds against people in that scenario. But I couldn't guarantee that. I would never urge someone to ignore that bill and see what happens.And it's such an off-putting thing that, from my perspective, is beneath of the company. And let's be clear, I see this behavior at times on Google Cloud, and I see it on Azure as well. This is not something that is unique to AWS, but they are the 800-pound gorilla in the space, and that's important. Or as I just to mention right now, like, as I—because I was about to give you crap for this, too, but if I go to grafana.com, it says, and I quote, “Play around with the Grafana Stack. Experience Grafana for yourself, no registration or installation needed.”Good. I was about to yell at you if it's, “Oh, just give us your credit card and go ahead and start spinning things up and we won't charge you. Honest.” Even your free account does not require a credit card; you're doing it right. That tells me that I'm not going to get a giant surprise bill.Richard: You have no idea how much thought and work went into our free offering. There was a lot of math involved.Corey: None of this is easy, I want to be very clear on that. Pricing is one of the hardest things to get right, especially in cloud. And it also, when you get it right, it doesn't look like it was that hard for you to do. But I fix [sigh] I people's AWS bills for a living and still, five or six years in, one of the hardest things I still wrestle with is pricing engagements. It's incredibly nuanced, incredibly challenging, and at least for services in the cloud space where you're doing usage-based billing, that becomes a problem.But glancing at your pricing page, you do hit the two things that are incredibly important to me. The first one is use something for free. As an added bonus, you can use it forever. And I can get started with it right now. Great, when I go and look at your pricing page or I want to use your product and it tells me to ‘click here to contact us.' That tells me it's an enterprise sales cycle, it's got to be really expensive, and I'm not solving my problem tonight.Whereas the other side of it, the enterprise offering needs to be ‘contact us' and you do that, that speaks to the enterprise procurement people who don't know how to sign a check that doesn't have to commas in it, and they want to have custom terms and all the rest, and they're prepared to pay for that. If you don't have that, you look to small-time. When it doesn't matter what price you put on it, you wind up offering your enterprise tier at some large number, it's yeah, for some companies, that's a small number. You don't necessarily want to back yourself in, depending upon what the specific needs are. You've gotten that right.Every common criticism that I have about pricing, you folks have gotten right. And I definitely can pick up on your fingerprints on a lot of this. Because it sounds like a weird thing to say of, “Well, he's the Director of Community, why would he weigh in on pricing?” It's, “I don't think you understand what community is when you ask that question.”Richard: Yes, I fully agree. It's super important to get pricing right, or to get many things right. And usually the things which just feel naturally correct are the ones which took the most effort and the most time and everything. And yes, at least from the—like, I was in those conversations or part of them, and the one thing which was always clear is when we say it's free, it must be free. When we say it is forever free, it must be forever free. No games, no lies, do what you say and say what you do. Basically.We have things where initially you get certain pro features and you can keep paying and you can keep using them, or after X amount of time they go away. Things like these are built in because that's what people want. They want to play around with the whole thing and see, hey, is this actually providing me value? Do I want to pay for this feature which is nice or this and that plugin or what have you? And yeah, you're also absolutely right that once you leave these constraints of basically self-serve cloud, you are talking about bespoke deals, but you're also talking about okay, let's sit down, let's actually understand what your business is: what are your business problems? What are you going to solve today? What are you trying to solve tomorrow?Let us find a way of actually supporting you and invest into a mutual partnership and not just grab the money and run. We have extremely low churn for, I would say, pretty good reasons. Because this thing about our users, our customers being successful, we do take it extremely seriously.Corey: It's one of those areas that I just can't shake the feeling is underappreciated industry-wide. And the reason I say that this is your fingerprints on it is because if this had been wrong, you have a lot of… we'll call them idiosyncrasies, where there are certain things you absolutely will not stand for, and misleading people and tricking them into paying money is high on that list. One of the reasons we're friends. So yeah, but I say I see your fingerprints on this, it's yeah, if this hadn't been worked out the way that it is, you would not still be there. One other thing that I wanted to call out about, well, I guess it's a confluence of pricing and logging in the rest, I look at your free tier, and it offers up to 50 gigabytes of ingest a month.And it's easy for me to sit here and compare that to other services, other tools, and other logging stories, and then I have to stop and think for a minute that yeah, discs have gotten way bigger, and internet connections have gotten way faster, and even the logs have gotten way wordier. I still am not sure that most people can really contextualize just how much logging fits into 50 gigs of data. Do you have any, I guess, ballpark examples of what that looks like? Because it's been long enough since I've been playing in these waters that I can't really contextualize it anymore.Richard: Lord of the Rings is roughly five megabytes. It's actually less. So, we're talking literally 10,000 Lord of the Rings, which you can just shove in us and we're just storing this for you. Which also tells you that you're not going to be reading any of this. Or some of it, yes, but not all of it. You need better tooling and you need proper tooling.And some of this is more modern. Some of this is where we actually pushed the state of the art. But I'm also biased. But I, for myself, do claim that we did push the state of the art here. But at the same time you come back to those absolute fundamentals of how humans deal with data.If you look back basically as far as we have writing—literally 6000 years ago, is the oldest writing—humans have always dealt with information with the state of the world in very specific ways. A, is it important enough to even write it down, to even persist it in whatever persistence mechanisms I have at my disposal? If yes, write a detailed account or record a detailed account of whatever the thing is. But it turns out, this is expensive and it's not what you need. So, over time, you optimize towards only taking down key events and only noting key events. Maybe with their interconnections, but fundamentally, the key events.As your data grows, as you have more stuff, as this still is important to your business and keeps being more important to—or doesn't even need to be a business; can be social, can be whatever—whatever thing it is, it becomes expensive, again, to retain all of those key events. So, you turn them into numbers and you can do actual math on them. And that's this path which you've seen again, and again, and again, and again, throughout humanity's history. Literally, as long as we have written records, this has played out again, and again, and again, and again, for every single field which humans actually cared about. At different times, like, power networks are way ahead of this, but fundamentally power networks work on metrics, but for transient load spike, and everything, they have logs built into their power measurement devices, but those are only far in between. Of course, the main thing is just metrics, time-series. And you see this again, and again.You also were sysadmin in internet-related all switches have been metrics-based or metrics-first for basically forever, for 20, 30 years. But that stands to reason. Of course the internet is running at by roughly 20 years scale-wise in front of the cloud because obviously you need the internet because as you wouldn't be having a cloud. So, all of those growing pains why metrics are all of a sudden the thing, “Or have been for a few years now,” is basically, of course, people who were writing software, providing their own software services, hit the scaling limitations which you hit for Internet service providers two decades, three decades ago. But fundamentally, you have this complete system. Basically profiles or distributed tracing depending on how you view distributed tracing.You can also argue that distributed tracing is key events which are linked to each other. Logs sit firmly in the key event thing and then you turn this into numbers and that is metrics. And that's basically it. You have extremes at the and where you can have valid, depending on your circumstances, engineering trade-offs of where you invest the most, but fundamentally, that is why those always appear again in humanity's dealing with data, and observability is no different.Corey: I take a look at last month's AWS bill. Mine is pretty well optimized. It's a bit over 500 bucks. And right around 150 of that is various forms of logging and detecting change in the environment. And on the one hand, I sit here, and I think, “Oh, I should optimize that,” because the value of those logs to me is zero.Except that whenever I have to go in and diagnose something or respond to an incident or have some forensic exploration, they then are worth an awful lot. And I am prepared to pay 150 bucks a month for that because the potential value of having that when the time comes is going to be extraordinarily useful. And it basically just feels like a tax on top of what it is that I'm doing. The same thing happens with application observability where, yeah, when you just want the big substantial stuff, yeah, until you're trying to diagnose something. But in some cases, yeah, okay, then crank up the verbosity and then look for it.But if you're trying to figure it out after an event that isn't likely or hopefully won't recur, you're going to wish that you spent a little bit more on collecting data out of it. You're always going to be wrong, you're always going to be unhappy, on some level.Richard: Ish. You could absolutely be optimizing this. I mean, for $500, it's probably not worth your time unless you take it as an exercise, but outside of due diligence where you need specific logs tied to—or specific events tied to specific times, I would argue that a lot of the problems with logs is just dealing with it wrong. You have this one extreme of full-text indexing everything, and you have this other extreme of a data lake—which is just a euphemism of never looking at the data again—to keep storage vendors happy. There is an in between.Again, I'm biased, but like for example, with Loki, you have those same label sets as you have on your metrics with Prometheus, and you have literally the same, which means you only index that part and you only extract on ingestion time. If you don't have structured logs yet, only put the metadata about whatever you care about extracted and put it into your label set and store this, and that's the only thing you index. But it goes further than just this. You can also turn those logs into metrics.And to me this is a path of optimization. Where previously I logged this and that error. Okay, fine, but it's just a log line telling me it's HTTP 500. No one cares that this is at this precise time. Log levels are also basically an anti-pattern because they're just trying to deal with the amount of data which I have, and try and get a handle on this on that level whereas it would be much easier if I just counted every time I have an HTTP 500, I just up my counter by one. And again, and again, and again.And all of a sudden, I have literally—and I did the math on this—over 99.8% of the data which I have to store just goes away. It's just magic the way—and we're only talking about the first time I'm hitting this logline. The second time I'm hitting this logline is functionally free if I turn this into metrics. It becomes cheap enough that one of the mantras which I have, if you need to onboard your developers on modern observability, blah, blah, blah, blah, blah, the whole bells and whistles, usually people have logs, like that's what they have, unless they were from ISPs or power companies, or so; there they usually start with metrics.But most users, which I see both with my Grafana and with my Prometheus [unintelligible 00:38:46] tend to start with logs. They have issues with those logs because they're basically unstructured and useless and you need to first make them useful to some extent. But then you can leverage on this and instead of having a debug statement, just put a counter. Every single time you think, “Hey, maybe I should put a debug statement,” just put a counter instead. In two months time, see if it was worth it or if you delete that line and just remove that counter.It's so much cheaper, you can just throw this on and just have it run for a week or a month or whatever timeframe and done. But it goes beyond this because all of a sudden, if I can turn my logs into metrics properly, I can start rewriting my alerts on those metrics. I can actually persist those metrics and can more aggressively throw my logs away. But also, I have this transition made a lot easier where I don't have this huge lift, where this day in three months is to be cut over and we're going to release the new version of this and that software and it's not going to have that, it's going to have 80% less logs and everything will be great and then you missed the first maintenance window or someone is ill or what have you, and then the next Big Friday is coming so you can't actually deploy there. I mean Black Friday. But we can also talk about deploying on Fridays.But the thing is, you have this huge thing, whereas if you have this as a continuous improvement process, I can just look at, this is the log which is coming out. I turn this into a number, I start emitting metrics directly, and I see that those numbers match. And so, I can just start—I build new stuff, I put it into a new data format, I actually emit the new data format directly from my code instrumentation, and only then do I start removing the instrumentation for the logs. And that allows me to, with full confidence, with psychological safety, just move a lot more quickly, deliver much more quickly, and also cut down on my costs more quickly because I'm just using more efficient data types.Corey: I really want to thank you for spending as much time as you have. If people want to learn more about how you view the world and figure out what other personal attacks they can throw your way, where's the best place for them to find you?Richard: Personal attacks, probably Twitter. It's, like, the go-to place for this kind of thing. For actually tracking, I stopped maintaining my own website. Maybe I'll do again, but if you go on github.com/ritchieh/talks, you'll find a reasonably up-to-date list of all the talks, interviews, presentations, panels, what have you, which I did over the last whatever amount of time. [laugh].Corey: And we will, of course, put links to that in the [show notes 00:41:23]. Thanks again for your time. It's always appreciated.Richard: And thank you.Corey: Richard Hartmann, Director of Community at Grafana Labs. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment. And then when someone else comes along with an insulting comment they want to add, we'll just increment the counter by one.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About MattMatt is a Sr. Architect in Belfast, an AWS DevTools Hero, Serverless Architect, Author and conference speaker. He is focused on creating the right environment for empowered teams to rapidly deliver business value in a well-architected, sustainable and serverless-first way.You can usually find him sharing reusable, well architected, serverless patterns over at cdkpatterns.com or behind the scenes bringing CDK Day to life.Links Referenced: Previous guest appearance: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/slinging-cdk-knowledge-with-matt-coulter/ The CDK Book: https://thecdkbook.com/ Twitter: https://twitter.com/NIDeveloper TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the best parts about, well I guess being me, is that I can hold opinions that are… well, I'm going to be polite and call them incendiary, and that's great because I usually like to back them in data. But what happens when things change? What happens when I learn new things?Well, do I hold on to that original opinion with two hands at a death grip or do I admit that I was wrong in my initial opinion about something? Let's find out. My guest today returns from earlier this year. Matt Coulter is a senior architect since he has been promoted at Liberty Mutual. Welcome back, and thanks for joining me.Matt: Yeah, thanks for inviting me back, especially to talk about this topic.Corey: Well, we spoke about it a fair bit at the beginning of the year. And if you're listening to this, and you haven't heard that show, it's not that necessary to go into; mostly it was me spouting uninformed opinions about the CDK—the Cloud Development Kit, for those who are unfamiliar—I think of it more or less as what if you could just structure your cloud resources using a programming language you claim to already know, but in practice, copy and paste from Stack Overflow like the rest of us? Matt, you probably have a better description of what the CDK is in practice.Matt: Yeah, so we like to say it's imperative code written in a declarative way, or declarative code written in an imperative way. Either way, it lets you write code that produces CloudFormation. So, it doesn't really matter what you write in your script; the point is, at the end of the day, you still have the CloudFormation template that comes out of it. So, the whole piece of it is that it's a developer experience, developer speed play, that if you're from a background that you're more used to writing a programming language than a YAML, you might actually enjoy using the CDK over writing straight CloudFormation or SAM.Corey: When I first kicked the tires on the CDK, my first initial obstacle—which I've struggled with in this industry for a bit—is that I'm just good enough of a programmer to get myself in trouble. Whenever I wind up having a problem that StackOverflow doesn't immediately shine a light on, my default solution is to resort to my weapon of choice, which is brute force. That sometimes works out, sometimes doesn't. And as I went through the CDK, a couple of times in service to a project that I'll explain shortly, I made a bunch of missteps with it. The first and most obvious one is that AWS claims publicly that it has support in a bunch of languages: .NET, Python, there's obviously TypeScript, there's Go support for it—I believe that went generally available—and I'm sure I'm missing one or two, I think? Aren't I?Matt: Yeah, it's: TypeScript, JavaScript, Python Java.Net, and Go. I think those are the currently supported languages.Corey: Java. That's the one that I keep forgetting. It's the block printing to the script that is basically Java cursive. The problem I run into, and this is true of most things in my experience, when a company says that we have deployed an SDK for all of the following languages, there is very clearly a first-class citizen language and then the rest that more or less drift along behind with varying degrees of fidelity. In my experience, when I tried it for the first time in Python, it was not a great experience for me.When I learned just enough JavaScript, and by extension TypeScript, to be dangerous, it worked a lot better. Or at least I could blame all the problems I ran into on my complete novice status when it comes to JavaScript and TypeScript at the time. Is that directionally aligned with what you've experienced, given that you work in a large company that uses this, and presumably, once you have more than, I don't know, two developers, you start to take on aspects of a polyglot shop no matter where you are, on some level?Matt: Yeah. So personally, I jump between Java, Python, and TypeScript whenever I'm writing projects. So, when it comes to the CDK, you'd assume I'd be using all three. I typically stick to TypeScript and that's just because personally, I've had the best experience using it. For anybody who doesn't know the way CDK works for all the languages, it's not that they have written a custom, like, SDK for each of these languages; it's a case of it uses a Node process underneath them and the language actually interacts with—it's like the compiled JavaScript version is basically what they all interact with.So, it means there are some limitations on what you can do in that language. I can't remember the full list, but it just means that it is native in all those languages, but there are certain features that you might be like, “Ah,” whereas, in TypeScript, you can just use all of TypeScript. And my first inclination was actually, I was using the Python one and I was having issues with some compiler errors and things that are just caused by that process. And it's something that talking in the cdk.dev Slack community—there is actually a very active—Corey: Which is wonderful, I will point out.Matt: [laugh]. Thank you. There is actually, like, an awesome Python community in there, but if you ask them, they would all ask for improvements to the language. So, personally if someone's new, I always recommend they start with TypeScript and then branch out as they learn the CDK so they can understand is this a me problem, or is this a problem caused by the implementation?Corey: From my perspective, I didn't do anything approaching that level of deep dive. I took a shortcut that I find has served me reasonably well in the course of my career, when I'm trying to do something in Python, and you pull up a tutorial—which I'm a big fan of reading experience reports, and blog posts, and here's how to get started—and they all have the same problem, which is step one, “Run npm install.” And that's “Hmm, you know, I don't recall that being a standard part of the Python tooling.” It's clearly designed and interpreted and contextualized through a lens of JavaScript. Let's remove that translation layer, let's remove any weird issues I'm going to have in that transpilation process, and just talk in the language it written in. Will this solve my problems? Oh, absolutely not, but it will remove a subset of them that I am certain to go blundering into like a small lost child trying to cross an eight-lane freeway.Matt: Yeah. I've heard a lot of people say the same thing. Because the CDK CLI is a Node process, you need it no matter what language you use. So, if they were distributing some kind of universal binary that just integrated with the languages, it would definitely solve a lot of people's issues with trying to combine languages at deploy time.Corey: One of the challenges that I've had as I go through the process of iterating on the project—but I guess I should probably describe it for those who have not been following along with my misadventures; I write blog posts about it from time to time because I need a toy problem to kick around sometimes because my consulting work is all advisory and I don't want to be a talking head-I have a Twitter client called lasttweetinaws.com. It's free; go and use it. It does all kinds of interesting things for authoring Twitter threads.And I wanted to deploy that to a bunch of different AWS regions, as it turns out, 20 or so at the moment. And that led to a lot of interesting projects and having to learn how to think about these things differently because no one sensible deploys an application simultaneously to what amounts to every AWS region, without canary testing, and having a phased rollout in the rest. But I'm reckless, and honestly, as said earlier, a bad programmer. So, that works out. And trying to find ways to make this all work and fit together led iteratively towards me discovering that the CDK was really kind of awesome for a lot of this.That said, there were definitely some fairly gnarly things I learned as I went through it, due in no small part to help I received from generous randos in the cdk.dev Slack team. And it's gotten to a point where it's working, and as an added bonus, I even mostly understand what he's doing, which is just kind of wild to me.Matt: It's one of those interesting things where because it's a programming language, you can use it out of the box the way it's designed to be used where you can just write your simple logic which generates your CloudFormation, or you can do whatever crazy logic you want to do on top of that to make your app work the way you want it to work. And providing you're not in a company like Liberty, where I'm going to do a code review, if no one's stopping you, you can do your crazy experiments. And if you understand that, it's good. But I do think something like the multi-region deploy, I mean, with CDK, if you'd have a construct, it takes in a variable that you can just say what the region is, so you can actually just write a for loop and pass it in, which does make things a lot easier than, I don't know, try to do it with a YAML, which you can pass in parameters, but you're going to get a lot more complicated a lot quicker.Corey: The approach that I took philosophically was I wrote everything in a region-agnostic way. And it would be instantiated and be told what region to run it in as an environment variable that CDK deploy was called. And then I just deploy 20 simultaneous stacks through GitHub Actions, which invoke custom runners that runs inside of a Lambda function. And that's just a relatively basic YAML file, thanks to the magic of GitHub Actions matrix jobs. So, it fires off 20 simultaneous processes and on every commit to the main branch, and then after about two-and-a-half minutes, it has been deployed globally everywhere and I get notified on anything that fails, which is always fun and exciting to learn those things.That has been, overall, just a really useful experiment and an experience because you're right, you could theoretically run this as a single CDK deploy and then wind up having an iterate through a list of regions. The challenge I have there is that unless I start getting into really convoluted asynchronous concurrency stuff, it feels like it'll just take forever. At two-and-a-half minutes a region times 20 regions, that's the better part of an hour on every deploy and no one's got that kind of patience. So, I wound up just parallelizing it a bit further up the stack. That said, I bet they are relatively straightforward ways, given the async is a big part of JavaScript, to do this simultaneously.Matt: One of the pieces of feedback I've seen about CDK is if you have multiple stacks in the same project, it'll deploy them one at a time. And that's just because it tries to understand the dependencies between the stacks and then it works out which one should go first. But a lot of people have said, “Well, I don't want that. If I have 20 stacks, I want all 20 to go at once the way you're saying.” And I have seen that people have been writing plugins to enable concurrent deploys with CDK out of the box. So, it may be something that it's not an out-of-the-box feature, but it might be something that you can pull in a community plug-in to actually make work.Corey: Most of my problems with it at this point are really problems with CloudFormation. CloudFormation does not support well, if at all, secure string parameters from the AWS Systems Manager parameter store, which is my default go-to for secret storage, and Secrets Manager is supported, but that also cost 40 cents a month per secret. And not for nothing, I don't really want to have all five secrets deployed to Secrets Manager in every region this thing is in. I don't really want to pay $20 a month for this basically free application, just to hold some secrets. So, I wound up talking to some folks in the Slack channel and what we came up with was, I have a centralized S3 bucket that has a JSON object that lives in there.It's only accessible from the deployment role, and it grabs that at deploy time and stuffs it into environment variables when it pushes these things out. That's the only stateful part of all of this. And it felt like that is, on some level, a pattern that a lot of people would benefit from if it had better native support. But the counterargument that if you're only deploying to one or two regions, then Secrets Manager is the right answer for a lot of this and it's not that big of a deal.Matt: Yeah. And it's another one of those things, if you're deploying in Liberty, we'll say, “Well, your secret is unencrypted at runtime, so you probably need a KMS key involved in that,” which as you know, the costs of KMS, it depends on if it's a personal solution or if it's something for, like, a Fortune 100 company. And if it's personal solution, I mean, what you're saying sounds great that it's IAM restricted in S3, and then that way only at deploy time can be read; it actually could be a custom construct that someone can build and publish out there to the construct library—or the construct hub, I should say.Corey: To be clear, the reason I'm okay with this, from a security perspective is one, this is in a dedicated AWS account. This is the only thing that lives in that account. And two, the only API credentials we're talking about are the application-specific credentials for this Twitter client when it winds up talking to the Twitter API. Basically, if you get access to these and are able to steal them and deploy somewhere else, you get no access to customer data, you get—or user data because this is not charge for anything—you get no access to things that have been sent out; all you get to do is submit tweets to Twitter and it'll have the string ‘Last Tweet in AWS' as your client, rather than whatever normal client you would use. It's not exactly what we'd call a high-value target because all the sensitive to a user data lives in local storage in their browser. It is fully stateless.Matt: Yeah, so this is what I mean. Like, it's the difference in what you're using your app for. Perfect case of, you can just go into the Twitter app and just withdraw those credentials and do it again if something happens, whereas as I say, if you're building it for Liberty, that it will not pass a lot of our Well-Architected reviews, just for that reason.Corey: If I were going to go and deploy this at a more, I guess, locked down environment, I would be tempted to find alternative approaches such as having it stored encrypted at rest via KMS in S3 is one option. So, is having global DynamoDB tables that wind up grabbing those things, even grabbing it at runtime if necessary. There are ways to make that credential more secure at rest. It's just, I look at this from a real-world perspective of what is the actual attack surface on this, and I have a really hard time just identifying anything that is going to be meaningful with regard to an exploit. If you're listening to this and have a lot of thoughts on that matter, please reach out I'm willing to learn and change my opinion on things.Matt: One thing I will say about the Dynamo approach you mentioned, I'm not sure everybody knows this, but inside the same Dynamo table, you can scope down a row. You can be, like, “This row and this field in this row can only be accessed from this one Lambda function.” So, there's a lot of really awesome security features inside DynamoDB that I don't think most people take advantage of, but they open up a lot of options for simplicity.Corey: Is that tied to the very recent announcement about Lambda getting SourceArn as a condition key? In other words, you can say, “This specific Lambda function,” as opposed to, “A Lambda in this account?” Like that was a relatively recent Advent that I haven't fully explored the nuances of.Matt: Yeah, like, that has opened a lot of doors. I mean, the Dynamo being able to be locked out in your row has been around for a while, but the new Lambda from SourceArn is awesome because, yeah, as you say, you can literally say this thing, as opposed to, you have to start going into tags, or you have to start going into something else to find it.Corey: So, I want to talk about something you just alluded to, which is the Well-Architected Framework. And initially, when it launched, it was a whole framework, and AWS made a lot of noise about it on keynote stages, as they are want to do. And then later, they created a quote-unquote, “Well-Architected Tool,” which let's be very direct, it's the checkbox survey form, at least the last time I looked at it. And they now have the six pillars of the Well-Architected Framework where they talk about things like security, cost, sustainability is the new pillar, I don't know, absorbency, or whatever the remainders are. I can't think of them off the top of my head. How does that map to your experience with the CDK?Matt: Yeah, so out of the box, the CDK from day one was designed to have sensible defaults. And that's why a lot of the things you deploy have opinions. I talked to a couple of the Heroes and they were like, “I wish it had less opinions.” But that's why whenever you deploy something, it's got a bunch of configuration already in there. For me, in the CDK, whenever I use constructs, or stacks, or deploying anything in the CDK, I always build it in a well-architected way.And that's such a loaded sentence whenever you say the word ‘well-architected,' that people go, “What do you mean?” And that's where I go through the six pillars. And in Liberty, we have a process, it used to be called SCORP because it was five pillars, but not SCORPS [laugh] because they added sustainability. But that's where for every stack, we'll go through it and we'll be like, “Okay, let's have the discussion.” And we will use the tool that you mentioned, I mean, the tool, as you say, it's a bunch of tick boxes with a text box, but the idea is we'll get in a room and as we build the starter patterns or these pieces of infrastructure that people are going to reuse, we'll run the well-architected review against the framework before anybody gets to generate it.And then we can say, out of the box, if you generate this thing, these are the pros and cons against the Well-Architected Framework of what you're getting. Because we can't make it a hundred percent bulletproof for your use case because we don't know it, but we can tell you out of the box, what it does. And then that way, you can keep building so they start off with something that is well documented how well architected it is, and then you can start having—it makes it a lot easier to have those conversations as they go forward. Because you just have to talk about the delta as they start adding their own code. Then you can and you go, “Okay, you've added these 20 lines. Let's talk about what they do.” And that's why I always think you can do a strong connection between infrastructure-as-code and well architected.Corey: As I look through the actual six pillars of the Well-Architected Framework: sustainability, cost optimization, performance, efficiency, reliability, security, and operational excellence, as I think through the nature of what this shitpost thread Twitter client is, I am reasonably confident across all of those pillars. I mean, first off, when it comes to the cost optimization pillar, please, don't come to my house and tell me how that works. Yeah, obnoxiously the security pillar is sort of the thing that winds up causing a problem for this because this is an account deployed by Control Tower. And when I was getting this all set up, my monthly cost for this thing was something like a dollar in charges and then another sixteen dollars for the AWS config rule evaluations on all of the deploys, which is… it just feels like a tax on going about your business, but fine, whatever. Cost and sustainability, from my perspective, also tend to be hand-in-glove when it comes to this stuff.When no one is using the client, it is not taking up any compute resources, it has no carbon footprint of which to speak, by my understanding, it's very hard to optimize this down further from a sustainability perspective without barging my way into the middle of an AWS negotiation with one of its power companies.Matt: So, for everyone listening, watch as we do a live well-architected review because—Corey: Oh yeah, I expect—Matt: —this is what they are. [laugh].Corey: You joke; we should do this on Twitter one of these days. I think would be a fantastic conversation. Or Twitch, or whatever the kids are using these days. Yeah.Matt: Yeah.Corey: And again, if so much of it, too, is thinking about the context. Security, you work for one of the world's largest insurance companies. I shitpost for a living. The relative access and consequences of screwing up the security on this are nowhere near equivalent. And I think that's something that often gets lost, per the perfect be the enemy of the good.Matt: Yeah that's why, unfortunately, the Well-Architected Tool is quite loose. So, that's why they have the Well-Architected Framework, which is, there's a white paper that just covers anything which is quite big, and then they wrote specific lenses for, like, serverless or other use cases that are shorter. And then when you do a well-architected review, it's like loose on, sort of like, how are you applying the principles of well-architected. And the conversation that we just had about security, so you would write that down in the box and be, like, “Okay, so I understand if anybody gets this credential, it means they can post this Last Tweet in AWS, and that's okay.”Corey: The client, not the Twitter account, to be clear.Matt: Yeah. So, that's okay. That's what you just mark down in the well-architected review. And then if we go to day one on the future, you can compare it and we can go, “Oh. Okay, so last time, you said this,” and you can go, “Well, actually, I decided to—” or you just keep it as a note.Corey: “We pivoted. We're a bank now.” Yeah.Matt: [laugh]. So, that's where—we do more than tweets now. We decided to do microtransactions through cryptocurrency over Twitter. I don't know but if you—Corey: And that ends this conversation. No no. [laugh].Matt: [laugh]. But yeah, so if something changes, that's what the well-architected reviews for. It's about facilitating the conversation between the architect and the engineer. That's all it is.Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: And the lens is also helpful in that this is a serverless application. So, we're going to view it through that lens, which is great because the original version of the Well-Architected Tool is, “Oh, you built this thing entirely in Lambda? Have you bought some reserved instances for it?” And it's, yeah, why do I feel like I have to explain to AWS how their own systems work? This makes it a lot more streamlined and talks about this, though, it still does struggle with the concept of—in my case—a stateless app. That is still something that I think is not the common path. Imagine that: my code is also non-traditional. Who knew?Matt: Who knew? The one thing that's good about it, if anybody doesn't know, they just updated the serverless lens about, I don't know, a week or two ago. So, they added in a bunch of more use cases. So, if you've read it six months ago, or even three months ago, go back and reread it because they spent a good year updating it.Corey: Thank you for telling me that. That will of course wind up in next week's issue of Last Week in AWS. You can go back and look at the archives and figure out what week record of this then. Good work. One thing that I have learned as well as of yesterday, as it turns out, before we wound up having this recording—obviously because yesterday generally tends to come before today, that is a universal truism—is it I had to do a bit of refactoring.Because what I learned when I was in New York live-tweeting the AWS Summit, is that the Route 53 latency record works based upon where your DNS server is. Yeah, that makes sense. I use Tailscale and wind up using my Pi-hole, which lives back in my house in San Francisco. Yeah, I was always getting us-west-1 from across the country. Cool.For those weird edge cases like me—because this is not the common case—how do I force a local region? Ah, I'll give it its own individual region prepend as a subdomain. Getting that to work with both the global lasttweetinaws.com domain as well as the subdomain on API Gateway through the CDK was not obvious on how to do it.Randall Hunt over at Caylent was awfully generous and came up with a proof-of-concept in about three minutes because he's Randall, and that was extraordinarily helpful. But a challenge I ran into was that the CDK deploy would fail because the way that CloudFormation was rendered in the way it was trying to do stuff, “Oh, that already has that domain affiliated in a different way.” I had to do a CDK destroy then a CDK deploy for each one. Now, not the end of the world, but it got me thinking, everything that I see around the CDK more or less distills down to either greenfield or a day one experience. That's great, but throw it all away and start over is often not what you get to do.And even though Amazon says it's always day one, those of us in, you know, real companies don't get to just treat everything as brand new and throw away everything older than 18 months. What is the day two experience looking like for you? Because you clearly have a legacy business. By legacy, I of course, use it in the condescending engineering term that means it makes actual money, rather than just telling really good stories to venture capitalists for 20 years.Matt: Yeah. We still have mainframes running that make a lot of money. So, I don't mock legacy at all.Corey: “What's that piece of crap do?” “Well, about $4 billion a year in revenue. Perhaps show some respect.” It's a common refrain.Matt: Yeah, exactly. So yeah, anyone listening, don't mock legacy because as Corey says, it is running the business. But for us when it comes to day two, it's something that I'm actually really passionate about this in general because it is really easy. Like I did it with CDK patterns, it's really easy to come out and be like, “Okay, we're going to create a bunch of starter patterns, or quickstarts”—or whatever flavor that you came up with—“And then you're going to deploy this thing, and we're going to have you in production and 30 seconds.” But even day one later that day—not even necessarily day two—it depends on who it was that deployed it and how long they've been using AWS.So, you hear these stories of people who deployed something to experiment, and they either forget to delete, it cost them a lot of money or they tried to change it and it breaks because they didn't understand what was in it. And this is where the community starts to diverge in their opinions on what AWS CDK should be. There's a lot of people who think that at the minute CDK, even if you create an abstraction in a construct, even if I create a construct and put it in the construct library that you get to use, it still unravels and deploys as part of your deploy. So, everything that's associated with it, you don't own and you technically need to understand that at some point because it might, in theory, break. Whereas there's a lot of people who think, “Okay, the CDK needs to go server side and an abstraction needs to stay an abstraction in the cloud. And then that way, if somebody is looking at a 20-line CDK construct or stack, then it stays 20 lines. It never unravels to something crazy underneath.”I mean, that's one pro tip thing. It'd be awesome if that could work. I'm not sure how the support for that would work from a—if you've got something running on the cloud, I'm pretty sure AWS [laugh] aren't going to jump on a call to support some construct that I deployed, so I'm not sure how that will work in the open-source sense. But what we're doing at Liberty is the other way. So, I mean, we famously have things like the software accelerator that lets you pick a pattern or create your pipelines and you're deployed, but now what we're doing is we're building a lot of telemetry and automated information around what you deployed so that way—and it's all based on Well-Architected, common theme. So, that way, what you can do is you can go into [crosstalk 00:26:07]—Corey: It's partially [unintelligible 00:26:07], and partially at a glance, figure out okay, are there some things that can be easily remediated as we basically shift that whole thing left?Matt: Yeah, so if you deploy something, and it should be good the second you deploy it, but then you start making changes. Because you're Corey, you just start adding some stuff and you deploy it. And if it's really bad, it won't deploy. Like, that's the Liberty setup. There's a bunch of rules that all go, “Okay, that's really bad. That'll cause damage to customers.”But there's a large gap between bad and good that people don't really understand the difference that can cost a lot of money or can cause a lot of grief for developers because they go down the wrong path. So, that's why what we're now building is, after you deploy, there's a dashboard that'll just come up and be like, “Hey, we've noticed that your Lambda function has too little memory. It's going to be slow. You're going to have bad cold starts.” Or you know, things like that.The knowledge that I have had the gain through hard fighting over the past couple of years putting it into automation, and that way, combined with the well-architected reviews, you actually get me sitting in a call going, “Okay, let's talk about what you're building,” that hopefully guides people the right way. But I still think there's so much more we can do for day two because even if you deploy the best solution today, six months from now, AWS are releasing ten new services that make it easier to do what you just did. So, someone also needs to build something that shows you the delta to get to the best. And that would involve AWS or somebody thinking cohesively, like, these are how we use our products. And I don't think there's a market for it as a third-party company, unfortunately, but I do think that's where we need to get to, that at day two somebody can give—the way we're trying to do for Liberty—advice, automated that says, “I see what you're doing, but it would be better if you did this instead.”Corey: Yeah, I definitely want to spend more time thinking about these things and analyzing how we wind up addressing them and how we think about them going forward. I learned a lot of these lessons over a decade ago. I was fairly deep into using Puppet, and came to the fair and balanced conclusion that Puppet was a steaming piece of crap. So, the solution was that I was one of the very early developers behind SaltStack, which was going to do everything right. And it was and it was awesome and it was glorious, right up until I saw an environment deployed by someone else who was not as familiar with the tool as I was, at which point I realized hell is other people's use cases.And the way that they contextualize these things, you craft a finely balanced torque wrench, it's a thing of beauty, and people complain about the crappy hammer. “You're holding it wrong. No, don't do it that way.” So, I have an awful lot of sympathy for people building platform-level tooling like this, where it works super well for the use case that they're in, but not necessarily… they're not necessarily aligned in other ways. It's a very hard nut to crack.Matt: Yeah. And like, even as you mentioned earlier, if you take one piece of AWS, for example, API Gateway—and I love the API Gateway team; if you're listening, don't hate on me—but there's, like, 47,000 different ways you can deploy an API Gateway. And the CDK has to cover all of those, it would be a lot easier if there was less ways that you could deploy the thing and then you can start crafting user experiences on a platform. But whenever you start thinking that every AWS component is kind of the same, like think of the amount of ways you're can deploy a Lambda function now, or think of the, like, containers. I'll not even go into [laugh] the different ways to run containers.If you're building a platform, either you support it all and then it sort of gets quite generic-y, or you're going to do, like, what serverless cloud are doing though, like Jeremy Daly is building this unique experience that's like, “Okay, the code is going to build the infrastructure, so just build a website, and we'll do it all behind it.” And I think they're really interesting because they're sort of opposites, in that one doesn't want to support everything, but should theoretically, for their slice of customers, be awesome, and then the other ones, like, “Well, let's see what you're going to do. Let's have a go at it and I should hopefully support it.”Corey: I think that there's so much that can be done on this. But before we wind up calling it an episode, I had one further question that I wanted to explore around the recent results of the community CDK survey that I believe is a quarterly event. And I read the analysis on this, and I talked about it briefly in the newsletter, but it talks about adoption and a few other aspects of it. And one of the big things it looks at is the number of people who are contributing to the CDK in an open-source context. Am I just thinking about this the wrong way when I think that, well, this is a tool that helps me build out cloud infrastructure; me having to contribute code to this thing at all is something of a bug, whereas yeah, I want this thing to work out super well—Docker is open-source, but you'll never see me contributing things to Docker ever, as a pull request, because it does, as it says on the tin; I don't have any problems that I'm aware of that, ooh, it should do this instead. I mean, I have opinions on that, but those aren't pull requests; those are complete, you know, shifts in product strategy, which it turns out is not quite done on GitHub.Matt: So, it's funny I, a while ago, was talking to a lad who was the person who came up with the idea for the CDK. And CDK is pretty much the open-source project for AWS if you look at what they have. And the thought behind it, it's meant to evolve into what people want and need. So yes, there is a product manager in AWS, and there's a team fully dedicated to building it, but the ultimate aspiration was always it should be bigger than AWS and it should be community-driven. Now personally, I'm not sure—like you just said it—what the incentive is, given that right now CDK only works with CloudFormation, which means that you are directly helping with an AWS tool, but it does give me hope for, like, their CDK for Terraform, and their CDK for Kubernetes, and there's other flavors based on the same technology as AWS CDK that potentially could have a thriving open-source community because they work across all the clouds. So, it might make more sense for people to jump in there.Corey: Yeah, I don't necessarily think that there's a strong value proposition as it stands today for the idea of the CDK becoming something that works across other cloud providers. I know it technically has the capability, but if I think that Python isn't quite a first-class experience, I don't even want to imagine what other providers are going to look like from that particular context.Matt: Yeah, and that's from what I understand, I haven't personally jumped into the CDK for Terraform and we didn't talk about it here, but in CDK, you get your different levels of construct. And is, like, a CloudFormation-level construct, so everything that's in there directly maps to a property in CloudFormation, and then L2 is AWS's opinion on safe defaults, and then L3 is when someone like me comes along and turns it into something that you may find useful. So, it's a pattern. As far as I know, CDK for Terraform is still on L1. They haven't got the rich collection—Corey: And L4 is just hiring you as a consultant—Matt: [laugh].Corey: —to come in fix my nonsense for me?Matt: [laugh]. That's it. L4 could be Pulumi recently announced that you can use AWS CDK constructs inside it. But I think it's one of those things where the constructs, if they can move across these different tools the way AWS CDK constructs now work inside Pulumi, and there's a beta version that works inside CDK for Terraform, then it may or may not make sense for people to contribute to this stuff because we're not building at a higher level. It's just the vision is hard for most people to get clear in their head because it needs articulated and told as a clear strategy.And then, you know, as you said, it is an AWS product strategy, so I'm not sure what you get back by contributing to the project, other than, like, Thorsten—I should say, so Thorsten who wrote the book with me, he is the number three contributor, I think, to the CDK. And that's just because he is such a big user of it that if he sees something that annoys him, he just comes in and tries to fix it. So, the benefit is, he gets to use the tool. But he is a super user, so I'm not sure, outside of super users, what the use case is.Corey: I really want to thank you for, I want to say spending as much time talking to me about this stuff as you have, but that doesn't really go far enough. Because so much of how I think about this invariably winds up linking back to things that you have done and have been advocating for in that community for such a long time. If it's not you personally, just, like, your fingerprints are all over this thing. So, it's one of those areas where the entire software developer ecosystem is really built on the shoulders of others who have done a lot of work that came before. Often you don't get any visibility of who those people are, so it's interesting whenever I get to talk to someone whose work I have directly built upon that I get to say thank you. So, thank you for this. I really do appreciate how much more straightforward a lot of this is than my previous approach of clicking in the console and then lying about it to provision infrastructure.Matt: Oh, no worries. Thank you for the thank you. I mean, at the end of the day, all of this stuff is just—it helps me as much as it helps everybody else, and we're all trying to do make everything quicker for ourselves, at the end of the day.Corey: If people want to learn more about what you're up to, where's the best place to find you these days? They can always take a job at Liberty; I hear good things about it.Matt: Yeah, we're always looking for people at Liberty, so come look up our careers. But Twitter is always the best place. So, I'm @NIDeveloper on Twitter. You should find me pretty quickly, or just type Matt Coulter into Google, you'll get me.Corey: I like it. It's always good when it's like, “Oh, I'm the top Google result for my own name.” On some level, that becomes an interesting thing. Some folks into it super well, John Smith has some challenges, but you know, most people are somewhere in the middle of that.Matt: I didn't used to be number one, but there's a guy called the Kangaroo Kid in Australia, who is, like, a stunt driver, who was number one, and [laugh] I always thought it was funny if people googled and got him and thought it was me. So, it's not anymore.Corey: Thank you again for, I guess, all that you do. And of course, taking the time to suffer my slings and arrows as I continue to revise my opinion of the CDK upward.Matt: No worries. Thank you for having me.Corey: Matt Coulter, senior architect at Liberty Mutual. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and leave an angry comment as well that will not actually work because it has to be transpiled through a JavaScript engine first.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About TimTim's tech career spans over 20 years through various sectors. Tim's initial journey into tech started as a US Marine. Later, he left government contracting for the private sector, working both in large corporate environments and in small startups. While working in the private sector, he honed his skills in systems administration and operations for large Unix-based datastores.Today, Tim leverages his years in operations, DevOps, and Site Reliability Engineering to advise and consult with clients in his current role. Tim is also a father of five children, as well as a competitive Brazilian Jiu-Jitsu practitioner. Currently, he is the reigning American National and 3-time Pan American Brazilian Jiu-Jitsu champion in his division.Links Referenced:Twitter: https://twitter.com/elchefe TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it's still somebody's problem. And a big part of that responsibility is app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you're actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That's S-N-Y-K.co/screamCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. A bit of a sad episode Today. I am joined by Duckbill Group principal cloud economist, Tim Banks, but by the time this publishes, he will have left the Duckbill nest, as it were. Tim, thank you for joining me, and can I just start by saying, this is sad?Tim: It is. I have really enjoyed being with Duckbill and I will never forget that message you sent me. It's like, “Hey, would you like to do this?” And I was like, “Boy would I.” It's been a fantastic ride and I have enjoyed working with a friend. And I'm glad that we remain friends to this day and always will be, so far as I can tell.Corey: Yes, yes. What you can't see while recording this, I'm actually sitting in the same room as Tim with a weapon pointed at him to make sure that he stays exactly on message. Yeah, I kid. There's been a lot that's happened over the last year. We only got to spend time together in person once at re:Invent. I think because re:Invent is such a blur for me, I don't remember who the hell I talk to.Someone can walk up and say, “Oh yeah, we met at re:Invent,” and I'll nod and say, “Oh yeah,” and I will have no recollection of that whatsoever. But you don't argue with people. But I do distinctly remember hanging out with you there. But since then, it's been a purely distributed company, purely distributed work.Tim: Yeah, that's the only time I've seen you since I've worked here. It's the only time I met Mike. But it's weird because it's like, someone you work with you see every day virtually and talk to, and then you actually get to, like, IRL them and like, “Oh, wow. I had all these, kind of, conceptions of, you know, what you are or who you are as a person, and then you get to, like, check yourself. Was I right? Was I wrong?” I was like, “Oh, you're taller than I thought; you're shorter than I thought,” you know, whatever it was.But I think the fun part about it was we all end up being so close by the nature of how we work that it was just like going back and seeing family after a while; you already know who they are and how they are and about them. So, it felt good, but it felt familiar. That's a great feeling to have. To me, that's a sign of a very successful distributed culture.Corey: Yeah, it's weird the kinds of friendships we've built during the pandemic. When I was in New York for the summit, I got to meet Linda Haviv at AWS for the first time, despite spending the past year or so talking to her repeatedly. As I referred to her the entire time I was in New York, this is Linda, my new old friend because that is exactly how it felt. It's the idea of meeting someone in person that you've had a long-term ongoing friendship with. It's just a really—it's a strange way Everything's new but it's not, all at the same time.It reminds me of the early days of the internet culture where I had more friends online than off, which in my case was not hard. And finally meeting them, some people were exactly like they were described and others were nothing at all like they presented. Now that we have Zoom and this constant level of Slack chatter and whatnot, it's become a lot easier to get a read on what someone is like, I think.Tim: I think so too, you know, we've gotten away—and I think largely because of the pandemic—of just talking about work at work, right? The idea of embracing, you know, almost a cliche of the whole person. But it's become a very necessary thing as people have dealt with pandemic, social upheaval, political climates, and whatever, while they're working from home. You can't compartmentalize that safely in perpetuity, right? So, you do end up getting to know people very well, especially in what their concerns are, what their anxieties are, what makes them happy, what makes them sad, things that go on in their lives.You bring all that to your distributed culture because it's not like you leave it at the door, when you walk out. You're not walking out anymore; you're walking to another room, and it's hard to walk away from those things in this day and age. And we shouldn't have to, right? I feel like for a successful and nurturing culture—whatever it is, whether it's tech culture, whether it's whatever kind of work culture—you can't say, “I only want your productivity and nothing else about you,” and expect people to sustain that. So, you see these companies are, like, you know, “We don't have political discussions. We don't have personal discussions. We're just about the work.” I'm like, “All right, well, that's not going to last.” A person cannot just be an automaton in perpetuity and expect them to grow and thrive.Corey: And this is why you're leaving. And I want to give that a little context because without, sounds absolutely freaking horrifying. You've been a strong advocate for an awful lot of bringing the human to work, on your philosophy around leadership, around management. And you've often been acting in that capacity throughout, I would say, the majority of your career. But here at The Duckbill Group, we don't have a scale of team where you being the director of the team or leader of the team is going to happen in anything approaching the near or mid-term.And so, much of your philosophy is great and all because it's easy to sit here at a small company and start talking about, “Oh, this is how you should be doing it.” You have the opportunity to wind up making a much deeper impact on a lot more people from a management perspective, but you do in fact, need a team to manage as opposed to sitting around there, “Oh, yeah. Who do you manage?” “This one person and I'm doing all of these things to make their life and job awesome.” It's like, “Yeah, how many hours a week are you spending in one-on-ones?” “20 to 25.”Okay, maybe you need a slightly larger team so you can diffuse that out a little bit. And we are definitely sad to be losing you; super excited to see where you wind up going next. This has been a long time coming where there are things that you have absolutely knocked out of the park here at The Duckbill Group, but you also have that growing—from what I picked up on anyway—need to set a good management example. And lord knows this industry needs more of those. So first, sad to lose you. Secondly, very excited for where you wind up next and what they're in for, even though it has a strong likelihood that they don't know the half of it yet.Tim: One of the things that I like about The Duckbill Group and how my time here has been is the first thing that I was asked in the interview was very sincere, like, “Well, what's your next job?” And I was very clear. It's like, “After this, I want to be a director or VP of engineering because I would like to be a force multiplier, right?” I would like to make engineering orgs better. I would like to make engineering practices better. I want to make the engineers better, right?And not by driving KPIs and not by management, right, not administrative functions. I want to do it via leadership. I want to do it by setting examples, making safe places for people, making people feel like they're important and invested in, nurturing them, right? I've said this before I—this analogy was getting me somewhere else and I love, it's like, if I plant a tree and I want it to grow apples, right, I'm not going to sit there and put a number down of apples it's expected to produce, and then put it on a performance plan if it doesn't get that number of apples, right? I need to nurture the tree, I need to fertilize it, I need to protect it, I need to keep it safe, I need to keep it safe from the elements, I need to make sure that it doesn't have parasites, I need to take care of that tree.And if that tree grows and it's healthy and it's thriving, it will produce, right? But I'm not—I can't just expect apples if I'm not taking care of the tree. Now, people are not trees, but you still have to take care of the people if you want them to do things. And if you can't take care of the people, if you can't manage the environment that they're in to make it safe, if you can't give them the things they need to be successful, then you're just going to be holding numbers over someone and expecting to hit them.And that doesn't work. That's not something that's sustainable. And it doesn't really—it's not even about how much you pay them. You must pay them well, right, but it has to be more than just that if you want people to succeed. And that doesn't necessarily mean—like, one thing is at the Duckbill Group I love, succeeding doesn't necessarily mean that I'm going to stay at—or your engineer is going to stay at one place in perpetuity. If you mentor and train and coach and give an engineer opportunity to grow and thrive and what they do is they go to another job for a title increase and a pay increase or something like that, you did your job.Corey: A lot of companies love to tell that lie and they almost convince themselves of it where I look at your resume, and great you have not generally crossed the two-year mark at companies for the last decade. I never did until I started at this place. But we magically always liked to pretend in job interviews that, “Oh yeah, this is my forever job—” like you're a rescue dog getting adopted or something, “—and I'm going to work here for 25 years and get a gold watch and a pension at the end of it.” It's lunacy. I have never seen the value in lying to ourselves like that, which is why we start our interviews with, “What's the job after this and how do we help you get there?”It's important that we ask those questions and acknowledge that reality. And the downside to it—if you can call it a downside—is you've got to live by it. It's not just words, you can slop onto an interview questionnaire; you actually have to mean it. People can see through insincerity.Tim: And it's one of the things, like, if you run an org and you grow your people and you don't have a place for them to grow into, you should expect and encourage them to find those opportunities elsewhere. It is not reasonable, I feel like, as a leader for you expect people to stay in a place where they have grown past or grown out of. You need to either need to give them a new pot to grow into or you need to let them move elsewhere and thrive and grow. And moving elsewhere—like, if you have a retention problem where you can't retain anybody, that's a problem, but if you have your junior engineers who become senior engineers at other places, right, and everyone leaves on good terms, and they got the role and you gave them a great recommendation and they give glowing recommendations to you, there's nothing wrong with that. That's not a failure; that's success.Corey: One bit of I would say pushback that I suspect you might get when talking to people about what's next is that, “Well, you are just a consultant, on some level, for a year.” You always know that someone is really arguing in good faith when they describe what you did with the word ‘just,' but we'll skip past that part. And it's, “You're just a consultant. What would you possibly know about team management and team dynamics?” And there is a little bit of truth to that insofar as the worst place in the world to get management advice is very clearly on Twitter.It turns out that most interpersonal scenarios are, one, far too personal to wind up tweeting about, and two, do not lend themselves to easy solutions that succinctly fit within 280 characters. Imagine that. The counter-argument though, is that you have—correctly from where I sit—identified a number of recurring dynamics on teams that you have encountered and worked with deeply as a large number of engagements. And these are recurring things, I want to be clear. So, I'm not talking about one particular client. If you're one of our clients and listen to this thinking that we're somehow subtweeting you with our voices—I don't know what that is; subwoofing, maybe?Tim: [crosstalk 00:12:05]—Corey: Is that what a subwoofer is? I'm not an audio person.Tim: Throwing shade, we'll just say—call it throwing shade.Corey: Yeah, we're not throwing shade at any one person, team, or group in particular; these are recurring things. Tim, what have you seen?Tim: And so, I think the biggest thing I see is folks that are on the precipice of a big technological change, right, and there is an extraordinary amount of anxiety, right? I've seen a number of customers through our engagements that, “We are moving away from this legacy platform,” or from this thing that we have been doing for X amount of time. And everyone has staked the other domains, staked out their areas of expertise and control and we're going to change that. And the solution to that is not a technical solution. You don't fix that by Helm charts, or Terraform, or CloudFormation. You fix that by conversations, and you fix that by listening. You fix that by finding ways to reassure folks and giving them confidence in their ability to adjust and thrive in a new environment.If you take somebody who's been, you know, an Oracle admin for 20 years, and you going to say, “Great. Now, you're going to learn, you're going to do this an RDS,” that's a whole new animal, and folks feel like, well, you know, I can't learn something new like that? Well, yeah you can. If you can learn Oracle, you can learn anything. I firmly believe that.But that's one of the conversations we have, it's never, almost never a technical problem folks have. We need to reassure people, right? And so, folks who reach out to us, it's typically folks who are trying to get their organizations in that direction. Another thing we see sometimes is that we find that there's a disconnect between leadership and the engineers. They have either different priorities or different understandings of what's going on. And we come in to solve a problem, which may be cost but that's not the problem we actually solve. The problem we actually solve is fixing this communication bridges between management and leadership.And that's almost an every time occurred. At some point or another, there's some disconnect there. And that's the best part of the job. Like, the reason I do this consulting gig is not because I want to bang away at code. If I've had to do that, that's an anomaly for sure because I want to have these conversations.And people want to have these conversations; they want to get these problems solved and sometimes they don't know how to. And that is the common thing, I think, through all of our customers. Like, we need some amount of expertise to help us find solutions to these things that aren't necessarily technical problems. And I think that's where we run into problems as an industry, right, where we think a lot of things are technical problems or have technical solutions, and they don't. There are people problems. They're—Corey: Here at The Duckbill Group, we're basically marriage counseling for engineering and finance in many cases.Tim: We really are.Corey: This is why were people not software.Tim: Yeah. And I will say this very firmly and you can quote me on this: like, you cannot replace us. You cannot replace the kind of engagements we do with software. You can't. Can't be done, right? Software is not empathetic.Corey: There are a whole series of questions we ask our clients at the start of an engagement and the answers to those questions change what we ask them going forward. In fact, even the level-setting in the conversation that we have at the start of that changes the nature of those. We're not reading from a list; we're trying to build an understanding. There is a process around what we do, but it's not process that can ever be scoped down to the point where it's just a list of questions or a questionnaire that isn't maddening for people to fill out because it's so deeply and clearly misses the mark around context of what they're actually doing.Tim: Mm-hm. Our engaged with their conversations. That's all they are. They're really in-depth conversations where we're going to start asking questions and we're going to ask questions about those answers. We're start pulling out strings and kicking over rocks and seeing what we find.And that's the kind of thing that, you know, you would expect anyone to do who's coming in and saying, “Okay, we have a problem. Now, let's figure it out.” Right? Well, you can't just look at something on the surface, and say, “Oh, I know what this is.” Right? You know, for someone to say, “Oh, I know how to fix this,” when they walk in is the surest way to know that someone doesn't know what they're talking about, right?Corey: Oh, easiest thing in the world is to walk in and say, “This is broken and wrong.” That can translate directly to, “Hi, I am very junior. Please feed my own ass to me.” Because no one shows up at work thinking they're going to do a crap job today on purpose. There's a reason things are the way that they are.Tim: Mm-hm. And that's the biggest piece of context we get from our customers is we can understand what the best practices are. You can go Google them right now and say, “This is the ten things you're supposed to do all the time,” right? And we would be really, really crappy consultants if we just read off that list, right? We need to have context: does this thing make sense? Is this the best practice? Maybe, but we want to know why you did it this way.And after you tell us that way, I'm like, “You know what? I would do it the exact same way for this use case.” And that's great. We can say like, “This is the best way to do that. Good job.” It's atypical; it's unusual, but it solves the problems that you need solving.And that's where I think a lot of people miss. Like, you know, you can go—and not to throw shade at AWS's Trusted Advisor, but we're going to throw shade at AWS's Trusted Advisor—and the fact that it will give you—Corey: It is Plausible Advisor at absolute best.Tim: [laugh]. It will give you suggestions that have no context. And a lot of the automated AI things that will recommend that you do this and this and this and this are pretty much all the same. And they have no context because they don't understand what you're trying to do. And that's what makes the difference between people. There's these people problems.And so, one of the things that I think is really interesting is that we have moved into doing a shorter engagement style that is very short. It's very quick, it's very kind of almost tactical, but we go in, we look at your bill, we ask you some questions, and we're going to give you a list of suggestions that are going to save you a significant amount of money right away, right? So, a lot of times, folks when they need quick wins, or they don't really need us to deep-dive into all their DynamoDB access patterns, right? They just want like, “Hey, what are the five things we can do to save us some money?” And we're like, “Well, here they are. And here's what we think they're going to save you.” And folks who really enjoyed that type of engagement. And it's one of my favorite ones to do.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: I can also predict that people are going to have questions for you—probably inane—of, well, you were a consultant, how are your actual technical chops? And I love answering these questions with data. So, I have here pulled up the last six months of The Duckbill Group's AWS bills. And for those who are unaware, every cloud economist has their own dedicated test account for testing out strange things that we come across. And again, can the correct answer in many consulting engagements is, “I don't know, but I'll find out.”Well, this is how we find out. We run tests and learn these things ourselves. I suppose we could extend this benefit, if you want to call it that, to people who aren't cloud economists but I'm not entirely sure what, I don't know, an audio engineer is going to do with an AWS account that isn't, you know, kind of horrifying. To the audio engineer that is editing this podcast, my condolences if you take that as a slight, and if there is something you would use an AWS account for, please let me know. We'll come talk about it here.But back to topic, looking at the last six months of your bill for your account—that's right, a ritualistic shaming of the AWS bill—in January you spent $16.06. In February, you spent 44 cents. And you realized that was too high, so back in March, you then spent 19 cents. And then $3.01 back in April. May wound up $10.02, and now you're $9.84 as of June. July has not yet finalized as of this recording.And what I want to highlight—and what that tells me when I look at these types of bills—and I assure you as the world's leading self-described expert in AWS billing, I'm right; listening to me is a best practice on these things—that shows the exact opposite of a steady-state workload. There's a lot of dynamism to those giant swings because we don't have cloud economists who are going to just run these things steady-state for the rest of our lives. Those are experiments of building and testing out new and exciting things in a whole bunch of very weird, very strange ways. Whenever I wind up talking to someone in one of the overarching AWS services at AWS and I pull up my account, a common refrain is, “Wow, you use an awful lot of services.” Right. I'm not just sitting here run and EC2 instances forever. Imagine that. And your account is a perfect microcosm of that entire philosophy.Tim: Well, I don't know all the answers, right? And I will never profess all the answers. And before I say, “You should do this—” or maybe I will say, “You might be able to do this. Let me go save as possible.” [laugh]. Right? And so, just let me just see, can you do this? Does this work? No, I guess it doesn't. Or AWS docu—especially, “The AWS documentation says this. Let me see if that's actually the case.”Corey: I don't believe that they intend to lie, but—Tim: No.Corey: —they also certainly don't get it correct all the time.Tim: And to be fair, they have, what, 728 services by this point, and that's a lot of documentation you're not going to get—Corey: Three more have launched since the start of this recording.Tim: I—yeah, actually—well, by the time this hits, they're probably going to have 22. But we'll [laugh] see. But yeah, no. And that's fine. And they're not going to have every use case, and every edge, kind of like, concern handled, and so that's why we need to kick the tires a little bit.And what I think more than anything else is, you know, sometimes we just do things out of convenience. Like, “Well, I don't want to run this on this; let me just fire it up because it's not my money.” [laugh]. But we also want to be fairly concerned about you know, how we do things. You don't want to run a fleet of z1ds, obviously.But there is a certain amount of tire-kicking and infrastructure spinning up that you have to do in order to maintain freshness, right? And it's not a thing where I'm going to say, “Oh, I know YAML off the top of my head, and I need to do—you know, I'm up to speed on every single possible API call that you can make.” No. My technical prowess has always been in architecture and operations. So, I think when we have these conversations, folks mostly tend to be impressed by not only business acumen and strategy, but also being able to get down to the weeds and talking with the developers and the engineers about the minutia. And you will have seen you know, the feedback that I've gotten about my technical prowess has always been good. You know, I can hang with anybody, I feel like.Corey: I would agree wholeheartedly. It's been really interesting watching you in conversations, internally and with our clients, where you will just idly bust out something fricking brilliant out of left field. And most of the time, I don't think you even realize it. It's just one of those things that makes intuitive and instinctive sense to you. And you basically just leave people stunned and their scribbling notes and trying to wrap their heads around what you just said.And it's adorable because sometimes you wind up almost, like, looking embarrassed, like, “Did I say something rude and not realize it? Like, I wasn't trying to be insulting.” It's like, “Nope, nope. You're just doing your thing, Tim. Just keep on doing it. That's fine.”Tim: Yeah, it's funny because, like you, one of the things that I've really enjoyed about it is, like, we'll just start bouncing ideas off of each other and come up with something brilliant. “Yeah, let's do that.” And then, “Okay, this is now a thing.” And it's like, you know, there's something to be said about being around smart people. So, it's not just me coming up with something brilliant; these are almost always fruits of a conversation and discussion being had, and then you formulate something great in your head.But again, this is why I love the aspect of talking and having conversations with people, so that way you can come up with something kind of brilliant. None of this is done in a silo. Like we're not really, really good at what we do because we don't rely or talk to or have conversations with other people.Corey: One thing that you did that I think is one of the most transformative things that has happened in company history in some respects has been when you started, and for the first half of your tenure here, we had two engagement types that we would wind up giving our consulting clients. There's contract negotiation, where we help companies negotiate their long-term commitment contracts with AWS—and we're effective at it and that's fun; that's basically what you would more or less expected to be—and the other is our cost optimization project engagements. And those tend to look six to eight weeks where we wind up going in deep-dives into the intricacies of an organization's AWS accounts, bills, strategy, growth plan, et cetera, et cetera, et cetera, to an exhaustive level of detail. And in an interest of being probably overly transparent here, I didn't like working on those engagements myself. I like coming in, finding the big things that will be transformative to reduce the bills—it's like solving a puzzle—and then the relatively in-depth analysis for things that are a relatively paltry portion of the AWS bill does not really lead me to enjoying the work very much.And I beat my head against that one for years. And you busted out one day with an idea that became our third type of engagement, which is the first pass, where we charge significantly less for the engagement and it essentially distills down into you get us to talk to your engineering teams for a day. Bring us any questions, give us access in advance to these things, and we will basically go on a whirlwind guided tour and lay waste to your AWS bill and highlight different opportunities that we see to optimize these things. And it has been an absolute smash success. People love the engagements.Very often, it leads to that second full-bore engagement that I was describing earlier, but it also aligns very well with the way that I like to think about these things. I'm a great consultant, specifically because once I've delivered the value, I like to leave. Whereas as an employee, I just sort of linger around, and then I go cause problems and other people's departments—ideally, not on purpose, but you know, I am me—and this really emphasizes that and keeps me moving quickly. I really, really like that engagement style and I have you to thank for coming up with the idea and finding a way to do it that didn't either not resonate with the market—in which case, we're not selling a damn thing—or wound up completely eviscerating the value of the longer-term deep-dive engagements, and you threaded that needle perfectly.Tim: I thank you; I appreciate that. There was this kind of vacuum that I saw where, both from a cost and from a resource point where six to eight weeks is a long time for an engineering org to dedicate to any one thing, especially if that one thing isn't directly making money. But engineering orgs are also very interested in saving money. But it's especially in smaller orgs where that velocity is very important, they don't have six to eight weeks for that. They can't dedicate the resources to those deep-dives all the time, and all the conversations we—and when we do a COP, it is exhaustive. We are exploring every avenue to almost an absurd level, right?And that's not the right engagement for a lot of orgs, right? So, coming in and saying, “Hey, you know, this is a quick one; these are the things that you can do. This is 90% of the savings you're going to realize. These things: bam, bam, bam, bam, bam.” Right?And then we give it to the folks and we let them work on it, and then they're like, “Hey, we need this because we want to negotiate EDP,” or, “We need this because, you know, we're just trying to make sure that our costs are in line so we can be more agile, so we can do this project, or whatever.” Right? And then there are a lot of other orgs that do need that exhaustive kind of thing, larger orgs especially, right? Larger, more complex orgs, orgs that are trying to maybe—like, if you're trying to make a play to get acquired, you want to get this very, very in-depth study so you know all your liabilities and all your assets, so that way you can fix those problems and make it very attractive for someone to buy you, right? Or orgs that just have, like, we are not having an impending EDP; we have a lot of time to be able to focus on these things, and we can build this into the roadmap, right?Then we can do a very exhaustive study of those things. But for a lot of times, people are just like, “Look, I just need to save X amount of money on my AWS bill and can you do that?” Well, sure. We can go in there and have those conversations and give you a lot of savings. And I'm very much in the camp of, you know, ‘perfect is the enemy of good.' I don't have to save down to the nth penny on your DynamoDB bill. But if I can, shave—cut it in half, that's great. Most people are very happy about those kinds of things. And that's a very routine finding for us.Corey: One other aspect that I really liked about it, too, is that it let us move down market a bit, away from companies that are spending millions of dollars a month. Because yeah, the ROI for those customers is a slam dunk on virtually any engagement that we could put together, but what about the smaller companies, the ones that are not spending that much money, yet? They've never felt great talk to them and say, “Oh, just go screw up your AWS bill some more. Then, then you will absolutely be able to generate some value. Maybe turn off MFA and post your credentials to GitHub or something. That'll speed up the process nicely.”That's terrible advice and we can't do it. But this enables us to move down to smaller companies that are earlier in their cloud estate build-out or are growing organically rather than trying to do a giant migration as sort of greenfield growth approach. I really, really like our ability to help companies that are a bit earlier in their cloud journey, as well as in smaller environments, just because I guess, on some level, for me, at least, when you see enormous multimillion-dollar levels of spend, the misconfigurations are generally less fun to find; they're less exciting. Because, yeah at a small scale, you can screw up and your Managed NAT Gateway bill is a third of your spend. When you're spending $80 million a year, you're not wasting that kind of money on Managed NAT Gateways because that misconfiguration becomes visible from frickin' orbit.So, someone has already found that stuff. And it's always then it's almost certainly EC2, RDS, and storage. Great. Then there's some weird data transfer stuff and it starts to look a lot more identical. Smaller accounts, at least from my perspective, tend to have a lot more of interesting things to learn hiding in the shadows.Tim: Oh, absolutely. And I think the impact that you make for the future for small companies much higher, right? You go in there and you have an engagement, you can say, “Okay, I understand the business reason why you did this here, but if you make these changes—bam, bam, bam—12 to 18 months and on, right, this is going to make a huge difference in your business. You're going to save a tremendous amount of money and you're going to be much more agile.”You did this thing because it worked for the POC, it worked for the MVP, right? That's great, but before it gets too big and becomes load-bearing technical debt, let's make some changes to put you in a better position, both for cost optimization and an architectural future that you don't have to then break a bone that's already set to try and fix it. So, getting in there before there's a tremendous load on their architecture—or rather on their infrastructure, it's super, super fun because you know that when you've done this, you have given that company more runway, or you've given them the things they need to actually be more successful, and so they can focus their time and efforts on growth and not on trying to stop the bleeding with their AWS bill.Corey: Tim, it's been an absolute pleasure to work with you. I'm going to miss working with you, but we are definitely going to remain in touch. Where can people find you to follow along with your continuing adventures?Tim: The best way to find me is on Twitter, I am @elchefe—E-L-C-H-E-F-E. And yeah, I will definitely keep in touch with you, Corey. Again, you have been a tremendous friend and I really appreciate you, your insights, and your honesty. Our partners are friends with each other and I do not think that they will let us ever drift too far apart. So.Corey: No, I think it is pretty clear that we are basically going to be both of their plus-ones forever.Tim: [laugh]. I think so.Corey: I'm just waiting for them when they pulled the prank of dressing us the exact same way because our styles are somewhat different, and I'm pretty sure that there's not a whole lot of convergence where we both wind up looking great. So, it's going to be hilarious regardless of what direction it goes in.Tim: Well, you do have velour tracksuits too, right?Corey: Not yet, but please don't tell that to Bethany.Tim: [laugh].Corey: Tim, it has been an absolute pleasure.Tim: The pleasure has been all mine, Corey. I really appreciate it.Corey: Tim Banks, for one last time, principal cloud economist at The Duckbill Group. I am Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and an insulting comment that says that we are completely wrong in our approach to management and the real answer is as follows, making sure to keep that answer less than 280 characters.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About JeffJeff Smith has been in the technology industry for over 20 years, oscillating between management and individual contributor. Jeff currently serves as the Director of Production Operations for Basis Technologies (formerly Centro), an advertising software company headquartered in Chicago, Illinois. Before that he served as the Manager of Site Reliability Engineering at Grubhub.Jeff is passionate about DevOps transformations in organizations large and small, with a particular interest in the psychological aspects of problems in companies. He lives in Chicago with his wife Stephanie and their two kids Ella and Xander.Jeff is also the author of Operations Anti-Patterns, DevOps Solutions with Manning publishing. (https://www.manning.com/books/operations-anti-patterns-devops-solutions) Links Referenced: Basis Technologies: https://basis.net/ Operations Anti-Patterns: https://attainabledevops.com/book Personal Site: https://attainabledevops.com LinkedIn: https://www.linkedin.com/in/jeffery-smith-devops/ Twitter: https://twitter.com/DarkAndNerdy Medium: https://medium.com/@jefferysmith duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by our friends at Fortinet. Fortinet's partnership with AWS is a better-together combination that ensures your workloads on AWS are protected by best-in-class security solutions powered by comprehensive threat intelligence and more than 20 years of cybersecurity experience. Integrations with key AWS services simplify security management, ensure full visibility across environments, and provide broad protection across your workloads and applications. Visit them at AWS re:Inforce to see the latest trends in cybersecurity on July 25-26 at the Boston Convention Center. Just go over to the Fortinet booth and tell them Corey Quinn sent you and watch for the flinch. My thanks again to my friends at Fortinet.Corey: Let's face it, on-call firefighting at 2am is stressful! So there's good news and there's bad news. The bad news is that you probably can't prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the fun things about doing this show for long enough is that you eventually get to catch up with people and follow up on previous conversations that you've had. Many years ago—which sounds like I'm being sarcastic, but is increasingly actually true—Jeff Smith was on the show talking about a book that was about to release. Well, time has passed and things have changed. And Jeff Smith is back once again. He's the Director of Product Operations at Basis Technologies, and the author of DevOps Anti-Patterns? Or what was the actual title of the book it was—Jeff: Operations Anti-Patterns.Corey: I got hung up in the anti-patterns part because it's amazing. I love the title.Jeff: Yeah, Operations Anti-Patterns, DevOps Solutions.Corey: Got you. Usually in my experience, alway been operations anti-patterns, and here I am to make them worse, probably by doing something like using DNS as a database or some godforsaken thing. But you were talking about the book aspirationally a few years ago, and now it's published and it has been sent out to the world. And it went well enough that they translated it to Japanese, I believe, and it has seen significant uptick. What was your experience of it? How did it go?Jeff: You know, it was a great experience. This is definitely the first book that I've written. And the Manning process was extremely smooth. You know, they sort of hold your hand through the entire process. But even after launch, just getting feedback from readers and hearing how it resonated with folks was extremely powerful.I was surprised to find out that they turned it into an audiobook as well. So, everyone reaches out and says, “Did you read the audiobook? I was going to buy it, but I wasn't sure.” I was like, “No, unfortunately, I don't read it.” But you know, still cool to have it out there.Corey: My theory has been for a while now that no one wants to actually write a book; they want to have written a book. Now that you're on the other side, how accurate is that? Are you in a position of, “Wow, sure glad that's done?” Or are you, “That was fun. Let's do it again because I like being sad all the time.” I mean, you do work Kubernetes for God's sake. I mean, there's a bit of masochism inherent to all of us in this space.Jeff: Yeah. Kubernetes makes me cry a little bit more than the writing process. But it's one of the things when you look back on it, you're like, “Wow, that was fun,” but not in the heat of the moment, right? So, I totally agree with the sentiment that people want to have written a book but not actually gone through the process. And that's evident by the fact that how many people try to start a book on their own without a publisher behind them, and they end up writing it for 15 years. The process is pretty grueling. The feedback is intense at first, but you start to get into a groove and you—I could see, you know, in a little while wanting to write another book. So, I can see the appeal.Corey: And the last time you were on the show, I didn't really bother to go in a particular topical direction because, what's the point? It didn't really seem like it was a top-of-mind issue to really bring up because what's it matter; it's a small percentage of the workforce. Now I feel like talking about remote work is suddenly taking on a bit of a different sheen than it was before the dark times arrived. Where do you land on the broad spectrum of opinions around the idea of remote work, given that you have specialized in anti-patterns, and well, as sarcastic as I am, I tend to look at almost every place I've ever worked is expressing different anti-patterns from time to time. So, where do you land on the topic?Jeff: So, it's funny, I started as a staunch office supporter, right? I like being in the office. I like collaborating in person; I thought we were way more productive. Since the pandemic, all of us are forced into remote work, I've hired almost half of my team now as remote. And I am somewhat of a convert, but I'm not on the bandwagon of remote work is just as good or is better as in person work.I've firmly landed in the camp of remote work is good. It's got its shortcomings, but it's worth the trade off. And I think acknowledging what those trade-offs are important to keeping the team afloat. We just recently had a conversation with the team where we were discussing, like, you know, there's definitely been a drop in productivity over the past six months to a year. And in that conversation, a lot of the things that came up were things that are different remote that were better in person, right, Slack etiquette—which is something, you know, I could talk a little bit about as well—but, you know, Slack etiquette in terms of getting feedback quickly, just the sort of camaraderie and the lack of building that camaraderie with new team members as they come on board and not having those rituals to replace the in-person rituals. But through all that, oddly enough, no one suggested going back into the office. [laugh].Corey: For some strange reason, yeah. I need to be careful what I say here, I want to disclaim the position that I'm in. There is a power imbalance and nothing I say is going to be able to necessarily address that because I own the company and if my team members are listening to this, they're going to read a lot into what I say that I might not necessarily intend. But The Duckbill Group, since its founding, has been a fully distributed company. My business partner lives in a different state than I do so there's never been the crappy version of remote, which is, well, we're all going to be in the same city, except for Theodore. Theodore is going to be timezones away and then wonder why he doesn't get to participate in some of the conversations where the real decisions get made.Like that's crappy. I don't like that striated approach to things. We don't have many people who are co-located in any real sense, nor have we for the majority of the company's life. But there are times when I am able to work on a project in a room with one of my colleagues, and things go a lot more smoothly. As much as we want to pretend that video is the same, it quite simply isn't.It is a somewhat poor substitute for the very high bandwidth of a face-to-face interaction. And yes, I understand this is also a somewhat neurotypical perspective, let's be clear with that as well, and it's not for everyone. But I think that for the base case, a lot of the remote work advocates are not being fully, I guess, honest with themselves about some of the shortcomings remote has. That is where I've mostly landed on this. Does that generally land with where you are?Jeff: Yeah, that's exactly where I'm at. I completely agree. And when we take work out of the equation, I think the shortcomings lay themselves bare, right? Like I was having a conversation with a friend and we were like, well, if you had a major breakup, right, I would never be like, “Oh, man. Grab a beer and hop on Zoom,” right? [laugh]. “Let's talk it out.”No, you're like, hey, let's get in person and let's talk, right? We can do all of that conversation over Zoom, but the magic of being in person and having that personal connection, you know, can't be replaced. So, you know, if it's not going to work, commiserating over beers, right? I can't imagine it's going to work, diagramming some complex workflows and trying to come to an answer or a solution on that. So again, not to say that, you know, remote work is not valuable, it's just different.And I think organizations are really going to have to figure out, like, okay, if I want to entice people back into the office, what are the things that I need to do to make this realistic? We've opened the floodgates on remote hiring, right, so now it's like, okay, everyone's janky office setup needs to get fixed, right? So, I can't have a scenario where it's like, “Oh, just point your laptop at the whiteboard, right?” [laugh]. Like that can't exist, we have to have office spaces that are first-class citizens for our remote counterparts as well.Corey: Right because otherwise, the alternative is, “Great, I expect you to take the home that you pay for and turn it into an area fit for office use. Of course, we're not going to compensate you for that, despite the fact that, let's be realistic, rent is often larger than the AWS bill.” Which I know, gasp, I'm as shocked as anyone affected by that, but it's true. “But oh, you want to work from home? Great. That just means you can work more hours.”I am not of the school of thought where I consider time in the office to be an indicator of anything meaningful. I care if the work gets done and at small-scale, this works. Let me also be clear, we're an 11-person company. A lot of what I'm talking about simply will not scale to companies that are orders of magnitude larger than this. And from where I sit, that's okay. It doesn't need to.Jeff: Right. And I think a lot of the things that you talk about will scale, right? Because in most scenarios, you're not scaling it organizationally so much as you are with a handful of teams, right? Because when I think about all the different teams I interact with, I never really interact with the organization as a whole, I interact with my little neighborhood in the organization. So, it is definitely something that scales.But again, when it comes to companies, like, enticing people back into the office, now that I'm talking about working from home five days a week, I've invested in my home setup. I've got the monitor I want, I've got the chair that I want, I've got the mouse and keyboard that I want. So, you're going to bring me back to the office so I can have some standard Dell keyboard and mouse with some janky, you know—maybe—21-inch monitor or something like that, right? Like, you really have to decide, like, okay, we're going to make the office a destination, we're going to make it where people want to go there where it's not just even about the collaboration aspect, but people can still work and be effective.And on top of that, I think how we look at what the office delivers is going to change, right? Because now when I go to the office now, I do very little work. It's connections, right? It's like, you know, “Oh, I haven't seen you in forever. Let's catch up.” And a lot of that stuff is valuable. You know, there's these hallway conversations that exist that just weren't happening previously because how do I accidentally bump into you on Slack? [laugh]. Right, it has to be much more it of a—Corey: Right. It takes some contrivance to wind up making that happen. I remember back in the days of working in offices, I remember here in San Francisco where we had unlimited sick time and unlimited PTO, I would often fake a sick day, but just stay home and get work done. Because I knew if I was in the office, I'd be constantly subjected to drive-bys the entire time of just drive-by requests, people stopping by to ask, “Oh, can you just help me with this one thing,” that completely derails my train of thought. Then at the end of the day, they'd tell me, “You seem distractible and you didn't get a lot of work done.”It's, “Well, no kidding. Of course not. Are you surprised?” And one of the nice things about starting your own company—because there are a lot of downsides, let me be very clear—one of the nice things is you get to decide how you want to work. And that was a study in, first, amazement, and then frustration.It was, “All right, I just landed a big customer. I'm off to the races and going to take this seriously for a good six to twelve months. Great sky's the limit, I'm going to do up my home office.” And then you see how little money it takes to have a nice chair, a good standing desk, a monitor that makes sense and you remember fighting tooth-and-nail for nothing that even approached this quality at companies and they acted like it was going to cost them 20-grand. And here, it's two grand at most, when I decorated this place the first time.And it was… “What the hell?” Like, it feels like the scales fall away from your eyes, and you start seeing things that you didn't realize were a thing. Now I worry that five years in, there's no way in the world I'm ever fit to be an employee again, so this is probably the last job I'll ever have. Just because I've basically made myself completely unemployable across six different axes.Jeff: [laugh]. And I think one of the things when it comes to, like, furniture, keyboard, stuff like that, I feel like part of it was just, like, this sort of enforced conformity, right, that the office provided us the ability to do. We can make sure everyone's got the same monitor, the same keyboard that way, when it breaks, we can replace it easily. In a lot of organizations that I've been in, you know, that sort of like, you know, even if it was the same amount or ordering a custom keyboard was a big exception process, right? Like, “Oh, we've got to do a whole thing.” And it's just like, “Well, it doesn't have to be that complicated.”And like you said, it doesn't cost much to allow someone to get the tools that they want and prefer and they're going to be more productive with. But to your point really quickly about work in the office, until the pandemic, I personally didn't recognize how difficult it actually was to get work done in the office. I don't think I appreciated it. And now that I'm remote, I'm like, wow, it is so much easier for me to close this door, put my headphones on, mute Slack and go heads down. You know, the only drive-by I've got is my wife wondering if I want to go for a walk, and that's usually a text message that I can ignore and come back to later.Corey: The thing that just continues to be strange for me and breaks in some of the weirdest ways has just been the growing awareness of how much of office life is unnecessary and ridiculous. When you're in the office every day, you have to find a way to make it work and be productive and you have this passive-aggressive story of this open office, it's for collaboration purposes. Yeah, I can definitively say that is not true. I had a boss who once told me that there was such benefits to working in an open plan office that if magically it were less expensive to give people individual offices, he would spare the extra expense for open plan. That was the day I learned he would lie to me while looking me in the eye. Because of course you wouldn't.And it's for collaboration. Yeah, it means two loud people—often me—are collaborating and everyone else wears noise-canceling headphones trying desperately to get work done, coming in early, hours before everyone else to get things done before people show up and distracted me. What the hell kind of day-to-day work environment is that?Jeff: What's interesting about that, though, is those same distractions are the things that get cited as being missed from the perspective of the person doing the distracting. So, everyone universally hates that sort of drive-by distractions, but everyone sort of universally misses the ability to say like, “Hey, can I just pull on your ear for a second and get your feedback on this?” Or, “Can we just walk through this really quickly?” That's the thing that people miss, and I don't think that they ever connect it to the idea that if you're not the interruptee, you're the interruptor, [laugh] and what that might do to someone else's productivity. So, you would think something like Slack would help with that, but in reality, what ends up happening is if you don't have proper Slack etiquette, there's a lot of signals that go out that get misconstrued, misinterpreted, internalized, and then it ends up impacting morale.Corey: And that's the most painful part of a lot of that too. Is that yeah, I want to go ahead and spend some time doing some nonsense—as one does; imagine that—and I know that if I'm going to go into an office or meet up with my colleagues, okay, that afternoon or that day, yeah, I'm planning that I'm probably not going to get a whole lot of deep coding done. Okay, great. But when that becomes 40 hours a week, well, that's a challenge. I feel like being full remote doesn't work out, but also being in the office 40 hours a week also feels a little sadistic, more than almost anything else.I don't know what the future looks like and I am privileged enough that I don't have to because we have been full remote the entire time. But what we don't spend on office space we spend on plane tickets back and forth so people can have meetings. In the before times, we were very good about that. Now it's, we're hesitant to do it just because it's we don't want people traveling before the feel that it's safe to do so. We've also learned, for example, when dealing with our clients, that we can get an awful lot done without being on site with them and be extraordinarily effective.It was always weird have traveled to some faraway city to meet with the client, and then you're on a Zoom call from their office with the rest of the team. It's… I could have done this from my living room.Jeff: Yeah. I find those sorts of hybrid meetings are often worse than if we were all just remote, right? It's just so much easier because now it's like, all right, three of us are going to crowd around one person's laptop, and then all of the things that we want to do to take advantage of being in person are excluding the people that are remote, so you got to do this careful dance. The way we've been sort of tackling it so far—and we're still experimenting—is we're not requiring anyone to come back into the office, but some people find it useful to go to the office as a change of scenery, to sort of, like break things up from their typical routine, and they like the break and the change. But it's something that they do sort of ad hoc.So, we've got a small group that meets, like, every Thursday, just as a day to sort of go into the office and switch things up. I think the idea of saying everyone has to come into the office two or three days a week is probably broken when there's no purpose behind it. So, my wife technically should go into the office twice a week, but her entire team is in Europe. [laugh]. So, what point does that make other than I am a body in a chair? So, I think companies are going to have to get flexible with this sort of hybrid environment.But then it makes you wonder, like, is it worth the office space and how many people are actually taking advantage of it when it's not mandated? We find that our office time centers around some event, right? And that event might be someone in town that's typically remote. That might be a particular project that we're working on where we want to get ideas and collaborate and have a workshop. But the idea of just, like, you know, we're going to systematically require people to be in the office x many days, I don't see that in our future.Corey: No, and I hope you're right. But it also feels like a lot of folks are also doing some weird things around the idea of remote such as, “Oh, we're full remote but we're going to pay you based upon where you happen to be sitting geographically.” And we find that the way that we've done this—and again, I'm not saying there's a right answer for everyone—but we wind up paying what the value of the work is for us. In many cases, that means that we would be hard-pressed to hire someone in the Bay Area, for example. On the other hand, it means that when we hire people who are in places with relatively low cost of living, they feel like they've just hit the lottery, on some level.And yeah, some of them, I guess it does sort of cause a weird imbalance if you're a large Amazon-scale company where you want to start not disrupting local economies. We're not hiring that many people, I promise. So, there's this idea of figuring out how that works out. And then where does the headquarters live? And well, what state laws do we wind up following on what we're doing? Just seems odd.Jeff: Yeah. So, you know, one thing I wanted to comment on that you'd mentioned earlier, too, was the weird things that people are doing, and organizations are doing with this, sort of, remote work thing, especially the geographic base pay. And you know, a lot of it is, how can we manipulate the situation to better us in a way that sounds good on paper, right? So, it sounds perfectly reasonable. Like, oh, you live in New York, I'm going to pay you in New York rates, right?But, like, you live in Des Moines, so I'm going to pay you Des Moines rates. And on the surface, when you just go you're like, oh, yeah, that makes sense, but then you think about it, you're like, “Wait, why does that matter?” Right? And then, like, how do I, as a manager, you know, level that across my employees, right? It's like, “Oh, so and so is getting paid 30 grand less. Oh, but they live in a cheaper area, right?” I don't know what your personal situation is, and how much that actually resonates or matters.Corey: Does the value that they provide to your company materially change based upon where they happen to be sitting that week?Jeff: Right, exactly. But it's a good story that you can tell, it sounds fair at first examination. But then when you start to scratch the surface, you're like, “Wait a second, this is BS.” So, that's one thing.Corey: It's like tipping on some level. If you can't afford the tip, you can't afford to eat out. Same story here. If you can't afford to compensate people the value that they're worth, you can't afford to employ people. And figure that out before you wind up disappointing people and possibly becoming today's Twitter main character.Jeff: Right. And then the state law thing is interesting. You know, when you see states like California adopting laws similar to, like, GDPR. And it's like, do you have to start planning for the most stringent possibility across every hire just to be safe and to avoid having to have this sort of patchwork of rules and policies based on where someone lives? You might say like, “Okay, Delaware has the most stringent employer law, so we're going to apply Delaware's laws across the board.” So, it'll be interesting to see how that sort of plays out in the long run. Luckily, that's not a problem I have to solve, but it'll be interesting to see how it shakes out.Corey: It is something we had to solve. We have an HR consultancy that helps out with a lot of these things, but the short answer is that we make sure that we obey with local laws, but the way that we operate is as if everyone were a San Francisco employee because that is—so far—the locale that, one, I live here, but also of every jurisdiction we've looked at in the United States, it tends to have the most advantageous to the employee restrictions and requirements. Like one thing we do is kind of ridiculous—and we have to do for me and one other person, but almost no one else, but we do it for everyone—is we have to provide stipends every month for electricity, for cellphone usage, for internet. They have to be broken out for each one of those categories, so we do 20 bucks a month for each of those. It adds up to 100 bucks, as I recall, and we call it good. And employees say, “Okay. Do we just send you receipts? Please don't.”I don't want to look at your cell phone bill. It's not my business. I don't want to know. We're doing this to comply with the law. I mean, if it were up to me, it would be this is ridiculous. Can we just give everyone $100 a month raise and call it good? Nope. The forms must be obeyed. So, all right.We do the same thing with PTO accrual. If you've acquired time off and you leave the company, we pay it out. Not every state requires that. But paying for cell phone access and internet access as well, is something Amazon is currently facing a class action about because they didn't do that for a number of their California employees. And even talking to Amazonians, like, “Well, they did, but you had to jump through a bunch of hoops.”We have the apparatus administratively to handle that in a way that employees don't. Why on earth would we make them do it unless we didn't want to pay them? Oh, I think I figured out this sneaky, sneaky plan. I'm not here to build a business by exploiting people. If that's the only way to succeed, and the business doesn't deserve to exist. That's my hot take of the day on that topic.Jeff: No, I totally agree. And what's interesting is these insidious costs that sneak up that employees tend to discount, like, one thing I always talk about with my team is all that time you're thinking about a problem at work, right, like when you're in the shower, when you're at dinner, when you're talking it over with your spouse, right? That's work. That's work. And it's work that you're doing on your time.But we don't account for it that way because we're not typing; we're not writing code. But, like, think about how much more effective as people, as employees, we would be if we had time dedicated to just sit and think, right? If I could just sit and think about a problem without needing to type but just critically think about it. But then it's like, well, what does that look like in the office, right? If I'm just sitting there in my chair like this, it doesn't look like I'm doing anything.But that's so important to be able to, like, break down and digest some of the complex problems that we're dealing with. And we just sort of write it off, right? So, I'm like, you know, you got to think about how that bleeds into your personal time and take that into account. So yeah, maybe you leave three hours early today, but I guarantee you, you're going to spend three hours throughout the week thinking about work. It's the same thing with these cellphone costs that you're talking about, right? “Oh, I've got a cell phone anyways; I've got internet anyways.” But still, that's something that you're contributing to the business that they're not on the hook for, so it seems fair that you get compensated for that.Corey: I just think about that stuff all the time from that perspective, and now that I you know, own the place, it's one of those which pocket of mine does it come out of? But I hold myself to a far higher standard about that stuff than I do the staff, where it's, for example, I could theoretically justify paying my internet bill here because we have business-class internet and an insane WiFi system because of all of the ridiculous video production I do. Now. It's like, like, if anyone else on the team was doing this, yes, I will insist we pay it, but for me, it just it feels a little close to the edge. So, it's one of those areas where I'm very conservative around things like that.The thing that also continues to just vex me, on some level, is this idea that time in a seat is somehow considered work. I'll never forget one of the last jobs I had before I started this place. My boss walked past me and saw that I was on Reddit. And, “Is that really the best use of your time right now?” May I use the bathroom when I'm done with this, sir?Yeah, of course it is. It sounds ridiculous, but one of the most valuable things I can do for The Duckbill Group now is go on the internet and start shit posting on Twitter, which sounds ridiculous, but it's also true. There's a brand awareness story there, on some level. And that's just wild to me. It's weird, we start treating people like adults, they start behaving that way. And if you start micromanaging them, they live up or down to the expectations you tend to hold. I'm a big believer in if I have to micromanage someone, I should just do the job myself.Jeff: Yeah. The Reddit story makes me think of, like, how few organizations have systematic ways of getting vital information. So, the first thing I think about is, like, security and security vulnerabilities, right? So, how does Basis Technologies, as an organization, know about these things? Right now, it's like, well, my team knows because we're plugged into Reddit and Twitter, right, but if we were gone Basis, right, may not necessarily get that information.So, that's something we're trying to correct, but it just sort of highlights the importance of freedom for these employees, right? Because yeah, I'm on Reddit, but I'm on /r/sysadmin. I'm on /r/AWS, right, I'm on /r/Atlassian. Now I'm finding out about this zero-day vulnerability and it's like, “Oh, guys, we got to act. I just heard about this thing.” And people are like, “Oh, where did this come from?” And it's like it came from my network, right? And my network—Corey: Mm-hm.Jeff: Is on Twitter, LinkedIn, Reddit. So, the idea that someone browsing the internet on any site, really, is somehow not a productive use of their time, you better be ready to itemize exactly what that means and what that looks like. “Oh, you can do this on Reddit but you can't do that on Reddit.”Corey: I have no boss now, I have no oversight, but somehow I still show up with a work ethic and get things done.Jeff: Right. [laugh].Corey: Wow, I guess I didn't need someone over my shoulder the whole time. Who knew?Jeff: Right. That's all that matters, right? And if you do it in 30 hours or 40 hours, that doesn't really matter to me, you know? You want to do it at night because you're more productive there, right, like, let's figure out a way to make that happen. And remote work is actually empowering us ways to really retain people that wasn't possible before I had an employee that was like, you know, I really want to travel. I'm like, “Dude, go to Europe. Work from Europe. Just do it. Work from Europe,” right? We've got senior leaders on the C-suite that are doing it. One of the chief—Corey: I'm told they have the internet, even there. Imagine that?Jeff: Yeah. [laugh]. So, our chief program officer, she was in Greece for four weeks. And it worked. It worked great. They had a process. You know, she would spent one week on and then one week off on vacation. But you know, she was able to have this incredible, long experience, and still deliver. And it's like, you know, we can use that as a model to say, like—Corey: And somehow the work got done. Wow, she must be amazing. No, that's the baseline expectation that people can be self-managing in that respect.Jeff: Right.Corey: They aren't toddlers.Jeff: So, if she can do that, I'm sure you can figure out how to code in China or wherever you want to visit. So, it's a great way to stay ahead of some of these companies that have a bit more lethargic policies around that stuff, where it's like, you know, all right, I'm not getting that insane salary, but guess what, I'm going to spend three weeks in New Zealand hanging out and not using any time off or anything like that, and you know, being able to enjoy life. I wish this pandemic had happened pre-kids because—Corey: Yeah. [laugh].Jeff: —you know, we would really take advantage of this.Corey: You and me both. It would have very different experience.Jeff: Yeah. [laugh]. Absolutely, right? But with kids in school, and all that stuff, we've been tethered down. But man, I you know, I want to encourage the young people or the single people on my team to just, like, hey, really, really embrace this time and take advantage of it.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it's still somebody's problem. And a big part of that responsibility is app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you're actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That's S-N-Y-K.co/screamCorey: One last topic I want to get into before we call it an episode is, I admit, I read an awful lot of books, it's a guilty pleasure. And it's easy to fall into the trap, especially when you know the author, of assuming that snapshot of their state of mind at a very fixed point in time is somehow who they are, like a fly frozen in amber, and it's never true. So, my question for you is, quite simply, what have you learned since your book came out?Jeff: Oh, man, great question. So, when I was writing the book, I was really nervous about if my audience was as big as I thought it was, the people that I was targeting with the book.Corey: Okay, that keeps me up at night, too. I have no argument there.Jeff: Yeah. You know what I mean?Corey: Please, continue.Jeff: I'm surrounded, you know, by—Corey: Is anyone actually listening to this? Yeah.Jeff: Right. [laugh]. So, after the book got finished and it got published, I would get tons of feedback from people that so thoroughly enjoyed the book, they would say things like, you know, “It feels like you were in our office like a fly on the wall.” And that was exciting, one, because I felt like these were experiences that sort of resonated, but, two, it sort of proved this thesis that sometimes you don't have to do something revolutionary to be a positive contribution to other people, right? So, like, when I lay out the tips and things that I do in the book, it's nothing earth-shattering that I expect Google to adopt. Like, oh, my God, this is the most unique view ever.But being able to talk to an audience in a way that resonates with them, that connects with them, that shows that I understand their problem and have been there, it was really humbling and enlightening to just see that there are people out there that they're not on the bleeding edge, but they just need someone to talk to them in a language that they understand and resonate with. So, I think the biggest thing that I learned was this idea that your voice is important, your voice matters, and how you tell your story may be the difference between someone understanding a concept and someone not understanding a concept. So, there's always an audience for you out there as you're writing, whether it be your blog post, the videos that you produce, the podcasts that you make, somewhere there's someone that needs to hear what you have to say, and the unique way that you can say it. So, that was extremely powerful.Corey: Part of the challenge that I found is when I start talking to other people, back in the before times, trying to push them into conference talks and these days, write blog posts, the biggest objection I get sometimes is, “Well, I don't have anything worth saying.” That is provably not true. One of my favorite parts about writing Last Week in AWS is as I troll the internet looking for topics about AWS that I find interesting, I keep coming across people who are very involved in one area or another of this ecosystem and have stories they want to tell. And I love, “Hey, would you like to write a guest post for Last Week in AWS?” It's always invite only and every single one of them has been paid because people die of exposure and I'm not about that exploitation lifestyle.A couple have said, “Oh, I can't accept payment for a variety of reasons.” Great. Pick a charity that you would like it to go to instead because we do not accept volunteer work, we are a for-profit entity. That is the way it works here. And that has been just one of the absolute favorite parts about what I do just because you get to sort of discover new voices.And what I find really neat is that for a lot of these folks, this is their start to writing and telling the story, but they don't stop there, they start telling their story in other areas, too. It leads to interesting career opportunities for them, it leads to interesting exposure that they wouldn't have necessarily had—again, not that they're getting paid in exposure, but the fact that they are able to be exposed to different methodologies, different ways of thinking—I love that. It's one of my favorite parts about doing what I do. And it seems to scale a hell of a lot better than me sitting down with someone for two hours to help them build a CFP that they wind up not getting accepted or whatnot.Jeff: Right. It's a great opportunity that you provide folks, too, because of, like, an instant audience, I think that's one of the things that has made Medium so successful as, like, a blogging platform is, you know, everyone wants to go out and build their own WordPress site and launch it, but then it like, you write your blog post and it's crickets. So, the ability for you to, you know, use your platform to also expose those voices is great and extremely powerful. But you're right, once they do it, it lights a fire in a way that is admirable to watch. I have a person that I'm mentoring and that was my biggest piece of advice I can give. It was like, you know, write. Just write.It's the one thing that you can do without anyone else. And you can reinforce your own knowledge of a thing. If you just say, you know, I'm going to teach this thing that I just learned, just the writing process helps you solidify, like, okay, I know this stuff. I'm demonstrating that I know it and then four years from now, when you're applying for a job, someone's like, “Oh, I found your blog post and I see that you actually do know how to set up a Kubernetes cluster,” or whatever. It's just extremely great and it—Corey: It's always fun. You're googling for how to do something and you find something you wrote five years ago.Jeff: Right, yeah. [laugh]. And it's like code where you're like, “Oh, man, I would do that so much differently now.”Corey: Since we last spoke, one of the things I've been doing is I have been on the hook to write between a one to two-thousand-word blog post every week, and I've done that like clockwork, for about a year-and-a-half now. And I was no slouch at storytelling before I started doing that. I've given a few hundred conference talks in the before times. And I do obviously long Twitter threads in the past and I write reports a lot. But forcing me to go through that process every week and then sit with an editor and go ahead and get it improved, has made me a far better writer, it's made me a better storyteller, I am far better at articulating my point of view.It is absolutely just unlocking a host of benefits that I would have thought I was, oh, I passed all this. I'm already good at these things. And I was, but I'm better now. I think that writing is one of those things that people need to do a lot more of.Jeff: Absolutely. And it's funny that you mentioned that because I just recently, back in April, started to do the same thing I said, I'm going to write a blog post every week, right? I'm going to get three or four in the can, so that if life comes up and I miss a beat, right, I'm not actually missing the production schedule, so I have a steady—and you're right. Even after writing a book, I'm still learning stuff through the writing process, articulating my point of view.It's just something that carries over, and it carries over into the workforce, too. Like, if you've ever read a bad piece of documentation, right, that comes from—Corey: No.Jeff: Right? [laugh]. That comes from an inability to write. Like, you know, you end up asking these questions like who's the audience for this? What is ‘it' in this sentence? [laugh].Corey: Part of it too, is that people writing these things are so close to the problem themselves that the fact that, “Well, I'm not an expert in this.” That's why you should write about it. Talk about your experience. You're afraid everyone's going to say, “Oh, you're a fool. You didn't understand how this works.”Yeah, my lived experiences instead—and admittedly, I have the winds of privilege of my back on this—but it's also yeah, I didn't understand that either. It turns out that you're never the only person who has trouble with a concept. And by calling it out, you're normalizing it and doing a tremendous service for others in your shoes.Jeff: Especially when you're not an expert because I wrote some documentation about the SSL process and it didn't occur to me that these people don't use the AWS command line, right? Like, you know, in our organization, we sort of mask that from them through a bunch of in-house automation. Now we're starting to expose it to them and simple things like oh, you need to preface the AWS command with a profile name. So, then when we're going through the setup, we're like, “Oh. What if they already have an existing profile, right?” Like, we don't want to clobber that.SSo, it just changed the way you write the documentation. But like, that's not something that initially came to mind for me. It wasn't until someone went through the docs, and they're like, “Uh, this is blowing up in a weird way.” And I was like, “Oh, right. You know, like, I need to also teach you about profile management.”Corey: Also, everyone has a slightly different workflow for the way they interact with AWS accounts, and their shell prompts, and the way they set up local dev environments.Jeff: Yeah, absolutely. So, not being an expert on a thing is key because you're coming to it with virgin eyes, right, and you're able to look at it from a fresh perspective.Corey: So, much documentation out there is always coming from the perspective of someone who is intimately familiar with the problem space. Some of the more interesting episodes that I have, from a challenge perspective, are people who are deep technologists in a particular area and they love they fallen in love with the thing that they are building. Great. Can you explain it to the rest of us mere mortals so that we can actually we can share your excitement on this? And it's very hard to get them to come down to a level where it's coherent to folks who haven't spent years thinking deeply about that particular problem space.Jeff: Man, the number one culprit for that is, like, the AWS blogs where they have, like, a how-to article. You follow that thing and you're like, “None of this is working.” [laugh]. Right? And then you realize, oh, they made an assumption that I knew this, but I didn't right?So, it's like, you know, I didn't realize this was supposed to be, like, a handwritten JSON document just jammed into the value field. Because I didn't know that, I'm not pulling those values out as JSON. I'm expecting that just to be, like, a straight string value. And that has happened more and more times on the AWS blog than I can count. [laugh].Corey: Oh, yeah, very often. And then there's other problems, too. “Oh, yeah. Set up your IAM permissions properly.” That's left as an exercise for the reader. And then you wonder why everything's full of stars. Okay.Jeff: Right. Yep, exactly, exactly.Corey: Ugh. It's so great to catch up with you and see what you've been working on. If people want to learn more, where's the best place to find you?Jeff: So, the best place is probably my website, attainabledevops.com. That's a place where you can find me on all the other places. I don't really update that site much, but you can find me on LinkedIn, Twitter, from that jumping off point, links to the book are there if anyone's interested in that. Perfect stocking stuffers. Mom would love it, grandma would love it, so definitely, definitely buy multiple copies of that.Corey: Yeah, it's going to be one of my two-year-old's learning to read books, it'd be great.Jeff: Yeah, it's perfect. You know, you just throw it in the crib and walk away, right? They're asleep at no time. Like I said, I've also been taking to, you know, blogging on Medium, so you can catch me there, the links will be there on Attainable DevOps as well.Corey: Excellent. And that link will of course, be in the show notes. Thank you so much for being so generous with your time. I really do appreciate it. And it's great to talk to you again.Jeff: It was great to catch up.Corey: Really was. Jeff Smith, Director of Product Operations at Basis Technologies. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice or smash the like and subscribe buttons on the YouTubes, whereas if you've hated this podcast, do the exact same thing—five-star review, smash the buttons—but also leave an angry, incoherent comment that you're then going to have edited and every week you're going to come back and write another incoherent comment that you get edited. And in the fullness of time, you'll get much better at writing angry, incoherent comments.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About SharoneI'm Sharone Zitzman, a marketing technologist and open source community builder, who likes to work with engineering teams that are building products that developers love. Having built both the DevOps Israel and Cloud Native Israel communities from the ground up, today I spend my time finding the places where technology and people intersect and ensuring that this is an excellent experience. You can find my talks, articles, and employment experience at rtfmplease.dev. Find me on Twitter or Github as @shar1z.Links Referenced: Personal Twitter: https://twitter.com/shar1z Website: https://rtfmplease.dev LinkedIn: https://www.linkedin.com/in/sharonez/ @TLVCommunity: https://twitter.com/TLVcommunity @DevOpsDaysTLV: https://twitter.com/devopsdaystlv TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: DoorDash had a problem as their cloud native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, competence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud/C-H-R-O-N-O-S-P-H-E-R-E.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance. To learn more visit: snark.cloud/deployandgoCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn and I have been remiss by not having today's guest on years ago because back before I started this ridiculous nonsense that, well, whatever it is you'd call what I do for a living, I did other things instead. I did the DevOps, which means I was sad all the time. And the thing that I enjoyed was the chance to go and speak on conference stages. One of those stages, early on in my speaking career, was at DevOpsDays Tel Aviv.My guest today is Sharone Zitzman, who was an organizer of DevOpsDays Tel Aviv, who started convincing me to come back. And today is in fact, in the strong tradition here of making up your own job titles in ways that make people smile, she is the Chief Manual Reader at RTFM Please Ltd. Sharone, thank you for joining me.Sharone: Thank you for having me, Corey. Israelis love the name of my company, but Americans think it has a lot of moxie and chutzpah. [laugh].Corey: It seems a little direct and aggressive. It's like, oh, good, you are familiar with how this is going to go. There's something to be said for telling people what you do on the tin upfront. I've never been a big fan of trying to hide that. I mean, the first iteration of my company was the Quinn Advisory Group because I thought, you know, let's make it look boring and sedate and like I can talk to finance people. And yeah, that didn't last more than ten seconds of people talking to me.Also, in hindsight, the logo of a big stylized Q. Yeah, I would have had to change that anyway, for the whole QAnon nonsense because I don't want to be mistaken for that particular brand of nuts.Sharone: Yeah, I decided to do away with the whole formalities and upfront, just go straight [laugh]. For the core of who we are, Corey; you are very similar in that. So, yes. Being a dev first company, I thought the developers would appreciate such a title and name for my company. And I have to give a shout out here to Avishai Ish-Shalom, who's my friend from the community who you also know from the DevOpsDays community.Corey: Oh, yeah @nukemberg on Twitter—Sharone: Yes exactly.Corey: For those who are not familiar.Sharone: [laugh]. Yep. He coined the name.Corey: The problem that I found is that people when they start companies or they manage their careers, they don't bias for the things that they're really good at. And it took me a long time to realize this, I finally discovered, “Ah, what am I the best at? That's right, getting myself fired for my personality, so why don't I build a business where that stops being a liability?” So, I started my own company. And I can tell this heroic retcon of what happened, but no, it's because I had nowhere else to go at that point.And would you hire me? Think about this for a minute. You, on the other hand, had options. You are someone with a storied history in community building, in marketing to developers without that either coming across as insincere or that marked condescending accent that so many companies love to have of, “Oh, you're a developer. Let me look at you and get down on my hands and knees like we're going camping and tell a story in ways that actively and passively insult you.”No, you have always gotten that pitch-perfect. The world was your oyster. And for some godforsaken reason, you looked around and decided, “Ah, I'm going to go out independently because you know what I love? Worrying.” Because let's face it, running your own company is an exercise in finding new and exciting things to worry about that 20 minutes ago, you didn't know existed. I say this from my own personal experience. Why would you ever do such a thing?Sharone: [laugh]. That's a great question. It was a long one, but a good one. And I do a thing where I hit the mic a lot because I also have. I can't control my hand motions.Corey: I too speak with my hands. It's fine.Sharone: [laugh]. Yeah, so it's interesting because I wanted to be independent for a really long time. And I wasn't sure, you know, if it was something that I could do if I was a responsible enough adult to even run my own company, if I could make it work, if I could find the business, et cetera. And I left the job in December 2020, and it was the first time that I hadn't figured out what I was doing next yet. And I wanted to take some time off.And then immediately, like, maybe a week after I started to get a lot of, like, kind of people reaching out. And I started to interview places and I started to look into possibly being a co-founder at places and I started to look at all these different options. And then just, I was like, “Well…. This is an opportunity, right? Maybe I should finally—that thing that's gnawing at the back of my head to see if, like, you know if I should go for this dream that I've always wanted, maybe now I can just POC it and see if, you know, it'll work.”And it just, like, kind of exploded on me. It was like there was so much demand, like, I just put a little, like, signal out to the world that this is something that I'm interested in doing, and everyone was like, “Ahh, I need that.” [laugh]. I wanted to take a quarter off and I signed my first clients already on February 1st, which was, like, a month after. I left in December and that—it was crazy. And since then, I've been in business. So, yeah. So, and since then, it's also been a really crazy ride; I got to discover some really exciting companies. So.Corey: How did you get into this? I found myself doing marketing-adjacent work almost entirely by accident. I started the newsletter and this podcast, and I was talking to sponsors periodically and they'd come back with, “Here's the thing we want you to talk about in the sponsor read.” And it's, “Okay, you want to give people a URL to go to that has four sub-directories and entire UTM code… okay, have you considered, I don't know, not?” And because so much of what they were talking about did not resonate.Because I have the engineering background, and it was, I don't understand what your company does and you're spending all your time talking about you instead of my painful problem. Because as your target market, I don't give the slightest of shits about you, I care about my problem, so tell me how you're going to solve my problem and suddenly I'm all ears. Spend the whole time talking about you, and I could not possibly care less and I'll fast-forward through the nonsense. That was my path to it. How did you get into it?Sharone: How did I get into it? It's interesting. So, I started my journey in typical marketing, enterprise B2B marketing. And then at GigaSpaces, we kickstarted the open-source project Cloudify, and that's when I found myself leading this project as the open-source community team leader, building, kind of, the community from the ground floor. And I discovered a whole new world of, like, how to build experience into your marketing, kind of making it really experiential and making sure that everyone has a really, really easy and frictionless way of using your product, and that the product—putting the product at the center and letting it speak for itself. And then you discover this whole new world of marketing where it's—and today, you know, it has more of a name and a title, PLG, and people—it has a whole methodology and practice, but then it was like we were—Corey: PLG? I'm unfamiliar with the acronym. I thought tech was bad for acronyms.Sharone: Right? [laugh]. So, product-led growth. But then, you know, like, kind of wasn't solidified yet. And so, a lot of what we were doing was making sure that developers had a really great experience with the product then it kind of sold itself and marketed itself.And then you understood what they wanted to hear and how they wanted to consume the product and how they wanted it to be and to learn about it and to kind of educate themselves and get into it. And so, a lot of the things that I learned in the context of marketing was very guerilla, right, from the ground up and kind of getting in front of people and in the way they wanted to consume it. And that taught me a lot about how developers consume technology, the different channels that they're involved in, and the different tools that they need in order to succeed, and the different, you know, all the peripheral experience, that makes marketing really, really great. And it's not about what you're selling to somebody; it's making your product shine and making the experience shine, making them ensure that it's a really, really easy and frictionless experience. You know, I like how [Donald Bacon 00:08:00] says it; he calls it, like, mean time to hello world, and that to me is the best kind of marketing, right? When you enable people to succeed very, very quickly.Corey: Yeah, there's something to be said for the ring of authenticity and the rest. Periodically I'll promote guest episodes on this, where it's a sponsored episode where people get up and they talk about what they're working on. And they're like, “Great. So, here's the sales pitch I want to give,” and it's no you won't because first, it won't work. And secondly, I'm sorry, whether it's a promoted episode or not, I will not publish something that isn't good because I have a reputation to uphold here.And people run into challenges an awful lot when they're trying to effectively tell their story. If you have a startup that was founded by an engineer, for example, as so many of these technical startups were, the engineer is often so deeply and profoundly in love with this problem space and the solution and the rest, but if they talk about that, no one cares about the how. I mean, I fix AWS bills, and people don't care—as a general rule—how I do that at all if they're in my target market. They don't care if it's through clever optimization, amazing tooling, doing it on-site, or taking hostages in Seattle. They care about their outcome much more than they ever do about the how.The only people who care about the how are engineers who very often are going to want to build it themselves, or work for you, or start a competitor. And it doesn't resonate in quite the same way. It's weird because all these companies are in slightly different spaces; all of them tend to do slightly different things—or very different things—but so many of the challenges that I see in the way that they're articulating what they do to customers rhymes with one another.Sharone: Yeah. So, I agree completely that developers will talk often about how it works. How it works. How does it work under the hood? What are the bits and bytes, you know?Like, nobody cares about how it works. People care about how will this make my life better, right? How will this improve my life? How will this change my life? [laugh]. As an operations engineer, if I'm, you know, crunching through logs, how will this tool change that? What my days look like? What will my on-call rotation look like? What will—you know, how are you changing my life for the better?So, I think that that's the question. When you learn how to crystallize the answer to that question and you hit it right on the mark—you know, and it takes a long time to understand the market, and to understand the buying persona, and t—and there's so much that you have to do in the background, and so much research you have to do to understand who is that person that needs to have that question answered? But once you do and you crystallize that answer, it lands. And that's the fun part about marketing, really trying to understand the person who's going to consume your product and how you can help them understand that you will make their life better.Corey: Back when I was starting out as a consultant myself, I would tell stories that I had seen in the AWS billing environment, and I occasionally had clients reach out to me, “Hey, why don't you tell our story in public?” It's, “Because that wasn't your story. That was something I saw on six different accounts in the same month. It is something that everyone is feeling.” It's, people think that you're talking about them.So, with that particular mindset on this, without naming specific companies, what themes are you seeing emerging? What are companies getting wrong when they are attempting and failing to market effectively to developers?Sharone: So, exactly what we're talking about in terms of the product pitch, in that they're talking at developers from this kind of marketing speak and this business language that, you know, developers often—you know, unless a company does a really, really good job of translating, kind of, the business value—which they should do, by the way—to engineers, but oftentimes, it's a little bit far from them in the chain, and so it's very hard for them to understand the business fluff. If you talk to them in bits and bytes of this is what my day-to-day developer workflow looks like and if we do these things, it'll cut down the time that I'm working on these things, it'll make these things easier, it'll help streamline whatever processes that are difficult, remove these bottlenecks, and help them understand, like I said, how it improves their life.But the things that I've seen breakdown is also in the authenticity, right? So obviously, the world is built on a lot of the same gimmicks and it's just a matter of whether you're doing it right or not, right? So, there's so much content out there and webcasts and webinars, and I don't know what and podcasts and whatever it is, but a lot of the time, people, their most valuable asset is their time. And if you end up wasting their time, without it being, like, really deeply valuable—if you're going to write content, make sure that there is a valuable takeaway; if you're going to create a webinar, make sure that somebody learned something. That if they're investing their time to join your marketing activities, make sure that they come away with something meaningful and then they'll really appreciate you.And it's the same idea behind the whole DevOpsDays movement with the law of mobility and open spaces that people if they find value, they'll join this open space and they'll participate meaningfully and they'll be a part of your event, and they'll come back to your event from year to year. But if you're not going to provide that tangible value that somebody takes away, and it's like, okay, well, I can practically apply this in my specific tech stack without using your tool, without having to have this very deterministic or specific kind of tech stack that they're talking about. You want to give people something—or even if it is, but even how to do it with or without, or giving them, like, kind of practical tools to try it. Or if there's an open-source project that they can check out first, or some kind of lean utility that gives them a good indication of the value that this will give them, that's a lot more valuable, I think. And practically understandable to somebody who wants to eventually consume your product or use your products.Corey: The way that I see things, at least in the past couple of years, the pandemic has sharpened an awful lot of the messaging that needs to happen. Because in most environments, you're sitting at a DevOpsDays in the front row or whatnot, and it's time for the sponsor talks and someone gets up and starts babbling and wasting your time, most people are not going to get up and leave. Okay, they will in Israel, but in most places, they're not going to get up and leave, whereas in pandemic land, it's you are one tab away from something I actually want—Sharone: Exactly.Corey: To be doing, so if you become even slightly boring, it's not going to go well. So, you have to be on message, you have to be on point or no one cares. People are like, “Oh, well what if we say the wrong thing and people wind up yelling about us on Twitter?” It's like unless it is for something horrifying, you should be so lucky because people are then talking about you. The failure mode isn't that people don't like your product, it's no one talks about it.Sharone: Yeah. No such thing as bad publicity [crosstalk 00:14:32] [laugh]—Corey: Oh, there very much is such a thing is bad publicity. Like, “I could be tweeting about your product most days,” is apparently a version of that, according to some folks. But it's a hard problem to solve for. And one of the things that continually surprises me is the things I'm still learning about this entire industry. The reason that people sponsor this show—and the rates they pay, to be direct—have little bearing to the actual size of the audience—as best we can tell; lies, damn lies, and podcast statistics; if you're listening to this, let me know. I'd love to know if anyone listens to this nonsense—but when you see all of that coming out, why are we able to charge the rates that we do?It's because the long-term value of someone who is going to buy a long-term subscription or wind up rolling out something like ChaosSearch or whatnot that is going to be a fundamental tenet of their product, one prospect becoming a customer pays for anything, I can sell a company, it will sponsor—they can pay me to sponsor for the next ten years, as opposed to the typical mass-market audience where well, I'm here to sling Casper mattresses today or something. It's a different audience and there's a different perception there. People are starting to figure out the value of—in an age where tracking is getting harder and harder to do and attribution will drive you nuts, instead of go where your audience is. Go where the people who care about the problem that you have and will experience that problem are going to hang out. And it always is wild to me to see companies missing out on that.It's, “Okay, so you're going to do a $25 million billboard ad in spotted in airports around the world talking about your company… but looking at your billboard, it makes no sense. I don't understand what it's there for.” Even as a brand awareness play, it fails because your logo is tiny in the corner or something. It's you spent that much money on ads, and maybe a buck on messaging because it seems like with all that attention you just bought, you had nothing worthwhile to say. That's the cardinal sin to me at least.Sharone: Yeah. One thing that I found—and back to our community circuit and things that we've done historically—but that's one thing that, you know, as a person comes from community, I've seen so much value, even from the smaller events. I mean, today, like with Covid and the pandemic and everything has changed all the equilibrium and the way things are happening. But some meetups are getting smaller, face-to-face events are getting smaller, but I've had people telling me that even from small, 30 to 40 people events, they'll go up and they'll do a talk and great, okay, a talk; everybody does talks, but it's like, kind of, the hallway track or the networking that you do after the talk and you actually talk to real users and hear their real problems and you tap into the real community. And some people will tell me like, I had four concrete leads from a 30-person meet up just because they didn't even know that this was a real challenge, or they didn't know that there was a tool that solves this problem, or they didn't understand that this can actually be achieved today.Or there's so many interesting technologies and emerging technologies. I'm privileged to be able to be at the forefront of that and discover it all, and I if I could, I would drop names of all of the awesome companies that work for me, that I work with, and just give them a shout out. But really, there's so many amazing companies doing, like, developer metrics, and all kinds of troubleshooting and failure analysis that's, like, deeply intelligent—and you're going to love this one: I have a Git replacement client apropos to your closing keynote of DevOpsDays 2015—and tapping into the communities and tapping into the real users.And sometimes, you know, it's just a matter of really understanding how developers are working, what processes look like, what workflows look like, what teams look like, and being able to architect your products and things around real use cases. And that you can only discover by really getting in front of actual users, or potential users, and learning from them and feedback loops, and that's the little core behind DevRel and developer advocacy is really understanding your actual users and your consumers, and encouraging them to you know, give you feedback and try things, and beta programs and a million things that are a lot more experiential today that help you understand what your users need, eventually, and how to actually architect that into your products. And that's the important part in terms of marketing. And it's a whole different marketing set. It's a whole different skill set. It's not talking at people, it's actually… ingesting and understanding and hearing and implementing and bringing it into your products.Corey: And it takes time. And you have to make yourself synonymous with a painful problem. And those problems are invariably very point-in-time specific. I don't give a crap about log aggregation today, but in two weeks from now, when I'm trying to chase down 18 different Lambdas function trying to figure out what the hell's broken this week, I suddenly will care very much about log aggregation. Who was that company that's in that space that's doing interesting things? And maybe it's Cribl, for example; they do a lot of stuff in that space and they've been a good sponsor. Great.I start thinking about those things in that light because it is—when I started having these problems, it sticks in your head and it resonates. And there's value and validity to that, but you're never going to be able to attribute that either, which is where people often lose their minds. Because for anything even slightly complicated—you're going to be selling things to big bank—great, good on you. Most of those customers are not going to go and spin up a trial in the dead of night. They're going to hear about you somewhere and think, “Ohh, this is interesting.”They're going to talk about a meeting, they're going to get approval, and at that point, you have long since lost any tracking opportunity there. So, the problem is that by saying it like this, as someone who is a publisher, let's be very clear here, it sounds like you're trying to justify your entire business model. I feel like that half the time, but I've been reassured by people who are experts in doing these things, like, oh, yeah, we have data on this; it's working. So, the alternative is either I accept that they're right or I sit here and arrogantly presume I know more about marketing than people who've devoted their entire careers to it. I'm not that bold. I am a white guy in tech, but not that much.Sharone: Yeah, I mean, the DevRel measurement problem is a known problem. We have people like [unintelligible 00:20:21] who have written about it. We have [Sarah Drasner 00:20:23], we have a million people that have written really, really great content about how do you really measure DevRel and the quality. And one of the things that I liked, Philipp Krenn, the dev advocate at Elastic once said in one of his talks that, you know, “If you're measuring your developer advocates on leads, you're a marketing organization. If you're measuring them on revenue, you're a sales organization. It's about reach, engagement, and awareness, and a lot of things that it's much, much harder to measure.”And I can say that, like, once upon a time, I used to try and attribute it at Cloudify. Like, I remember thinking, like, “Okay, maybe I could really track this back to, you know, the first touch that I actually had with this user.” It's really, really difficult, but I do remember, like, when we used to go out into the events and we were really active in the OpenStack community, in the DevOps community, and many other things, and I remember, like, even after events, like, you get all those lead gen emails. All I would say now is, like, “Hey, if you missed us at the booth, you know, and you want still want a t-shirt, you know, reach out and I'll ship it to you.” And some of those eventually, after we continued the relationship, and we, you know, when we were friends and community friends, six months later, when they moved to their next role at their next job, they were like, “Oh, now I have an opportunity to use Cloudify and I'm going to check it out.”And it's very long relationship that you have to cultivate. It has to be, you know, mutual. You have to be, you have to give be giving something and eventually is going to come back to you. Good deeds come back to you. So, I—that's my credo, by the way, good deeds come back to you. I believe in that and I try to live by that.Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: So, I have one last question for you and it is pointed and the reason I buried it this deep in the episode is so that if I open with it, I will get letters and I'm hoping to get fewer of them. But I met you, again, at DevOpsDays Tel Aviv, and it was glorious. And then you said, “This is fun. Come help me organize it next year.”And I, like an idiot said, “Sure, that sounds awesome because I love going to conferences and it's great. So, what's involved?” “Oh, a whole bunch of meetings.” “Okay, great.” “And planning”—things I'm terrible at—“Okay.” And then the big day finally arrives where, “Great, when do we get to get on stage and tell a story?” Like, “That's the neat part. We don't.” So, I have to ask, given that it is all behind-the-scenes work that is fairly thankless unless you really screw it up because then it's very visible, what is the point of being so involved in the community?Sharone: Wow, that's a big question, Corey.Corey: It really is.Sharone: [laugh].Corey: Because you've been involved in community for a long time and you're very good at it.Sharone: It's true. It's true. Appreciate it, thank you. So, for me, first of all, I enjoy, kind of, the people aspect of it, absolutely. And that people aspect of it actually has played out in so many different ways.Corey: Oh, you mean great people, and also me.Sharone: [laugh]. Particularly you, Corey, and we will bring you back. [laugh]. And we will make sure you chop wood and carry water because eventually it'll fill your soul, you'll see. [laugh] one of the things that really I have had the privilege and honor, and having come out of, like, kind of all my community work is really the network I've built and the people that I've met.And I've learned so much and I've grown so much, but I've also had the opportunity to connect people, connect things that you wouldn't imagine, un—seemingly-related things. So, there are so many friends of mine that have grown up with me in this community, it's been already ten years now, and a lot of folks have now been going on to new adventures and are looking to kickstart their new startup and I can connect them to this investor, I can connect them to this other person who is maybe a good, you know, partner for their startup, and hiring opportunities, and something—I've had this, like, privilege of kind of being able to connect Israel to the outer world and other things and the global kind of community, and also bring really intelligent folks into the community. And this has just created this amazing flywheel of opportunity that I'm really happy to be at the center of. And I think I've grown as a person, I think our community has grown, has learned, and there's a lot of value in that, I think, yeah. We got to meet wonderful folks like you, Corey. [laugh].Corey: It has its moments. Again, you're one of those rarities in that it's almost become a trope in VC land where VCs always like, “How may I be useful?” And it's this self-serving transparent thing. Every single time you have deigned to introduce me to someone, it's been a productive conversation and I'm always glad I took the meeting. That is no small thing.A lot of people say, “I'm good at community,” which is sort of cover for, “I'm not good at anything,” but in your case, it—Sharone: [laugh]. [I'm an entrepreneur 00:24:48].—Corey: Is very much not true. Oh, yeah. I'm a big believer that ‘entrepreneur' and ‘hero' and other terms like that are things people call you; you don't call yourself that. It always feels weird for, “Oh, he's an entrepreneur.” It's like, that's a pretty lofty word for shitposting, but okay, we'll roll with it.It doesn't work that way. You've clearly invested long-term in a building reputation for yourself by building a name for yourself in the space, and I know that whenever you reach out to me as a result, you are not there to waste my time or shill some bullshit. It is always something that is going to, even if I don't love every aspect of it or agree with the core of the message you're sending, great, it is never not going to be worth my time, which is why I'm so glad I got the chance to talk to you this show.Sharone: I appreciate that. It's something that I really believe in, I don't want to waste people's time and I really only will connect folks or only really will reach out to someone if I do think that there's something meaningful for both sides. It's never only what's in it for me, also. I also want to make sure that there's something in it for the other person and it's something that makes sense and it's meaningful for both sides. I've had the opportunity of meeting such interesting folks, and sometimes it's just like, “You must meet. [laugh]. You will love each other.” You will have so much to do together or it's so much collaboration opportunity.And so yeah, I really am that type of person. And I'll even say from a personal perspective, you know, I know a lot of people, and I've even been asked from the flip side, “Okay, is this a toxic manager? Or is this a, you know, a good hire? Is this”—and I tried to provide really authentic input so people make the right decisions, or make, you know, the right contacts, or make—and that's something I really value. And I managed to build trust with a lot of really great folks—Corey: And also me—Sharone: —and it's come back to me, also. And—[laugh] and particularly you, again. [laugh].Corey: If people want to learn more about how you see the world and the space and otherwise bask in your wisdom, where's the best place to find you?Sharone: So, I'm on Twitter as @shar1z, which is SharoneZ. Basically, everyone thinks it's such a smart, or I don't know what, like, or an esoteric screen name. And I'm like, no, it's just my name, I just—the O-N-E is… the one. [laugh].So yes, shar1z on Twitter, but also my website, rtfmplease.dev, you can reach out, there's a contact form there. You can find me on the web anywhere—LinkedIn. Reach out, I answer almost all my DMs when I can. It's very rare that I don't answer DMs. Maybe there'll be a slight lag, but I do. And I really do like when folks reach out to me. I do like it when people try and make contact.Corey: And you can also be found, of course, wherever find DevOps products are sold, on stage apparently.Sharone: [laugh]. The DevOps community, that's right. @TLVCommunity, @DevOpsDaysTLV—don't out me. All those are—yes, those are also handles that I run on Twitter, it's true.Corey: Excellent.Sharone: So, when you see them all retweeting the same tweet, yes, it's happening within same five minutes, it's me.Corey: Oh, that would have made it way easier to go viral. My God, I should have just thought of that earlier.Sharone: [laugh].Corey: Thank you so much for your time. I appreciate it.Sharone: Thank you, Corey, for having me. It's been a privilege and honor being on your show and I really do think that you are doing wonderful things in the cloud space. You're teaching us, and we're all learning, and you—keep up the good work.Corey: Well, thank you. I appreciate that.Sharone: I also want to add that on proposed marketing and whatever, I do actually listen to all of your openings of all of your shows because they're not fluffy and I like that you do, like, kind of a deep explanation, a deep technical explanation of what your sponsoring product does, and it gives a lot more insight into why is this important. So, I think you're doing that right. So, anybody who's sponsoring this show, listen. Corey knows what he's doing.Corey: Well, thank you. I appreciate that. Yay, “I know what I'm doing.” That one's going in the testimonial kit. My God.Sharone: [laugh]. That's the name of this episode, “Corey knows what he's doing.”Corey: We're going to roll with it, you know. No take-backsies. Sharone Zitzman, Chief Manual Reader at RTFM Please. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review of your podcast platform of choice, or if it's on the YouTubes smash the like and subscribe buttons, whereas if you've hated this show, exact same thing—five-star review wherever you happen to find it, smash both the buttons—but also leave an insulting comment telling me that I'm completely wrong which then devolves into an 18-page diatribe about exactly how your nonsense, bullshit product is built and works.Sharone: [laugh].Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About WesleyWesley Faulkner is a first-generation American, public speaker, and podcaster. He is a founding member of the government transparency group Open Austin and a staunch supporter of racial justice, workplace equity, and neurodiversity. His professional experience spans technology from AMD, Atlassian, Dell, IBM, and MongoDB. Wesley currently works as a Developer Advocate, and in addition, co-hosts the developer relations focused podcast Community Pulse and serves on the board for SXSW.Links Referenced: Twitter: https://twitter.com/wesley83 Polywork: https://polywork.com/wesley83 Personal Website: https://www.wesleyfaulkner.com/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Finding skilled DevOps engineers is a pain in the neck! And if you need to deploy a secure and compliant application to AWS, forgettaboutit! But that's where DuploCloud can help. Their comprehensive no-code/low-code software platform guarantees a secure and compliant infrastructure in as little as two weeks, while automating the full DevSecOps lifestyle. Get started with DevOps-as-a-Service from DuploCloud so that your cloud configurations are done right the first time. Tell them I sent you and your first two months are free. To learn more visit: snark.cloud/duplo. Thats's snark.cloud/D-U-P-L-O-C-L-O-U-D.Corey: What if there were a single place to get an inventory of what you're running in the cloud that wasn't "the monthly bill?" Further, what if there were a way to compare that inventory to what you were already managing via Terraform, Pulumi, or CloudFormation, but then automatically add the missing unmanaged or drifted parts to it? And what if there were a policy engine to immediately flag and remediate a wide variety of misconfigurations? Well, stop dreaming and start doing; visit snark.cloud/firefly to learn more.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I am joined again for a second time this year by Wesley Faulkner. Last time we spoke, he was a developer advocate. And since then, as so many have, he's changed companies. Wesley, thank you for joining me again. You're the Head of Community at SingleStore, now. Congrats on the promotion.Wesley: Thank you. It's been a very welcome change. I love developer advocates and developer advocacy. But I love people, too, so it's almost, I think, very analogous to the ebbs and flow that we all have gone through, through the pandemic, and leaning into my strong suits.Corey: It's a big deal having a ‘head of' in a role title, as opposed to Developer Advocate, Senior Developer Advocate. And it is a different role. It's easy to default into the world of thinking that it's a promotion. Management is in many ways orthogonal to what it takes to succeed in an actual role. And further, you're not the head of DevRel, or DevRelopers or whatever you want to call the term. You are instead the Head of Community. How tied is that to developer relations, developer advocacy, or other things that we are used to using as terms of art in this space?Wesley: If we're talking about other companies, I would say the Head of Community is something that's under the umbrella of developer relations, where it's just a peer to some of the other different elements or columns of developer relations. But in SingleStore specifically, I have to say that developer relations in terms of what you think about whole umbrella is very new to the company. And so, I consider myself the first person in the role of developer relations by being the Head of Community. So, a lot of the other parts are being bolted in, but under the focus of developer as a community. So, I'm liaisoning right now as helping with spearheading some of the design of the activities that the advocates do, as well as architecting the platform and the experiences of people coming in and experiencing SingleStore through the community's perspective.So, all that to say is, what I'm doing is extremely structured, and a lot of stuff that we're doing with the efficacy, I'm using some of my expertise to help guide that, but it's still something that's kind of like an offshoot and not well integrated at the moment.Corey: How has it changed the way that you view the function of someone who's advocating to developers, which is from my cynical perspective, “Oh, it's marketing, but we don't tell people it's marketing because they won't like it.” And yes, I know, I'll get emails about that. But how does it differ from doing that yourself versus being the head of the function of a company? Because leadership is a heck of a switch? I thought earlier in my career that oh, yeah, it's a natural evolution of being a mediocre engineer. Time to be a mediocre manager. And oh, no, no, I aspired to be a mediocre manager. It's a completely different skill set and I got things hilariously wrong. What's it like for you going through that shift?Wesley: First of all, it is kind of like advertising, and people may not think of it that way. Just to give an example, movie trailers is advertising. The free samples at the grocery store is advertising. But people love those because it gives an experience that they like in a package that they are accustomed to. And so, it's the same with developer relations; it's finding the thing that makes the experience worthwhile.On the community side, this is not new to me. I've done several different roles, maybe not in this combination. But when I was at MongoDB, I was a technical community manager, which is like a cog in the whole giant machine. But before that, in my other life, I managed social and community interactions for Walmart, and I had, at the slow period, around 65, but during the holidays, it would ramp up to 95 direct reports that I managed.It's almost—if you're a fan of The Princess Bride, it's different than fighting one person. Sometimes it's easier to fight, like, a squad or a gang of people. So, being Head of Community with such a young company is definitely a lot different than. In some ways, harder to deal with this type of community where we're just growing and emerging, rather than something more well-established.Corey: It probably gives you an interesting opportunity. Because back when I was doing engineering work as an SRE or whatever we call them in that era, it was, “Yeah, wow, my boss is terrible and has no idea what the hell they're doing.” So, then I found myself in the role, and it's, “Cool. Now, do all the things that you said you would do. Put up or shut up.”And it turns out that there's a lot you don't see that our strategic considerations. I completely avoided things like managing up or managing laterally or balancing trade-offs in different ways. Yeah, you're right. If you view the role of management as strictly being something that is between you and your direct reports, you can be an amazing manager from their perspective, but completely ineffective organizationally at accomplishing the goals that have been laid out for you.Wesley: Yeah. The good thing about being head of and the first head of is that you help establish those goals. And so, when you take a role with another company saying, “Hey, we have headcount for this,” and it's an established role, then you're kind of like streamlining into a process that's already underway. What's good about this role specifically, a ‘head of,' is that I help with not only designing what are the goals and the OKRs but deciding what the teams and what the team structure should look like. And so, I'm hiring for a specific position based on how it interacts with everything else.So, when I'm coming in, I don't say, “Well, what do you do?” Or, “How do you do it?” I said, “This is what needs to be done.” And that makes it so much easier just to say that if everything is working the way it should and to give marching orders based on the grand vision, instead of hitting the numbers this quarter or next quarter. Because what is core to my belief, and what's core, too, of how I approach things is at the heart of what I'm trying to do, which is really great, in terms of making something that didn't exist before.Corey: The challenge, too, is that everyone loves to say—and I love to see this at different ways—is the evolution and understanding of the DevRel folks who I work with and I have great relationships with realizing that you have to demonstrate business value. Because I struggle with this my entire career where I know intrinsically, that if I get on stage and tell a story about a thing that is germane to what my company does, that good things are going to happen. But it's very hard to do any form of attribution to it. In a different light, this podcast is a great example of this.We have sponsors. And people are listening. Ideally, they aren't fast-forwarding through sponsor messages; I do have interesting thoughts about the sponsors that I put into these ads. And that's great, but I also appreciate that people are driving while they're listening to this, and they are doing the dishes, they are mowing the lawn, and hopefully not turning up the volume too loudly so it damages their hearing. And the idea that they're going to suddenly stop any of those things and go punch in the link that I give is a little out to lunch there.Instead, it's partially brand awareness and it is occasionally the, “Wait. That resonates exactly with the problem that I have.” So, they get to work or they get back in front of a computer and the odds are terrific they're not going to punch in that URL of whatever I wound up giving; they're going to type in whatever phrases they remember and the company name into Google. Now—and doing attribution on something like that is very hard.It gets even more hard when we're talking about something that is higher up the stack that requires a bit more buy-in than individual developers. There's often a meeting or two about it. And then someone finally approaches the company to have a conversation. Now, does it work? Yes. There are companies that are sponsoring this stuff that spend a lot of time, effort, and money on that.I don't know how you do that sort of attribution; I don't pretend to know, but I know that it works. Because these people whose entire job is making sure that it does tell me it does. So, I smile, I nod, and that's great. But it's very hard to wind up building out a direct, “If you spend X dollars sponsoring this, you will see Y dollars in response.” But in the DevOps world, when your internal doing these things, well, okay because to the company, I look an awful lot like an expensive developer except I don't ever write production code.And then—at least in the before times—“So, what does your job do? Because looking at the achievements and accomplishments last quarter, it looks an awful lot like you traveled to exotic places on the company dime, give talks that are of only vague relevance to what we do, and then hang out at parties with your friends? Nice job, how can I get that?” But it's also first on the chopping block when okay, how do we trim expenses go? And I think it's a mistake to do that. I just don't think that story of the value of developer relations is articulated super-well. And I say that, but I don't know how to do a much better job of it myself.Wesley: Well, that's why corporate or executive buy-in is important because if they know from the get-go while you're there, it makes it a little bit easier to sell. But you do have to show that you are executing. So, there are always two parts to presenting a story, and that's one, the actual quantitative, like, I've done this many talks—so that output part—I've written this many blog posts, or I've stood up this many events that people can attend to. And then there's the results saying, people did read this post, people did show up to my event, people did listen to my talk that I gave. But you also need to give the subjective ones where people respond back and say, “I loved your talk,” or, “I heard you on Corey's podcast,” or, “I read your blog posts,” because even though you might not understand that it goes all the way down in a conversion funnel to a purchase, you can least use that stand-in to say there's probably, like, 20, 30 people behind this person to have that same sentiment, so you can see that your impact is reaching people and that it's having some sort of lasting effect.That said, you have to keep it up. You have to try to increase your output and increase your sphere of influence. Because when people go to solve their problem, they're going to look into their history and their own Rolodex of saying what was the last thing that I heard? What was the last thing that's relevant?There is a reason that Pepsi and Coke still do advertising. It's not because people don't know those brands, but being easily recalled, or a center of relevance based on how many touchpoints or how many times that you've seen them, either from being on American Idol and the logo facing the camera, or seeing a whole display when you go into the grocery store. Same with display advertising. All of this stuff works hand in hand so that you can be front-of-mind with the people and the decision-makers who will make that decision. And we went through this through the pandemic where… that same sentiment, it was like, “You just travel and now you can't travel, so we're just going to get rid of the whole department.”And then those same companies are hunting for those people to come back or to rebuild these departments that are now gone because maybe you don't see what we do, but when it's gone, you definitely notice a dip. And that trust is from the top-up. You have to do not just external advocacy, but you have to do internal advocacy about what impacts you're having so that at least the people who are making that decision can hopefully understand that you are working hard and the work is paying off.Corey: Since the last time that we spoke, you've given your first keynote, which—Wesley: Yes.Corey: Is always an interesting experience to go through. It was at a conference called THAT Conference. And I feel the need to specify that because otherwise, we're going to wind up with a ‘who's on first' situation. But THAT Conference is the name.Wesley: Specify THAT. Yes.Corey: Exactly. Better specify THAT. Yes. So, what was your keynote about? And for a bit of a behind-the-scenes look, what was that like for you?Wesley: Let me do the behind-the-scenes because it's going to lead up to actual the execution.Corey: Excellent.Wesley: So, I've been on several different podcasts. And one of the ones that I loved for years is one called This Week in Tech with Leo Laporte. Was a big fan of Leo Laporte back in the Screen Saver days back in TechTV days. Loved his opinion, follow his work. And I went to a South by Southwest… three, four years ago where I actually met him.And then from that conversation, he asked me to be on his show. And I've been on the show a handful of times, just talking about tech because I love tech. Tech is my passion, not just doing it, but just experiencing and just being on either side of creating or consuming. When I moved—I moved recently also since, I think, from the last time I was on your show—when I moved here to Wisconsin, the organizer of THAT Conference said that he's been following me for a while, since my first appearance on This Week in Tech, and loved my outlook and my take on things. And he approached me to do a keynote.Since I am now Wisconsin—THAT Conference is been in Wisconsin since inception and it's been going on for ten years—and he wanted me to just basically share my knowledge. Clean slate, have enough time to just say whatever I wanted. I said, “Yes, I can do that.” So, my experience on my end was like sheer excitement and then quickly sheer terror of not having a framework of what I was going to speak on or how I was going to deliver it. And knowing as a keynote, that it would be setting the tone for the whole conference.So, I decided to talk on the thing that I knew the most about, which was myself. Talked about my journey growing up and learning what my strengths, what my weaknesses are, how to navigate life, as well as the corporate jungle, and deciding where I wanted to go. Do I want to be the person that I feel like I need to be in order to be successful, which when we look at structures and examples and the things that we hold on a pedestal, we feel that we have to be perfect, or we have to be knowledgeable, and we have to do everything, well rounded in order to be accepted. Especially being a minority, there's a lot more caveats in terms of being socially acceptable to other people. And then the other path that I could have taken, that I chose to take, was to accept my things that are seen as false, but my own quirkiness, my own uniqueness and putting that front and center about, this is me, this is my person that over the years has formed into this version of myself.I'm going to make sure that is really transparent and so if I go anywhere, they know what they're getting, and they know what they're signing up for by bringing me on board. I have an opinion, I will share my opinion, I will bring my whole self, I won't just be the person that is technical or whimsical, or whatever you're looking for. You have to take the good with the bad, you have to take the I really understand technology, but I have ADHD and I might miss some deadlines. [laugh].Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on premises, private cloud, and they just announced a fully managed service on AWS and Azure called BigAnimal, all one word.Don't leave managing your database to your cloud vendor because they're too busy launching another half dozen manage databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications, including Oracle, to the cloud.To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: I have a very similar philosophy, and how I approach these things where it's there is no single speaking engagement that I can fathom even being presented to me, let alone me accepting that is going to be worth me losing the reputation I have developed for authenticity. It's you will not get me to turn into a shill for whatever it is that I am speaking in front of this week. Conversely, whether it's a paid speaking engagement or not, I have a standing policy of not using a platform that is being given to me by a company or organization to make them look foolish. In other words, I will not make someone regret inviting me to speak at their events. Full stop.And I have spoken at events for AWS; I have spoken at events for Oracle, et cetera, et cetera, and there's no company out there that I'm not going to be able to get on stage and tell an entertaining and engaging story, but it requires me to dunk on them. And that's fine. Frankly, if there is a company like that where I could not say nice things about them—such as Facebook—I would simply decline to pursue the speaking opportunity. And that is the way that I view it. And very few companies are on that list, to be very honest with you.Now, there are exceptions to this, if you're having a big public keynote, I will do my traditional live-tweet the keynote and make fun of people because that is, A, expected and, B, it's live-streamed anywhere on the planet I want to be sitting at that point in time, and yeah, if you're saying things in public, you can basically expect that to be the way that I approach these things. But it's a nuanced take, and that is something that is not fully understood by an awful lot of folks who run events. I'll be the first to admit that aspects of who and what I am mean that some speaking engagements are not open to me. And I'm okay with that, on some level, I truly am. It's a different philosophy.But I do know that I am done apologizing for who I am and what I'm about. And at some point that required a tremendous amount of privilege and a not insignificant willingness to take a risk that it was going to work out all right. I can't imagine going back anymore. Now, that road is certainly not what I would recommend to everyone, particularly folks earlier in their career, particularly for folks who don't look just like I do and have a failure mode of a board seat and a book deal somewhere, but figuring out where you will and will not compromise is always an important thing to get straight for yourself before you're presented with a situation where you have to make those decisions, but now there's a whole bunch of incentive to decide in one way or another.Wesley: And that's a journey. You can't just skip sections, right? You didn't get to where you are unless you went through the previous experience that you went through. And it's true for everyone. If you see those success books or how-to books written by people who are extremely rich, and, like, how to become successful and, like, okay, well, that journey is your own. It doesn't make it totally, like, inaccessible to everyone else, but you got to realize that not everyone can walk that path. And—Corey: You were in the right place at the right time, an early employee at a company that did phenomenally well and that catapulted you into reach beyond the wildest dreams of avarice territory. Good for you, but fundamentally, when you give talks like that as a result, what it often presents as is, “I won the lottery, and here's how you can too.” It doesn't work that way. The road you walked was unique to you and that opportunity is closed, not open anyone else, so people have to find their own paths.Wesley: Yeah, and lightning doesn't strike in the same place twice. But there are some things where you can understand some fundamentals. And depending on where you go, I think you do need to know yourself, you do need to know—like, be able to access yourself, but being able to share that, of course, you have to be at a point where you feel comfortable. And so, even if you're in a space where you don't feel that you can be your authentic self or be able to share all parts of you, you yourself should at least know yourself and then make that decision. I agree that it's a point of privilege to be able to say, “Take me how I am.”I'm lucky that I've gotten here, not everyone does, and just because you don't doesn't mean that you're a failure. It just means that the world hasn't caught up yet. People who are part of marginalized society, like, if you are, let's say trans, or if you are even gay, you take the same person, the same stance, the same yearning to be accepted, and then transport it to 50 years ago, you're not safe. You will not necessarily be accepted, or you may not even be successful. And if you have a lane where you can do that, all the power to you, but not everyone could be themselves, and you just need to make sure that at least you can know yourself, even if you don't share that with the world.Corey: It takes time to get there, and I think you're right that it's impossible to get there without walking through the various steps. It's one of the reasons I'm somewhat reluctant to talk overly publicly about my side project gig of paid speaking engagements, for instance, is that the way to get those is you start off by building a reputation as a speaker, and that takes an awful lot of time. And speaking at events where there's no budget even to pay you a speaking fee out of anyway. And part of what gets the keynote invitations to, “Hey, we want you to come and give a talk,” is the fact that people have seen you speak elsewhere and know what you're about and what to expect. Here's a keynote presented by someone who's never presented on stage before is a recipe for a terrifying experience, if not for the speaker or the audience, definitely [laugh] for the event organizers because what if they choke.?Easy example of this, even now hundreds of speaking engagements in, the adrenaline hit right before I go on stage means that sometimes my knees shake a bit before I walk out on stage. I make it a point to warn the people who are standing with me backstage, “Oh, this is a normal thing. Don't worry, it is absolutely expected. It happens every time. Don't sweat it.”And, like, “Thank you for letting us know. That is the sort of thing that's useful.” And then they see me shake, and they get a little skeptical. Like, I thought this guy was a professional. What's the story and I walk on stage and do my thing and I come back. Like, “That was incredible. I was worried at the beginning.” “I told you, we all have our rituals before going on stage. Mine is to shake like a leaf.”But the value there is that people know what to generally expect when I get on stage. It's going to have humor, there's going to be a point interwoven throughout what I tend to say, and in the case of paid speaking engagements, I always make sure I know where the boundaries are of things I can make fun of a big company for. Like, I can get on stage and make fun of service naming or I can make fun of their deprecation policy or something like that, but yeah, making fun of the way that they wind up handling worker relations is probably not going to be great and it could get the person who championed me fired or centered internally. So, that is off the table.Like, even on this podcast, for example, I sometimes get feedback from listeners of, “Well, you have someone from company X on and you didn't beat the crap out of them on this particular point.” It's yeah, you do understand that by having people on the show I'm making a tacit agreement not to attack them. I'm not a journalist. I don't pretend to be. But if I beat someone up with questions about their corporate policy, yeah, very rarely do I have someone who is in a position in those companies to change that policy, and they're certainly not authorized to speak on the record about those things.So, I can beat them up on it, they can say, “I can't answer that,” and we're not going to go anywhere. What is the value of that? It looks like it's not just gotcha journalism, but ineffective gotcha journalism. It doesn't work that way. And that's never been what this show is about.But there's that consistent effort behind the scenes of making sure that people will be entertained, will enjoy what they're seeing, but also are not going to deeply regret giving me a microphone, has always been the balancing act, at least for me. And I want to be clear, my style is humor. It is not for everyone. And my style of humor has a failure mode of being a jerk and making people feel bad, so don't think that my path is the only or even a recommended way for folks who want to get more into speaking to proceed.Wesley: You also mention, though, about, like, punching up versus punching down. And if you really tear down a company after you've been invited to speak, what you're doing is you're punching down at the person who booked you. They're not the CEO; they're not the owner of the company; they're the person who's in charge of running an event or booking speakers. And so, putting that person and throwing them under the bus is punching down because now you're threatening their livelihood, and it doesn't make any market difference in terms of changing the corporate's values or how they execute. So yeah, I totally agree with you in that one.And, like you were saying before, if there's a company you really thought was abhorrent, why speak there? Why give them or lend your reputation to this company if you absolutely feel that it's something you don't want to be associated with? You can just choose not to do that. For me, when I look at speaking, it is important for me to really think about why I'm speaking as well. So, not just the company who's hiring me, but the audience that I'll be serving.So, if I'm going to help with inspiring the next generation of developers, or helping along the thought of how to make the world a better place, or how people themselves can be better people so that we can just change the landscape and make it a lot friendlier, that is also its own… form of compensation and not just speaking for a speaker's fee. So, I do agree that you need to not just be super Negative Nancy, and try to fight all fights. You need to embrace some of the good things and try to make more of those experiences good for everyone, not just the people who are inviting you there, but the people who are attending. And when I started speaking, I was not a good speaker as well. I made a lot of mistakes, and still do, but I think speaking is easier than some people think and if someone truly wants to do it, they should go ahead and get started.What is the saying? If there's something is truly important, you'll be bad at it [laugh] and you'll be okay with it. I started speaking because of my role as a developer advocate. And if you just do a Google search for ‘CFPs,' you can start speaking, too. So, those who are not public speakers and want to get into it, just Google ‘CFP' and then start applying.And then you'll get better at your submissions, you'll get better at your slides, and then once you get accepted, then you'll get better at preparing, then you'll get better at actually speaking. There's a lot of steps between starting and stopping and it's okay to get started doing that route. The other thing I wanted to point out is I feel public speaking is the equivalent of lifting your own bodyweight. If you can do it, you're one of the small few of the population that is willing to do so or that can do it. If you start public speaking, that in itself is an accomplishment and an experience that is something that is somewhat enriching. And being bad at it doesn't take the passion away from you. If you just really want to do it, just keep doing it, even if you're a bad speaker.Corey: Yeah. The way to give a great talk because you have a bunch of terrible talks first.Wesley: Yeah. And it's okay to do that.Corey: And it's not the in entirety of community. It's not even a requirement to be involved with the community. If you're one of those people that absolutely dreads the prospect of speaking publicly, fine. I'm not suggesting that, oh, you need to get over that and get on stage. That doesn't help anyone. Don't do the things you dread doing because you know that it's not going to go well for you.That's the reason I don't touch actual databases. I mean, come on, let's be realistic. I will accidentally the data, and then we won't have a company anymore. So, I know what things I'm good at and things I'm not. I also don't do hostage negotiations, for obvious reasons.Wesley: And also, here's a little, like, secret tip. If you really want to do public speaking and you start doing public speaking and you're not so good at it from other peoples' perspective, but you still love doing it and you think you're getting better, doing public speaking is one of those things where you can say that you do it and no one will really question how good you are at it. [laugh]. If you're just in casual conversation, it's like, “Hey, I wrote a book.” People like, “Oh, wow. This person wrote the book on blah, blah, blah.”Corey: It's a self-published book that says the best way to run Kubernetes. It's a single page; it says, “Don't.” In 150-point type. “The end.” But I wrote a book.Wesley: Yeah.Corey: Yeah.Wesley: People won't probe too much and it'll help you with your development. So, go ahead and get started. Don't worry about doing that thing where, like, I have to be the best before I can present it. Call yourself a public speaker. Check, done.Corey: Always. We are the stories we tell, and nowhere is it more true than in the world of public speaking. I really want to thank you for taking the time out of your day to speak with me about this for a second time in a single year. Oh, my goodness. If people want to learn more about what you're up to, where can they find you?Wesley: I'm on Twitter, @wesley83 on Twitter. And you can find me also on PolyWork. So, polywork.com/wesley83. Or just go to wesleyfaulkner.com which redirects you there. I list pretty much everything that I am working on and any upcoming speaking opportunities, hopefully when they release that feature, will also be on that Polywork page.Corey: Excellent. And of course, I started Polywork recently, and I'm at thoughtleader.cloud because of course I am, which is neither here nor there. Thank you so much for taking the time to speak about this side of the industry that we never really get to talk about much, at least not publicly and not very often.Wesley: Well, thank you for having me on the show. And I wanted to take some time to say thank you for the work that you're doing. Not just elevating voices like myself, but talking truth to power, like we mentioned before, but being yourself and being a great representation of how people should be treating others: being honest without being mean, being snarky without being rude. And other companies and other people who've given me a chance, and given me a platform, I wanted to say thank you to you too, and I wouldn't be here unless it was people like you acknowledging the work that I've been doing.Corey: All it takes is just recognizing what you're doing and acknowledging it. People often want to thank me for this stuff, but it's just, what, for keeping my eyes open? I don't know, I feel like it's just the job; it's not something that is above and beyond any expected normal behavior. The only challenge is I look around the industry and I realize just how wrong that impression is, apparently. But here we are. It's about finding people doing interesting work and letting them tell their story. That's all this podcast has ever tried to be.Wesley: Yeah. And you do it. And doing the work is part of the reward, and I really appreciate you just going through the effort. Even having your ears open is something that I'm glad that you're able to at least know who the people are and who are making noises—or making noise to raise their profile up and then in turn, sharing that with the world. And so, that's a great service that you're providing, not just for me, but for everyone.Corey: Well, thank you. And as always, thank you for your time. Wesley Faulkner, Head of Community at SingleStore. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a rambling comment telling me exactly why DevRel does not need success metrics of any kind.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About LizLiz Rice is Chief Open Source Officer with cloud native networking and security specialists Isovalent, creators of the Cilium eBPF-based networking project. She is chair of the CNCF's Technical Oversight Committee, and was Co-Chair of KubeCon + CloudNativeCon in 2018. She is also the author of Container Security, published by O'Reilly.She has a wealth of software development, team, and product management experience from working on network protocols and distributed systems, and in digital technology sectors such as VOD, music, and VoIP. When not writing code, or talking about it, Liz loves riding bikes in places with better weather than her native London, and competing in virtual races on Zwift.Links: Isovalent: https://isovalent.com/ Container Security: https://www.amazon.com/Container-Security-Fundamental-Containerized-Applications/dp/1492056707/ Twitter: https://twitter.com/lizrice GitHub: https://github.com/lizrice Cilium and eBPF Slack: http://slack.cilium.io/ CNCF Slack: https://cloud-native.slack.com/join/shared_invite/zt-11yzivnzq-hs12vUAYFZmnqE3r7ILz9A TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the interesting things about hanging out in the cloud ecosystem as long as I have and as, I guess, closely tied to Amazon as I have been, is that you learned that you never quite are able to pronounce things the way that people pronounce them internally. In-house pronunciations are always a thing. My guest today is Liz Rice, the Chief Open Source Officer at Isovalent, and they're responsible for, among other things, the Cilium open-source project, which is around eBPF, which I can only assume is internally pronounced as ‘Ehbehpf'. Liz, thank you for joining me today and suffering my pronunciation slings and arrows.Liz: I have never heard ‘Ehbehpf' before, but I may have to adopt it. That's great.Corey: You also are currently—in a term that is winding down if I'm not misunderstanding—you were the co-chair of KubeCon and CloudNativeCon at the CNCF, and you are also currently on the technical oversight committee for the foundation.Liz: Yeah, yeah. I'm currently the chair, in fact, of the technical oversight committee.Corey: And now that Amazon has joined, I assumed that they had taken their horrible pronunciation habits, like calling AMIs ‘Ah-mies' and whatnot, and started spreading them throughout the ecosystem with wild abandon.Liz: Are we going to have to start calling CNCF ‘Ka'Nff' or something?Corey: Exactly. They're very frugal, by which I mean they never buy a vowel. So yeah, it tends to be an ongoing challenge. Joking and all the rest aside, let's start, I guess, at the macro view. The CNCF does an awful lot of stuff, where if you look at the CNCF landscape, for example, like, I think some of my jokes on the internet go a bit too far, but you look at this thing and last time I checked, there were something like four or 500 different players in various spaces.And it's a very useful diagram, don't get me wrong by any stretch of the imagination, but it also is one of those things that is so staggeringly vast that I've got a level with you on this one, given my old, ancient sysadmin roots, “The hell with it. I'm going to run some VMs in a three-tiered architecture just like grandma and grandpa used to do,” and call it good. Not really how the industry is evolved, but it's overwhelming.Liz: But that might be the right solution for your use case so, you know, don't knock it if it works.Corey: Oh, yeah. If it's a terrible architecture and it works, is it really that terrible of an architecture? One wonders.Liz: Yeah, yeah. I mean, I'm definitely not one of those people who thinks, you know, every solution has the same—you know, is solved by the same hammer, you know, all problems are not the same nail. So, I am a big fan of a lot of the CNCF projects, but that doesn't mean to say I think those are the only ways to deploy software. You know, there are plenty of things like Lambda are a really great example of something that is super useful and very applicable for lots of applications and for lots of development teams. Not necessarily the right solution for everything. And for other people, they need all the bells and whistles that something like Kubernetes gives them. You know, horses for courses.Corey: It's very easy for me to make fun of just about any company or service or product, but the thing that always makes me set that aside and get down to brass tacks has been, “Okay, great. You can build whatever you want. You can tell whatever glorious marketing narrative you wish to craft, but let's talk to a real customer because once we do that, then if you're solving a problem that someone is having in the wild, okay, now it's no longer just this theoretical exercise and PowerPoint. Now, let's actually figure out how things work when the rubber meets the road.”So, let's start, I guess, with… I'll leave it to you. Isovalent are the creators of the Cilium eBPF-based networking project.Liz: Yeah.Corey: And eBPF is the part of that I think I'm the most familiar with having heard the term. Would you rather start on the company side or on the eBPF side?Liz: Oh, I don't mind. Let's—why don't we start with eBPF? Yeah.Corey: Cool. So easy, ridiculous question. I know that it's extremely important because Brendan Gregg periodically gets on stage and tells amazing stories about this; the last time he did stuff like that, I went stumbling down into the rabbit hole of DTrace, and I have never fully regretted doing that, nor completely forgiven him. What is eBPF?Liz: So, it stands for extended Berkeley Packet Filter, and we can pretty much just throw away those words because it's not terribly helpful. What eBPF allows you to do is to run custom programs inside the kernel. So, we can trigger these programs to run, maybe because a network packet arrived, or because a particular function within the kernel has been called, or a tracepoint has been hit. There are tons of places you can attach these programs to, or events you can attach programs to.And when that event happens, you can run your custom code. And that can change the behavior of the kernel, which is, you know, great power and great responsibility, but incredibly powerful. So Brendan, for example, has done a ton of really great pioneering work showing how you can attach these eBPF programs to events, use that to collect metrics, and lo and behold, you have amazing visibility into what's happening in your system. And he's built tons of different tools for observing everything from, I don't know, memory use to file opens to—there's just endless, dozens and dozens of tools that Brendan, I think, was probably the first to build. And now this sort of new generations of eBPF-based tooling that are kind of taking that legacy, turning them into maybe more, going to say user-friendly interfaces, you know, with GUIs, and hooking them up to metrics platforms, and in the case of Cilium, using it for networking and hooking it into Kubernetes identities, and making the information about network flows meaningful in the context of Kubernetes, where things like IP addresses are ephemeral and not very useful for very long; I mean, they just change at any moment.Corey: I guess I'm trying to figure out what part of the stack this winds up applying to because you talk about, at least to my mind, it sounds like a few different levels all at once: You talk about running code inside of the kernel, which is really close to the hardware—it's oh, great. It's adventures in assembly is almost what I'm hearing here—but then you also talk about using this with GUIs, for example, and operating on individual packets to run custom programs. When you talk about running custom programs, are we talking things that are a bit closer to, “Oh, modify this one field of that packet and then call it good,” or are you talking, “Now, we launch Microsoft Word.”Liz: Much more the former category. So yeah, let's inspect this packet and maybe change it a bit, or send it to a different—you know, maybe it was going to go to one interface, but we're going to send it to a different interface; maybe we're going to modify that packet; maybe we're going to throw the packet on the floor because we don't—there's really great security use cases for inspecting packets and saying, “This is a bad packet, I do not want to see this packet, I'm just going to discard it.” And there's some, what they call ‘Packet of Death' vulnerabilities that have been mitigated in that way. And the real beauty of it is you just load these programs dynamically. So, you can change the kernel or on the fly and affect that behavior, just immediately have an effect.If there are processes already running, they get instrumented immediately. So, maybe you run a BPF program to spot when a file is opened. New processes, existing processes, containerized processes, it doesn't matter; they'll all be detected by your program if it's observing file open events.Corey: Is this primarily used from a security perspective? Is it used for—what are the common use cases for something like this?Liz: There's three main buckets, I would say: Networking, observability, and security. And in Cilium, we're kind of involved in some aspects of all those three things, and there are plenty of other projects that are also focusing on one or other of those aspects.Corey: This is where when, I guess, the challenge I run into the whole CNCF landscape is, it's like, I think the danger is when I started down this path that I'm on now, I realized that, “Oh, I have to learn what all the different AWS services do.” This was widely regarded as a mistake. They are not Pokémon; I do not need to catch them all. The CNCF landscape applies very similarly in that respect. What is the real-world problem space for which eBPF and/or things like Cilium that leverage eBPF—because eBPF does sound fairly low-level—that turn this into something that solves a problem people have? In other words, what is the problem that Cilium should be the go-to answer for when someone says, “I have this thing that hurts.”Liz: So, at one level, Cilium is a networking solution. So, it's Kubernetes CNI. You plug it in to provide connectivity between your applications that are running in pods. Those pods have to talk to each other somehow and Cilium will connect those pods together for you in a very efficient way. One of the really interesting things about eBPF and networking is we can bypass some of the networking stack.So, if we are running in containers, we're running our applications in containers in pods, and those pods usually will have their own networking namespace. And that means they've got their own networking stack. So, a packet that arrives on your machine has to go through the networking stack on that host machine, go across a virtual interface into your pod, and then go through the networking stack in that pod. And that's kind of inefficient. But with eBPF, we can look at the packet the moment it's come into the kernel—in fact in some cases, if you have the right networking interfaces, you can do it while it's still on the network interface card—so you look at that packet and say, “Well, I know what pod that's destined for, I can just send it straight there.” I don't have to go through the whole networking stack in the kernel because I already know exactly where it's going. And that has some real performance improvements.Corey: That makes sense. In my explorations—we'll call it—with Kubernetes, it feels like the universe—at least at the time I went looking into it—was, “Step One, here's how to wind up launching Kubernetes to run a blog.” Which is a bit like using a chainsaw to wind up cutting a sandwich. Okay, massively overpowered but I get the basic idea, like, “Okay, what's project Step Two?” It's like, “Oh, great. Go build Google.”Liz: [laugh].Corey: Okay, great. It feels like there's some intermediary steps that have been sort of glossed over here. And at the small-scale that I kicked the tires on, things like networking performance never even entered the equation; it was more about get the thing up and running. But yeah, at scale, when you start seeing huge numbers of containers being orchestrated across a wide variety of hosts that has serious repercussions and explains an awful lot. Is this the sort of thing that gets leveraged by cloud providers themselves, is it something that gets built in mostly on-prem environments, or is it something that rides in, almost, user-land for most of these use cases that customers coming to bringing to those environments? I'm sorry, users, not customers. I'm too used to the Amazonian phrasing of everyone as a customer. No, no, they are users in an open-source project.Liz: [laugh]. Yeah, so if you're using GKE, the GKE Dataplane V2 is using Cilium. Alibaba Cloud uses Cilium. AWS is using Cilium for EKS Anywhere. So, these are really, I think, great signals that it's super scalable.And it's also not just about the connectivity, but also about being able to see your network flows and debug them. Because, like you say, that day one, your blog is up and running, and day two, you've got some DNS issue that you need to debug, and how are you going to do that? And because Cilium is working with Kubernetes, so it knows about the individual pods, and it's aware of the IP addresses for those pods, and it can map those to, you know, what's the pod, what service is that pod involved with. And we have a component of Cilium called Hubble that gives you the flows, the network flows, between services. So, you know, we've probably all seen diagrams showing Service A talking to Service B, Service C, some external connectivity, and Hubble can show you those flows between services and the outside world, regardless of how the IP addresses may be changing underneath you, and aggregating network flows into those services that make sense to a human who's looking at a Kubernetes deployment.Corey: A running gag that I've had is that one of the drawbacks and appeals of Kubernetes, all at once, is that it lets you cosplay as a cloud provider, even if you don't happen to work for one of them. And there's a bit of truth to it, but let's be serious here, despite what a lot of the cloud providers would wish us to believe via a bunch of marketing, there's a tremendous number of data center environments out there, hybrid environments, and companies that are in those environments are not somehow laggards, or left behind technologically, or struggling to digitally transform. Believe it or not—I know it's not a common narrative—but large companies generally don't employ people who lack critical thinking skills and strategic insight. There's usually a reason that things are the way that they are and when you don't understand that my default approach is that, oh context that gets missing, so I want to preface this with the idea there is nothing wrong in those environments. But in a purely cloud-native environment—which means that I'm very proud about having no single points of failure as I have everything routing to a single credit card that pays the cloud providers—great. What is the story for Cilium if I'm using, effectively, the managed Kubernetes options that Name Any Cloud Provider will provide for me these days? Is it at that point no longer for me or is it something that instead expresses itself in ways I'm not seeing, yet?Liz: Yeah, so I think, as an open-source project—and it is the only CNI that's at incubation level or beyond, so you know, it's CNCF-supported networking solution; you can use it out of the box, you can use it for your tiny blog application if you've decided to run that on Kubernetes, you can do so—things start to get much more interesting at scale. I mean, that… continuum between you know, there are people purely on managed services, there are people who are purely in the cloud, hybrid cloud is a real thing, and there are plenty of businesses who have good reasons to have some things in their own data centers, something's in the public cloud, things distributed around the world, so they need connectivity between those. And Cilium will solve a lot of those problems for you in the open-source, but also, if you're telco scale and you have things like BGP networks between your data centers, then that's where the paid versions of Cilium, the enterprise versions of Cilium, can help you out. And, as Isovalent, that's our business model to have, like—we fully support or we contribute a lot of resources into the open-source Cilium, and we want that to be the best networking solution for anybody, but if you are an enterprise who wants those extra bells and whistles, and the kind of scale that, you know, a telco, or a massive retailer, or a large media organization, or name your vertical, then we have solutions for that as well. And I think it was one of the really interesting things about the eBPF side of it is that, you know, we're not bound to just Kubernetes, you know? We run in the kernel, and it just so happens that we have that Kubernetes interface for allocating IP addresses to endpoints that happened to be pods. But—Corey: So, back to my crappy pile of VMs—because the hell with all this newfangled container nonsense—I can still benefit from something like Cilium?Liz: Exactly, yeah. And there's plenty of people using it for just load-balancing, which, why not have an eBPF-based high-performance load balancer?Corey: Hang on, that's taking me a second to work my way through. What is the programming language for eBPF? It is something custom?Liz: Right. So, when you load your BPF program into the kernel, it's in the form of eBPF bytecode. There are people who write an eBPF bytecode by hand; I am not one of those people.Corey: There are people who used to be able to write Sendmail configs without running through the M four preprocessor, and I don't understand those people either.Liz: [laugh]. So, our choices are—well, it has to be a language that can be compiled into that bytecode, and at the moment, there are two options: C, and more recently, Rust. So, the C code, I'm much more familiar with writing BPF code in C, it's slightly limited. So, because these BPF programs have to be safe to run, they go through a verification process which checks that you're not going to crash the kernel, that you're not going to end up in some hardware loop, and basically make your machine completely unresponsive, we also have to know that BPF programs, you know, they'll only access memory that they're supposed to and that they can't mess up other processes. So, there's this BPF verification step that checks for example that you always check that a pointer isn't nil before you dereference it.And if you try and use a pointer in your C code, it might compile perfectly, but when you come to load it into the kernel, it gets rejected because you forgot to check that it was non-null before.Corey: You try and run it, the whole thing segfaults, you see the word ‘fault' there and well, I guess blameless just went out the window there.Liz: [laugh]. Well, this is the thing: You cannot segfault in the kernel, you know, or at least that's a bad [day 00:19:11]. [laugh].Corey: You say that, but I'm very bad with computers, let's be clear here. There's always a way to misuse things horribly enough.Liz: It's a challenge. It's pretty easy to segfault if you're writing a kernel module. But maybe we should put that out as a challenge for the listener, to try to write something that crashes the kernel from within an eBPF because there's a lot of very smart people.Corey: Right now the blood just drained from anyone who's listening, in the kernel space or the InfoSec space, I imagine.Liz: Exactly. Some of my colleagues at Isovalent are thinking, “Oh, no. What's she brought on here?” [laugh].Corey: What have you done? Please correct me if I'm misunderstanding this. So, eBPF is a very low-level tool that requires certain amounts of braining in order [laugh] to use appropriately. That can be a heavy lift for a lot of us who don't live in those spaces. Cilium distills this down into something that is all a lot more usable and understandable for folks, and then beyond that, you wind up with Isovalent, that winds up effectively productizing and packaging this into something that becomes a lot more closer to turnkey. Is that directionally accurate?Liz: Yes, I would say that's true. And there are also some other intermediate steps, like the CLI tools that Brendan Gregg did, where you can—I mean, a CLI is still fairly low-level, but it's not as low-level as writing the eBPF code yourself. And you can be quite in-dep—you know, if you know what things you want to observe in the kernel, you don't necessarily have to know how to write the eBPF code to do it, but if you've got these fairly low-level tools to do it. You're absolutely right that very few people will need to write their own… BPF code to run in the kernel.Corey: Let's move below the surface level of awareness; the same way that most of us don't need to know how to compile our own kernel in this day and age.Liz: Exactly.Corey: A few people very much do, but because of their hard work, the rest of us do not.Liz: Exactly. And for most of us, we just take the kernel for granted. You know, most people writing applications, it doesn't really matter if—they're just using abstractions that do things like open files for them, or create network connections, or write messages to the screen, you don't need to know exactly how that's accomplished through the kernel. Unless you want to get into the details of how to observe it with eBPF or something like that.Corey: I'm much happier not knowing some of the details. I did a deep dive once into Linux system kernel internals, based on an incredibly well-written but also obnoxiously slash suspiciously thick O'Reilly book, Linux Systems Internalsand it was one of those, like, halfway through, “Can I please be excused? My brain is full.” It's one of those things that I don't use most of it on a day-to-day basis, but it's solidified by understanding of what the computer is actually doing in a way that I will always be grateful for.Liz: Mmm, and there are tens of millions of lines of code in the Linux kernel, so anyone who can internalize any of that is basically a superhero. [laugh].Corey: I have nothing but respect for people who can pull that off.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.In your day job, quote-unquote—which is sort of a weird thing to say, given that you are working at an open-source company; in fact, you are the Chief Open Source Officer, so what you're doing in the community, what you're exploring on the open-source project side of things, it is all interrelated. I tend to have trouble myself figuring out where my job starts and stops most weeks; I'm sympathetic to it. What inspired you folks to launch a company that is, “Ah, we're going to be in the open-source space?” Especially during a time when there's been a lot of pushback, in some respects, about the evolution of open-source and the rise of large cloud providers, where is open-source a viable strategy or a tactic to get to an outcome that is pleasing for all parties?Liz: Mmm. So, I wasn't there at the beginning, for the Isovalent journey, and Cilium has been around for five or six years, now, at this point. I very strongly believe in open-source as an effective way of developing technology—good technology—and getting really good feedback and, kind of, optimizing the speed at which you can innovate. But I think it's very important that businesses don't think—if you're giving away your code, you cannot also sell your code; you have to have some other thing that adds value. Maybe that's some extra code, like in the Isovalent example, the enterprise-related enhancements that we have that aren't part of the open-source distribution.There's plenty of other ways that people can add value to open-source. They can do training, they can do managed services, there's all sorts of different—support was the classic example. But I think it's extremely important that businesses don't just expect that I can write a bunch of open-source code, and somehow magically, through building up a whole load of users, I will find a way to monetize that.Corey: A bunch of nerds will build my product for me on nights and weekends. Yeah, that's a bit of an outmoded way of thinking about these things.Liz: Yeah exactly. And I think it's not like everybody has perfect ability to predict the future and you might start a business—Corey: And I have a lot of sympathy for companies who originally started with the idea of, “Well, we are the project leads. We know this code the best, therefore we are the best people in the world to run this as a service.” The rise of the hyperscale cloud providers has called that into significant question. And I feel for them because it's difficult to completely pivot your business model when you're already a publicly-traded company. That's a very fraught and challenging thing to do. It means that you're left with a bunch of options, none of them great.Cilium as a project is not that old, neither is Isovalent, but it's new enough in the iterative process, that you were able to avoid that particular pitfall. Instead, you're looking at some level of making this understandable and useful to humans, almost the point where it disappears from their level of awareness that they need to think about. There's huge value in something like that. Do you think that there is a future in which projects and companies built upon projects that follow this model are similarly going to be having challenges with hyperscale cloud providers, or other emergent threats to the ecosystem—sorry, ‘threat' is an unfair and unkind word here—but changes to the ecosystem, as we see the world evolving in ways that most of us did not foresee?Liz: Yeah, we've certainly seen some examples in the last year or two, I guess, of companies that maybe didn't anticipate, and who necessarily has a crystal ball to anticipate how cloud providers might use their software? And I think in some cases, the cloud providers has not always been the most generous or most community-minded in their approach to how they've done that. But I think for a company, like Isovalent, our strong point is talent. It would be extremely rare to find the level of expertise in, you know, what is a pretty specialized area. You know, the people at Isovalent who are working on Cilium are also working on eBPF itself, and that level of expertise is, I think, pretty unrivaled.So, we're in such a new space with eBPF, we've only in the last year or so, got to the point where pretty much everyone is running a kernel that's new enough to use eBPF. Startups do have a kind of agility that I think gives them an advantage, which I hope we'll be able to capitalize on. I think sometimes when businesses get upset about their code being used, they probably could have anticipated it. You know, if it's open-source, people will use your software, and you have to think of that.Corey: “What do you mean you're using the thing we gave away for free and you're not paying us to use it?”Liz: Yeah.Corey: “Uh, did you hear what you just said?” Some of this was predictable, let's be fair.Liz: Yeah, and I think you really have to, as a responsible business, think about, well, what does happen if they use all the open-source code? You know, is that a problem? And as far as we're concerned, everybody using Cilium is a fantastic… thing. We fully welcome everyone using Cilium as their data plane because the vast majority of them would use that open-source code, and that would be great, but there will be people who need that extra features and the expertise that I think we're in a unique position to provide. So, I joined Isovalent just about a year ago, and I did that because I believe in the technology, I believe in the company, I believe in, you know, the foundations that it has in open-source.It's a very much an open-source first organization, which I love, and that resonates with me and how I think we can be successful. So, you know, I don't have that crystal ball. I hope I'm right, we'll find out. We should do this again, you know, a couple of years and see how that's panning out. [laugh].Corey: I'll book out the date now.Liz: [laugh].Corey: Looking back at our conversation just now, you talked about open-source, and business strategy and how that's going to be evolving. We talked about the company, we talked about an incredibly in-depth, technical product that honestly goes significantly beyond my current level of technical awareness. And at no point in any of those aspects of the conversation did you talk about it in a way that I did not understand, nor did you come off in any way as condescending. In fact, you wrote an O'Reilly book on Container Security that's written very much the same way. How did you learn to do that? Because it is, frankly, an incredibly rare skill.Liz: Oh, thank you. Yeah, I think I have never been a fan of jargon. I've never liked it when people use a complicated acronym, or really early days in my career, there was a bit of a running joke about how everything was TLAs. And you think, well, I understand why we use an acronym to shorten things, but I don't think we need to assume that everybody knows what everything stands for. Why can't we explain things in simple language? Why can't we just use ordinary terms?And I found that really resonates. You know, if I'm doing a presentation or if I'm writing something, using straightforward language and explaining things, making sure that people understand the, kind of, fundamentals that I'm going to build my explanation on. I just think that has a—it results in people understanding, and that's my whole point. I'm not trying to explain something to—you know, my goal is that they understand it, not that they've been blown away by some kind of magic. I want them to go away going, “Ah, now I understand how this bit fits with that bit,” or, “How this works.” You know?Corey: The reason I bring it up is that it's an incredibly undervalued skill because when people see it, they don't often recognize it for what it is. Because when people don't have that skill—which is common—people just write it off as oh, that person's a bad communicator. Which I think is a little unfair. Being able to explain complex things simply is one of the most valuable yet undervalued skills that I've found in this entire space.Liz: Yeah, I think people sometimes have this sort of wrong idea that vocabulary and complicated terms are somehow inherently smarter. And if you use complicated words, you sound smarter. And I just don't think that's accessible, and I don't think it's true. And sometimes I find myself listening to someone, and they're using complicated terms or analogies that are really obscure, and I'm thinking, but could you explain that to me in words of one syllable? I don't think you could. I think you're… hiding—not you [laugh]. You know, people—Corey: Yeah. No, no, that's fair. I'll take the accusation as [unintelligible 00:31:24] as I can get it.Liz: [laugh]. But I think people hide behind complex words because they don't really understand them sometimes. And yeah, I would rather people understood what I'm saying.Corey: To me—I've done it through conference talks, but the way I generally learn things is by building something with them. But the way I really learn to understand something is I give a conference talk on it because, okay, great. I can now explain Git—which was one of my early technical talks—to folks who built Git. Great. Now, how about I explain it to someone who is not immersed in the space whatsoever? And if I can make it that accessible, great, then I've succeeded. It's a lot harder than it looks.Liz: Yeah, exactly. And one of the reasons why I enjoy building a talk is because I know I've got a pretty good understanding of this, but by the time I've got this talk nailed, I will know this. I might have forgotten it in six months time, you know, but [laugh] while I'm giving that talk, I will have a really good understanding of that because the way I want to put together a talk, I don't want to put anything in a talk that I don't feel I could explain. And that means I have to understand how it works.Corey: It's funny, this whole don't give talks about things you don't understand seems like there's really a nouveau concept, but here we are, we're [working on it 00:32:40].Liz: I mean, I have committed to doing talks that I don't fully understand, knowing that—you know, with the confidence that I can find out between now and the [crosstalk 00:32:48]—Corey: I believe that's called a forcing function.Liz: Yes. [laugh].Corey: It's one of those very high-risk stories, like, “Either I'm going to learn this in the next three months, or else I am going to have some serious egg on my face.”Liz: Yeah, exactly, definitely a forcing function. [laugh].Corey: I really want to thank you for taking so much time to speak with me today. If people want to learn more, where can they find you?Liz: So, I am online pretty much everywhere as lizrice, and I am on Twitter. I'm on GitHub. And if you want to come and hang out, I am on the Cilium and eBPF Slack, and also the CNCF Slack. Yeah. So, come say hello.Corey: There. We will put links to all of that in the [show notes 00:33:28]. Thank you so much for your time. I appreciate it.Liz: Pleasure.Corey: Liz Rice, Chief Open Source Officer at Isovalent. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment containing an eBPF program that on every packet fires off a Lambda function. Yes, it will be extortionately expensive; almost half as much money as a Managed NAT Gateway.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About LauraLaura leads the research program at Jeli.io.  She has a Master's degree in Human Factors & Systems Safety and a PhD in Cognitive Systems Engineering. Her doctoral work focused on distributed incident response practices in DevOps teams responsible for critical digital services. She was a researcher with the SNAFU Catchers Consortium from 2017-2020 and her research interests lie in resilience engineering, coordination design and enabling adaptive capacity across distributed work teams. As a backcountry skier and alpine climber, she also studies cognition & resilient performance in high risk, high consequence mountain environments.  Links: Howie: The Post-Incident Guide: https://www.jeli.io/howie-the-post-incident-guide/ Jeli: https://www.jeli.io Twitter: https://twitter.com/lauramdmaguire TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the things that's always been a treasure and a joy in working in production environments is things breaking. What do you do after the fact? How do you respond to that incident?Now, very often in my experience, you dive directly into the next incident because no one has time to actually fix the problems but just spend their entire careers firefighting. It turns out that there are apparently alternate ways. My guest today is Laura Maguire who leads the research program at Jeli, and her doctoral work focused on distributed incident response in DevOps teams responsible for critical digital services. Laura, thank you for joining me.Laura: Happy to be here, Corey, thanks for having me.Corey: I'm still just trying to wrap my head around the idea of there being a critical digital service, as someone whose primary output is, let's be honest, shitposting. But that's right, people do use the internet for things that are a bit more serious than making jokes that are at least funny only to me. So, what got you down this path? How did you get to be the person that you are in the industry and standing in the position you hold?Laura: Yeah, I have had a long circuitous route to get to where I am today, but one of the common threads is about safety and risk and how do people manage safety and risk? I started off in natural resource industries, in mountain safety, trying to understand how do we stop things from crashing, from breaking, from exploding, from catching fire, and how do we help support the people in those environments? And when I went back to do my PhD, I was tossed into the world of software engineers. And at first I thought, now, what do firefighters, pilots, you know, emergency room physicians have to do with software engineers and risk in software engineering? And it turns out, there's actually a lot, there's a lot in common between the types of people who handle real-time failures that have widespread consequences and the folks who run continuous deployment environments.And so one of the things that the pandemic did for us is it made it immediately apparent that digital service delivery is a critical function in society. Initially, we'd been thinking about these kinds of things as being financial markets, as being availability of electronic health records, communication systems for disaster recovery, and now we're seeing things like communication and collaboration systems for schools, for businesses, this helps keep society functioning.Corey: What makes part of this field so interesting is that the evolution in the space where, back when I first started my career about a decade-and-a-half ago, there was a very real concern in my first Linux admin gig when I accidentally deleted some of the data from the data warehouse that, “Oh, I don't have a job anymore.” And I remember being surprised and grateful that I still did because, “Oh, you just learned something. You going to do it again?” “No. Well, not like that exactly, but probably some other way, yeah.”And we have evolved so far beyond that now, to the point where when that doesn't happen after an incident, it becomes almost noteworthy in its own right and it blows up on social media. So, the Overton window of what is acceptable disaster response and incident management, and how we learn from those things has dramatically shifted even in the relatively brief window of 15 years. And we're starting to see now almost a next-generation approach to this. One thing that you were, I believe the principal author behind is Howie: The Post-Incident Guide, which is a thing that you have up on jeli.io—that's J-E-L-I dot I-O—talking about how to run post-incident investigations. What made you decide to write something like this?Laura: Yeah, so what you described at the beginning there about this kind of shift from blameless—blameful-type approaches to incident response to thinking more broadly about the system of work, thinking about what does it mean to operate in continuous deployment environments is really fundamental. Because working in these kinds of worlds, we don't have an established knowledge base about how these systems work, about how they break because they're continuously changing, the knowledge, the expertise required to manage them is continuously changing. And so that shift towards a blameless or blame-aware post-incident review is really important because it creates this environment where we can actually share knowledge, share expertise, and distribute more of our understandings of how these systems work and how they break. So that, kind of, led us to create the Howie Guide—the how we got here post-incident guide. And it was largely because companies were kind of coming from this position of, we find the person who did the thing that broke the system and then we can all rest easy and move forward. And so it was really a way to provide some foundation, introduce some ideas from the resilience engineering literature, which has been around for, you know, the last 30 or 40 years—Corey: It's kind of amazing, on some level, how tech as an industry has always tried to reinvent things from first principles. I mean, we figured out long before we started caring about computers in the way we do that when there was an incident, the right response to get the learnings from it for things like airline crashes—always a perennial favorite topic in this space for conference talks—is to make sure that everyone can report what happened in a safe way that's non-accusatory, but even in the early-2010s, I was still working in environments where the last person to break production or break the bill had the shame trophy hanging out on their desk, and it would stay there until the next person broke it. And it was just a weird, perverse incentive where it's, “Oh if I broke something, I should hide it.”That is absolutely the most dangerous approach because when things are broken, yes, it's generally a bad thing, so you may as well find the silver lining in it from my point of view and figure out, okay, what have we learned about our systems as a result of the way that these things break? And sometimes the things that we learn are, in fact, not that deep, or there's not a whole lot of learnings about it, such as when the entire county loses power, computers don't work so well. Oh, okay. Great, we have learned that. More often, though, there seem to be deeper learnings.And I guess what I'm trying to understand is, I have a relatively naive approach on what the idea of incident response should look like, but it's basically based on the last time I touched things that were production-looking, which was six or seven years ago. What is the current state of the art that the advanced leaders in the space as they start to really look at how to dive into this? Because I'm reasonably certain it's not still the, “Oh, you know, you can learn things when your computers break.” What is pushing the envelope these days?Laura: Yeah, so it's kind of interesting. You brought up incident response because incident response and incident analysis are the, sort of like, what do we learn from those things are very tightly coupled. What we can see when we look at someone responding in real-time to a failure is, it's difficult to detect all of the signals; they don't pop up and wave a little flag and say, like, “I am what's broken.” There's multiple compounding and interacting factors. So, there's difficulty in the detection phase; diagnosis is always challenging because of how the systems are interrelated, and then the repair is never straightforward.But when we stop and look at these kinds of things after the fact, of really common theme emerges, and that it's not necessarily about a specific technical skill set or understanding about the system, it's about the shared, distributed understanding of that. And so to put that in plain speak, it's what do you know that's important to the problem? What do I know that's important to the problem? And then how do we collectively work together to extract that specific knowledge and expertise, and put that into practice when we're under time pressure, when there's a lot of uncertainty, when we've got the VP DMing us and being like, “When's the system going to be back up?” and Twitter's exploding with unhappy customers?So, when we think about the cutting edge of what's really interesting and relevant, I think organizations are starting to understand that it's how do we coordinate and we collaborate effectively? And so using incident analysis as a way to recognize not only the technical aspects of what went wrong but the social aspects of that as well. And the teamwork aspects of that is really driving some innovation in this space.Corey: It seems to me, on some level, that the increasing sophistication of what environments look like is also potentially driving some of these things. I mean, again, when you have three web servers and one of them's broken, okay, it's a problem; we should definitely jump on that and fix it. But now you have thousands of containers running hundreds of microservices for some Godforsaken reason because what we decided this thing that solves the problem of 500 engineers working on the same repository is a political problem, so now we're going to use microservices for everything because, you know, people. Great. But then it becomes this really difficult to identify problem of what is actually broken?And past a certain point of scale, it's no longer a question of, “Is it broken?” so much as, “How broken is it at any given point in time?” And getting real-time observability into what's going on does pose more than a little bit of a challenge.Laura: Yeah, absolutely. So, the more complexity that you have in the system, the more diversity of knowledge and skill sets that you have. One person is never going to know everything about the system, obviously, and so you need kind of variability in what people know, how current that knowledge is, you need some people who have legacy knowledge, you have some people who have bleeding edge, my fingers were on the keyboard just moments ago, I did the last deploy, that kind of variability in whose knowledge and skill sets you have to be able to bring to bear to the problem in front of you. One of the really interesting aspects, when you step back and you start to look really carefully about how people work in these kinds of incidents, is you have folks that are jumping, get things done, probe a lot of things, they look at a lot of different areas trying to gather information about what's happening, and then you have people who sit back and they kind of take a bit of a broader view, and they're trying to understand where are people trying to find information? Where might our systems not be showing us what's going on?And so it takes this combination of people working in the problem directly and people working on the problem more broadly to be able to get a better sense of how it's broken, how widespread is that problem, what are the implications, what might repair actually look like in this specific context?Corey: Do you suspect that this might be what gives rise, sometimes, to it seems middle management's perennial quest to build the single pane of glass dashboard of, “Wow, it looks like you're poking around through 15 disparate systems trying to figure out what's going on. Why don't we put that all on one page?” It's a, “Great, let's go tilt at that windmill some more.” It feels like it's very aligned with what you're saying. And I just, I don't know where the pattern comes from; I just know I see it all the time, and it drives me up a wall.Laura: Yeah, I would call that pattern pretty common across many different domains that work in very complex, adaptive environments. And that is—like, it's an oversimplification. We want the world to be less messy, less unstructured, less ad hoc than it often is when you're working at the cutting edge of whatever kind of technology or whatever kind of operating environment you're in. There are things that we can know about the problems that we are going to face, and we can defend against those kinds of failure modes effectively, but to your point, these are very largely unstructured problem spaces when you start to have multiple interacting failures happening concurrently. And so Ashby, who back in 1956 started talking about, sort of, control systems really hammered this point home when he was talking about, if you have a world where there's a lot of variability—in this case, how things are going to break—you need a lot of variability in how you're going to cope with those potential types of failures.And so part of it is, yes, trying to find the right dashboard or the right set of metrics that are going to tell us about the system performance, but part of it is also giving the responders the ability to, in real-time, figure out what kinds of things they're going to need to address the problem. So, there's this tension between wanting to structure unstructured problems—put those all in a single pane of glass—and what most folks who work at the frontlines of these kinds of worlds know is, it's actually my ability to be flexible and to be able to adapt and to be able to search very quickly to gather the information and the people that I need, that are what's really going to help me to address those hard problems.Corey: Something I've noticed for my entire career, and I don't know if it's just unfounded arrogance, and I'm very much on the wrong side of the Dunning-Kruger curve here, but it always struck me that the corporate response to any form of outage has is generally trending toward oh, we need a process around this, where it seems like the entire idea is that every time a thing happens, there should be a documented process and a runbook on how to perform every given task, with the ultimate milestone on the hill that everyone's striving for is, ah, with enough process and enough runbooks, we can then eventually get rid of all the people who know all this stuff works, and basically staff at up with people who'd know how to follow a script and run push the button when told to buy the instruction manual. And that's always rankled, as someone who got into this space because I enjoy creative thinking, I enjoy looking at the relationships between things. Cost and architecture are the same thing; that's how I got into this. It's not due to an undying love of spreadsheets on my part. That's my business partner's problem.But it's this idea of being able to play with the puzzle, and the more you document things with process, the more you become reliant on those things. On some level, it feels like it ossifies things to the point where change is no longer easily attainable. Is that actually what happens, or am I just wildly overstating the case? Either as possible. Or a third option, too. You're the expert; I'm just here asking ridiculous questions.Laura: Yeah, well, I think it's a balance between needing some structure, needing some guidelines around expected actions to take place. This is for a number of reasons. One, we talked about earlier about how we need multiple diverse perspectives. So, you're going to have people from different teams, from different roles in the organization, from different levels of knowledge, participating in an incident response. And so because of that, you need some form of script, some kind of process that creates some predictability, creates some common ground around how is this thing going to go, what kinds of tools do we have at our disposal to be able to either find out what's going on, fix what's going on, get the right kinds of authority to be able to take certain kinds of actions.So, you need some degree of process around that, but I agree with you that too much process and the idea that we can actually apply operational procedures to these kinds of environments is completely counterproductive. And what it ends up doing is it ends up, kind of, saying, “Well, you didn't follow those rules and that's why the incident went the way it did,” as opposed to saying, “Oh, these rules actually didn't apply in ways that really matter, given the problem that was faced, and there was no latitude to be able to adapt in real-time or to be able to improvise, to be creative in how you're thinking about the problem.” And so you've really kind of put the responders into a bit of a box, and not given them productive avenues to, kind of, move forward from. So, having worked in a lot of very highly regulated environments, I recognize there's value in having prescription, but it's also about enabling performance and enabling adaptive performance in real-time when you're working at the speeds and the scales that we are in this kind of world.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance query accelerator for the Oracle MySQL Database Service, although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLAP and OLTP—don't ask me to pronounce those acronyms again—workloads directly from your MySQL database and eliminate the time-consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: Yeah, and let's be fair, here; I am setting up something of a false dichotomy. I'm not suggesting that the answer is oh, you either are mired in process, or it is the complete Wild West. If you start a new role and, “Great. How do I get started? What's the onboarding process?” Like, “Step one, write those docs for us.”Or how many times have we seen the pattern where day-one onboarding is, “Well, here's the GitHub repo, and there's some docs there. And update it as you go because this stuff is constantly in motion.” That's a terrible first-time experience for a lot of folks, so there has to be something that starts people off in the right direction, a sort of a quick guide to this is what's going on in the environment, and here are some directions for exploration. But also, you aren't going to be able to get that to a level of granularity where it's going to be anything other than woefully out of date in most environments without resorting to draconian measures. I feel like—Laura: Yeah.Corey: —the answer is somewhere in the middle, and where that lives depends upon whether you're running Twitter for Pets or a nuclear reactor control system.Laura: Yeah. And it brings us to a really important point of organizational life, which is that we are always operating under constraints. We are always managing trade-offs in this space. It's very acute when you're in an incident and you're like, “Do I bring the system back up but I still don't know what's wrong or do I leave it down a little bit longer and I can collect more information about the nature of the problem that I'm facing?”But more chronic is the fact that organizations are always facing this need to build the next thing, not focus on what just happened. You talked about the next incident starting and jumping in before we can actually really digest what just happened with the last incident; these kinds of pressures and constraints are a very normal part of organizational life, and we are balancing those trade-offs between time spent on one thing versus another as being innovating, learning, creating change within our environment. The reason why it's important to surface that is that it helps change the conversation when we're doing any kind of post-incident learning session.It's like, oh, it allows us to surface things that we typically can't say in a meeting. “Well, I wasn't able to do that because I know that team has a code freeze going on right now.” Or, “We don't have the right type of, like, service agreement to get our vendor on the phone, so we had to sit and wait for the ticket to get dealt with.” Those kinds of things are very real limiters to how people can act during incidents, and yet, don't typically get brought up because they're just kind of chronic, everyday things that people deal with.Corey: As you look across the industry, what do you think that organizations are getting, I guess, it's the most wrong when it comes to these things today? Because most people are no longer in the era of, “All right. Who's the last person to touch it? Well, they're fired.” But I also don't think that they're necessarily living the envisioned reality that you described in the Howie Guide, as well as the areas of research you're exploring. What's the most common failure mode?Laura: Hmm. I got to tweak that a little bit to make it less about the failure mode and more about the challenges that I see organizations facing because there are many failure modes, but some common issues that we see companies facing is they're like, “Okay, we buy into this idea that we should start looking at the system, that we should start looking beyond the technical thing that broke and more broadly at how did different aspects of our system interact.” And I mean, both people as a part of the system, I mean processes part of the system, as well as the software itself. And so that's a big part of why we wrote the Howie Guide, is because companies are struggling with that gap between, “Okay, we're not entirely sure what this means to our organization, but we're willing to take steps to get there.” But there's a big gap between recognizing that and jumping into the academic literature that's been around for many, many years from other kinds of high-risk, high-consequence type domains.So, I think some of the challenges they face is actually operationalizing some of these ideas, particularly when they already have processes and practices in place. There's ideas that are very common throughout an organization that take a long time to shift people's thinking around, the implicit biases or orientations towards a problem that we as individuals have, all of those kinds of things take time. You mentioned the Overton window, and that's a great example of it is intolerable in some organizations to have a discussion about what do people know and not know about different aspects of the system because there's an assumption that if you're the engineer responsible for that, you should know everything. So, those challenges, I think, are quite limiting to helping organizations move forward. Unfortunately, we see not a lot of time being put into really understanding how an incident was handled, and so typically, reviews get done on the side of the desk, they get done with a minimal amount of effort, and then the learnings that come out of them are quite shallow.Corey: Is there a maturity model, where it makes sense to begin investing in this, whereas if you've do it too quickly, you're not really going to be able to ship your MVP and see what happens; if you go too late, you have a globe-spanning service that winds up being down all the time so no one trusts it. What is the sweet spot for really started to care about incident response? In other words, how do people know that it's time to start taking this stuff more seriously?Laura: Ah. Well… you have kids?Corey: Oh, yes. One and four. Oh yeah.Laura: Right—Corey: Demons. Little demons whom I love very much.Laura: [laugh]. They look angelic, Corey. I don't know what you're talking about. Would you not teach them how to learn or not teach them about the world until they started school?Corey: No, but it would also be considered child abuse at this age to teach them about the AWS bill. So, there is a spectrum as far as what is appropriate learnings at what stage.Laura: Yeah, absolutely. So, that's a really good point is that depending on where you are at in your operation, you might not have the resources to be able to launch full-scale investigations. You may not have the complexity within your system, within your teams, and you don't have the legacy to, sort of, draw through, to pull through, that requires large-scale investigations with multiple investigators. That's really why we were trying to make the Howie Guide very applicable to a broad range of organizations is, here are the tools, here are the techniques that we know can help you understand more about the environment that you're operating in, the people that you're working with, so that you can level up over time, you can draw more and more techniques and resources to be able to go deeper on those kinds of things over time. It might be appropriate at an early stage to say, hey, let's do these really informally, let's pull the team together, talk about how things got set up, why choices were made to use the kinds of components that we use, and talk a little bit more about why someone made a decision they did.That might be low-risk when you're small because y'all know each other, largely you know the decisions, those conversations can be more frank. As you get larger, as more people you don't know are on those types of calls, you might need to handle them differently so that people have psychological safety, to be able to share what they knew and what they didn't know at the time. It can be a graduated process over time, but we've also seen very small, early-stage companies really treat this seriously right from the get-go. At Jeli, I mean, one of our core fundamentals is learning, right, and so we do, we spend time on sharing with each other, “Oh, my mental model about this was X. Is that the same as what you have?” “No.” And then we can kind of parse what's going on between those kinds of things. So, I think it really is an orientation towards learning that is appropriate any size or scale.Corey: I really want to thank you for taking the time to speak with me today. If people want to learn more about what you're up to, how you view these things and possibly improve their own position on these areas, where can they find you?Laura: So, we have a lot of content on jeli.io. I am also on Twitter at—Corey: Oh, that's always a mistake.Laura: [laugh]. @lauramdmaguire. And I love to talk about this stuff. I love to hear how people are interpreting, kind of, some of the ideas that are in the resilience engineering space. Should I say, “Tweet at me,” or is that dangerous, Corey?Corey: It depends. I find that the listeners to this show are all far more attractive than the average, and good people, through and through. At least that's what I tell the sponsors. So yeah, it should be just fine. And we will of course include links to those in the [show notes 00:27:11].Laura: Sounds good.Corey: Thank you so much for your time. I really appreciate it.Laura: Thank you. It's been a pleasure.Corey: Laura Maguire, researcher at Jeli. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please give a five-star review on your podcast platform of choice along with an angry, insulting comment that I will read just as soon as I get them all to display on my single-pane-of-glass dashboard.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About MilesAs Chief Technology Officer at SADA, Miles Ward leads SADA's cloud strategy and solutions capabilities. His remit includes delivering next-generation solutions to challenges in big data and analytics, application migration, infrastructure automation, and cost optimization; reinforcing our engineering culture; and engaging with customers on their most complex and ambitious plans around Google Cloud.Previously, Miles served as Director and Global Lead for Solutions at Google Cloud. He founded the Google Cloud's Solutions Architecture practice, launched hundreds of solutions, built Style-Detection and Hummus AI APIs, built CloudHero, designed the pricing and TCO calculators, and helped thousands of customers like Twitter who migrated the world's largest Hadoop cluster to public cloud and Audi USA who re-platformed to k8s before it was out of alpha, and helped Banco Itau design the intercloud architecture for the bank of the future.Before Google, Miles helped build the AWS Solutions Architecture team. He wrote the first AWS Well-Architected framework, proposed Trusted Advisor and the Snowmobile, invented GameDay, worked as a core part of the Obama for America 2012 “tech” team, helped NASA stream the Curiosity Mars Rover landing, and rebooted Skype in a pinch.Earning his Bachelor of Science in Rhetoric and Media Studies from Willamette University, Miles is a three-time technology startup entrepreneur who also plays a mean electric sousaphone.Links: SADA.com: https://sada.com Twitter: https://twitter.com/milesward Email: miles@sada.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I am joined today, once again by my friend and yours, Miles Ward, who's the CTO at SADA. However, he is, as I think of him, the closest thing the Google Cloud world has to Corey Quinn. Now, let's be clear, not the music and dancing part that is Forrest Brazeal, but Forrest works at Google Cloud, whereas Miles is a reasonably salty third-party. Miles, thank you for coming back and letting me subject you to that introduction.Miles: Corey, I appreciate that introduction. I am happy to provide substantial salt. It is easy, as I play brass instruments that produce my spit in high volumes. It's the most disgusting part of any possible introduction. For the folks in the audience, I am surrounded by a collection of giant sousaphones, tubas, trombones, baritones, marching baritones, trumpets, and pocket trumpets.So, Forrest threw down the gauntlet and was like, I can play a keyboard, and sing, and look cute at the same time. And so I decided to fail at all three. We put out a new song just a bit ago that's, like, us thanking all of our customers and partners, covering Kool & the Gang “Celebration,” and I neither look good, [laugh] play piano, or smiling, or [capturing 00:01:46] any of the notes; I just play the bass part, it's all I got to do.Corey: So, one thing that I didn't get to talk a lot about because it's not quite in my universe, for one, and for another, it is during the pre re:Invent—pre:Invent, my nonsense thing—run up, which is Google Cloud Next.Miles: Yes.Corey: And my gag a few years ago is that I'm not saying that Google is more interested in what they're building and what they're shipping, but even their conference is called Next. Buh dum, hiss.Miles: [laugh].Corey: So, I didn't really get to spend a lot of attention on the Google Cloud releases that came out this year, but given that SADA is in fact the, I believe, largest Google Cloud partner on the internet, and thus the world—Miles: [unintelligible 00:02:27] new year, three years in a row back, baby.Corey: Fantastic. I assume someone's watch got stuck or something. But good work. So, you have that bias in the way that I have a bias, which is your business is focused around Google Cloud the way that mine is focused on AWS, but neither of us is particularly beholden to that given company. I mean, you do have the not getting fired as partner, but that's a bit of a heavy lift; I don't think I can mouth off well enough to get you there.So, we have a position of relative independence. So, you were tracking Google Next, the same way that I track re:Invent. Well, not quite the same way I track re:Invent; there are some significant differences. What happened at Cloud Next 2021, that the worst of us should be paying attention to?Miles: Sure. I presented 10% of the material at the first re:Invent. There are 55 sessions; I did six. And so I have been at Cloud events for a really long time and really excited about Google's willingness to dive into demos in a way that I think they have been a little shy about. Kelsey Hightower is the kind of notable deep exception to that. Historically, he's been ready to dive into the, kind of, heavy hands-on piece but—Corey: Wait, those were demos? [Thought 00:03:39] was just playing Tetris on stage for the love of it.Miles: [laugh]. No. And he really codes all that stuff up, him and the whole team.Corey: Oh, absol—I'm sorry. If I ever grow up, I wish to be Kelsey Hightower.Miles: [laugh]. You and me both. So, he had kind of led the charge. We did a couple of fun little demos while I was there, but they've really gotten a lot further into that, and I think are doing a better job of packaging the benefits to not just developers, but also operators and data scientists and the broader roles in the cloud ecosystem from the new features that are being launched. And I think, different than the in-person events where there's 10, 20,000, 40,000 people in the audience paying attention, I think they have to work double-hard to capture attention and get engineers to tune in to what's being launched.But if you squint and look close, there are some, I think, very interesting trends that sit in the back of some of the very first launches in what I think are going to be whole veins of launches from Google over the course of the next several years that we are working really hard to track along with and make sure we're extracting maximum value from for our customers.Corey: So, what was it that they announced that is worth paying attention to? Now, through the cacophony of noise, one announcement that [I want to note 00:04:49] was tied to Next was the announcement that GME group, I believe, is going to be putting their futures exchange core trading systems on Google Cloud. At which point that to me—and I know people are going to yell at me, and I don't even slightly care—that is the last nail in the coffin of the idea that well, Google is going to turn this off in a couple years. Sorry, no. That is not a thing that's going to happen. Worst case, they might just stop investing it as aggressively as they are now, but even that would be just a clown-shoes move that I have a hard time envisioning.Miles: Yeah, you're talking now over a dozen, over ten year, over a billion-dollar commitments. So, you've got to just really, really hate your stock price if you're going to decide to vaporize that much shareholder value, right? I mean, we think that, in Google, stock price is a material fraction of the recognition of the growth trajectory for cloud, which is now basically just third place behind YouTube. And I think you can do the curve math, it's not like it's going to take long.Corey: Right. That requires effectively ejecting Thomas Kurian as the head of Google Cloud and replacing him with the former SVP of Bad Decisions at Yahoo.Miles: [laugh]. Sure. Google has no shyness about continuing to rotate leadership. I was there through three heads of Google Cloud, so I don't expect that Thomas will be the last although I think he may well go down in history as having been the best. The level of rotation to the focuses that I think are most critical, getting enterprise customers happy, successful, committed, building macroscale systems, in systems that are critical to the core of the business on GCP has grown at an incredible rate under his stewardship. So, I think he's doing a great job.Corey: He gets a lot of criticism—often from Googlers—when I wind up getting the real talk from them, which is, “Can you tell me what you really think?” Their answer is, “No,” I'm like, “Okay, next question. Can I go out and buy you eight beers and then”— and it's like, “Yeah.” And the answer that I get pretty commonly is that he's brought too much Oracle into Google. And okay, that sounds like a bad thing because, you know, Oracle, but let's be clear here, but what are you talking about specifically? And what they say distills down to engineers are no longer the end-all be-all of everything that Google Cloud. Engineers don't get to make sales decisions, or marketing decisions, or in some cases, product decisions. And that is not how Google has historically been run, and they don't like the change. I get it, but engineering is not the only hard thing in the world and it's not the only business area that builds value, let's be clear on this. So, I think that the things that they don't like are in fact, what Google absolutely needs.Miles: I think, one, the man is exceptionally intimidating and intentionally just hyper, hyper attentive to his business. So, one of my best employees, Brad [Svee 00:07:44], he worked together with me to lay out what was the book of our whole department, my team of 86 people there. What are we about? What do we do? And like I wanted this as like a memoriam to teach new hires as got brought in. So, this is, like, 38 pages of detail about our process, our hiring method, our promotional approach, all of it. I showed that to my new boss who had come in at the time, and he thought some of the pictures looked good. When we showed it to TK, he read every paragraph. I watched him highlight the paragraphs as he went through, and he read it twice as fast as I can read the thing. I think he does that to everybody's documents, everywhere. So, there's a level of just manual rigor that he's brought to the practice that was certainly not there before that. So, that alone, it can be intimidating for folks, but I think people that are high performance find that very attractive.Corey: Well, from my perspective, he is clearly head and shoulders above Adam Selipsky, and Scott Guthrie—the respective heads of AWS and Azure—for one key reason: He is the only one of those three people who follows me on Twitter. And—Miles: [laugh].Corey: —honestly, that is how I evaluate vendors.Miles: That's the thing. That's the only measure, yep. I've worked on for a long time with Selipsky, and I think that it will be interesting to see whether Adam's approach to capital allocation—where he really, I think, thinks of himself as the manager of thousands of startups, as opposed to a manager of a global business—whether that's a more efficient process for creating value for customers, then, where I think TK is absolutely trying to build a much more unified, much more singular platform. And a bunch of the launches really speak to that, right? So, one of the product announcements that I think is critical is this idea of the global distributed cloud, Google Distributed Cloud.We started with Kubernetes. And then you layer on to that, okay, we'll take care of Kubernetes for you; we call that Anthos. We'll build a bunch of structural controls and features into Anthos to make it so that you can really deal with stuff in a global way. Okay, what does that look like further? How do we get out into edge environments? Out into diverse hardware? How do we partner up with everybody to make sure that, kind of like comparing Apple's approach to Google's approach, you have an Android ecosystem of Kubernetes providers instead of just one place you can buy an outpost. That's generally the idea of GDC. I think that's a spot where you're going to watch Google actually leverage the muscle that it already built in understanding open-source dynamics and understanding collaboration between companies as opposed to feeling like it's got to be built here. We've got to sell it here. It's got to have our brand on it.Corey: I think that there's a stupendous and extreme story that is still unfolding over at Google Cloud. Now, re:Invent this year, they wound up talking all about how what they were rolling out was a focus on improving primitives. And they're right. I love their managed database service that they launched because it didn't exist.Miles: Yeah Werner's slide, “It's primitives, not frameworks.” I was like, I think customers want solutions, not frameworks or primitives. [laugh]. What's your plan?Corey: Yeah. However, I take a different perspective on all of this, which is that is a terrific spin on the big headline launches all missed the re:Invent timeline, and… oops, so now we're just going to talk about these other things instead. And that's great, but then they start talking about industrial IOT, and mainframe migrations, and the idea of private 5G, and running fleets of robots. And it's—Miles: Yeah, that's a cool product.Corey: Which one? I'm sorry, they're all very different things.Miles: Private 5G.Corey: Yeah, if someone someday will explain to me how it differs from Wavelength, but that's neither here nor there. You're right, they're all interesting, but none of them are actually doing the thing that I do, which is build websites, [unintelligible 00:11:31] looking for web services, it kind of says it in the name. And it feels like it's very much broadening into everything, and it's very difficult for me to identify—and if I have trouble that I guarantee you customers do—of, which services are for me and which are very much not? In some cases, the only answer to that is to check the pricing. I thought Kendra, their corporate information search thing was for me, then it's 7500 bucks a month to get started with that thing, and that is, “I can hire an internal corporate librarian to just go and hunt through our Google Drive.” Great.Miles: Yeah.Corey: So, there are—or our Dropbox, or our Slack. We have, like, five different information repositories, and this is how corporate nonsense starts, let me assure you.Miles: Yes. We call that luxury SaaS, you must enjoy your dozens of overlapping bills for, you know, what Workspace gives you as a single flat rate.Corey: Well, we have [unintelligible 00:12:22] a lot of this stuff, too. Google Drive is great, but we use Dropbox for holding anything that touches our customer's billing information, just because I—to be clear, I do not distrust Google, but it also seems a little weird to put the confidential billing information for one of their competitors on there to thing if a customer were to ask about it. So, it's the, like, I don't believe anyone's doing anything nefarious, but let's go ahead and just make sure, in this case.Miles: Go further man. Vimeo runs on GCP. You think YouTube doesn't want to look at Vimeo stats? Like they run everything on GCP, so they have to have arrived at a position of trust somehow. Oh, I know how it's called encryption. You've heard of encryption before? It's the best.Corey: Oh, yes. I love these rumors that crop up every now and again that Amazon is going to start scanning all of its customer content, somehow. It's first, do you have any idea how many compute resources that would take and to if they can actually do that and access something you're storing in there, against their attestations to the contrary, then that's your story because one of them just makes them look bad, the other one utterly destroys their entire business.Miles: Yeah.Corey: I think that that's the one that gets the better clicks. So no, they're not doing that.Miles: No, they're not doing that. Another product launch that I thought was super interesting that describes, let's call it second place—the third place will be the one where we get off into the technical deep end—but there's a whole set of coordinated work they're calling Cortex. So, let's imagine you go to a customer, they say, “I want to understand what's happening with my business.” You go, “Great.” So, you use SAP, right? So, you're a big corporate shop, and that's your infrastructure of choice. There are a bunch of different options at that layer.When you set up SAP, one of the advantages that something like that has is they have, kind of, pre-built configurations for roughly your business, but whatever behaviors SAP doesn't do, right, say, data warehousing, advanced analytics, regression and projection and stuff like that, maybe that's somewhat outside of the core wheelhouse for SAP, you would expect like, oh okay, I'll bolt on BigQuery. I'll build that stuff over there. We'll stream the data between the two. Yeah, I'm off to the races, but the BigQuery side of the house doesn't have this like bitching menu that says, “You're a retailer, and so you probably want to see these 75 KPIs, and you probably want to chew up your SKUs in exactly this way. And here's some presets that make it so that this is operable out of the box.”So, they are doing the three way combination: Consultancies plus ISVs plus Google products, and doing all the pre-work configuration to go out to a customer and go I know what you probably just want. Why don't I just give you the whole thing so that it does the stuff that you want? That I think—if that's the very first one, this little triangle between SAP, and Big Query, and a bunch of consultancies like mine, you have to imagine they go a lot further with that a lot faster, right? I mean, what does that look like when they do it with Epic, when they go do it with Go just generally, when they go do it with Apache? I've heard of that software, right? Like, there's no reason not to bundle up what the obvious choices are for a bunch of these combinations.Corey: The idea of moving up the stack and offering full on solutions, that's what customers actually want. “Well, here's a bunch of things you can do to wind up wiring together to build a solution,” is, “Cool. Then I'm going to go hire a company who's already done that is going to sell it to me at a significant markup because I just don't care.” I pay way more to WP Engine than I would to just run WordPress myself on top of AWS or Google Cloud. In fact, it is on Google Cloud, but okay.Miles: You and me both, man. WP Engine is the best. I—Corey: It's great because—Miles: You're welcome. I designed a bunch of the hosting on the back of that.Corey: Oh, yeah. But it's also the—I—well, it costs a little bit more that way. Yeah, but guess what's not—guess what's more expensive than that bill, is my time spent doing the care and feeding of this stuff. I like giving money to experts and making it their problem.Miles: Yeah. I heard it said best, Lego is an incredible business. I love their product, and you can build almost any toy with it. And they have not displaced all other plastic toy makers.Corey: Right.Miles: Some kids just want to buy a little car. [laugh].Corey: Oh, yeah, you can build anything you want out of Lego bricks, which are great, which absolutely explains why they are a reference AWS customer.Miles: Yeah, they're great. But they didn't beat all other toy companies worldwide, and eliminate the rest of that market because they had the better primitive, right? These other solutions are just as valuable, just as interesting, tend to have much bigger markets. Lego is not the largest toy manufacturer in the world. They are not in the top five of toy manufacturers in the world, right?Like, so chasing that thread, and getting all the way down into the spots where I think many of the cloud providers on their own, internally, had been very uncomfortable. Like, you got to go all the way to building this stuff that they need for that division, inside of that company, in that geo, in that industry? That's maybe, like, a little too far afield. I think Google has a natural advantage in its more partner-oriented approach to create these combinations that lower the cost to them and to customers to getting out of that solution quick.Corey: So, getting into the weeds of Google Next, I suppose, rather than a whole bunch of things that don't seem to apply to anyone except the four or five companies that really could use it, what things did Google release that make the lives of people building, you know, web apps better?Miles: This is the one. So, I'm at Amazon, hanging out as a part of the team that built up the infrastructure for the Obama campaign in 2012, and there are a bunch of Googlers there, and we are fighting with databases. We are fighting so hard, in fact, with RDS that I think we are the only ones that [Raju 00:17:51] has ever allowed to SSH into our RDS instances to screw with them.Corey: Until now, with the advent of RDS Custom, meaning that you can actually get in as root; where that hell that lands between RDS and EC2 is ridiculous. I just know that RDS can now run containers.Miles: Yeah. I know how many things we did in there that were good for us, and how many things we did in there that were bad for us. And I have to imagine, this is not a feature that they really ought to let everybody have, myself included. But I will say that what all of the Googlers that I talk to, you know, at the first blush, were I'm the evil Amazon guy in to, sort of, distract them and make them build a system that, you know, was very reliable and ended up winning an election was that they had a better database, and they had Spanner, and they didn't understand why this whole thing wasn't sitting on Spanner. So, we looked, and I read the white paper, and then I got all drooly, and I was like, yes, that is a much better database than everybody else's database, and I don't understand why everybody else isn't on it. Oh, there's that one reason, but you've heard of it: No other software works with it, anywhere in the world, right? It's utterly proprietary to Google. Yes, they were kind—Corey: Oh, you want to migrate it off somewhere else, or a fraction of it? Great. Step one, redo your data architecture.Miles: Yeah, take all of my software everywhere, rewrite every bit of it. And, oh all those commercial applications? Yeah, forget all those, you got, too. Right? It was very much where Google was eight years ago. So, for me, it was immensely meaningful to see the launch at Next where they described what they are building—and have now built; we have alpha access to it—a Postgres layer for Spanner.Corey: Is that effectively you have to treat it as Postgres at all times, or is it multimodal access?Miles: You can get in and tickle it like Spanner, if you want to tickle it like Spanner. And in reality, Spanner is ANSI SQL compliant; you're still writing SQL, you just don't have to talk to it like a REST endpoint, or a GRPC endpoint, or something; you can, you know, have like a—Corey: So, similar to Azure's Cosmos DB, on some level, except for the part where you can apparently look at other customers' data in that thing?Miles: [laugh]. Exactly. Yeah, you will not have a sweeping discovery of incredible security violations in the structure Spanner, in that it is the control system that Google uses to place every ad, and so it does not suck. You can't put a trillion-dollar business on top of a database and not have it be safe. That's kind of a thing.Corey: The thing that I find is the most interesting area of tech right now is there's been this rise of distributed databases. Yugabyte—or You-ji-byte—Pla-netScale—or PlanetScale, depending on how you pronounce these things.Miles: [laugh]. Yeah, why, why is G such an adversarial consonant? I don't understand why we've all gotten to this place.Corey: Oh, yeah. But at the same time, it's—so you take a look at all these—and they all are speaking Postgres; it is pretty clear that ‘Postgres-squeal' is the thing that is taking over the world as far as databases go. If I were building something from scratch that used—Miles: For folks in the back, that's PostgreSQL, for the rest of us, it's okay, it's going to be, all right.Corey: Same difference. But yeah, it's the thing that is eating the world. Although recently, I've got to say, MongoDB is absolutely stepping up in a bunch of really interesting ways.Miles: I mean, I think the 4.0 release, I'm the guy who wrote the MongoDB on AWS Best Practices white paper, and I would grab a lot of customer's and—Corey: They have to change it since then of, step one: Do not use DocumentDB; if you want to use Mongo, use Mongo.Miles: Yeah, that's right. No, there were a lot of customers I was on the phone with where Mongo had summarily vaporized their data, and I think they have made huge strides in structural reliability over the course of—you know, especially this 4.0 launch, but the last couple of years, for sure.Corey: And with all the people they've been hiring from AWS, it's one of those, “Well, we'll look at this now who's losing important things from production?”Miles: [laugh]. Right? So, maybe there's only actually five humans who know how to do operations, and we just sort of keep moving around these different companies.Corey: That's sort of my assumption on these things. But Postgres, for those who are not looking to depart from the relational model, is eating the world. And—Miles: There's this, like, basic emotional thing. My buddy Martin, who set up MySQL, and took it public, and then promptly got it gobbled up by the Oracle people, like, there was a bet there that said, hey, there's going to be a real open database, and then squish, like, the man came and got it. And so like, if you're going to be an independent, open-source software developer, I think you're probably not pushing your pull requests to our friends at Oracle, that seems weird. So instead, I think Postgres has gobbled up the best minds on that stuff.And it works. It's reliable, it's consistent, and it's functional in all these different, sort of, reapplications and subdivisions, right? I mean, you have to sort of squint real hard, but down there in the guts of Redshift, that's Postgres, right? Like, there's Postgres behind all sorts of stuff. So, as an interface layer, I'm not as interested about how it manages to be successful at bossing around hardware and getting people the zeros and ones that they ask for back in a timely manner.I'm interested in it as a compatibility standard, right? If I have software that says, “I need to have Postgres under here and then it all will work,” that creates this layer of interop that a bunch of other products can use. So, folks like PlanetScale, and Yugabyte can say, “No, no, no, it's cool. We talk Postgres; that'll make it so your application works right. You can bring a SQL alchemy and plug it into this, or whatever your interface layer looks like.”That's the spot where, if I can trade what is a fairly limited global distribution, global transactional management on literally ridiculously unlimited scalability and zero operations, I can handle the hard parts of running a database over to somebody else, but I get my layer, and my software talks to it, I think that's a huge step.Corey: This episode is sponsored in part by my friends at Cloud Academy. Something special just for you folks. If you missed their offer on Black Friday or Cyber Monday or whatever day of the week doing sales it is—good news! They've opened up their Black Friday promotion for a very limited time. Same deal, $100 off a yearly plan, $249 a year for the highest quality cloud and tech skills content. Nobody else can get this because they have a assured me this not going to last for much longer. Go to CloudAcademy.com, hit the "start free trial" button on the homepage, and use the Promo code cloud at checkout. That's c-l-o-u-d, like loud, what I am, with a “C” in front of it. It's a free trial, so you'll get 7 days to try it out to make sure it's really a good fit for you, nothing to lose except your ignorance about cloud. My thanks again for sponsoring my ridiculous nonsense.Corey: I think that there's a strong movement toward building out on something like this. If it works, just because—well, I'm not multiregion today, but I can easily see a world in which I'd want to be. So, great. How do you approach the decision between—once this comes out of alpha; let's be clear. Let's turn this into something that actually ships, and no, Google that does not mean slapping a beta label on it for five years is the answer here; you actually have to stand behind this thing—but once it goes GA—Miles: GA is a good thing.Corey: Yeah. How do you decide between using that, or PlanetScale? Or Yugabyte?Miles: Or Cockroach or or SingleStore, right? I mean, there's a zillion of them that sit in this market. I think the core of the decision making for me is in every team you're looking at what skills do you bring to bear and what problem that you're off to go solve for customers? Do the nuances of these products make it easier to solve? So, I think there are some products that the nature of what you're building isn't all that dependent on one part of the application talking to another one, or an event happening someplace else mattering to an event over here. But some applications, that's, like, utterly critical, like, totally, totally necessary.So, we worked with a bunch of like Forex exchange trading desks that literally turn off 12 hours out of the day because they can only keep it consistent in one geographical location right near the main exchanges in New York. So, that's a place where I go, “Would you like to trade all day?” And they go, “Yes, but I can't because databases.” So, “Awesome. Let's call the folks on the Spanner side. They can solve that problem.”I go, “Would you like to trade all day and rewrite all your software?” And they go, “No.” And I go, “Oh, okay. What about trade all day, but not rewrite all your software?” There we go. Now, we've got a solution to that kind of problem.So like, we built this crazy game, like, totally other end of the ecosystem with the Dragon Ball Z people, hysterical; your like—you literally play like Rock, Paper, Scissors with your phone, and if you get a rock, I throw a fireball, and you get a paper, then I throw a punch, and we figure out who wins. But they can play these games like Europe versus Japan, thousands of people on each side, real-time, and it works.Corey: So, let's be clear, I have lobbied a consistent criticism at Google for a while now, which is the Google Cloud global control plane. So, you wind up with things like global service outages from time to time, you wind up with this thing is now broken for everyone everywhere. And that, for a lot of these use cases, is a problem. And I said that AWS's approach to regional isolation is the right way to do it. And I do stand by that assessment, except for the part where it turns out there's a lot of control plane stuff that winds up single tracking through us-east-1, as we learned in the great us-east-1 outage of 2021.Miles: Yeah, when I see customers move from data center to AWS, what they expect is a higher count of outages that lasts less time. That's the trade off, right? There's going to be more weird spurious stuff, and maybe—maybe—if they're lucky, that outage will be over there at some other region they're not using. I see almost exactly the same promise happening to folks that come from AWS—and in particular from Azure—over onto GCP, which is, there will be probably a higher frequency of outages at a per product level, right? So, like sometimes, like, some weird product takes a screw sideways, where there is structural interdependence between quite a few products—we actually published a whole internal structural map of like, you know, it turns out that Cloud SQL runs on top of GCE not on GKE, so you can expect if GKE goes sideways, Cloud SQL is probably not going to go sideways; the two aren't dependent on each other.Corey: You take the status page and Amazon FreeRTOS in a region is having an outage today or something like that. You're like, “Oh, no. That's terrible. First, let me go look up what the hell that is.” And I'm not using it? Absolutely not. Great. As hyperscalers, well, hyperscale, they're always things that are broken in different ways, in different locations, and if you had a truly accurate status page, it would all be red all the time, or varying shades of red, which is not helpful. So, I understand the challenge there, but very often, it's a partition that is you are not exposed to, or the way that you've architected things, ideally, means it doesn't really matter. And that is a good thing. So, raw outage counts don't solve that. I also maintain that if I were to run in a single region of AWS or even a single AZ, in all likelihood, I will have a significantly better uptime across the board than I would if I ran it myself. Because—Miles: Oh, for sure.Corey: —it is—Miles: For sure they're way better at ops than you are. Me, right?Corey: Of course.Miles: Right? Like, ridiculous.Corey: And they got that way, by learning. Like, I think in 2022, it is unlikely that there's going to be an outage in an AWS availability zone by someone tripping over a power cable, whereas I have actually done that. So, there's a—to be clear in a data center, not an AWS facility; that would not have flown. So, there is the better idea of of going in that direction. But the things like Route 53 is control plane single-tracking through the us-east-1, if you can't make DNS changes in an outage scenario, you may as well not have a DR plan, for most use cases.Miles: To be really clear, it was a part of the internal documentation on the AWS side that we would share with customers to be absolutely explicit with them. It's not just that there are mistakes and accidents which we try to limit to AZs, but no, go further, that we may intentionally cause outages to AZs if that's what allows us to keep broader service health higher, right? They are not just a blast radius because you, oops, pulled the pin on the grenade; they can actually intentionally step on the off button. And that's different than the way Google operates. They think of each of the AZs, and each of the regions, and the global system as an always-on, all the time environment, and they do not have systems where one gets, sort of, sacrificed for the benefit of the rest, right, or they will intentionally plan to take a system offline.There is no planned downtime in the SLA, where the SLAs from my friends at Amazon and Azure are explicit to, if they choose to, they decide to take it offline, they can. Now, that's—I don't know, I kind of want the contract that has the other thing where you don't get that.Corey: I don't know what the right answer is for a lot of these things. I think multi-cloud is dumb. I think that the idea of having this workload that you're going to seamlessly deploy to two providers in case of an outage, well guess what? The orchestration between those two providers is going to cause you more outages than you would take just sticking on one. And in most cases, unless you are able to have complete duplication of not just functionality but capacity between those two, congratulations, you've now just doubled your number of single points of failure, you made the problem actively worse and more expensive. Good job.Miles: I wrote an article about this, and I think it's important to differentiate between dumb and terrifyingly shockingly expensive, right? So, I have a bunch of customers who I would characterize as rich, as like, shockingly rich, as producing businesses that have 80-plus percent gross margins. And for them, the costs associated with this stuff are utterly rational, and they take on that work, and they are seeing benefits, or they wouldn't be doing it.Corey: Of course.Miles: So, I think their trajectory in technology—you know, this is a quote from a Google engineer—it's just like, “Oh, you want to see what the future looks like? Hang out with rich people.” I went into houses when I was a little kid that had whole-home automation. I couldn't afford them; my mom was cleaning house there, but now my house, I can use my phone to turn on the lights. Like—Corey: You know, unless us-east-1 is having a problem.Miles: Hey, and then no Roomba for you, right? Like utterly offline. So—Corey: Roomba has now failed to room.Miles: Conveniently, my lights are Philips Hue, and that's on Google, so that baby works. But it is definitely a spot where the barrier of entry and the level of complexity required is going down over time. And it is definitely a horrible choice for 99% of the companies that are out there right now. But next year, it'll be 98. And the year after that, it'll probably be 97. [laugh].And if I go inside of Amazon's data centers, there's not one manufacturer of hard drives, there's a bunch. So, that got so easy that now, of course you use more than one; you got to do—that's just like, sort of, a natural thing, right? These technologies, it'll move over time. We just aren't there yet for the vast, vast majority of workloads.Corey: I hope that in the future, this stuff becomes easier, but data transfer fees are going to continue to be a concern—Miles: Just—[makes explosion noise]—Corey: Oh, man—Miles: —like, right in the face.Corey: —especially with the Cambrian explosion of data because the data science folks have successfully convinced the entire industry that there's value in those mode balancer logs in 2012. Okay, great. We're never deleting anything again, but now you've got to replicate all of that stuff because no one has a decent handle on lifecycle management and won't for the foreseeable future. Great, to multiple providers so that you can work on these things? Like, that is incredibly expensive.Miles: Yeah. Cool tech, from this announcement at Next that I think is very applicable, and recognized the level of like, utter technical mastery—and security mastery to our earlier conversation—that something like this requires, the product is called BigQuery Omni, what Omni allows you to do is go into the Google Cloud Console, go to BigQuery, say I want to do analysis on this data that's in S3, or in Azure Blob Storage, Google will spin up an account on your behalf on Amazon and Azure, and run the compute there for you, bring the result back. So, just transfer the answers, not the raw data that you just scanned, and no work on your part, no management, no crapola. So, there's like—that's multi-cloud. If I've got—I can do a join between a bunch of rows that are in real BigQuery over on GCP side and rows that are over there in S3. The cross-eyedness of getting something like that to work is mind blowing.Corey: To give this a little more context, just because it gets difficult to reason about these things, I can either have data that is in a private subnet in AWS that traverses their horribly priced Managed NAT Gateways, and then goes out to the internet and sent there once, for the same cost as I could take that same data and store it in S3 in their standard tier for just shy of six full months. That's a little imbalanced, if we're being direct here. And then when you add in things like intelligent tiering and archive access classes, that becomes something that… there's no contest there. It's, if we're talking about things that are now approaching exabyte scale, that's one of those, “Yeah, do you want us to pay by a credit card?”—get serious. You can't at that scale anyway—“Invoice billing, or do we just, like, drive a dump truck full of gold bricks and drop them off in Seattle?”Miles: Sure. Same trajectory, on the multi-cloud thing. So, like a partner of ours, PacketFabric, you know, if you're a big, big company, you go out and you call Amazon and you buy 100 gigabit interconnect on—I think they call theirs Direct Connect, and then you hook that up to the Google one that's called Dedicated Interconnect. And voila, the price goes from twelve cents a gig down to two cents a gig; everybody's much happier. But Jesus, you pay the upfront for that, you got to set the thing up, it takes days to get deployed, and now you're culpable for the whole pipe if you don't use it up. Like, there are charges that are static over the course of the month.So, PacketFabric just buys one of those and lets you rent a slice of it you need. And I think they've got an incredible product. We're working with them on a whole bunch of different projects. But I also expect—like, there's no reason the cloud providers shouldn't be working hard to vend that kind of solution over time. If a hundred gigabit is where it is now, what does it look like when I get to ten gigabit? When I get to one gigabit? When I get to half gigabit? You know, utility price that for us so that we get to rational pricing.I think there's a bunch of baked-in business and cost logic that is a part of the pricing system, where egress is the source of all of the funding at Amazon for internal networking, right? I don't pay anything for the switches that connect to this machine to that machine, in region. It's not like those things are cheap or free; they have to be there. But the funding for that comes from egress. So, I think you're going to end up seeing a different model where you'll maybe have different approaches to egress pricing, but you'll be paying like an in-system networking fee.And I think folks will be surprised at how big that fee likely is because of the cost of the level of networking infrastructure that the providers deploy, right? I mean, like, I don't know, if you've gone and tried to buy a 40 port, 40 gig switch anytime recently. It's not like they're those little, you know, blue Netgear ones for 90 bucks.Corey: Exactly. It becomes this, [sigh] I don't know, I keep thinking that's not the right answer, but part of it also is like, well, you know, for things that I really need local and don't want to worry about if the internet's melting today, I kind of just want to get, like, some kind of Raspberry Pi shoved under my desk for some reason.Miles: Yeah. I think there is a lot where as more and more businesses bet bigger and bigger slices of the farm on this kind of thing, I think it's Jassy's line that you're, you know, the fat in the margin in your business is my opportunity. Like, there's a whole ecosystem of partners and competitors that are hunting all of those opportunities. I think that pressure can only be good for customers.Corey: Miles, thank you for taking the time to speak with me. If people want to learn more about you, what you're up to, your bad opinions, your ridiculous company, et cetera—Miles: [laugh].Corey: —where can they find you?Miles: Well, it's really easy to spell: SADA.com, S-A-D-A dot com. I'm Miles Ward, it's @milesward on Twitter; you don't have to do too hard of a math. It's miles@sada.com, if you want to send me an email. It's real straightforward. So, eager to reach out, happy to help. We've got a bunch of engineers that like helping people move from Amazon to GCP. So, let us know.Corey: Excellent. And we will, of course, put links to this in the [show notes 00:37:17] because that's how we roll.Miles: Yay.Corey: Thanks so much for being so generous with your time, and I look forward to seeing what comes out next year from these various cloud companies.Miles: Oh, I know some of them already, and they're good. Oh, they're super good.Corey: This is why I don't do predictions because like, the stuff that I know about, like, for example, I was I was aware of the Graviton 3 was coming—Miles: Sure.Corey: —and it turns out that if your—guess what's going to come up and you don't name Graviton 3, it's like, “Are you simple? Did you not see that one coming?” It's like—or if I don't know it's coming and I make that guess—which is not the hardest thing in the world—someone would think I knew and leaked. There's no benefit to doing predictions.Miles: No. It's very tough, very happy to do predictions in private, for customers. [laugh].Corey: Absolutely. Thanks again for your time. I appreciate it.Miles: Cheers.Corey: Myles Ward, CTO at SADA. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and be very angry in your opinion when you write that obnoxious comment, but then it's going to get lost because it's using MySQL instead of Postgres.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About AaronI am a Cloud Focused Product Management and Technical Product Ownership Consultant. I have worked on several Cloud Products & Services including resale, management & governance, cost optimisation, platform management, SaaS, PaaS. I am also recognised as a AWS Community Builder due to my work building cloud communities cross-government in the UK over the last 3 years. I have extensive commercial experience dealing with Cloud Service Providers including AWS, Azure, GCP & UKCloud. I was the Single Point of Contact for Cloud at the UK Home Office and was the business representative for the Home Office's £120m contract with AWS. I have been involved in contract negotiation, supplier relationship management & financial planning such as business cases & cost management.I run a IT Consultancy called Embue, specialising in Agile, Cloud & DevOps consulting, coaching and training. Links: Twitter: https://twitter.com/AaronBoothUK LinkedIn: https://www.linkedin.com/in/aaronboothuk/ Embue: https://embue.co.uk Publicgood.cloud: https://publicgood.cloud TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open-source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers, and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com.Corey: This episode is sponsored in part by our friends at Rising Cloud, which I hadn't heard of before, but they're doing something vaguely interesting here. They are using AI, which is usually where my eyes glaze over and I lose attention, but they're using it to help developers be more efficient by reducing repetitive tasks. So, the idea being that you can run stateless things without having to worry about scaling, placement, et cetera, and the rest. They claim significant cost savings, and they're able to wind up taking what you're running as it is in AWS with no changes, and run it inside of their data centers that span multiple regions. I'm somewhat skeptical, but their customers seem to really like them, so that's one of those areas where I really have a hard time being too snarky about it because when you solve a customer's problem and they get out there in public and say, “We're solving a problem,” it's very hard to snark about that. Multus Medical, Construx.ai and Stax have seen significant results by using them. And it's worth exploring. So, if you're looking for a smarter, faster, cheaper alternative to EC2, Lambda, or batch, consider checking them out. Visit risingcloud.com/benefits. That's risingcloud.com/benefits, and be sure to tell them that I said you because watching people wince when you mention my name is one of the guilty pleasures of listening to this podcast.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. So, when I went to re:Invent last year, I discovered a whole bunch of things I honestly was a little surprised to discover. One of those things is my guest today, Aaron Booth, who's a cloud consultant with an emphasis on sustainability. Now, you see a number of consultants at things like re:Invent, but what made Aaron interesting was that this was apparently his first time visiting the United States, and he started with not just Las Vegas, but Las Vegas to attend re:Invent. Aaron, thank you for joining me, and honestly, I'm a little surprised you survived.Aaron: Yeah, I think one of the things about going to Las Vegas or Nevada is no one really prepared me for how dry it was. I ended up walking out of re:Invent with my fingers, like, bleeding, and everything else. And there was so much about America that I didn't expect, but that was one thing I wish somebody had warned me about. But yeah, it was my first time in the US, first time at re:Invent, and I really enjoyed it. It was probably the best investment in myself and my business that I think I've done so far.Corey: It's always strange to look at a place that you live and realize, oh, yeah, this is far away for someone else. What would their experience be of coming and learning about the culture we have here? And then you go to Las Vegas, and it's easy to forget there are people who live there. And even the people who live there do not live on the strip, in the casinos, at loud, obnoxious cloud conferences. So, it feels like it's one of those ideas of oh, I'm going to go to a movie for the first time and then watching something surreal, like Memento or whatnot, that leaves everyone very confused. Like, “Is this what movies are like?” “Well, this one, but no others are quite like that.” And I feel that way about Las Vegas and re:Invent, simultaneously.Aaron: I mean, talking about movies, before it came to the US and before I came to Vegas, I was like, “Oh, how can I prepare myself for this trip?” I ended up watching Fear and Loathing in Las Vegas. And I don't know if you ever seen it, with Johnny Depp, but it's probably not the best representation, or the most modern representation what Vegas would be like. And I think halfway through the conference, went down to Fremont Street in the old downtown. And they have this massive, kind of, free block screen in the sky that is lit up and doing all these animations. And you're just thinking, “What world am I on?” And it kind of is interesting as well, from a point of view of, we're at this tech conference; it's in Vegas; what is the reason for that? And there's obviously lots of different things. We want people to have fun, but you know, it is an interesting place to put 30,000 people, especially during a pandemic.Corey: It really is. I imagine it's going to have to stay there because in a couple more years, you're going to need a three block long screen just to list all of the various services that AWS offers because they don't believe in turning anything off. Now, it would be remiss for me not to ask you, what was announced at re:Invent that got you the most, let's call it excited, I guess? What got you enthusiastic? What are you happy to start working with more?Aaron: I think from my perspective, there's a few different announcements. The first one that comes to mind is the stuff of AWS Amplify Studio, and that's taken this, kind of, no-code Figma designs and turn into a working front end. And it's really interesting for me to think about, okay, what is the point of cloud? Why are we moving forward in the world, especially in technology? And, you know, abstracting a lot of stuff we worry about today to simple drag-and-drop tools is probably going to be the next big thing for most of the world.You know, we've come from a privileged position in the West where we follow technology along the whole of the journey, where now we have an opportunity to open this out to many more regions, and many more AWS customers, for example. But for me, as a small business owner—I've run multiple businesses—there's a lot of effort you put into, okay, I need to set up a business, and a website, and newsletter, or whatever else. But the more you can just turn that into, “I've got an idea, and I can give it to people with one click,” you'll enable a lot more business and a lot more future customers as well.Corey: I was very excited about that one, too, just from a perspective of I want to drag and drop something together to make a fairly crappy web app, that sounds like the thing that I could use to do that. No, that feels a lot more like what Honeycode is trying to be, as opposed to the Amplify side of the world, which is still very focused on React. Which, okay, that makes sense. There's a lot of front end developers out there, and if you're trying to get into tech today and are asking what language should I learn, I would be very hard-pressed to advise you pick anything that isn't JavaScript because it is front end, it is back end, it runs slash eats the world. And I've just never understood it. It does not work the way that I think about computers because I'm old and grumpy. I have high hopes of where it might go, but so far I'm looking at it's [sigh] it's not what I want it to be, yet. And maybe that's just because I'm weird.Aaron: Well, I mean, you know, you mentioned part of the problem really is two different competing AWS services themselves, which with a business like AWS and their product strategy being the word, “Yes,” you know, you're never really going to get a lot of focus or forward direction with certain products. And hopefully, there'll be the next, no-code tool announced in re:Invent in a few years' time, which is exactly what we're looking for, and gives startup founders or small businesses drag-and-drop tools. But for now, there's going to be a lot of competing services.Corey: There's so much out there that it's almost impossible to wind up contextualizing re:Invent as a single event. It feels like it's too easy to step back and say, “Oh, okay. I'm here to build websites”—is what we're talking about now in the context of Amplify—and then they start talking about mainframes. And then they start talking about RoboRunner to control 10,000 robots at once. And I'm looking around going, “I don't have problems that feel a lot like that. What's the deal?”Aaron: I think even just, like you said in perspective of re:Invent is like, when you go to an event like this, that you can't experience everything and you probably have a very specific focus of, you know, what am I here to do. And I was really surprised—again, my first time at a big tech conference, as well as Vegas and the US is, how important it was just to meet people and how valuable that was. First time I met you, and you know, going from somebody who's probably very likely interacted with you on Twitter before the event to being on this podcast and having a great conversation now is kind of crazy to think that the value you can get out of it. I mean, in terms of over services, and areas of re:Invent that I found interesting was the announcement of the new sustainability pillar, as part of the well-architected framework. You know, I've tried to use that before in previous workplaces, and it has been useful. You know, I'm hoping it is more useful in the future, and the cynical part of me worries about whether the whole point of putting this as part of a well-architected framework review where the customer is supposed to do it is Amazon passing the buck for sustainability. But it's an interesting way forward for what we care about.Corey: An interesting quirk of re:Invent—to me—has always been that despite there being tens of thousands of people there are always a few folks that you wind up running into again and again and again throughout the week. One year for me it was Ben Kehoe; this trip it was you where we kept finding ourselves at the same events, we kept finding ourselves at the same restaurants, and we had three or four meals together as a result, and it was a blast talking to you. And I was definitely noticing that sustainability was a topic that you kept going back to a bunch of different ways. I mean previously, before starting your current consulting company, you did a lot of work in the government—specifically the UK Government, for those who are having trouble connecting the fact this is the first time in America to the other thing. Like, “Wow, you can be far away and work for the government?” It's like, we have more than one on this planet, as it turns out.Yes, it was a fun series of conversations, and I am honestly a little less cynical about the idea of the sustainability pillar, in no small part due to the conversations that we had together. I initially had the cynical perspective of here's how to make your cloud infrastructure more sustainable. It's, isn't that really a “you” problem? You're the cloud provider. I can't control how you get energy on the markets, how you wind up handling heat issues, how you address water issues from your data center outflows, et cetera. It seems to me that the only thing I can really do is use the services you give me, and then it becomes a “you” problem. You have a more nuanced take on it.Aaron: I think there's a log of different things to think about when it comes to sustainability. One of the main ones is, from my perspective, you know, I worked at the UK Home Office in the UK, and we'd been using cloud for about six or seven years. And just looking at how we use clouds as an enterprise organization, one of the things I really started to see was these different generations of cloud and you've got aspects of legacy infrastructure, almost, that we lifted-and-shifted in the early days, versus maybe stuff would run on serverless now. And you know, that's one element, from a customer is how you control your energy usage is actually the use of servers, how efficient your code is, and there's definitely a difference between stringing together EC2 and S3 buckets compared to using serverless or Lambda functions.Corey: There's also a question of scale. When I'm trying to build something out of Lambda functions, and okay, which region is the most cost effective way to run this thing? The Google search for that will have a larger climate impact than any decision I can make at the scale that I operate at. Whereas if you're a company running tens of thousands of instances at any given point in time and your massive scale, then yeah, the choices you make are going to have significant impact. I think that a problem AWS has always struggled with has been articulating who needs to care about what, when.If you go down the best practices for security and governance and follow the white papers, they put out as a one-person startup trying to build an idea this evening, just to see if it's viable, you're never going to get anywhere. If you ignore all those things, and now you're about to go public as a bank, you're going to have a bad time, but at what point do you have to start caring about these different things in different ways? And I don't think we know the answer yet, from a sustainability perspective.Aaron: I think it's interesting in some senses, that sustainability is only just enter the conversation when it comes to stuff we care about in businesses and enterprises. You know, we all know about risk registers, and security reviews, and all those things, but sustainability, while we've, kind of, maybe said nice public statements, and put things on our website, it's not really been a thing that's, okay, this is how we're going to run our business, and the thing we care about as number one. You know, Amazon always says security is job zero, but maybe one day someone will be saying sustainability is our job zero. And especially when it comes down to, sort of, you know, the ethics of running a business and how you want that to be run, whether it is going to be a capitalistic VC-funded venture to extract wealth from citizens and become a billionaire versus creating something that's a bit more circular, and gives back as sustainability might be a key element of what you care about when you make decisions.Corey: The challenge that I find as well is, I don't know how you can talk about the relative sustainability impact of various cloud services within the AWS umbrella without, effectively, AWS explaining to you what their margins are on different services, in many respects. Power usage is the primary driver of this and that determines the cost of running things. It is very clear that it is less expensive and more efficient to run more modern hardware than older hardware, so we start seeing, okay, wow, if I start seeing those breakdowns, what does that say about the margin on some of these products and services? And I don't think they want to give that level of transparency into their business, just because as soon as someone finds out just how profitable Managed NAT gateways are, my God, everything explodes.Aaron: I think it's interesting from a cloud provider or hyperscaler perspective, as well, is, you know, what is your USP? And I think Amazon is definitely not saying sustainability is their USP right now, and I think you know, there are other cloud providers, like Azure for example, who basically can provide you a Power BI plugin; if you just log in with your Cloud account details, it will show you a sustainability dashboard and give you more of this information that you might be looking for, whereas Amazon currently doesn't offer anything like that automated. And even having conversations with your account team or trying to get hold of the right person, Amazon isn't going to go anywhere at the moment, just because maybe that's the reason why we don't want to talk about it: It's too sensitive. I'm sure that'll change because of the public statements they've made at re:Invent now and previously of, you know, where they're going in terms of energy usage. They want to be carbon neutral by 2025, so maybe it'll change to next re:Invent, we'll get the AWS Sustainability Explorer add-on for [unintelligible 00:15:23] or 12—Corey: Oh no.Aaron: —tools to do the same thing [laugh].Corey: In the Google Cloud Console, you click around, and there are green leafs next to some services and some regions, and it's, on the one hand, okay, I appreciate the attention that is coming from. On the other hand, it feels like you're shaming me for putting things in a region that I've already built things out in when there weren't these green leafs here, and I don't know that I necessarily want to have that conversation with my entire team because we can't necessarily migrate at this point. And let's also be clear, here, I cannot fathom a scenario in which running your own data centers is ever going to be more climate-friendly than picking a hyperscaler.Aaron: And I think that's sort of, you know, we all might think about is, at the end of the day, if your sustainability strategy for your business is to go all-in-on cloud, and bet horse on AWS or another cloud provider, then, at the end of the day, that's going to be viable. I know, from the, sort of, hands-on stuff I've done with our own data centers, you can never get it as efficient as what some of these cloud providers are doing. And I mean, look at Microsoft. The fact that they're putting some of their data centers under the sea to use that as a cooling mechanism, and kind of all the interesting things that they're able to do because they can invest at scale, you're never going to be able to do that with the cupboard beyond the desks in your local office to make it more efficient or sustainable.Corey: There are definite parallels between Cloud economics and sustainability because as mentioned, I worship at the altar of Our Lady of Turn that Shit Off because that's important. If you don't have a workload running and it doesn't exist, it has no climate impact. Mostly. I'm sure there are corner cases. But that does lead to the question then of okay, what is the climate sustainability impact, for example, of storing a petabyte of data and EBS versus in S3?And that has architectural impact as well, and there's also questions of how often does it move because when you move it, Lord knows there is nothing more dear than the price of data transfer for data movement. And in order to answer those questions, they're going to start talking a lot more about their architecture. I believe that is why Peter DeSantis's keynote talked so much about—finally—the admission of what we sort of known for ages now that they use erasure coding to make S3 as durable yet inexpensive, as it is. That was super interesting. Without that disclosure, it would have been pretty clear as soon as they start publishing sustainability numbers around things like that.Aaron: And I think is really interesting, you know, when you look at your business and make decisions like that. I think the first thing to start with is do you need that data at all? What's a petabyte of data are going to do? Unless it's for serious compliance reasons for, you know, the sector or the business that you're doing, the rest of it is, you know, you've got to wonder how long is that relevant for. And you know, even as individuals, we could delete junk mail and take things off our internal emails, it's the same thing of businesses, what you're doing with this data.But it is interesting, when you look at some of the specific services, even just the tiering of S3, for example, put that into Glacier instead of keeping it on S3 general. And I think you've talked about this before, I think cost the same to transfer something in and out of Glacier as just to hold it for a month. So, at the end of the day, you've got to make these decisions in the right way, and you know, with the right goals in mind, and if you're not able to make these decisions or you need help, then that's where, you know, people like us come in to help you do this.Corey: There's also the idea of—when I was growing up, the thing they always told us about being responsible was, “Oh, turn out the lights when you're not in the room.” Great. Well, cloud economics starts to get in that direction, too. If you have a job that fires off once a day at two in the morning and it stops at four in the morning, you should not be running those instances the other 22 hours of the day. What's the deal here?And that becomes an interesting expiratory area just as far as starting to wonder, okay, so you're telling me that if I'm environmentally friendly, I'm also going to save money? Let's be clear people, in many cases—in a corporate sense—care about sustainability only insofar as that don't get yelled out about it. But when it comes to saving money, well, now you've got the power of self-interest working for you. And if you can dress them both up and do the exact same things and have two reasons to do it. That feels like it could in some respects, be an accelerator towards achieving both outcomes.Aaron: Definitely. I think, you know, at the end of the day, we all want to work on things that are going to hopefully make the world a better place. And if you use that as a way of motivating, not just yourself as a business, but the workforce and the people that you want to work for you, then that is a really great goal as well. And I think you just got to look at companies that are in this world and not doing very great things that maybe they end up paying more for engineers. I think I read an interesting article the other day about Facebook is basically offering almost double or 150 percent of over salaries because it feels like a black mark on the soul to work for that company. And if there is anything—maybe it's not greenwashing per se, but if you can just make your business a better place, then that could be something that you can hopefully attract other like-minded people with.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of, “Hello World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And let me be clear here, it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications, or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: One would really like to hope that the challenge, of course, is getting there in such a way that it, well, I guess makes sense, is probably the best way to frame it. These are still early days, and we don't know how things are going to wind up… I guess, it playing out. I have hopes, I have theories, but I just don't know.Aaron: I mean, even looking at Cloud as a concept, how long we've all worked with this now ranges probably from fifteen to five, and for me the last six years, but you got to think looking at the outages at the end of last year at Amazon, that [unintelligible 00:21:57], very close to re:Invent, that impacted a lot of different workloads, not just if you were hosted in us-west or east-1, but actually for a lot of the regional services that actually were [laugh]… discovered to be kind of integral to these regions. You know, one AZ going down can impact single-sign-on logins around the world. And let's see what Amazon looks like in ten years' time as well because it could be very different.Corey: Do you find that as you talk to folks, both in government and in private sector, that there is a legitimate interest in the sustainability story? Or is it the self-serving cynical perspective that I've painted?Aaron: I mean, a lot of my experience is biased towards the public sector, so I'll start with that. In terms of the public sector, over the last few years, especially in the UK, there's been a lot more focus on sustainability as part of your business cases and your project plans for when you're making new services or building new things. And one of the things they've recently asked every government department in the UK to do is come up with a sustainability strategy for their technology. And that's been something that a lot of people have been working on as part of something called the One Gov Cloud Strategy Working Groups—which in the UK, we do love an abbreviation, so [laugh] a bit of a long name—but I think there's definitely more of an interest in it.In terms of the private sector, I'm not too sure if that's something that people are prioritizing. A lot of the focus I kind of come across as either, we want to focus on enterprise customers, so we're going to offer migration professional services, or you're a new business and you're starting to go up and already spending a couple a hundred pounds, or thousands of pounds a month. And at that scale, it's probably not going to be something you need to worry about right now.Corey: I want to talk a little bit about how you got into tech in the first place because you told me elements of this story, and I generally find them to be—how do I put this?—they strain the bounds of credulity. So, how did you wind up in this ridiculous industry?Aaron: I mean, hoping as I explain them, you don't just think I'm a liar. I have got a Scouse accent, so you're probably predisposed towards it. But my journey into tech was quite weird, I guess, in the sense that when I was 16—I was, again, like I said, born in Liverpool and didn't really know what I wanted to do in the world, and had no idea what the hell to do. So, I was at college, and kind of what happened to me there is I joined, like, an entrepreneurship club and was like, “Okay, I'll start my own business and do something interesting.” And I went to a conference at college, and there was a panel with Richard Branson and other few of business leaders, and I stood up and asked the question said, you know, “I'm 16. I want to start a business. Where can I get money to start a business?”And the panel answered with kind of a couple of different things, but one of them was, “Get a job.” The other one was, “Get money off your parents.” And I was kind of like, “Oh, a bit weird. I've got a job already. You know, I would ask my parents put their own benefits.”And asked the woman with the microphone, “Can I say something back?” And she said, “No.” So, being… a young person, I guess, and just I stood back up and said, you know, “You're in Liverpool. You've kind of come to one of the poorest cities in some sense in the UK, and you kind of—I've already got a job. What can I really do?”And that's when Richard Branson turned round and said, “Well, what is it you want to do?” And I said, “I make really good cheesecakes and I want to sell them to people.” And after that sort of exchange, he said he'd give me the money. So, he gave me 200 pounds to start my own business. And that was just, kind of like, this whirlwind of what the hell's going on here?But for me, it's one of those moments in my life, which I think back on, and honestly, it's like one of these ten [left 00:25:15] moments of, you know, I didn't stand back up and say something, if I didn't join the entrepreneurship club, like, I just wouldn't be in the position I am right now. And it was also weird in the sense that I said at the start of the story, I didn't know what I wanted to do in my life. This was the first time that anyone had ever said to me, “I trust you to do something, and here's 200 pounds to do it.” And it was such a small thing, and a small moment that basically got me to where I am today. And kind of a condensed version of that is, you know, after that event, I started volunteering for a charity who—a, sort of, magazine launch, and then applied for the civil service and progressed through six to eight years of the civil service.And it was because of that moment, and that experience, and that confidence boost, where I was like, “Oh, I actually can do something with my life.” And I think tech, and I think a lot of people talk about this is, it can be a bit of a crazy whirlwind, and to go from that background into, you know, working with great people and earning great money is a bit of a crazy thing sometimes.Corey: Is there another path that you might have gone down instead and completely missed out on, for lack of a better term—and not missed out. You probably would have been far happier not working in tech; I know I would have been—but as far as trying to figure out, like, what does the road not taken look like for you?Aaron: I'm not too sure, really. And at the time, I was working in a club. I was like 16, 17 years old, working in a nightclub in Liverpool for five pounds an hour, and was doing that while I was studying, and that was almost like, what was in my mind at the time. When it came to the end of college, I was applying for universities, I got in on, like, a second backup course, and that was the only thing to do was food science. And it was like, I can't imagine coming out of university three years after that, studying something that's not really that relevant to a lot of industries, and trying to find a good job. It could have just been that I was working in a supermarket for minimum wage after I came out for uni trying to find what I wanted to do in the world. And, yeah, I'm really glad that I kind of ended up where I am now.Corey: As you take a look at what you want your career to be about in the broad sweep of things, what is it that drives you? What is it that makes you, for example, decide to spend the previous portion of career working in public service? That is a very, shall we say, atypical path—I say, as someone who lives in San Francisco and is surrounded by people who want to make the world a better place, but all those paths just coincidentally would result in them also becoming billionaires along the way.Aaron: I mean, it is interesting. You know, one of the things that worked for the civil service for so long, is the fact that I did want to do more than just make somebody else more money. And you know, there are not really a lot of ways you can do that and make a good wage for yourself. And I think early on in your career, working for somewhere like the civil service or federal government can be a little bit of that opportunity. And especially with some of the government's focus on tech these days, and investments—you know, I joined through an apprenticeship scheme and then progressed on to a digital leadership scheme, you know, they were guided schemes to help me become a better leader and improve my skills.And I think I would have probably not gone to the same position if I just got the tech job or my first engineering job somewhere else. I think, if I was to look at the future and where do I want to go, what do I care about? And, you know, you ask me, sort of, this question at re:Invent, and it took me a few days to really figure out, but one of the things when I talk about making the world a better place is thinking about how you can start businesses that give back to people in local areas, or kind of solve problems and kind of keep itself running a bit like a trust does, [laugh], if only that keeping rich people running. And a lot of the time, like, you've highlighted is coincidentally these things that we try and solve whether it's, like, a new app or a new thing that does something seems to either be making money for VCs, reinventing things that we already have, or just trying to make people billionaires rather than trying to make everyone rise up and—high tide rise all ships, is the saying. And there are a few people that do this, a few CEOs who take salaries the same as everyone else in the business. And I think that's hopefully you know, as I grow my own business and work on different things in the future, is how can I just help people live better lives?Corey: It's a big question, and it's odd in that I don't find that most people asking it tend to find themselves going toward government work so much as they do NGOs, and nonprofits, and things that are very focused on specific things.Aaron: And it can be frustrating in some sense is that, you know, you look at the landscape of NGOs, and charities, and go, “Why are they involved in solving this problem?” You know, one of the big problems we have in the UK is the use of food banks where people who don't have enough money, whether they receive benefits or not, have to go and get food which is donated just by people of the UK and people who donate to these charities. You know, at the end of the day, I'm really interested in government, and public sector work, and potentially one day, being a bit more involved in policy elements of that, is how can we solve these problems with broad brushstrokes, whether it's technology advancements, or kind of policy decisions? And one of the interesting things that I got close to a few times, but I don't think we've ever really solved is stuff like how can we use Agile to build policy?How can we iterate on what that policy might look like, get customers or citizens of countries involved in those conversations, and measure outcomes, and see whether it's successful afterwards. And a lot of the time, policies and decisions are just things that come out of politicians minds, and it'd be interesting to see how we can solve some of these problems in the world with stuff like Agile methodologies or tech practices.Corey: So, it's easy to sit and talk about these things in the grand sweep of how the world could be or how it should look, but for those of us who think in more, I guess, tactical terms, what's a good first step?Aaron: I think from my point of view, and you know, meeting so many people at re:Invent, and just have my eyes opened of these great conversations we can have a great people and get things changed, one of the things that I'm looking at starting next year is a podcast and a newsletter, around the use of public cloud for public good. And when I say that, it does cover elements of sustainability, but it is other stuff like how do we use Cloud to deliver things in the public sector and NGOs and charities? And I think having more conversations like that would be really interesting. Obviously, that's just the start of a conversation, and I'm sure when I speak to more people in the future, more opportunities and more things might come out of it. But I'd just love to speak to more people about stuff like this.Corey: I want to thank you for spending so much time to speak with me today about… well, the wide variety of things, and of course, spending as much time as you did chatting with me at re:Invent in person. If people want to learn more, where can they find you?Aaron: So yep, got a few social media handles on Twitter, I'm @AaronBoothUK. On LinkedIn is the same, forward slash aaronboothuk, and I've also got the website for my consultancy, which is embue.co.uk—E-M-B-U-E dot co dot uk. And for the newsletter, it's publicgood.cloud.Corey: And we will, of course, include links to that in the [show notes 00:32:11]. Thank you so much for taking the time to speak with me. I really do appreciate it.Aaron: Thank you so much for having me.Corey: Aaron Booth, cloud consultant with an emphasis on sustainability. I'm Cloud Economist Corey Quinn with an emphasis on optimizing bills. And this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that you will then kickstart the coal-burning generator under your desk to wind up posting.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About NipunNipun Agarwal is Vice President, MySQL HeatWave and Advanced Development, Oracle. His interests include distributed data processing, machine learning, cloud technologies and security. Nipun was part of the Oracle Database team where he introduced a number of new features. He has been awarded over 170 patents.Links:HeatWave: https://oracle.com/heatwave TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You could build you go ahead and build your own coding and mapping notification system, but it takes time, and it sucks! Alternately, consider Courier, who is sponsoring this episode. They make it easy. You can call a single send API for all of your notifications and channels. You can control the complexity around routing, retries, and deliverability and simplify your notification sequences with automation rules. Visit courier.com today and get started for free. If you wind up talking to them, tell them I sent you and watch them wince—because everyone does when you bring up my name. Thats the glorious part of being me. Once again, you could build your own notification system but why on god's flat earth would you do that?Corey: This episode is sponsored in part by our friends at VMware. Let's be honest—the past year has been far from easy. Due to, well, everything. It caused us to rush cloud migrations and digital transformation, which of course means long hours refactoring your apps, surprises on your cloud bill, misconfigurations and headache for everyone trying manage disparate and fractured cloud environments. VMware has an answer for this. With VMware multi-cloud solutions, organizations have the choice, speed, and control to migrate and optimizeapplications seamlessly without recoding, take the fastest path to modern infrastructure, and operate consistently across the data center, the edge, and any cloud. I urge to take a look at vmware.com/go/multicloud. You know my opinions on multi cloud by now, but there's a lot of stuff in here that works on any cloud. But don't take it from me thats: VMware.com/go/multicloud and my thanks to them again for sponsoring my ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today's promoted episode is slightly off the beaten track. Normally in tech, we tend to find folks that have somewhere between an 18 to 36-month average tenure at companies. And that's great, however, let's do the exact opposite of that today. My guest is Nipun Agarwal, who's the VP of MySQL HeatWave and Advanced Development at Oracle, where you've been an employee for 27 years, is it?Nipun: That's absolutely right. 27 years and that was my first job out of school. So, [laugh] yes.Corey: First, thank you for joining me. It is always great to talk to people who have focused on an area that I only make fun of from a distance, in this case, databases which, you know, DNS works well enough for most use cases, but occasionally customers have other constraints. You are clearly at or damn near at the top of your field. In my pre-show research, I was able to unearth that you have—what is it now, 170, 180 filed patents that have been issued?Nipun: That's right. 180 issued patents. [laugh].Corey: You clearly know what you're doing when it comes to databases.Nipun: Thank you for the opportunity. Yes, thank you.Corey: So, being a VP at Oracle, but starting off as your first job as almost a mailroom to the executive suite style story, we don't see those anymore. In most companies, it very much feels like the path to advance is to change jobs to other companies. It's still interesting seeing that that's not always the path forward, for some folks. I think that the folks who have been in companies for a long time need more examples and role models to look at in that sense, just because it is such an uncommon narrative these days. You're not bouncing around between four companies.Nipun: Yeah. I've been lucky enough to have joined Oracle, and although I had been at Oracle, I've been on multiple teams at Oracle and there has been a great opportunity of talent, colleagues, and projects, where even to this day, I feel that I have a lot more to learn. And there are opportunities within the company to learn and to grow. So no, I've had an awesome ride.Corey: Let's dive in a little bit to something that's been making the rounds recently, specifically you've released something called HeatWave, which has been boasting some, frankly, borderline unbelievable performance benchmarks, and of course, everyone loves to take a crack at Oracle for a variety of reasons, so Twitter is very angry. But I've learned at some point, through the course of my career, to disambiguate Twitter's reactions from what's actually happening out there. So, let's start at the beginning. What is HeatWave?Nipun: HeatWave is an in-memory query accelerator for MySQL. It accelerates complex, long-running, analytic queries. The interesting thing about HeatWave is, with HeatWave we now have a single MySQL database which can run all your applications, whether they're OLTP, whether they're mixed workloads, or whether they're analytics, without having to move the data out of MySQL. Because in the past, people would need to move the data from MySQL to some other database running analytics, so people would end up with two different databases. With this single database, no need for moving the data, and all existing tools and applications which worked with MySQL continue to work, except they will be much faster. That's what HeatWave is.Corey: The benchmarks that you are publishing are fairly interesting to me, specifically, the ones that I've seen are, you've classified HeatWave as six-and-a-half times faster than Amazon Redshift, seven times faster than Snowflake, nine times faster than BigQuery, and a number of other things, and fourteen hundred times faster than Amazon Aurora. And what's interesting to me about the things that you're naming is they're not all data-warehouse style stuff. Aurora, for example, is Amazon's interpretation of an in-house developed managed database service named after a Disney Princess. And it tends to be aimed at things that are not necessarily massive scale. What is the sweet spot, I guess, of HeatWaves data sizes when it comes to really being able to shine?Nipun: So, there are two aspects where our customers are going to benefit from HeatWave. One characteristics is the data size, but the other characteristics is the complexity of the queries. So, let's first do the comparison with Aurora—and that's a very good question—the 1400 times comparison we have shown, yes, if you take the TPC-H queries on a four terabyte workload and if you run them, that's what you're going to see. Now, the interesting thing is this: not only is it 1400 times faster it's also at half the price because for most of these systems, if you throw more gear, if you throw more hardware, the performance would vary. So, it's very important to go with how much of performance and at what price.So, for pure analytics—say, for four terabytes—is 1400 times faster at half the price. So, if it provides truly 800 times better price performance compared to Aurora for pure analytics. Now, let's take the other extreme. 100 gigabytes—which is a much smaller, your bread and butter database—and this is for mixed workloads. So, something like a CH-benCHmark, which has a combination of say, some TPC-C transactions, and then some added IPP-CH queries, which—the CH benCHmark.Here we have 42 times advantage price performance over Aurora because we are 42% of the cost, less than half the cost of Aurora and for the complex queries, we are about 18 times faster, and for pure OLTP, we are at par. So, the aggregate comes out to be about 42 times better. So, the mileage varies depending upon the data size and depending upon the complexity of the queries. So, in the case of Aurora, it will be anywhere from 42 times better price performance all the way to 2800.Corey: Does this have an upper bound, for example? Like, if we take a look at something like Redshift or something like Snowflake, where they're targeting petabyte-scale workloads at some point, that becomes a very different story for a lot of companies out there. Is that something that this can scale to, or is there a general reasonable upper bound of, okay, once you're above X number of terabytes, it's probably good to start looking at tiering data out or looking at a different solution?Nipun: We designed HeatWave primarily for those customers who had to move the data out of MySQL database into some other database for running analytics. The upper bound for the data in the MySQL database is 64 terabytes. Based on the demand and such we are seeing, we support 32 terabytes processing in HeatWave at any given point in time. You can still have 64 terabytes in the MySQL database, but the amount of data you can load into the HeatWave cluster at any given point in time is 32 terabytes.Corey: Which is completely reasonable. I would agree with you from not having much database exposure myself in the traditional sense, but from a cloud economics standpoint alone, anytime you have to move data to a different database for a different workload, you're instantly jacking costs through the roof. Even if it's just the raw data volumes, you now have to store it in two different places instead of one. Plus, in many cases, the vaguearities of data transfer pricing in many places wind up meaning that you're paying money to move things out, there's a replication story, there's a sync factor, and then it just becomes a management overhead problem. If there's a capacity to start using the data where it is in more intelligent ways, that alone has a massive economic wind, just from a time it takes your team to not have to focus on changing infrastructure and just going ahead to run the queries. If you want to start getting into the weeds of all the different ways something like this is an economic win, there's a lot of angles to look at it from.Nipun: That's an excellent point and I'm very glad you brought it up. So, now let's take the other set of benchmarks we were talking about: Snowflake. So, HeatWave is seven times faster and one-fifth the cost; it's about 35 times better price performance. Compared to let's say Redshift AQUA, six-and-a-half times faster at half the cost, so 13 times better price performance. And it goes on and on.Now, these numbers I was quoting is for 10 terabytes TPC-H queries. And the point which you said is very, very valid. When we are talking about the cost for these other systems, it's only the cost for analytics without including the cost of the source database or without including the cost of moving the data or managing to different databases. Whereas when you're talking about the cost of HeatWave, this is the cost which includes the cost of both transaction processing as well as the analytics. So, it's a single database; all the cost is included, whereas, for these other vendors, it's only the cost of the analytic database. So, the actual cost to a user is probably going to be much higher with these other databases. So, the price performance advantage with HeatWave will perhaps be even higher.Corey: Tell me a little bit about how it works. I mean, it's easy to sit here and say, “Oh, it's way faster and it's better in a bunch of benchmark stuff,” and we will get into that in a little bit, but it's described primarily as an in-memory query accelerator. Naively, I think, “Oh, it's just faster because instead of having data that lives on disk, it winds up having some of it live in RAM. Well, that seems simple and straightforward.” Like, oh, yeah, I'm going to go on a limb and assume that there aren't 160 patents tied to the idea that RAM is faster than disk. There's clearly a lot more going on. How does this work? What is it foundationally?Nipun: So, the thing to realize is HeatWave has been built from the ground up for the cloud and it is optimized for the Oracle Cloud. So, let's take these things one at a time. When I say designed from the ground up for the cloud, we have actually invented and implemented new algorithms for distributed query processing, which is what gives us such a good advantage in terms of operations like joint processing, window functions, aggregations. So, we have come up—invented, implemented new algorithms for distributed query processing. Secondly, we have designed it for the cloud.And by that what I mean is, A, we have a lot of emphasis on scalability, that it scales to thousands of cores with a very, very good scale factor, which is very important for the cloud. The next angle about the cloud is that not only have we optimized it for the cloud, but we have gone with commodity cloud services, meaning, for instance, when you're looking at the storage, we are looking at the least expensive price. So, for instance, we use object store; you don't use, for instance, locally attached SSDs because that will be expensive. Similarly, for compute: instead of using Intel, we use AMD chips because they are less expensive. Similarly, networking: standard networking.And all of this has been optimized for the specific Oracle Cloud infrastructure shapes we have, for the specific VMs we use, for the specific networking bandwidth we get, for the object store bandwidth and such; so that's the third piece, optimized for OCI. And the last bit is pervasive use of machine learning in the service. So, a combination of these four things: designed for the cloud, using commodity cloud services, optimized for the quality cloud infrastructure, and finally the pervasive use of machine learning is what gives us very good performance, very good scale, at a very inexpensive price.Corey: I want to dig into the idea of the pervasive use of machine learning. In many cases, machine learning is the answer to how do I wind up bilking a bunch of VCs out of money? And Oracle is not a venture-backed company at this stage of its existence, it is a very large, publicly-traded entity; you have no need to do that. And I would also further accept that this is one of those bounded problem spaces where something that looks machine-learning-like could do very well. Is that based upon what it observes and learns from data access patterns? Is it something that it learns based from a specific workload in question? What is the gathering, and is it specific to individual workloads that a given customer has, or is it holistically across all of the database workloads that you see in Oracle Cloud?Nipun: So, there are multiple parts to this question. The first thing is—and I think as you're noting—that with the cloud, we have a lot more opportunity for automation because we know exactly what is the hardware stack, we know the software stack, we know the configuration parameters.Corey: Oh yes, hell is other people's data centers, for sure.Nipun: [laugh]. And the approach we have taken for automation is machine-learning-based automation because one of the big advantages is that we can have a model which is tailored to a specific instance and as you run more queries, as you run more workloads, the system gets more intelligent. And we can talk about that maybe later about, like, specific things which make it very, very compelling. The third thing, I think, which you were alluding to, is that there are two aspects in machine learning: data, and the models or the algorithms. So, the first thing is, we have made a lot of enhancements, both to the MySQL engine as well as HeatWave, to collect new kinds of data.And by new kinds of data, I mean, that not only do we collect statistics of data, but we collect statistics of, say, the queries: what was the compilation time? What was the execution time? And then, based on this data which we're collecting, we have then come up with very advanced algorithms—machine learning algorithms—which are, again, a lot of them, there is, like, you know, patterns or [IP 00:14:13] which we have built on top of the existing state of art. So, for instance, taking these statistics and extrapolating them on larger data sizes. That's completely an innovation which we did in-house.How do we sample a very small percentage of the data and still be accurate? And finally, how do we come up with these machine learning models which are accurate without hiring an army of engineers? That's because we invented our AutoML, which is very efficient. So, that's basically the ecosystem of the machine learning which we have, which has been used to provide this.Corey: It's easy for folks to sit there and have a bunch of problems with Oracle for a variety of reasons, some of which are no longer germane, some of which are, I'm not here to judge. But I think it's undeniable—though it sometimes gets eclipsed by people's knee-jerk reactions—the reason that Oracle is in so many companies that it is in is because it works. You folks have been pioneers in the database space for a very long time and that's undeniable. If it didn't deliver performance that was untouchable for a long time, it would not have gotten to the point where you now are, where it is the database of record for an awful lot of shops. And I know it's somehow trendy, sometimes, for the startup set to think, “Oh, big companies are slow and awful. All innovation comes out of small, scrappy startups here.”But your customers are not fools. They made intelligent decisions based upon constraints that they're working within and problems that they need to solve. And you still have an awful lot of customers that are not getting off of Oracle anytime soon because it works. It's one of those things that I think is nuanced and often missed. But I do feel the need to ask about the lock-in story. Today, HeatWave is available only on the managed MySQL service in Oracle Cloud, correct?Nipun: Correct.Corey: Is there any licensing story tied to that? In other words, “Well, if I'm going to be using this, I need to wind up making a multi-year commitment. I need to get certain support things, as well,” the traditional on-premises Oracle story. Or is this an actual cloud service, in that you pay for what you use while you use it, and when you turn it off, you're done? In theory. In practice, we know in cloud economics, no one ever turns anything off until the company goes out of business.Nipun: So, it's exactly the letter what you said that this is a managed service. It's pay as you go, you pay only for what you consume, and if you decide to move on, there's absolutely no license or anything that is holding you back. The second thing—and I'm glad you brought it up—about the vendor lock-in. One of the very important things to realize about HeatWave is, A, it's just an accelerator for MySQL, but in the process of doing so, we have not introduced any proprietary syntax. So, if customers have the MySQL application running on some other cloud, they can very easily migrate to OCI and try MySQL HeatWave.But for whatever reason, if they don't like it, and they want to move out, there is absolutely nothing which is holding them back. So, the ease of which they can come in with the same ease they can walk out because we don't have any vendor lock-in. There is absolutely no proprietary extensions to HeatWave.Corey: There is the counter-argument as far as lock-in goes, and we see this sometimes with companies we talk to that were considering Google Cloud Spanner, as an example. It's great, and you can use it in a whole bunch of different places and effectively get ACID-compliance-like behavior across multiple regions, and you don't have to change any of the syntax of what it is you're using except the lock-in there is one of a strategic or software architecture lock-in because there's nothing else quite like that in the universe, which means that if you're going to migrate off of the single cloud where that's involved, you have to re-architect a lot, and that leads to a story of lock-in. I'm curious as to whether you're finding that customers are considering that as far as the performance that you're giving for MySQL querying is apparently unparalleled in the rest of the industry; that leads to a sort of lock-in itself when people get used to that kind of responsiveness and build applications that expect that kind of tolerances. At some point, if there's nothing else in the industry like it, does that means that they find themselves de-facto locked in?Nipun: If you were to talk about some functionality which we are offering which no one else is offering, perhaps you could, kind of, make that case. But that's not the case for performance because when we are so much faster—so suppose I said, okay, we are so much faster; we are six-and-a-half times faster than Redshift at half the cost. Well, if someone wanted the same performance, they can absolutely do it Redshift on a much larger cluster, and pay a lot more. So, if they want the best performance at the best price, they can come to Oracle Cloud; if they want the same performance but they will have to pay more, they can go anywhere else. So, I don't think that's a vendor lock-in at all.That's a value which we are bringing in that for the same performance, we are much cheaper. Or you can have that kind of a balance that we are faster and cheaper. So, there is no lock-in. So, it's not to say that, okay, we have made some extensions to MySQL which are only available in our cloud. That is not at all the case.Now, for some other vendors and for some other applications—you brought up Spanner; that's one. But we have had multiple customers of MySQL who, when they were trying Google BigQuery, they mentioned this aspect that, okay, Google BigQuery had these proprietary extensions and they feel locked in. That is not the case at all with HeatWave.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: I do want to call out, just because it seems like there's a lies, damned lies, and database benchmarks story here where, for example, Azure for a while was doing a campaign where they were five times less expensive for database workloads than AWS until you scratched beneath the surface and realize it's because they're playing ridiculous games with licensing, making it very expensive to run a Microsoft SQL Server on anything that wasn't Azure. Customers are not necessarily as credulous as they once were when it comes to benchmarking. And Oracle for a long time hasn't really done benchmarking, and in fact, has actively discouraged it. For HeatWave, you've not only published benchmarks, which okay, vendors can say anything they want, and I'm going to wait until I see independent returns, but you put not just the benchmarks, but data sets, and your entire methodology onto GitHub as well. What led to that change? That seems like the least Oracle-like thing I could possibly imagine.Nipun: I couldn't take credit for the idea. The idea actually was from our Chief Marketing Officer, that was really his idea. But here is the reason why it makes a lot more sense for us to do it for MySQL HeatWave. MySQL is pervasive; pretty much any cloud vendor you can think about has a MySQL-based managed service. And obviously, MySQL runs on premise, like a lot of customers and applications do it.Corey: That's one of the baseline building blocks of any environment. I don't even need to be in the cloud; I can get MySQL working somewhere. Everyone has it, and if not, why don't you? And I can build it in a VM myself in 20 minutes.Nipun: That's right.Corey: It is a de-facto standard.Nipun: That's right. So, given that is the case and many other cloud vendors are innovating on top of it—which is great—how do you compare the innovation or the value proposition of Cloud Vendor A with us? So, for that, what we felt was that it is very important and very fair that we publish our scripts so that people can run those same scripts with a HeatWave, as well as with other cloud offerings, and make a determination for themselves. So, given the popularity of MySQL and given that pretty much all cloud vendors provide an offering of MySQL, and many of them have enhanced it, in order for customers to have an apples-to-apples comparison, it is imperative that we do this.Corey: I haven't run benchmarks myself just yet, just because it turns out, there's a lot of demands on my time and also, as mentioned, I'm not a deep database expert, unless it comes to DNS. And we keep waiting for people to come back with, “Aha. Here's why you're completely comprised of liars.” And I haven't heard any of that. I've heard edges and things here about, “Well, if you add an index over here, it might speed things up a bit,” but nothing that leads me to believe that it is just a marketing story.It is a great marketing story, but things like this fall apart super quickly in the event that it doesn't stand up to engineering scrutiny. And it's been out long enough that I would have fully expected to have heard about it. Lord knows if anyone is listening and has thoughts on this, I will be getting some letters after this episode, I expect. But I've come to expect those; please feel free to reach out. I'm always thrilled to do follow-up episodes and address things like this.When does it make sense from your perspective for someone to choose HeatWave on top of the Oracle Cloud MySQL service instead of using some of the other things we've talked about: Aurora, Redshift, Snowflake, et cetera? When does that become something that a customer should actively consider? Is it for net-new workloads? Should they consider it for migration stories? Should they run their database workloads in Oracle Cloud and keep other stuff elsewhere? What is the adoption path that you see that tends to lead to success?Nipun: All customers of MySQL, or all customers of any open-source database, those would be absolutely people who should consider MySQL HeatWave. For the very simple reason: first, regardless of the workload, whether it is OLTP only, or mixed workloads, or analytics, the cost is going to be significantly lower. I'll say at least it's going to be half the cost. In most of the cases, it's probably going to be less than half the cost. So, right off the bat, customers save half the cost by moving to MySQL HeatWave.And then depending upon the workload you have, as you have more complex queries, the performance advantage starts increasing. So, if you were just running only OLTP, if you only had transactions and you didn't have any complex queries—which is very unlikely for real-world applications, but even if that was the case, you're going to save 60% by going to MySQL HeatWave. But as you have more complex queries you will start finding that the net advantage you're going to get with performance is going to keep increasing and will go anywhere from 10 times aggregate to as much as 1400 times. So, all open-source, MySQL-based applications, they should consider moving. Then you mentioned about Snowflake, Redshift, and such; for all of them, it depends on what the source database is and what is it that they're trying to do.If they are moving data from, say, some open-source databases, if they are ETL-ing from MySQL, not only will MySQL HeatWave be much faster and much cheaper, but there's going to be a tremendous value proposition to the application because they don't need to have two different applications for two different databases. They can come back to MySQL, they can have a single database on which they can run all their applications. And then again, you have many of these cloud-native applications are born in the cloud where people may be looking for a simple database which does the job, and this is a great story—both in terms of cost as well as in terms of performance—and it's a single database for all your applications, significantly reduces the complexity for users.Corey: To turn the question around a little bit, what sort of workloads is MySQL HeatWave not a fit for? What sort of workloads are going to lead to a poor customer experience? Where, yeah, this is not a fit for that workload?Nipun: None, except in terms of the data size. So, if you have data sizes which are more than 64 terabytes, then yes, MySQL HeatWave is not a good fit. But if your data size is under 64 terabytes, you're going to win in all the cases by moving to MySQL HeatWave, given the functionality and capabilities of MySQL.Corey: I'd also like to point out that recently, HeatWave gained the MySQL Autopilot capability, which I believe is a lot of the machine learning technologies that you were speaking about a few minutes ago. Are there plans to continue to expand what HeatWave does and offer additional functionality? And—if you can talk about any of that. I know that roadmap is always something that is difficult to ask about, but it's clear that you're investing in this. Is your area of investment looking more like it's adding additional features? Is it continuing to improve existing performance? Something else entirely? And of course, we also accept you can't tell me any of [laugh] that has a valid answer.Nipun: Well, we just got started, so we just had our first [GF 00:27:03] HeatWave in December, and you saw that earlier this week we had our second major release of HeatWave. We are just getting started, so absolutely we are investing a lot in this area. But we are pretty much going to attempt all the things that you said. We have feedback from existing customers which is very high up on the priority list. And some of these are just one, say, class of enhancements which [unintelligible 00:27:25], can HeatWave handle larger sizes of data? Absolutely, we have done that; we will continue doing that.Second is, can HeatWave accelerate more constructs or more queries? Absolutely, we will do that. And then you have other kinds of capabilities which customers are asking which you can think of are, like you know, bigger features, which for instance, we announced the support for scale-out data storage which improves recovery time. Well, you're going to improve the recovery time or you're going to improve the time it takes to restart the database. And when I say improve, we are talking about not an improvement of 2X or 3X, but it's 100 times improvement for, let's say, a 10 terabyte data size.And then we have a very good roadmap which, I mean, it's a little far out that I can't say too much about it, but we will be adding a lot of very good new capabilities which will differentiate HeatWave even more, compared to the competitive services.Corey: You have very clearly forgotten more about databases than most of us are ever going to know. As you've been talking to folks about HeatWave, what do you find is the most common misunderstanding that folks like me tend to come away with when we're discussing the technology? What is it that is, I guess, a nuance that is often being missed in the industry's perspective as they evaluate the new technology?Nipun: One aspect is that many times, people just think about a service to be here some open-source code or some on-premise code which is being hosted as a managed service. Sure, there's a lot of value to having a managed service, don't get me wrong, but when you have innovations, particularly when you have spent years in years or decades of innovation for something which is optimized for the cloud, you have an architectural advantage which is going to pay dividends to customers for years and years to come. So, there is no substitute for that; if you have designed something for the cloud, it is going to do much better whether it's in terms of performance, whether it's in terms of scalability, whether it's in terms of cost. So, that's what people have to realize that it takes time, it takes investment, but when we start getting the payoff, it's going to be fairly big. And people have to think that okay, how many technologies or services are out there which have made this kind of investment?So, what I'm really excited about is, MySQL is the most popular database amongst developers in the world; we spend a lot of time, a lot of person-years investing over the last, you know, decade, and now we are starting to see the dividends. And from what we have seen so far, the response has been terrific. I mean, it's been really, really good response, and we are very excited about it.Corey: I want to thank you for taking so much time to speak with me today. If people want to learn more, where can they go?Nipun: Thank you very much for the opportunity. If they would like to know more, they can go to oracle.com/heatwavewhere we have a lot of details, including a technical brief, including all the details of the performance numbers we talked about, including a link to the GitHub where they can download the scripts. And we encourage them to download the scripts, see that they're able to reproduce the results we said, and then try their workloads. And they can find information as to how they can get free credits to try the service for free on their own and make up their mind themselves.Corey: [laugh]. Kicking the tires on something is a good way to form an opinion about it, very often. Thank you so much for being so generous with your time. I appreciate it.Nipun: Thank you.Corey: Nipun Agarwal, Vice President of MySQL HeatWave and Advanced Development at Oracle. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an insulting comment formatted as a valid SQL query.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About Forrest Forrest is a cloud educator, cartoonist, author, and Pwnie Award-winning songwriter. He currently leads the content marketing team at Google Cloud. You can buy his book, The Read Aloud Cloud, from Wiley Publishing or attend his talks at public and private events around the world.Links: The Cloud Bard Speaks: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/the-cloud-bard-speaks-with-forrest-brazeal/ The Read Aloud Cloud: https://www.amazon.com/Read-Aloud-Cloud-Innocents-Inside/dp/1119677629 The Cloud Resume Challenge Book: https://forrestbrazeal.gumroad.com/l/cloud-resume-challenge-book/launch-deal The Cloud Resume Challenge: https://cloudresumechallenge.dev Twitter: https://twitter.com/forrestbrazeal TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part my Cribl Logstream. Cirbl Logstream is an observability pipeline that lets you collect, reduce, transform, and route machine data from anywhere, to anywhere. Simple right? As a nice bonus it not only helps you improve visibility into what the hell is going on, but also helps you save money almost by accident. Kind of like not putting a whole bunch of vowels and other letters that would be easier to spell in a company name. To learn more visit: cribl.ioCorey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: Welcome to Screaming in the Cloud. I am Cloud Economist Corey Quinn, and as an industry, we stand on the precipice of change. There's an awful lot of movement lately. It feels like the real triggering event for this was when Andy Jassy ascended from being the CEO of AWS—the cloud computing division of Amazon—to being the CEO of all of Amazon, including things like not just AWS, but also the underpants store. Suddenly, we have people migrating between different cloud providers constantly.Today's guest is a change I would not have expected and didn't see coming. So, last year, on episode 127, called The Cloud Bard Speaks I had Forrest Brazeal from A Cloud Guru joining me. Forrest, welcome back.Forrest: Hey, thanks, Corey. Big fan of the show; always great to be here.Corey: At the time that we're recording this, you are unemployed, which is great because it's Screaming in the Cloud. Screaming at people on your day off is always fun. But by the time it airs, you'll have started your new job as the Head of Content for Google Cloud.Forrest: Yes. And of course, that's definitely a career change for me coming directly from A Cloud Guru, which was a wonderful place to be and it was exciting to be with them right up through their acquisition earlier this summer, but when it came time to make the next move, I ended up going to Google Cloud. I'll be starting there on Monday after this recording has been completed, and just really looking forward to helping tell the story of the cloud at a much bigger scale, something that I've been doing throughout my career with increasing levels of scale. It's exciting to do it at the level of an entire cloud provider.Corey: We'll get to the future in a minute, but I want to start by looking at the past. From my perspective, you were a consultant for a while at Trek10; we've talked about that before. You have an engineering background of building things with computers, at least presumably computers—you've been a big serverless advocate and I'm told that runs on computers somewhere, but I don't want to get into that particular debate—to the point where you were—I assume were, not are anymore—an AWS Serverless Hero?Forrest: Yes, that's right, and even going back prior to Trek10, my background is in enterprise software. I helped to migrate some of the world's largest enterprise applications from data centers to cloud when I was at Infor and continued to work on that kind of thing as a consultant later on. And in that time, I was working a lot with AWS, which was the only game in town for a lot of those years, right? You go back to 2014, 2015, I'm putting an enterprise app in the cloud, what am I going to put it on? Probably AWS if I'm serious about what I'm doing.But it's been amazing to see how the industry has grown and changed and the other options that have come along. And one of the cool things about my work in A Cloud Guru is that I really got a chance to branch out and expand, not just to AWS, but also to get a much better feel for the other cloud providers, for Azure and GCP, and even beyond to Oracle and some of the other vendors that are out there. And just to get a better understanding of how these different cloud providers thrive in different niches. So yes, it is absolutely a change for me; I obviously won't be an AWS Hero anymore, I'm having to close that chapter, sadly; I love those people and that program, but it is going to be a new and interesting change. I'm going to have to be back in learning mode, back in catch-up mode as I get busy on GCP.Corey: So, one thing that I think gets occluded with you because it definitely does with me is that you and I are both distinguishable personalities in the cloud community—historically AWS, let's be clear here—and you do your own custom songs; you write a newsletter that instead of snarky is insightful—of which I'm jealous—but it still has a personality that shines through; you wrote a children's book, The Read Aloud Cloud; you wound up having a new book that just came out last week for folks listening to this the day of release, called The Cloud Resume Challenge Book, if I'm getting the terms all in the right order?Forrest: Yeah, exactly.Corey: It's like naming cloud services only naming books instead? It's still challenging to keep all the words in the right order?Forrest: You know, I think it actually transcends industries; naming things is hard whether you're in computer science or not.Corey: Whereas making fun of things' names is a lot easier. It's something you did not do—to my understanding—as an employee of A Cloud Guru, The Cloud Resume Challenge, but it's something you did as a side project because it interested you. It's effectively, you want to get into tech, into cloud.Great. Here's a list of things I want you to do. And it ranges the gamut. And we talked about it before, but to my understanding it's, build a statically hosted website that winds up building your resume, and a blog post, and how to do all these things, CI/CD, frontend, backend, the works. It's a lot of work, but by the time you're done, you know a heck of a lot more about the cloud provider you're working with than you did when you started.Forrest: Yeah, not only do you know more than you did when you started, but quite frankly, you're going to know more than a lot of people who've even been doing this kind of thing for a couple of years. That's why we have people that take The Cloud Resume Challenge, who are not only aspiring cloud engineers but who have been doing this for a while, maybe even are hiring people, and they see this project and say, “Wow. That would look good on my resume. I've never actually sat down and plugged a frontend and a backend together on AWS,” and, “Maybe I've never had to actually sit down and think carefully about how I would build a CI/CD pipeline,” or, “I really want to get my hands dirty with Terraform,” or something like that. So, we see a whole range of people.I did a survey on this actually, and I found that about 40% of all the people who take The Cloud Resume Challenge have three years or more of professional IT experience. So, that should tell you how impressive it is, if you can figure this out as a brand new person to cloud. That's why we've seen so many of these folks change careers and go from things like plumbing, and working in a bank, and working in HR, and whatever else to starting roles, now, as cloud engineers and DevOps engineers. It's not entirely due to the challenge; not even mostly due to the challenge. These are folks who are self-motivated, quick learners, and are going to succeed no matter what, but The Cloud Resume Challenge was the thing that came on at the right time for them to build those skills and show what they had.Corey: And the fact that you put this together is incredibly uplifting for folks new to the field. And that's amazing, and it's great, and it's more content, the kind that I think that we need in this industry. You also launched a newsletter last week: the cloud jobs newsletter, which is fantastic. It's a pay-to-subscribe newsletter—which I've always debated experimenting with but never did—and lists curated jobs in the industry, sorted by level of experience required and things that you find personally interesting. You might have sponsored job listings in the future that you've already said would be clearly delineated from the others, which is the ethically right thing to do. You are seemingly everywhere in the cloud space.Forrest: Well, I mean look, I'm trying to give back. I've benefited from folks like yourself and others who have made time to help lift my career over the years, and I really want to be here to help others as well. The newsletter that you mentioned the Best Jobs in Cloud, it does have a small fee associated with it, but that's really just to help gate my [laugh] referrals so that they don't end up getting overwhelmed. You actually can get free access to the newsletter with the purchase of The Cloud Resume Challenge Book we talked about before. It's really intended to be a package deal where you prepare your resume by doing these projects, and there's a lot of other advice in that book about how to get yourself positioned for a great career in the cloud.And then you have this newsletter coming into your inbox every couple of weeks that lays out a list of jobs and they're broken down by, you know, these are jobs that are best for juniors, these are jobs where you're going to need some senior-level experience. Because what I found—and honestly, I've been kind of acting as a talent agent for a lot of engineers over the past several years as my network has grown, and I've tried to give back to others and help to connect folks who are eagerly trying to find great engineers for cool projects that are working on with folks who are eagerly looking for those opportunities. And what I've realized is whether you're a junior or whether you've been doing this for a long time, let's face it, most of us are not spending all of our time being those distinguishable personalities that you mentioned a minute ago. I like how you said distinguishable and not distinguished by the way; those are two very different words. But most of us are not spending our time doing that.You know, we're working engineers; we're working, right? We're not blogging and tweeting all the time and building these gigantic personal networks. So, it helps if you can have a trusted friend standing alongside you so that when you are thinking about maybe making a switch, or maybe you're not thinking about making a switch but you should be because of where the market is, that friend is coming alongside you and saying, “Hey, this is an awesome opportunity that I think you should consider checking out; why not just do the interview. Even if you're not really looking to move, it's always important to keep your skills fresh.” That's what this newsletter is designed to do. I hope that it'll be helpful for you, no matter where you are in your cloud career, as long as you're staying in the cloud space.Corey: And the fact that's how you view this is the answer to a question a lot of folks have asked me over drinks with theoretical conversations for years of, “Well, Corey, if you went to go work at one of these big cloud providers, it destroy everything you've built because how in the world could you be authentic while working for one of these companies?” And the answer is exactly what you're doing. It's, “Yeah, the people who pay you don't own you.” I cannot imagine that even Google could afford to buy your authenticity from you because once that's gone, you don't get it back, and you're one of those people in this space, that—I'm not entirely sure that you understand where you are in this space, so let me help enlighten you with that for a minute.Forrest: Oh, great. [laugh].Corey: Oh, yeah, like, the first thing I was starting to talk about that we have in common is that we do a lot of content, both of us and that sometimes occludes the very real fact that we have a distinct level of technical expertise, historically. You and I can both feel relatively deep technical questions about cloud services, but because our job doesn't have the word engineer in the title, it doesn't lead to the same type of recognition of that fact. But I want to be very clear: you are technically excellent at what you'll do. You also have a distinguished personality and brand in the space, and your authenticity is also unparalleled. When you say something is good, it is believed that it is because you say it, and the inverse is also true.You're also someone that is very clearly aligned with fighting for the user if you want to quote Tron. It's the, you're not here to shill for things that don't get people ahead in their careers; you're not here to prop things up just because that's where the money is blowing. Your position on this is unimpeachable. And I'm going to be clear here: I am more interested in Google Cloud now than I was before you made this announcement. That is the value of having someone like you aboard, and frankly, I'm astonished they managed to grab you. It shows a forward-looking ability that historically I have not associated with cloud marketing groups.Forrest: Yeah, well I mean, the space changes fast. And I think you've said this yourself as well, even with the services; you look away for six months and you look back and it's not the same industry you remember. And that actually is a challenge when you talk about that technical credibility because that can go away very, very quickly. So, it does require some constant effort to stay fresh on that, especially if you're not building every single day. But to your point about the forward-looking-ness of Google Cloud, I really am excited about that and that's honestly the biggest thing that attracted me to what they're doing.They clearly understand, I think, their position in the space. We know they're three out of three and trying to catch up, and because of that, they're able to [laugh] be really creative. They're able to make bold choices and try things that you might not try if you were trying to maintain a market-leading position. So, that's exciting to me. I'm a creative person, I like to do things that are outside the box and I think you can look forward to seeing some more outside-the-box things coming at Google Cloud here over the next couple of years.Corey: I'd be astounded if it were otherwise. The question I have for you is that ‘Head of Cloud' is not a junior role. That's not something entry-level that you're just going to pick some rando off of LinkedIn to fill. They're going to pick a different rando: you specifically as one of those randos. And to my understanding, you've never really touched Google Cloud in anger from a technical level before. Is that right? Am I dramatically misunderstanding, “Oh yeah, you don't remember the whole musical, and three-act stage play that you put on, and the music video, and the rock opera all about Google Cloud?” It's, “No, I must have been sick that week,” because that's the level of prolific you tend to be?Forrest: [laugh].Corey: What is your experience with it?Forrest: That's yet to come. So, check back on the Google Cloud rock opera; we'll see if that takes place. So no, I'm going to be learning about Google Cloud. This will be a chance for me to kind of start over a little bit from first principles. In another sense, I've been interacting with Google services for years.Keep in mind that Google Cloud is not just Google Cloud Platform, but it's G Suite as well, and there's a lot going on there. So, I definitely am going to be going back to being a beginner a little bit here. They do say if you can teach something to a beginner, you have to really understand it at an expert level. And I know that whether I'm doing this officially on behalf of Google or otherwise, I'm going to be continuing to try to help and educate folks wherever I can. So, it's going to be incumbent on me, if I want to keep doing that, to go deep quickly and continue to learn.I'm excited about that challenge. I've been doing a lot with AWS for a long time, I don't know everything. In fact, I know less every day with the amount that they're continuing to roll out, but this is a chance for me to expand, become a more well-rounded person to see how the other cloud lives. I'm taking that very seriously; I'm not going to be an expert overnight, but stick around, follow me. I'm going to be learning, I'm going to share what I learned, and maybe we'll all get a little better Google Cloud together.Corey: The thing I can't quite get past is that when you told me that you had resigned from A Cloud Guru, I want to be selfish here and say that there were two things that went through my mind. The first was, “Okay, it's probably AWS. I hope it's AWS,” because the alternative is you're going somewhere potentially independent, and I know you keep arguing with me on this point but you are one of the few people I could point out that could start something on the basis of cloud content with a personal brand that I would view as potentially being an audience split for what I do. And it's, “Oh, you're going to go work for a big cloud company. That's awesome. Is it AW—no, it's not.” And that one threw me for a different loop where it's, that is very odd because you have identified, clearly, publicly as the leading voice in AWS in many contexts. It just really surprised me. Did you consider looking at AWS as an alternative?Forrest: I mean first, I don't know that it's fair to say that I was a leading voice for AWS. There's many wonderful people that [crosstalk 00:14:13]—Corey: To be clear, Forrest, that was not a question. You are a leading voice in the community for AWS and understanding how it works. That is one of those things that no one knows their own reputation. This is one of those areas. Take it from me—a thought leader—that it's true. Please continue.Forrest: You have led my thoughts in that direction, so thanks for that, Corey. But to your question, Corey, regarding how did I decide what career move to make, and definitely was a challenge. And it was a struggle for me to say, well, I'm going to leave behind this warm, friendly AWS community that I know, and try something brand new. But it's not the first time I've done something like that in my career. You mentioned already that I spent a number of years as a very, very technical person and I identified strongly as an engineer.I had multiple degrees in computer science and I had worked as a frontend/backend software engineer, I'd worked as a database administrator, I'd worked as a cloud engineer, and a manager of cloud engineers, and I'd consulted for companies from startups all the way up to the Fortune 50, always on cloud and always very hands-on and writing code. I've never had a job where I didn't have an IDE open and wasn't writing code every day. And it was a tremendous shock to my system when I started moving away from that, moving a little bit more into the business side of cloud, learning more about marketing, learning how to impact the bottom line of a company in other ways. That was a real challenge, and I went through months where I kind of felt like I was having an identity crisis because if I'm not writing code if I didn't create YAML today, who am I? Can I call myself an engineer? What worth do I have? And I know a lot of folks have struggled with this, and a lot of times, I think that's what sometimes holds people back in their career, saying, “Well, I can only do what I've already done because I've identified myself so strongly with it.” So, I'm encouraging anyone who's listening, if you're at that point where you feel like, “I don't know if I can leave behind what I know because will I still be able to succeed?” I would encourage you to go ahead and take that step and commit to it if you really believe that you have an opportunity because growth is ultimately going to be a good thing for you. Getting outside your comfort zone and feeling those unpleasant cracks as you start to grow and change into a different person, that ultimately is a strength-building thing.If you're not growing, you're not struggling, you're not going to be the person that you want to be. So, tying all that back, I went through one round of that already, Corey, when I moved a little bit away from technical delivery. I'm about to go through a second round of that when I move away a little bit farther from the AWS community. I believe that's going to be a growth opportunity. But yeah, it's going to be hard.Corey: It really is. The idea of walking away from the thing that you've immersed yourself in is really an interesting thing to think about. Forgive me in advance for the next question; I have to ask it. As a part of your interview process at Google, do they make you write code in a Google Doc?Forrest: Not as a part of this interview process. I interviewed at Google years ago for a developer advocate position, actually, and made it all the way through their interview process, writing many lines of code in many Google Docs, but not this time.Corey: Yeah, I confess, I did the same with an SRE job many years ago at Google, and again, you are better at writing code than I am; I did not progress past this stage. But it was moot, honestly, because the way that the interview was conducted, the person I was talking to was so adversarial at the time and so, I got to be honest, condescending that I swore I would never put myself through that process again. But I was also under the impression that the ritualistic algorithmic hazing via whiteboarding code was sort of a requirement for every role at Google. So, things change, times change, people change. I'm gratified to know that was not a part of your interview process.Forrest: Well, I mean, I think it was more just about the role. My favorite whiteboard interview—Corey: Nonsense. Every accountant must be able to solve code on a whiteboard.Forrest: No, I don't think that's true. But my favorite whiteboard interview story and I'm sure you have a few, I remember being in an interview with someone—I won't say who it was or what company it was, but it wasn't not Google—it was some sort of problem where I was having to lay out, I don't know, a path for a robot to take through an environment or something like that. And I wrote the code, and it was fine. It was, like, iterative. It was what you would do if you had ten minutes to write something.And then the interviewer looked at the code, and he said, “Great, now write it again, but don't use any variables.” And I remember sitting there for a minute thinking, “In what professional context [laugh] would someone encourage you to do that in a pair programming situation?”Corey: Right. The response there is, “What the hell does your codebase in production look like?”Forrest: [laugh]. And of course, the answer is you're supposed to be using, like, the stack, and it's kind of like this thought exercise with the local stack. But even if you were to do that, the performance hit would be tremendous. It would not be a wise or logical way to actually write the code. So, it was a pure trivial, kind of like a just academic exercise that they were recommending. And I remember being really turned off by that. So, I guess if you're considering putting problems like that in your interview process, don't. They're not helpful.Corey: Yeah, I remember hearing at one point one of the Microsoft brain teasers which they've since done away with—credit where due—where someone was asked, “How would you go about finding out the weight of a Boeing 747?” And the person responded with the exact weight of a Boeing 747 because their previous job had been at Boeing for seven years. And that was apparently not what they were expecting to hear. But yeah, it's sort of an allegory as well for, first, this has no bearing on your ability to do the job, and two, expertise is important. There's a lot of ways I could try and Hacker News first principles my way through something like that, but the easier answer is for me to call someone at Boeing and ask them, or Google it, depending on exactly how precise I need to be and whether lives hang in the balance of the [laugh] answer to the question. That's a skill that seems lost somewhere, too.Forrest: Yeah, and this takes us all the way back to the conversation about The Cloud Resume Challenge, Corey. And why it works is it takes the burden of proof off of you in the interview, or the burden of proof off the interviewer to have to come up with some kind of trivial problem that you've done under time pressure, and instead, it lets the conversation flow naturally back to, “Well, what have you done? Tell me about a story about a problem that you have solved, a challenge you ran into, and how you got past it.” That's all work that has taken place prior to the interview that you've reflected on, that's built you as a person and as an engineer, even if you don't necessarily have professional experience. That's how I try to conduct interviews and I think it's a much healthier and more sustainable way to find people that you'll like to work with.Corey: Is this going to be your first outing at a giant multinational tech company?Forrest: No, although it will be my first time with a public company. When I worked at Infor, Infor was the largest privately owned software company in the world. I don't know if that's still technically true or not, but it'll be my first time with a publicly-traded company.Corey: Fantastic. The nice thing from my perspective is it gives me a little bit more context into what companies can and can't do, and how things are structured. It feels like your content—I mean, the music videos and things and whatnot that you do—I mean, you have something that I don't, which is commonly known as musical talent. And that's great. I can write funny lyrics, but you are not just able to write lyrics, you're able to perform, you're able to sing, the unanswered question for the entire interview right now is whether you can also dance. So, we're going to find that out at some point.Forrest: You would think that I could, Corey. I definitely seem like someone who should be able to tap dance. I regret to tell you that I can't, but I want to learn.Corey: For a lot of this, it's clearly you're doing this in front of your own piano with a microphone in front of you, doing it live, and having a—I don't know if it is a built-in webcam to a laptop that's sitting in front of you or something else, but—Forrest: I'm playing with that.Corey: Yeah, well don't take this the wrong way; it's not a high definition 4k camera, et cetera. It's the Lightning's—eh, it's your home office. You're comfortable there. It's not a studio. What I'm most excited about—from my perspective, I know what you're excited about—but you're now going to be producing content for Google and I checked the numbers in preparation for this interview.It's okay, can Google wind up affording a production house of some sort to work on your videos to upscale the production value of some of what you're doing? And I have checked; it is not the likeliest scenario—and I have no inside knowledge for those who are trying to trade on this—but yes, it turns out that Google could, in fact, shore up your content by buying you Disney.Forrest: I think that's technically true, and I do expect that to happen in the next three to six months, so that is completely inside information.Corey: Oh, exactly. Have reasonable expectations, but you could let it go as long as a year because that's when the first annual review cycle comes in and you want to give people time to let that clear through M&A and make sure that they are living up to their commitments to you, of course.Forrest: That's right, yeah. We're just about to go into the quiet period there. No, but kind of to that point, though, and you bring up the amateurish quality of a lot of these videos that I put together in terms of the lighting and the staging, and everything else. And I am doing a little bit to help with that. Like, it would be great if you could see—Corey: To be clear, that is not a criticism. I'm in the same boat as you are on this. It's—[laugh]—Forrest: So, far from a criticism, it's actually pretty deliberate. The fact of the matter is, there's something very raw, very authentic about just seeing someone sitting in their house, at their piano, playing and singing. There's no tricks, there's no edits, there's no glitz, there's no makeup team behind the scenes, there's no one who's involved with this other than just me caring a lot about something and sitting down and singing about it. And I think some of that is what helps come across to people and it helps these things travel. So yeah, I'm looking forward a lot to being able to collaborate with other fantastic people at Google, and I can't exactly promise what will come out of that, but I'm quite sure there will be more fun content to come.But I hope never to lose that, kind of, DIY sensibility. Because, again, my background is as an engineer, and the things I create, whether it's music, whether it's cartoons, whether it's books, or other things I write, I never want to lose that sense of just excitement about the technologies I'm working with and the fact that I get to use the tools that are available at my disposal to share them with you as directly and honestly and humanly as possible.Corey: Up next we've got the latest hits from Veem. Its climbing charts everywhere and soon its going to climb right into your heart. Here it is!Corey: No matter how hard you try, you're not able to hide the sheer joy you take from even talking about this sort of stuff, and I think that's a powerful lesson. For folks listening to this who want to expand into their own content story and approach things that they find interesting in a way that they enjoy, don't try and do what I do; don't try to do what Forrest does; do the thing that makes you happy. I would love to be able to sing, but I can't. I can write funny lyrics, but those don't do well in pure text form. I'm fortunate that I was able to construct a structure on my end where I can pay people who do know how to sing—like Adeem the Artist and many more—to participate in a lot of the things that I get to work on.But find the way that you want to express things and do you. You're only ever going to be second best at being Forrest or being Corey, but you're always going to be number one at being whoever you happen to be. I think that's a lesson that gets overlooked an awful lot.Forrest: Yeah, I've been playing with this thought for a while that the only real [moat 00:24:24] out there is originality, is your personality. Everything else can be cloned, but you are an individual. And I mean that to us specifically, Corey, and also the general ‘you' to anybody listening to this. So, find what makes you tick. It sounds like the most cliche device in the world, but another way, it's also the only useful advice that's out there.Corey: I want to be clear, you don't work there yet and I'm not here to effectively give undue praise to large companies, but I just want to say again how the sheer vision of hiring you is just astounding to me. That it makes perfect sense, don't get me wrong, but because I know that every large company, somewhere, at some point, internally has had a conversation of, “We really should hire Corey, except…” well, I've got to level with you, Corey without the except parts looks an awful lot like you.Forrest: Yeah, you know, you brought up earlier this idea that well, hopefully, Forrest doesn't lose his authenticity at Google. And one of the things that I appreciate about the team that I've talked to there so far, is that they really do understand the power of individuals and voices. And so that's not going to happen. You know, my authenticity is not for sale. And frankly, I'm useless without it, so it wouldn't be in anyone's best interest to buy it anyway. And that would be true for you as well, Corey. Whatever you end up doing, whether you someday ascend to the head of AWS Marketing, as is apparently your divine destiny, I know that—Corey: Well, I'm starting to worry that there's not too many people left in that org, so I'm worried people took me seriously and they think I've got this in hand or something.Forrest: You may be the last man standing for all we know. You may be able to go in and just, kind of, do this non-hostile takeover where there's just no one there to defend against you, anymore.Corey: Well, speaking about takeovers and whatnot, we talk about Google acquiring Disney so you now have a production studio on this. But let's talk about actual hard problems you're going to be solving there. Do you think you can bring back Google Reader?Forrest: That would be my dream. I have no inside knowledge of what would even be required to bring that off, but I think it's obvious that it's not just about that particular product that people like—because yes, you or I could go make a startup and create something that did what Google Reader did—but it's about what it represents. It's about the commitment that it would mean to Google's customers and to their products. So yeah, something like bring Google Reader back would be a wonderful thing for everyone that subscribes to Google but it would also be a fantastic storytelling element for Google as well. So yes, I'd be entirely in favor of something like that. I hope we can make it happen someday.Corey: Oh, as would I. YOu're in Brian Hall's org, correct?Forrest: Yes.Corey: Brian is a man who was the VP of Product Marketing over at AWS, went to Google for the same role, was sued by AWS under the auspices of a non-compete, which is just the most ridiculous thing in the world, and I want to be very clear here, you can say an awful lot about Brian Hall. I say an awful lot about Brian Hall. AWS says a lot about Brian Hall in very poorly conceived depositions and lawsuits that should never have been allowed to continue, and at least have an editor go over them, but that's a separate problem. But one thing you cannot say about Brian is that he is not incredibly intelligent. And the way that I find that manifesting is, I do not accept that he is someone with such a limited vision that he would be prepared to even entertain the idea of hiring you without giving you what amounts to effectively full creative control of the things you're going to be working on.You are not someone it would make any sense to hire and then try and shove into a box. That is my assessment of everything I've read on every conversation I've had with Googlers in the marketing org; it all speaks to something like this. Was that your impression during the interview? Specifically that you have carte blanche, not that Brian is smart. You're about to be in his org; you're obligated to say it. That's okay. We'll meet at the bar until the real Brian stories later but I'm talking about their remit here.Forrest: No, my authenticity is not for sale, but at the same time. I am a big fan of Brian's and have been since his AWS days, which was honestly one of the big reasons why I ended up joining his org. But yeah, to your question about what is that role going to look like, day to day, of course obviously, that remains to be seen, but it is my understanding that it will have a consultative element and that I will have some opportunity to help to drive some influence across some different teams. Something that I've learned as I've grown in my career a little bit and I've moved into more of management type of roles is that the people that report to you are such a small fraction of the overall influence that you should be having to be really successful in a role like that, any kind of leadership role, so much more of your leadership is going to happen indirectly and by influence, and it's going to happen slowly over time, as you build support for what you're doing and you start to show value and encourage other people to come around to your side. That's just the reality of making change in large organizations.And of course, this is by far the largest organization I've ever worked in, so I know it's going to take time. But my understanding is I do have a little bit of leeway to bring some of my ideas in, and I'm excited about that, and you can sort of judge for yourself, how successful I am, over time.Corey: My last question for you is that sort that has the potential to get you in trouble, except I think I'm going to agree with your answer to this. Do you believe that they're going to Google Reader Google Cloud?Forrest: If I believed that I wouldn't be joining? So obviously, no, I don't believe that.Corey: I have to confess that for the longest time, I was convinced that this was yet another Google misadventure, where they were going to dabble with it, sort of half-ass it, and then shut it down. Because that seems to be the fate of so many Google products out there. The first AWS service that entered beta was Simple Queuing Service. What is a queue but a messaging system, and we know how Google treats messaging products. Same problem; same story.I have to say over the last year or so, my perspective has evolved considerably. They are signing ten-year deals with very large banks; they are investing heavily in hiring, in R&D, in marketing clearly, in a bunch of different areas that are doing the right thing for the long-term. The financial analysts like to beat Google Cloud up because I think two quarters ago, they showed a $5 billion loss, either for the year or for the quarter, and, “It's not making money.” It's, “No. Given Google's position in the market, I'd be horrified if it were. The only way it shouldn't be turning a profit is if there's nowhere left to invest in the platform.”They're making the investments, they're doing the right things. And I have to say I've gone from, “I don't know if I would trust that without an exodus plan,” to, “Yeah, you should have a theoretical exodus plan the same way you should with any provider, but it's not the sort of thing that I feel the need to yank away on 30-days' notice.” I have crossed that bridge myself. In all sincerity, cheap, easy jokes aside, it's clear to me from what I've seen that Google Cloud is going to be around for the long term. Now, we are talking long-term in terms of tech companies, not 150-year-old companies based in Europe, but we can aspire to it. I expect it to outlive me, and not just because I have a big mouth and piss off large companies.Forrest: Yeah. Some of my closest friends and longest-tenured colleagues, people I've worked with for years are GCP engineers, people who are not working for GCP, but they're building on GCP services at various companies. And they always come to me and I've noticed a steady increase in this over the past, I would say 12 to 18 months where they say, “I love working on GCP. I love these services. I love the way the IAM is designed. I love the way the projects are put together. It just feels right. It feels natural to me. It scratches some sort of an itch in my engineering brain.”And then they pause and they say, “Why don't more people get this? Why don't more people understand this story?” That's a problem that I can help to solve. So, I'm really excited about helping to tell the story of Google Cloud. And yeah, that chapter is just about to be written.Corey: I can't wait to see what happens next. If people want to learn more about what you're up to, and how you're approaching these things, and sign up for your various newsletters, where's the entry point? Where can they find you?Forrest: I would say go to my Twitter. I'm on Twitter @forrestbrazeal and there'll be a link in my bio that has links to all the things we've mentioned: The Cloud Resume Challenge Book, my other extremely bizarre book about cloud which is called The Read Aloud Cloud. And there you can sign up for that Best Jobs in Cloud newsletter and all the other things we talked about. So, I'll see you there.Corey: I look forward to including those links in the [show notes 00:32:24]. That's how I wind up expressing my support for all of my guests' nonsense, but particularly yours. Forrest, thank you so much for taking the time to speak with me.Forrest: Much appreciated, Corey. Always a pleasure.Corey: Forrest Brazeal, currently unemployed, but by the time you listen to this, the Head of Content at Google Cloud. I am Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a long, obnoxious, insulting comment, and then rewrite the entire insulting comment without using vowels.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About Christina Christina Maslach, PhD, is a Professor of Psychology (Emerita) and a researcher at the Healthy Workplaces Center at the University of California, Berkeley.  She received her A.B. from Harvard, and her Ph.D. from Stanford.  She is best known as the pioneering researcher on job burnout, producing the standard assessment tool (the Maslach Burnout Inventory, MBI), books, and award-winning articles.  The impact of her work is reflected by the official recognition of burnout, as an occupational phenomenon with health consequences, by the World Health Organization in 2019.  In 2020, she received the award for Scientific Reviewing, for her writing on burnout, from the National Academy of Sciences.  Among her other honors are: Fellow of the American Association for the Advancement of Science (1991 -- "For groundbreaking work on the application of social psychology to contemporary problems"), Professor of the Year (1997), and the 2017 Application of Personality and Social Psychology Award (for her research career on job burnout).  Links: The Truth About Burnout: https://www.amazon.com/Truth-About-Burnout-Organizations-Personal/dp/1118692136 TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored in part by our friends at Lumigo. If you've built anything from serverless, you know that if there's one thing that can be said universally about these applications, it's that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications. It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You've created more problems for yourself; make one of them go away. To learn more, visit lumigo.io.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One subject that I haven't covered in much depth on this show has been a repeated request from the audience, and that is to talk a bit about burnout. So, when I asked the audience who I should talk to about burnout, there were really two categories of responses. The first was, “Pick me. I hate my job, and I'd love to talk about that.” And the other was, “You should speak to Professor Maslach.” Christina Maslach is a Professor of Psychology at Berkeley. She's a teacher and a researcher, particularly in the area of burnout. Professor, welcome to the show.Dr. Maslach: Well, thank you for inviting me.Corey: So, I'm going to assume from the outset that the reason that people suggest that I speak to you about burnout is because you've devoted a significant portion of your career to studying the phenomenon, and not just because you hate your job and are ready to go do something else. Is that directionally correct?Dr. Maslach: That is directionally correct, yes. I first stumbled upon the phenomenon back in the 1970s—which is, you know, 45, almost 50 years ago now—and have been fascinated with trying to understand what is going on.Corey: So, let's start at the very beginning because I'm not sure in, I guess, the layperson context that I use the term that I fully understand it. What is burnout?Dr. Maslach: Well, burnout as we have been studying it over many years, it's a stress phenomenon, okay, it's a response to stressors, but it's not just the exhaustion of stress. That's one component of it, but it actually has two other components that go along with it. One is this very negative, cynical, hostile attitude toward the job and the other people in it, you know, “Take this job and shove it,” kind of feeling. And usually, people don't begin their job like that, but that's where they go if they become more burned out.Corey: I believe you may have just inadvertently called out a decent proportion of the tech sector.Dr. Maslach: [laugh].Corey: Or at least, that might just be my internal cynicism rising to the foreground.Dr. Maslach: No, it's not. Actually, I have heard from a number of tech people over the past decades about just this kind of issue. And so I think it's particularly relevant. The third component that we see going along with this, it usually comes in a little bit later, but I've heard a lot about this from tech people as well, and that is that you begin to develop a very negative sense of your own self, and competence, and where you're going, and what you're able to do. So, the stress response of exhaustion, the negative cynicism towards the job, the negative evaluation of yourself, that's the trifecta of burnout.Corey: You've spent a lot of your early research at least focusing on, I guess, occupations that you could almost refer to as industrial, in some respects: working with heavy equipment, working with a variety of different professionals in very stressful situations. It feels weird, on some level, to say, “Oh, yeah, my job is very stressful. In that vein, I have to sit in front of a computer all day, and sometimes I have to hop on a meeting with people.” And it feels, on some level, like that even saying, “I'm experiencing burnout,” in my role is a bit of an overreach.Dr. Maslach: Yeah, that's an interesting point because, in fact, yes, when we think about OSHA, you know, and occupational risks and hazards, we do think about the chemicals, and the big equipment, and the hazards, so having more psychological and social risk factors, is something that probably a lot of people don't resonate to immediately and think, well, if you're strong, and if you're resilient, and whatever, you can—anybody can handle that, and that's really a test almost of your ability to do your work. But what we're finding is that it has its own hazards, psychological and social as well. And so, burnout is something that we've seen in a lot of more people-oriented professions, from the beginning. Healthcare has had this for a long time. Various kinds of social services, teaching, all of these other things. So, it's actually not a sign of weakness as some people might think.Corey: Right. And that's part of the challenge and, honestly, one of the reasons that I've stayed away from having in-depth discussions about the topic of burnout on the show previously is it feels that—rightly or wrongly, and I appreciate your feedback on this one either way—it feels like it's approaching the limits of what could be classified as mental health. And I can give terrible advice on how computers work—in fact, I do on a regular basis; it's kind of my thing—and that's usually not going to have any lasting impact on people who don't see through the humor part of that. But when we start talking about mental health, I'm cautious because it feels like an inadvertent story or advice that works for some but not all, has the potential to do a tremendous bit of damage, and I'm very cautious about that. Is burnout a mental health issue? Is it a medical issue that is recognized? Where does it start, okay does it stop on that spectrum?Dr. Maslach: It is not a medical issue—and the World Health Organization, which just came out with a statement about this in 2019 on burnout, they're recognizing it as an occupational risk factor—made it very clear that this is not a medical thing. It is not a medical disease, it doesn't have a certain set of medical diagnoses, although people tend to sometimes go there. Can it have physical health outcomes? In other words, if you're burning out and you're not sleeping well, and you're not eating well, and not taking care of yourself, do you begin to impair your physical health down the road? Yes.Could it also have mental health outcomes, that you begin to feel depressed, and anxious, and not knowing what to do, and afraid of the future? Yes, it could have those outcomes as well. So, it certainly is kind of like—I can put it this way, like a stepping stone in a path to potential negative health: physical health, or mental health issues. And I think that's one of the reasons why it is so important. But unfortunately, a lot of people still view it as somebody who's burned out isn't tough enough, strong enough, they're wimpy, they're not good enough, they're not a hundred percent.And so the stigma that is often attached to burnout, people not only indulge it, but they feel it directed towards them, and often they will try to hide the kinds of experiences they're having because they worry that they are going to be judged negatively, thrown under the bus, you know, let go from the job, whatever, if they talk about what's actually happening with them.Corey: What do you see, as you look around, I guess, the wide varieties of careers that are susceptible to burnout—which I have a sneaking suspicion based upon what you've said rounds to all of them—what do you think is the most misunderstood, or misunderstood aspects of burnout?Dr. Maslach: I think what's most misunderstood is that people assume that it is a problem of the individual person. And if somebody is burned out, then they've got to just take care of themselves, or take a break, or eat better, or get more sleep, all of those kinds of things which cope with stressors. What's not as well understood or focused on is the fact that this is a response to other stressors, and these stressors are often in the workplace—this is where I've been studying it—but in essentially in the larger social, physical environment that people are functioning in. They're not burning out all by themselves.There's a reason why they are feeling the kind of exhaustion, developing that cynicism, beginning to doubt themselves, that we see with burnout. So there, if you ever want to talk about preventing burnout, you really have to be focusing on what are the various kinds of things that seem to be causing the problem, and how do we modify those? Coping with stressors is a good thing, but it doesn't change the stressors. And so we really have to look at that, as well as what people can bring about, you know, taking care of themselves or trying to do the job better or differently.Corey: I feel like it's impossible to have a conversation like this without acknowledging the background of the past year that many of us have spent basically isolated, working from home. And for some folks, okay, they were working from home before, but it feels different now. At least that's the position I find myself in. Other folks are used to going into an office and now they're either isolated—and research shows that it has been worse, statistically, for single people versus married people, but married people are also trapped at home with their spouse, which sounds half-joking but it is very real. At some point, distance is useful.And it feels like everyone is sort of a bit at their wit's end. It feels like things are closer to being frayed, there's a constant sense that there's this, I guess, pervasive dread for the past year. Are you seeing that that has a potential to affect how burnout is being expressed or perceived?Dr. Maslach: I think it has, and one of the things that we clearly see is that people are using the word burnout, more and more and more and more. It's almost becoming the word du jour, and using it to describe, things are going wrong and it's not good. And it may be overstretching the use of burnout, but I think the reason of the popularity of the term is that it has this kind of very vivid imagery of things going up in smoke, and can't handle it, and flames licking at your heels, and all this sort of stuff so that they can do that. I even got a comment from a colleague in France just a few days ago, where they're talking about, “Is burnout the malady of the century?” you know, kind of thing. And it's being used a lot; it's sometimes maybe overused, but I think it's also striking a chord with people as a sign that things are going badly, and I don't know how to deal with it in some way.Corey: It also feels, on some level, for those of us who are trapped inside, it kind of almost feels like it's a tremendous expression of privilege because who am I to have a problem with this? Oh, I have to go inside and order a lot of takeout and spend time with my family. And I look at how folks who are nowhere near as privileged have to go and be essential workers and show up in increasingly dangerous positions. And it almost feels like burnout isn't something that I'm entitled to, if that makes sense.Dr. Maslach: [laugh]. Yeah. It's an interesting description of that because I think there are ways in which people are looking at their experience and dealing with it, and like many things in life, I find that all of these things are a bit of a double-edged sword; there's positive and there's negative aspects to them. And so when I've talked with some people about now having to work from home rather than working in their office, they're also bringing up, “Well, hey, I've noticed that the interviews I'm doing with potential clients are actually going a little better”—you know, this is from a law office—“And trying to figure out how—are we doing it differently so that people can actually relate to each other as human beings instead of the suit and tie in the big office? What's going on in terms of how we're doing the work that there may be actually a benefit here?”For others. It's been, “Oh, my gosh. I don't have to commute, but endless meetings and people are thinking I'm not doing my job, and I don't know how to get in touch, and how do we work together effectively?” And so there's other things that are much more difficult, in some sense. I think another thing that you have to keep in mind that it's not just about how you're doing your work, perhaps differently, or you're under different circumstances, but people, so many people have lost their jobs, and are worried that they may lose their jobs.That we're actually finding that people are going into overdrive and working harder and more hours as a way of trying to protect from being the next one who won't have any income at all. So, there's a lot of other dynamics that are going on as a result of the pandemic, I think, that we need to be aware of.Corey: One thing that I'd like to point out is that you are a Professor Emerita of Psychology at Berkeley, which means you presumably wound up formulating this based upon significant bodies of peer-reviewed research, as opposed to just coming up with a thesis, stating it as if it were fact, and then writing an entire series of books on it. I mean, that path, I believe, is called being a venture capitalist, but I may be mistaken on that front. How do you effectively study something like burnout? It feels like it is so subjective and situation-specific, but it has to have a normalization aspect to it.Dr. Maslach: Uh, yeah, that's a good point. I think, in fact, the first time I ever wrote about some of the stuff that I was learning about burnout back in the mid '70s—I think it was '75, '76 maybe—and it was in a magazine, it wasn't in a journal. It wasn't peer-reviewed because not even peer-reviewed journals would review this; they thought it was pop psychology, and eh. So, I would get, in those days, snail mail by the sackfuls from people saying, “Oh, my God. I didn't know anybody else felt like this. Let me tell you my story.”You know, kind of thing. And so that was really, after doing a lot of interviews with people, following them on the job when possible to, sort of, see how things were going, and then writing about the basic themes that were coming out of this, it turned out that there were a lot of people who responded and said, “I know that. I've been there. I'm experiencing it.” Even though each of them were sort of thinking, “I'm the only one. What's wrong with me? Everybody else seems fine.”And so part of the research in trying to get it out in whatever form you can is trying to share it because that gives you feedback from a wide variety of people, not only the peers reviewing the quality of the research, but the people who are actually trying to figure out how to deal effectively with this problem. So it's, how do I and my colleagues actually have a bigger, broader conversation with people from which we learn a lot, and then try and say, okay, and here's everything we've heard, and let's throw it back out and share it and see what people think.Corey: You have written several books on the topic, if I'm not mistaken. And one thing that surprises me is how much what you talk about in those books seems to almost transcend time. I believe your first was published in 1982—Dr. Maslach: Right.Corey: —if I'm not mistaken—Dr. Maslach: Yes.Corey: —and it's an awful lot of what it talks about still feels very much like it could be written today. Is this just part of the quintessential human experience? Or has nothing new changed in the last 200 years since the Industrial Revolution? How is it progressing, if at all, and what does the future look like?Dr. Maslach: Great questions and I don't have a good answer for you. But we have sort of struggled with this because if you look at older literature, if you even go back centuries, if you even go back in parts of the Bible or something, you're seeing phrases and descriptions sometime that says sounds a lot like burnout, although we're not using that term. So, it's not something that I think just somehow got invented; it wasn't invented in the '70s or anything like that. But trying to trace back those roots and get a better sense of what are we capturing here is fascinating, and I think we're still working on it.People have asked, well, where did the term ‘burnout' as opposed to other kinds of terms come from? And it's been around for a while, again, before the '70s or something. I mean, we have Graham Greene writing the novel A Burnt-Out Case, back in the early '60s. My dad was an engineer, rarefied gas dynamics, so he was involved with the space program and engineers talk about burnout all the time: ball bearings burn out, rocket boosters burn out. And when they started developing Silicon Valley, all those little startups and enterprises, they advertised as burnout shops. And that was, you know, '60s, into the '70s, et cetera, et cetera. So, the more modern roots, I think probably have some ties to that use of the term before I and other researchers even got started with it.Corey: This episode is sponsored by ExtraHop. ExtraHop provides threat detection and response for the Enterprise (not the starship). On-prem security doesn't translate well to cloud or multi-cloud environments, and that's not even counting IoT. ExtraHop automatically discovers everything inside the perimeter, including your cloud workloads and IoT devices, detects these threats up to 35 percent faster, and helps you act immediately. Ask for a free trial of detection and response for AWS today at extrahop.com/trial.Corey: This is one of those questions that is incredibly self-serving, and I refuse to apologize for it. How can I tell whether I'm suffering from burnout, versus I'm just a jerk with an absolutely terrible attitude? And that is not as facetious a question as it probably sounds like.Dr. Maslach: [laugh]. Yeah. Well, part of the problem for me—or the challenge for me—is to understand what it is people need to know about themselves. Can I take a diagnostic test which tells me if I am burned out or if I'm something else?Sort of the more important question is, what is feeling right and what is not feeling so good—or even wrong—about my experience? And usually, you can't figure that all out by yourself and you need to get other input from other people. And it could be a counselor or therapist, or it could be friends or colleagues who you have to be able to get to a point where we can talk about it, and hear each other, and get some feedback without putdowns, just sort of say, “Yeah, have you ever thought about the fact that when you get this kind of a task, you usually just go crazy for a while and not really settle down and figure out what you really need to do as opposed to what you think you have to do?” Part of this, are you bringing yourself in terms of the stress response, but what is it that you're not doing—or that you're doing not well—to figure out solutions, to get help or advice or better input from others. So, it takes time, but it really does take a lot of that kind of social feedback.So, when I said—if I can stay with it a little bit more—when I first was writing and publishing about and all these people were writing back saying, “I thought I was the only one,” that phenomenon of putting on a happy face and not letting anybody else see that you're going through some difficult challenges, or feeling bad, or depressed, or whatever is something we call pluralistic ignorance; means we don't have good knowledge about what is normal, or what is being shared, or how other people are because we're all pretending to put on the happy face, to pretend and make sure that everybody thinks we're okay and is not going to come after us. But if we all do that, then we all, together, are creating a different social reality that people perceive rather than actually what is happening behind that mask.Corey: It feels, on some level, like this is an aspect of the social media problem, where we're comparing our actual lives and all the bloopers that we see to other people's highlight reels because few people wind up talking very publicly about their failures.Dr. Maslach: Oh, yeah. Yeah. And often for good reason because they know they will be attacked and dumped. And there could be some serious consequences, and you just say, “I'm going to figure out what I'm going to do on my own.”But one of the things that when I work with people, and I'm asking them, “What do you think would help? What sort of things that don't happen could happen?” And so forth, one of the things that goes to the top of the list is having somebody else; a safe relationship, a safe place where we can talk, where we can unburden, where you're not going to spill the beans to everybody else, and you're getting advice, or you're getting a pat on the back, or a shoulder to cry on, and that you're there for them for the same kind of reason. So, it's a different form of what we think of as social network. It used to be that a network like that meant that you had other people, whether family, friends, neighbors, colleagues, whoever, that you knew, you could go to; a mentor, an advisor, a trusted ally, and that you would perform that role for them and other people, as well.And what has happened, I think, to add to the emphasis on burnout these days, is that those social connections, those trusts, between people has really been shredding, and, you know—or cut off or broken apart. And so people are feeling isolated, even if they're surrounded by a lot of other people, don't want to raise their hand, don't want to say, “Can we talk over coffee? I'm really having a bad day. I need some help to figure out this problem.” And so one of those most valuable resources that human beings need—which is other people—is, if we're working in environments where that gets pulled apart, and shredded, and it's lacking, that's a real risk factor for burnout.Corey: What are the things that contribute to burnout? It doesn't feel, based upon what you've said so far, that it's one particular thing. There has to be points of commonality between all of this, I have to imagine.Dr. Maslach: Yeah.Corey: Is it possible to predict that, oh, this is a scenario in which either I or people who are in this role are likely to become burned out faster?Dr. Maslach: Mm-hm. Yeah. Good question and I don't know if we have a final answer, but at this point, in terms of all the research that's been done, not just on burnout, but on much larger issues of health, and wellbeing, and stress, and coping, and all the rest of it, there are clearly six areas in which the fit between people and their job environment are critical. And if the fit is—or the match, or the balance—is better, they are going to be at less risk for burnout, they're more likely to be engaged with work.But if some real bad fits, or mismatches, occur in one or more of these areas, that raises the risk factor for burnout. So, if I can just mention those six quickly. And these are not in any particular order because I find that people assume the first one is the worst or the best, and it's not. Any rate, one of them has to do with that social environment I was just talking about; think of it as the workplace community. All the people whose paths you cross at various points—you know, coworkers, the people you supervise, your bosses, et cetera—so those social relationships, that culture, do you have a supportive environment which really helps people thrive? Can you trust people, there's respect, and all that kind of thing going on? Or is it really what people are now describing as a socially toxic work environment?A second area has to do with reward. And it turns out not so much salary and benefits, it's more about social recognition and the intrinsic reward you get from doing a good job. So, if you work hard, do some special things, and nothing positive happens—nobody even pats you on the back, nobody says, “Gee, why don't you try this new project? I think you're really good at it,” anything that acknowledges what you've done—it's a very difficult environment to work in. People who are more at risk of burnout, when I asked them, “What is a good day for you? A good day. A really good day.” And the answer is often, “Nothing bad happens.” But it's not the presence of good stuff happening, like people glad that you did such good work or something like that.Third area has to do with values—and this is one that also often gets ignored, but sometimes this is the critical bottom line—that you're doing work that you think is meaningful, where you're working has integrity, and you're in line with that as opposed to value conflicts and where you're doing things that you think are wrong: “I want to help people, I want to help cure patients, and here, I'm actually only supposed to be trying to help the hospital get more money.” When they have that kind of value conflict, this is often where they have to say, “I don't want to sell my soul and I'm leaving.”The fourth area is one of fairness. And this is really about that whatever the policies, the principles, et cetera, they're administered fairly. So, when things are going badly here—the mismatch—this is where discrimination lives, this is where glass ceilings are going on, that people are not being treated fairly in terms of the work they do, how they're promoted, or all of those kinds of things. So, that interpersonal respect, and, sort of, social justice is missing.The next two areas—the fifth and six—are probably the two that had been the most well-known for a long time. One has to do with workload and how manageable it is. Given the demands that you have, do you have sufficient resources, like time, and tools, and whatever other kind of teams support you need to get the job done. And control is about the amount of autonomy and the opportunities you have to perhaps improvise, or innovate, or correct, or figure out how to do the job better in some way. So, when people are having mismatches in work overload; a lack of control; you cannot improvise; where you have unfairness; where there is values that are just incompatible with what you believe is right, a sort of moral issue; where you're not getting any kind of positive feedback, even when it's deserved, for the kind of work you're doing; and when you're working in a socially toxic relationship where you can't trust people, you don't know who to turn to, people are having unresolved conflicts all the time. Those six areas are, those are the markers really of risk factors for burnout.Corey: I know that I'm looking back through my own career history listening to you recount those and thinking, “Oh, maybe I wasn't just a terrible employee in every one of those situations.”Dr. Maslach: Exactly.Corey: I'm sure a lot of it did come from me, I want to be very clear here. But there's also that aspect of this that might not just be a ‘me' problem.Dr. Maslach: Yeah. That's a good way of putting it. It's really in some sense, it's more of a ‘we' problem than a ‘me' problem. Because again, you're not working in isolation, and the reciprocal relationship you have with other people, and other policies, and other things that are happening in whatever workplace that is, is creating a kind of larger environment in which you and many others are functioning.And we've seen instances where people begin to make changes in that environment—how do we do this differently? How can we do this better, let's try it out for a while and see if this can work—and using those six areas, the value is not just, “Oh, it's really in bad shape. We have huge unfairness issues.” But then it says, “It would be better if we could figure out a way to get rid of that fairness problem, or to make a modification so that we have a more fair process on that.” So, they're like guideposts as well.As people start thinking through these six areas, you can sort of say, “What's working well, in terms of workload, what's working badly? Where do we run into problems on control? How do we improve the social relationships between colleagues who have to work together on a team?” They're not just markers of what's gone wrong, but they can—if you flip it around and look at it, let's look at the other end—okay is a path that we could get better? Make it right?Corey: If people want to learn more about burnout in general, and you're working in it specifically, where can they go to find your work and learn more about what you have to say?Dr. Maslach: Obviously, there's been a lot of articles, and now lots of things on the web, and in past books that I've written. And as you said, in many ways, they are still pretty relevant. The Truth About Burnout came out, oh gosh, '97. So, that's 25 years ago and it's still work.But my colleague, Michael Leiter from Canada, and I have just written up a new manuscript for a new book in which we really are trying to focus on sharing everything we have learned about, you know, what burnout has taught us, and put that into a format of a book that will allow people to really take what we've learned and figure out how does this apply? How can this be customized to our situation? So, I'm hoping that that will be coming out within the next year.Corey: And you are, of course, welcome back to discuss your book when it releases.Dr. Maslach: I would be honored if you would have me back. That would be a wonderful treat.Corey: Absolutely. But in return, I do expect a pre-release copy of the manuscript, so I have something intelligent to talk about.Dr. Maslach: [laugh]. Of course, of course.Corey: Thank you so much for your time. I really appreciate it.Dr. Maslach: Well, thank you for having me. I appreciate the opportunity to share this, especially during these times.Corey: Indeed. Professor Christina Maslach, Professor Emeritus of Psychology at Berkeley, I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an insulting comment telling me why you're burned out on this show.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About ScottScott is a web developer who has been blogging at https://hanselman.com for over a decade. He works in Open Source on ASP.NET and the Azure Cloud for Microsoft out of his home office in Portland, Oregon. Scott has three podcasts, http://hanselminutes.com for tech talk, http://thisdeveloperslife.com on developers' lives and loves, and http://ratchetandthegeek.com for pop culture and tech media. He's written a number of books and spoken in person to almost a half million developers worldwide.Links: Hanselminutes Podcast: https://www.hanselminutes.com/ Personal website: https://hanselman.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored in part by our friends at Lumigo. If you've built anything from serverless, you know that if there's one thing that can be said universally about these applications, it's that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications. It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You've created more problems for yourself; make one of them go away. To learn more, visit lumigo.io.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Scott Hanselman of Microsoft. He calls himself a partner program manager—or is called a partner program manager. But that feels like it's barely scraping the surface of who and what he is. Scott, thank you for joining me.Scott: [laugh]. Thank you for the introduction. I think my boss calls me that. It's just one of those HR titles; it doesn't really mean—you know, ‘program manager,' what does it even mean?Corey: I figure it means you do an awful lot of programming. One of the hardest questions is, you start doing different things—and Lord knows you do a lot of them—is that awful question that you wind up getting at cocktail parties of, “So, what is it you do exactly?” How do you answer that?Scott: Yeah, it's almost like, if you spent any time on Clubhouse recently, there was a wonderful comedian named Spunky Brewster on Instagram who had a whole thing where she talked about the introductions at the beginning of a Clubhouse thing, where it's like, you're a multi-hyphenate sandwich artist slash skydiver slash programmers slash whatever. One doesn't want to get too full of one's selves. I would say that I have for the last 30 years been a teacher and a professional enthusiast around computing and getting people excited about computing. And everything that I do, whether it be writing software, shipping software, or building community, hangs off of the fact that I'm an enthusiastic teacher.Corey: You really are. And you're also very hard to pin down. I mean, it's pretty clear to basically the worst half of the internet, that you're clearly a shill. The problem is defining exactly what you're a shill for. You're obviously paid by Microsoft, so clearly you push them well beyond the point when it would make sense to.You have a podcast that has been on for over 800 episodes—which puts this one to shame—called Hanselminutes, and that is, of course, something where you're shilling for your own podcast. You've recently started on TikTok, which I can only assume is what the kids are into these days. You're involved in so many different things and taking so many different positions, that it's very hard to pin down what is the stuff you're passionate about.Scott: I'm going to gently push back and say—Corey: Please do.Scott: That if one were to care to look at it holistically, I am selling enthusiasm around free and open-source software on primarily the Windows platform that I'm excited about, and I am selling empowerment for the next generation of people who want to do computing. Before I went to Microsoft, my blog and my podcast existed, and I was consistent in my, “Hey, have you heard the news?” Message to anyone who would listen. And I taught at both Portland Community College and Oregon Institute of Technology, teaching web services and history of the web and C# and all that kind of stuff. So, I'm one of those people where if you touch on a topic that I'm interested in, I'll be like, “Oh, my goodness, let's”—and I'll just like, you know, knock everything off the desk and I'm going to be like, “Okay, let's build a model, a working model of the solar system here, now. The orange is the sun.”And it's like, suddenly now we're talking about science, like Hank Green or whatever. My family will ask me, “Why isn't the remote control working?” And then I've taken it apart and I'm explaining to them how the infrared LED inside works. And, you know, how can you not be excited about all these things? And that's my whole thing about computing and the power that being able to program computers represents to me.Corey: I would agree with that. I'd say that one thing that is universal about everything you're involved in is the expression I heard that I love and am going to recapture has been, “Sending the elevator back down.”Scott: Oh, yeah. Throwing ladders, ropes, elevators. I am very blessed to have made it out of my neighborhood, and I am very hopeful that anyone who is in a situation that they do not want to be in could potentially use coding, programming, IT, computing as the great equalizer and that I can I could somehow lend my privilege to them to get the things done and solve the problems that they want to solve with computers.Corey: I'm sure that you've been asked ad nauseum about—you work in free and open-source software. You've been an advocate for this, effectively, for your entire career; did no one tell you you work at Microsoft? But that's old Microsoft in many respects. That's something that we've covered with a bunch of different guests previously from Microsoft, and it's honestly a little—it's becoming a bit of a tired trope. It was a really interesting conversation a few years back that, oh, it's clearly all just for show.Well, that is less and less obvious, and more tired and frankly bad take as time progresses. So, I want to go back a bit further into my own personal journey because it turns out that the number one reason to reach out to you for anything is tech support on various things. I don't talk about this often, but I started my career moonlighting as a Windows admin, back in the Windows 2003 server days; and it was an experience, and licensing was a colossal pain, and I finally had enough of it one day, in 2006, switched over to Unix administration on BSD, and got a Mac laptop, and that was really the last time that I used Windows in anger. Now, it's been 15 years since that happened, and I haven't really been tracking the Windows ecosystem. What have I missed?Scott: [laugh]. There's a lot there that you just said. So first, different people have their religions and they're excited about them, and I encourage everyone to be excited about the religion that they're excited about. It's great to be excited about your thing, but it's also really not cool to be a zealot about your thing. So hey, be excited about Windows, be excited about Linux, be excited about Mac.Just don't tell me that I'm going to heck because I didn't share your enthusiasm. Let's just be excited together and we can be friends together. I've worked on Linux at Nike, I've worked on Mac, I've worked on Windows, you know, I've been there before these things existed and I'll be there afterwards.Corey: Exactly. At some point being a zealot for a technology just sort of means you haven't been around the block enough to understand how it's going to break, how it's going to fail, how it's going to evolve, and it doesn't lead to a positive outcome for anyone. It fundamentally becomes a form of gatekeeping more than anything else, and I just don't have the stomach for it.Scott: Yeah. And ultimately, we're just looking for—you know, we got these smart rocks that we taught how to think with lightning, and they're running for loops for us. And maybe they're running them in the cloud, maybe they're running locally. So, I'm not really too worried about it. Windows is my thing of choice, but just, you know, one person's Honda is another person's Toyota; you get excited about the brand that you start out with.So, that's that. Currently, though, Windows has gone, at least in the last maybe 20 years, from one of those things where there's generational pain, and, like, “Microsoft killed my Pappy, and I'll never forgive you.” And it's like, yeah, there was some dumb stuff in the '90s with Internet Explorer, but as a somewhat highly placed middle manager at Microsoft, I've never been in an active mustache-twirling situation where I was behind closed doors and anyone thought anything nefarious. There's only a true, “What's the right thing for the customer? What is the right thing for the people?”My whole thing is to make it so developers can develop more easily on Windows, so I'm very fortunate to be helping some folks in a partnership between the Windows division and the developer division that I work in to make Windows kick butt when it comes to dev. Historically, the Windows terminal, or what's called cmd.exe which is run by a thing called the console host has sucked; it has lagged behind. So, if you drop out to the command line, you've got the, you know, the old, kind of, quote-unquote, “DOS shell” with a cmd processor—it's not really DOS—running in an old console host. And it's been there for gosh, probably early '90s. That sucks.But then you got PowerShell. And again, I want to juxtapose the difference between a console—or a terminal—and a shell. They're different things. There's lots of great third-party terminals in the ecosystem. There's lots of shells to choose from, whether it be PowerShell, PowerShell Core—now PowerShell 7.0—or the cmd, as well as bash, and Cygwin, and zsh, and fish.But the actual thing that paints the text on Windows has historically not been awesome. So, the new open-source Windows terminal has been the big thing. If you're a Machead and you use iTerm2, or Hyper, or things like that, you'll find it very comfortable. It's a tabbed terminal, split-screen, ripping fast, written in, you know, DirectX, C++ et cetera, et cetera, all open-source, and then it lets you do transparency, and background colors, and ligature fonts, and all the things that a great modern terminal would want to do. That is kind of the linchpin of making Windows awesome for developers, then gets even awesomer when you add in the ability that we're now shipping an actual Linux kernel, and I can run N number of Linuxes side-by-side, in multiple panes, all within the terminal.This getting to the point about juxtaposing the difference between a terminal and a console and a shell. So, I've got, on the machine, I'm talking to you on right now, on my third monitor, I've got Windows terminal open with PowerShell on Windows on the left, Ubuntu 18.04 LTS on the right, with the fish shell. And then I've got another Ubuntu 20.04 with bash, a standard bash shell.And I'm going and testing stuff in Docker, and running .NET in Docker, and getting ready to deploy my own podcast website up into Azure. And I'm doing it in a totally organic way. It's not like, “Oh, I'm just running a virtual machine.” No, it's integrated. That's what I think you'd be impressed with.Corey: That right there is the reason that I generally tended to shy away from getting back into the Windows ecosystem for the longest time—and this is not a slam on Windows, by any stretch of the—Scott: No of course. Sure, sure, sure.Corey: —imagination—my belief has always been that you operate within the environment as it's intended to be operated within, and it felt at the time, “Oh, install Cygwin, and get all this other stuff going, and run a VM to do it.” It felt like I was fighting upstream in some respects.Scott: Oh, yeah, that's a great point. Let's talk about that for a second. So—Corey: Let's do it.Scott: So, Cygwin is the GNU utilities that are written in a very nice portable C, but they are written against the Windows kernel. So, the example I like to use is ls, you type ls, you list out your directory, right? So, ls and dir are the same thing for this conversation. Which means that someone has to then call a system call—syscall in Linux, Windows kernel call in Windows—and say, “Hey, would you please enumerate these files, and then give me information about them, and check the metadata?” And that has to call the file system and then it's turtles all the way down.Cygwin isn't Linux. It's the bash and GNU utilities recompiled and compiled against the Windows stuff. So, it's basically putting a bash skin on Windows, but it's not Linux; it's bash. Okay? But WSL is actually Linux, and rather than firing up a big 30 gig Hyper-V, or VirtualBox, or Parallels virtual machine, which is, like, a moment—“I'm firing up the VM; call me in an hour when it comes back up.”—and when the VM comes up, it's, like, a square on your screen and now you're dealing with another thing to manage.The WSL stuff is actually a utility virtual machine built on a lower subsystem, the virtualization platform, and it starts in less than a second. You can start it faster than you can say, one one-thousand. And it goes instantly up, it automatically allocates and deallocates memory so that it's smart about memory, and it's running the actual Linux kernel, so it's not pretending to be Linux. So, if your goal is a Linux environment and you're a Linux developer, the time of Linux on the desktop is happening, in this case, on the Windows desktop. Where you get interesting stuff, and where I think your brain might explode is, imagine you're in the terminal, you're at the Linux file system at the bash prompt, and you type ‘notepad.exe.' What would you expect to happen? You'd expect it to try to find it in a Linux path and fail.Corey: Right. And then you're trying to figure out, am I in this environm—because you generally tend to run these things in the same-looking terminal, but then all the syntax changes as soon as you go back into the Windows native environment, you're having to deal with line-ending issues on a constant basis, and you just—Scott: Oh, yeah. All that stuff, where.Corey: And as soon as you ask for help because back in those days, I was looking primarily into using freenode as my primary source of support because I network staff on the network for the better part of a decade, and the answer is, “I'm having some trouble with Linux,” and the response is, “Oh, you're doing this within a Windows environment? Get a real computer, kid.” Because it's still IRC, and being condescending and rude to anyone who makes different choices than you do is apparently the way that was done back then.Scott: Well, today in 2020 because we don't want to just have light integration with Windows—and by light integration, like, I don't know if you remember firing up a virtual machine on Windows and then, like, copy-pasting a file, and we were all going like, “Oh, my God, that's amazing.” I drug the file in and then it did a little bit of magic and then moved the file from Windows into Linux. What we want is to blur the lines between the two so you can move comfortably. When you type explorer.exe or notepad.txt in Linux on Windows, Linux says no, and then Windows gets the chance, fires it up, and can access the Linux file system.And since Notepad now understands line endings, just happily, you can open up your .profile, your bash_profile, your csh file in Notepad, or—here's where it gets interesting—Visual Studio Code, and comfortably run your Windows apps, talking to your Linux file system, or in the—coming soon, and we've blogged about this and announced it at Build last year, run Linux GUI apps seamlessly so that I could have two browsers up, two Chromes, one Windows and one Linux, side-by-side, which is going to make web testing even that much easier. And I'm moving seamlessly between the two. Even cooler, I can type explorer.exe and then pass in dot, which represents the current folder, and if the current folder is the Linux file system, we seamlessly have a Plan 9 server—basically a file server that lets you access your Linux file system—from—Corey: Is it actually running Plan 9?Scott: It is a Plan 9 server.Corey: That is amazing. I'm sorry, that is a blast from the past.Scott: I'm glad. And we can run N number of Linuxes; this isn't just one Linux. I've got Kali Linux, two different Ubuntus, and I could tar up the user mode files on mine, zip them up, give them to you, and you could go and type ‘wsl–import,' and then have my Linux file system. Which means that we could make a custom Screaming in the Cloud distro, put it in the Windows Store, put it up on GitHub, build our own, and then the company could standardize on our Linux distro and run it on Windows.Corey: That is almost as terrible an idea as using a DNS service as a database.Scott: [laugh].Corey: I love it. I'm totally there for it.Scott: It's really nice because it's extremely—the point is, it has to have no friction, right? So, if you think about it this way, I just moved—I blogged about this; if people want to go and learn about it—I just moved my blog of 20 years off of a Windows Server 2008 server running under someone's desk at a host, into Azure. This is a multi-month-long migration. My blog, my main site, kind of the whole Hanselman ecosystem moved up in Azure. So, I had a couple things to deal with.Am I going to go from Windows to Linux? Am I going to go from a physical machine to a virtual machine? Am I going to go from a physical machine to a virtual machine to a Platform as a Service? And when I do that, well, how is that going to change the way that I write software? I was opening it in Visual Studio, pressing F5, and running it in IIS—the Internet Information Server for Windows—for the last 15, 20 years.How do I change that experience? Well, I like Visual Studio; I like pressing F5; I like interactive debugging sessions. But I also like saving money running Linux in the cloud, so how can I have the best of all those worlds? Because I wrote the thing in .NET, I moved into .NET 5, which runs everywhere, put together a Docker file, got full support for that in Visual Studio, moved it over into WSL so I can test it on both Windows and Linux.I can go into my folder on my WSL, my Windows subsystem for Linux, type code dot, open up Visual Studio Code. Visual Studio Code splits in half. The Windows client of Visual Studio Code runs on Windows; the server, the Visual Studio Code server, runs in WSL providing the bridge between the two worlds, and I can press F5 and have interactive debugging and now I'm a Linux developer even though I've never left Windows. Then I can right-click publish in Visual Studio to GitHub Actions, which will then throw it into the cloud, and I moved everything over into Azure, saved 30%, and everything's awesome. I'm still a Windows developer using Visual Studio. So, it's pretty much I don't know, non-denominational; kind of mixing the streams here.Corey: It is. And let me take it a step further. When I'm on the road, the only computer I bring with me these days—well, in the before times, let's be very realistic. Now, when ‘I'm on the road,' that means going to the kitchen for a snack—the only computer I bring with me is my iPad Pro, which means that everything I do has a distinct application. For when I want to get into my development environment, historically it was, use some terminal app—I'm a fan of Blink, but everyone has their own; don't email me.And everything else I tended to use looked an awful lot like a web app. If there wasn't a dedicated iOS app, it was certainly available via a web browser. Which leads me to the suspicion that we're almost approaching a post-operating-system world where the future development operating system begins to look an awful lot—and people are going to yell at me for this—Visual Studio Code.Scott: Mmm.Corey: It supports a bunch of remote activities now that GitHub Codespaces is available—at least to my account; I don't know if it's generally available yet—but I've been using it; I love it; everything it winds up doing is hosted remotely in Azure; I don't have to think about managing the infrastructure; it's just another tab within GitHub, and it works. My big problem is that I'm trying to shake, effectively, 20 years of muscle memory of wrestling with Vim, and it takes a little bit of a leap in order to become comfortable with something that's a more visually-oriented IDE.Scott: Why don't you use the VsVim, Jared Parsons Vim plugin for Visual Studio?Corey: I've never yet found a plugin that I like for something else to make it behave like Vim. Vimperator is a browser extension, all of it just tends to be unfortunate and annoying in different ways. For whatever reason, the way that I'm configured or built, it doesn't work for me in the same way. And it goes back to our previous conversation about using the native offering as it comes, rather than trying to make it look like something else.Scott: Okay. I would just offer to you and for other Vim people who might be listening, that VS Code Vim does have 2.5 million installs, over 2 million people happily using that. And they are—Corey: Come to find it only has 200,000 actual users; there was an installation bug and one person just kept trying over and over and over. I kid, I kid.Scott: No, seriously though, these are actual Vim-heads and Jared Parsons is a developer at Microsoft who is like, out of his cold dead hands you'll pull his Vim. So, there's solutions; whether you're Vim or Emacs, you know, we welcome all comers. But to your point, the Visual Studio, once it got split in half, where the language services, those services that provide context to Python, Ruby, C# C++ et cetera, once those extensions can be remoted, they can run on Windows, they can run on Linux, they can run on the cloud. So, VS Code being split in half as a client-server application has really made it shine. And for me, that means that I don't notice a difference, whether I'm running VS Code on Windows or running VS Code to a remote Linux install, or even using SSH and coding on Windows remotely to a Raspberry Pi.Corey: I love the idea. I've seen people do this, in some respects, back in the days of Code Server being a project on GitHub, and it took a fair bit of wrangling to get that to work in a way that wasn't scarily insecure and reliable. But once it was up and running, you could effectively plug a Raspberry Pi in underneath your iPad and effectively have a portable computer on the go that did local development. I'm looking at this and realizing the future doesn't look at all like what I thought it was going to, and it's really still kind of neat.Scott: Mm-hm.Corey: There's a lot of value in being able to make things like this more accessible, and the reason I'm excited about a lot of this, too, is that aligned with a generous free tier opportunity, which I don't know final pricing for things like GitHub Codespaces, suddenly the only real requirement is something that can render a browser and connect to the internet for an awful lot of folks to get started. It doesn't require a fancy local overpowered development machine the way a lot of things used to. And yes, I know; there are certain kinds of development that are changing in that respect, but it still feels to me like it has never been easier to get started with all of this technology than ever before, with a counterargument that there's so many different directions to go in. “Oh, I want to get started using Visual Studio Code or learning to write JavaScript. Great. How do I do this? Let me find a tutorial.” And you find 20 million tutorials, and then you're frozen with indecision. How do you get past that?Scott: Yeah, there is and always will be, unfortunately, a certain amount of analysis paralysis that occurs. I started a TikTok recently to try to help people to get involved in coding, and the number one question I get—and I mean, thousands and thousands of them—are like, “Where do I start?” Because everyone seems to think that if they pick the wrong language, that will be a huge mistake. And I can't think of a wrong language, you know? Like, what human language should I learn?You know, English, Chinese, Arabic, Japanese. Pick one and then learn another one if you can. Learn a couple. But I don't think there's a wrong language to learn because the basics of computer science are the basics of computer science. I think what we need to do is remind people that computers are computers no matter whether they're an Android phone or a Windows laptop, and that any forward motion at all is a good thing. I think a lot of people have analysis paralysis, and they're just afraid to pick stuff.Corey: I agree with what you're saying, but I'm also going to push back gently on what you're saying, as well. If someone who is new to the field was asking me what language to learn, I would be hard-pressed to recommend a language that was not JavaScript. I want to be clear, I do not understand or know JavaScript at all, but it's clear from what I'm seeing, that is, in many ways, the language of the future. It is how frontend is being interacted with; there are projects from every cloud provider that wind up managing infrastructure via JavaScript primitives. There are so many on-ramps for this, and the user experience for new folks is phenomenal compared to any language that I've worked with in my career. Would you agree with that or disagree with that assessment?Scott: So, I've written blog posts on this topic, and my answer is a little more ‘it depends.' I say that people should always learn JavaScript and one other language, preferably a systems language, which also may be JavaScript. But rather than thinking about things language-first, we think about things solutions-first. If someone says, “I want to do a lot of data science,” you don't learn JavaScript. If someone says, “I want to go and write an Android app,” yeah, you could do that in JavaScript, but JavaScript is not the answer to all questions.Just as the English language, while it may be the lingua franca, no pun intended, it is not the only language one should pick. I usually say, “Well, what do you want to do?” “Well, I want to write a video game for the Xbox.” Okay, well, you're probably not going to do that in JavaScript. “Oh, I want to do data science. I want to write an iPhone app.” JavaScript is the language you should learn if you're going to be doing things on the web, yes, but if you're going to be writing the backend for WhatsApp, then you're not going to do that JavaScript.Corey: This episode is sponsored by ExtraHop. ExtraHop provides threat detection and response for the Enterprise (not the starship). On-prem security doesn't translate well to cloud or multi-cloud environments, and that's not even counting IoT. ExtraHop automatically discovers everything inside the perimeter, including your cloud workloads and IoT devices, detects these threats up to 35 percent faster, and helps you act immediately. Ask for a free trial of detection and response for AWS today at extrahop.com/trial.Corey: Yeah, I think you're right. It comes down to what is the problem you're trying to solve for? Taking the analogy back to human languages, well, what is your goal? Is it just to say that you've learned a language and to understand, get a glimpse at another culture through its language? Yeah, there is no wrong answer. If it's that you want to go live in France one day and participate in French business discussions, I have a recommendation for you, and it's probably not Sanskrit.At some point, you have to align with what people want to do and the direction they're going in with the language selection. What I like about JavaScript is, frankly, it's incredible versatility as far as problems to which it can be applied. And without it, I think you're going to struggle as you enter the space. My first language was crappy Perl—slash bash because everyone does bash when you're a systems administrator—and then it has later evolved now to crappy Python as my language of choice. But I'm not going to be able to effectively do any frontend work in Python, nor would I attempt to do so.My way of handling frontend work now is to have the good sense to pay a professional. But if you're getting started today and you're not sure what you want to do in your career, my opinion has always been that if you think you know what you want to do in your career, there's a great chance you're going to be wrong, but pursuing the thing that you think you want to do will open other opportunities and doors, and present things to you that will catch your interest in a way you might not be able to anticipate. So, especially early on in careers, I like biasing for things that give increased options, that boost my optionality as far as what I'm going to be able to do.Scott: Okay. I think that's fair. I think that no one ever got fired for picking IBM; [laugh] no one ever jeopardized their career by choosing JavaScript. I do think it's a little more nuanced, as I mentioned.Corey: It absolutely is. I am absolutely willing to have a disagreement with you on that front. I think the thing that we're aligned on is that whatever you pick, make sure it's something you're interested in. Don't do it just for—like, “Well, I'm told I can make a lot of money doing X.” That feels like it's the worst reason to do things, in isolation.Scott: That's a tough one. I used to think that, too, but I am thinking that it's important to note and recognize that it is a valid reason to get into tech, not for the passion because for no other reason that I want to make a lot of money.Corey: Absolutely. I could not agree with you more, and that is… something I've gotten wrong in the past.Scott: Yeah. And I have been a fan of saying, you know, “Be passionate and work on these things on the side,” and all that kind of stuff. But all of those things involve a lot of assumptions and a lot of privileges that, you know, people have: that you have spare time and that you have a place to work on these things. I work on stuff on the side because it feeds my spirit. If you work on woodworking, or drones, or gardening on the side, you know, not everything you work on the side has to be steeped in hustle culture and having a startup, or something that you're doing on the side.Corey: Absolutely. If you're looking at a position of wanting to get into technology because it leads to a better financial outcome for you and that is what motivates you, you're not wrong.Scott: Exactly.Corey: The idea that, “Oh, you have to love it or you'll never succeed.” I think that some of the worst advice we ever wind up giving folks early in their career—particularly young people—is, ‘follow your passion.' That can be incredibly destructive advice in some contexts, depending upon what it is you want to do and what you want your life to look like.Scott: Yeah, exactly.Corey: One of the things that I've always been appreciative of from afar with Microsoft has been there's an entire developer ecosystem, and historically, it's focused on languages I can barely understand: ASP.NET, the C# is deep in that space, F#, I think, is now a thing as well. There's an entire ecosystem around this with Visual Studio the original, not Visual Studio Code—turns out naming is one of those things that no tech companies seems to get right—but it feels almost like there's an entire ecosystem there for those of us who spent significant time—and I'm speaking for myself here, not you—in the open-source community talking about things like Perl and whatnot, I never got much exposure to stuff like that. I would also classify Enterprise Java as being in that direction as well. Is there a bifurcation there that I'm not seeing, or was I just never talking to the right people? All the above? Maybe I was just—maybe I had blinders on; didn't realize it.Scott: There was a time when the Microsoft developer ecosystem meant write things for Windows, do things on Windows, use languages that Microsoft made and created. And now, with the rise of the cloud and with the rise of Software as a Service, Microsoft is a much simpler company, which is a funny thing to say for such a complicated company. Microsoft would love to run your for loop in the cloud for money. We don't care what language you use; we want you to use the language that makes you happy. Somewhere around five to seven years ago, in the developer division, we started optimizing for developer happiness.And that's why you can write Ruby, and Perl, and Python, and C, and C++ and C# and all those different things. Even C# now, and .NET, is owned by the .NET Foundation and not by Microsoft. Microsoft, of course, is one of the primary users, but we've got a lot of—Samsung is a huge contributor, Google is a huge contributor, Amazon Web Services is a big contributor to .NET.So, Microsoft's own zealotry towards—and bias towards our own languages has, kind of, gone away because Office is on iPhone, right? Like, anywhere that you are, we'll go there. So, we're really going where the customer is rather than trying to funnel the customer into where we want them to be, which is a really an inverted way of doing things over the way it was done 20, 30 years ago. In my opinion.Corey: This gets back to the idea of the Microsoft cultural transformation. It hasn't just been an internal transform; it's been something that is involved with how it's engaging with its customers, how it's engaging with the community, how it's becoming available in different ways to different folks. It's hard to tell where a lot of these things start and where a lot of these things stop. I don't pretend to be a Microsoft “fanboy,” quote-unquote, but I believe it is impossible to look at what has happened, especially in the world of cloud, and not at the very least respect what Microsoft has been able to achieve.Scott: Well, I came here to open source stuff. I'm surely not responsible for the transformation, I'm just a cog in the machine, but I can speak for the things that I own, like .NET and Visual Studio Community, and I think one of the things that we have gotten right is we are trying to create zero-distance products. You could be using Visual Studio Code, find a bug, suggest a feature, have a conversation in public with the PMs and devs that own the thing, get an insider's build a few days later, and see that promoted to production within a week or two. There is zero distance between you the consumer and the creator of the thing.And if you wanted to even fix the bug yourself, submit a pull request, and see that go into production, you could do that as well. You know, some of our best C# compiler folks are not working for Microsoft and they are giving improvements, they are making the product better. So, zero-distance in many ways, if you look at the other products at Microsoft, like PowerToys is a great thing, which is [unintelligible 00:32:06] an incubator for Windows features. We're adding stuff to the PowerToys open-source project like launchers, and a thing called FancyZones that is a window tiling manager, you know, features that prosumers and enthusiasts always wished Windows could have, they can now participate in, thereby creating a zero-distance product in Windows itself.Corey: And I want to point out as well that you are still Microsoft. You, the collective you. I suppose you personally; that is where your email address ends. But you're still Microsoft. This is still languages, and tools, and SDKs, and frameworks used by the largest companies in the world. This zero-distance approach is being done on things that service banks, who are famously not the earliest adopters of some code that I wrote last night; it's probably fine.Scott: Do you know what my job was before I came here?Corey: Tell me.Scott: I was the chief architect at a finance company that created software for banks. I was responsible for a quarter of the retail online banking systems in North America, built on .NET and open-source software. [laugh].Corey: So, you've lived that world. You've been that customer.Scott: Trying to convince a bank that open-source was a good idea in the early 2000s was non-trivial. You know, sitting around in 2003, 2004, talking about Agile, and you know, continuous integration, and build servers, and then going and saying, “Hey, you should use the software,” trying to deal with lawyers and explain to them the difference between the MIT, Apache, and GPL licenses and what it means to their bank was definitely a challenge. And working through those issues, it has been challenging. But open-source software now pervades. Just go and look at the license.txt in the Visual Studio Program Files folder to see all of the open-source software that is consumed by Visual Studio.Corey: One last topic that I want to get to before we call it a show is that you've spent a significant portion of your career, at least recently, focusing on, more or less, where the next generation of engineers, developers, et cetera, come from. And to that end, you've also started recently with TikTok, the social media platform. Are those two things related, first off, or am I making a giant pile of unwarranted assumption?Scott: [laugh]. I think that is a fair assumption. So, what's going on is I want to make sure that as I fade away and I leave the software industry in the next, you know, N number of years, that I'm setting up as many people as possible for success. That's where my career started when I was a professor, and that's hopefully where my career will end when I am a professor again. Hopefully, my retirement gig will have me teaching at some university somewhere.And in doing that, I want to find the next million developers, right? Where are they, the next 10 million developers? They're probably not on Twitter. They might be a lot of different places: they might be on Discord, they might be on Reddit, they might be on forums that I haven't found yet. But I have found, on TikTok, a very creative and for the most part kind and inclusive community.And both myself and also recently, the Visual Studio Code team have been hanging out there, and sharing our creativity, and having really interesting conversations about how you the listener can if not be a programmer, be a person that knows better the tools that are available to you to solve problems.Corey: So, I absolutely appreciate and enjoy the direction that you're going in, but again, people invite you to things and then spring technical support questions on you. Can you explain what TikTok is? I'm still trying to wrap my head around it because I turned around and discovered I was middle-aged one day.Scott: Sure. Well, I mean, I am an old man on TikTok, to be clear. TikTok, like Twitter, revels in its constraints. If you recall, there was a big controversy when Twitter went from 140 characters to 280 because people thought it was just letting the constraint that we were so excited about—which was artificial because it was the length of a standard message service text—Corey: I'm one of those people who bitterly protested it. I was completely wrong.Scott: Right? But the idea that something is constrained, that TikTok is either 15 seconds, or less than 60, it's similar to Vine in that it is a tiny video; what can I do in one minute? Additionally, before they allowed uploading of videos, everything was constrained within the TikTok editor, so people would do amazing and intricate 30 and 40 shot transitions within a 60 second period of time. But one of the things I find most unique about TikTok is you can reply to a text comment with a video. So, I make a video—maybe I do 60 seconds on how to be a software engineer—somebody replies in text, I can then reply to that text with a video, and then a TikTok creator can do what's called a stitch and reply to my video with a video.So, I could take 15 seconds of yours, a comment that you made, and say, “Oh, this is a great comment. Here's my thoughts on that comment.” Or we could even do a duet where you record a video and then I record one, side-by-side. And we either simulate that we're actually having a conversation, or I react to your video as well. Once you start teaching TikTok about yourself by liking things, you curate a very positive place for yourself.You might get on TikTok, not logged in, and it's dancing, and you might find some inappropriate things that you don't necessarily want to see, or you're not interested in, but one of the things that I've noticed as I talk about my home network and coding is people will say, “Oh, I finally found adjective TikTok; I finally found coding TikTok I finally found IT TikTok. Oh, I'm going to comment on your post because I want to stay on networking TikTok.” And then your feed isn't just a feed of the people that you follow, but it's a feed of all the things that TikTok thinks you're excited about. So, I am on this wonderful TikTok of linguistics and languages, and I'm learning about cultures, and I'm on indigenous TikTok, and I'm on networking TikTok. And the mix of creativity and the constraint of just 60 seconds has been, really, a joy. And I've only been there for about a month and I've blessed to have 80,000 people hanging out with me there.Corey: It sounds like you're quite the fan of the platform, which alone in isolation, is enough to get me to look at it in more depth.Scott: I am a fan of creativity. I would also say though, it's very addictive once you find your people. I've had to put screen time limits on my own phone to keep me from burning time there.Corey: That is all of tempting, provocative, and disturbing. I—Scott: You should hang out with me on YouTube, then. I just got my 100,000 YouTube Silver Play Button in the mail. That's where I spend my time doing my long-form. I just did, actually, 17 minutes on WSL and how to use Linux. That might be a good starter for you.Corey: It very well might. So, if people want to learn more about what you're up to, and how you think about the wide variety of things you're interested in, where can they find you?Scott: They should start at my last name dot com: Hanselman.com. They used to be able to Google for Scott, and I was in an epic battle with Scott brand toilet paper tissue, and then they trademarked the name Scott and now I'm somewhere in the distant second or third page. It was a tragedy. But as an early comer—Corey: Oh, my condolences.Scott: Yeah, oh my God. As an early comer to the internet, it was me and Scott Fly Rods on the first page, for many, many years. And then—Corey: If it helps, you and Scott Fly Rods are both on page two.Scott: Oh. Well, the tyranny of the Scott toilet paper conspiracy against me has been problematic.Corey: Exactly.Scott: [laugh].Corey: Thank you so much for taking the time to speak with me today. I really do appreciate it.Scott: It's my pleasure.Corey: Scott Hanselman, partner program manager at Microsoft and so much more. I'm Cloud Economist Corey Quinn. This is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a crappy comment that starts with a comment that gatekeeps a programming language so we know to ignore it.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Jay is joined by Eric from the Joshi Pod in the second half of the episode before that there really is not much going on this week is there? So as the mature and productive members of society that we are we some make motorbike noise and it is great a fun time had by all. Jay is not Corey & Corey Is not Jay. Check out The Joshi Pod : https://podcasts.apple.com/us/podcast/the-joshi-pod/id1486086093 Follow us on Twitter Eric Twitter : https://twitter.com/TheJoshiPod Podcast Twitter : https://twitter.com/SmackItDownPod Corey Gold Twitter : https://twitter.com/gold_corey Jay Silver Twitter : https://twitter.com/thevivalajady Other Podcasts that we are on. Red Leaf Retrocast : https://podcasts.apple.com/au/podcast/red-leaf-retrocast-gaming-anime-wrestling/id1193899321 Check out the real wrestling community of Facebook : https://www.facebook.com/TheRealWrestlingCommunity/

After Family issues Jay is back and ready to record a podcast, anyways Jay is NJPW bound and Corey Is going to a 21st. Podcast Twitter : https://twitter.com/SmackItDownPod Corey Gold Twitter : https://twitter.com/gold_corey Jay Silver Twitter : https://twitter.com/thevivalajady

I had multiple people on for the first time!Corey Is a automotive photographer and lordDank of the Danksquad Car Club!Phil is just a dude, playing a dude, disguised as a other dude with a subaru.IG: Danksquad.tm Philipondeznutz

Corey Is out on planet peacock & Jay is stuck somewhere between his notes and Good Things Music Festival BUT that will not stop them from bringing you a review of the week in wrestling & Wrestlemania-XXIV Podcast Twitter : https://twitter.com/SmackItDownPod Corey Gold Twitter : https://twitter.com/gold_corey Jay Silver Twitter : https://twitter.com/thevivalajady

Guest Co-hosts: Corey Roberts and Teymur Madjderey! Movie Review: "Funny People"! Movie News: Jeremy Renner is Mad Max? Rob Marshall is directing Pirates 4? Bruckheimer declares World War Robot! Comics: Jonathan goes nuts for Volume 2 of Atomic Robo from Red 5 Comics! Corey IS a comic... and fun at Stephen Seagal parties! Can Tyrese Gibson sell comic books? Video Games: Turtles in Time Reshelled hits XBL! Daisy Fuentes Pilates! PLUS! SAM takes on the Comic Con floor! Zack Haddad interviews the cast of Dead of Night! Teymur talks about being a geek in Germany!