Podcasts about Lambda

Meet A.J. Irving and her new book The Wishing Flower!An LGBTQ-inclusive story about understanding your peers, your feelings, and yourself, The Wishing Flower is a love letter to longing, belonging, and longing to belong. With stunning illustrations by Kip Alizadeh, The Wishing Flower will inspire readers to honor their wishes and show the world their truest selves.A.J. Irving grew up in Boise, Idaho, writing stories and daydreaming about becoming an author. Now, she writes picture books and poetry beneath an old elm tree in Salt Lake City. A.J. reads kidlit every day and dances every chance she gets. She is also the author of Dance Like a Leaf (Barefoot Books, 2020).Buy The Wishing FlowerHead to our Bookshop store or buy directly: https://bookshop.org/a/82376/9780593430446Connect with A.J. IrvingWebsite: ajirving.comTwitter: @aj_irvingInstagram: @aj_irvingFacebook: facebook.com/ajirvingauthorShow linksAs promised in the episode, links to:A Night Without Armor by Jewel (https://www.thriftbooks.com/w/a-night-without-armor--poems_jewel/581143/#edition=2334061&idiq=6502050) A Million Quiet Revolutions by Robin Gow (https://bookshop.org/a/82376/9780374388416).Become an Associate Producer!Become an Associate Producer of our podcast through a $20/month sponsorship on Patreon! A professionally recognized credit, you can gain access to Associate Producer meetings to help guide our podcast into the future! Get started today: patreon.com/thisqueerbookCreditsHost/Founder: J.P. Der BoghossianExecutive Producer: Jim PoundsAssociate Producers: Archie Arnold, Natalie Cruz, Paul Kaefer, Nicole Olila, Joe Perazzo, Bill Shay, and Sean SmithPatreon Subscribers: Awen Briem, Stephen D., Thomas Michna, and Gary Nygaard.Ask for the Pride Special at the Spectacle Shoppe!Until August 1st, the Spectacle Shoppe is offering you $250 off as a Pride special. For locations, visit: https://spectacleshoppe.com Join Me In Supporting Lambda LiteraryAs a Lambda Literary Fellow, I hope you can donate to Lambda's Writers Retreat for Emerging LGBTQ Voices. They're raising $56k to ensure every fellow attend!You can donate to the scholarship fundraising campaign by visiting lambdaliterary.org/writers-retreat & clicking on SUPPORT EMERGING WRITERS or by texting LITVOICES to 44-321. Support the show

Join us again as we get to know more about Davis Vaughn, Southwest District President, and hear about his experiences within the Fraternity.

Our guest today is Nathan Eckstein (he/they) who is a writer, playwright, and graduate student at the University of Minnesota pursuing a Masters degrees in Architecture. They are also a candidate for another masters degree in Science Research Practices.Nathan shares with us how the memoir I Am Not Myself These Days by Josh Kilmer-Purcell saved his life. What's extra special about this episode is that it was recorded live in front of an audience at Lush Lounge and Theater in Northeast Minneapolis. This was a Drag Edition of our podcast with performances by the Haus of Taylor: Connie Taylor, Maiden Taiwan, and a special performance by Eduardo (Nathan!). I Am Not Myself These Days follows Josh's life, as well as his drag persona Aqua's life. By day, he works at a So-Ho advertising agency. By night, he performs drag throughout New York. His is a life of vodka and a new relationship with Jack, a BDSM sex worker. The memoir is a tragic-comedy-romance, charting Josh's, Aqua's, and Jack's life as they navigate love, substance abuse, and New York's 90s subculture. Donate to the ACLU's Drag Defense FundDuring our live event, we fundraised for the ACLU's Drag Defense Fund. Join us by donating here: https://action.aclu.org/give/support-drag-defense-fundBuy I Am Not Myself These DaysVisit our Bookshop or buy it here: https://bookshop.org/a/82376/9780060817329Connect with Nathan and the Haus of TaylorOn Instagram:Nathan Eckstein: @notstraightnateConnie Taylor: @connie_taylormnMaiden Taiwan: @maiden.taiwan Also, shout out to photographer Nick Lents who shot the live event! Follow him on Instagram: @nicklentsWatch Nathan's Play: Technically Lovehttps://youtu.be/ZTBW3irw-_IBecome an Associate Producer!Become an Associate Producer of our podcast through a $20/month sponsorship on Patreon! A professionally recognized credit, you can gain access to Associate Producer meetings to help guide our podcast into the future! Get started today: patreon.com/thisqueerbookCreditsHost/Founder: J.P. Der BoghossianExecutive Producer: Jim PoundsAssociate Producers: Archie Arnold, Natalie Cruz, Paul Kaefer, Nicole Olilla, Joe Perazzo, Bill Shay, and Sean SmithPatreon Subscribers: Awen Briem, Stephen D., Thomas Michna, and Gary Nygaard.Ask for the Pride Special at the Spectacle Shoppe!Until August 1st, the Spectacle Shoppe is offering you $250 off as a Pride special. For locations, visit: https://spectacleshoppe.com Join Me In Supporting Lambda LiteraryAs a Lambda Literary Fellow, I hope you can donate to Lambda's Writers Retreat for Emerging LGBTQ Voices. They're raising $56k to ensure every fellow attend!You can donate to the scholarship fundraising campaign by visiting lambdaliterary.org/writers-retreat & clicking on SUPPORT EMERGING WRITERS or by texting LITVOICES to 44-321. Support the show

Brooke Sargent, Software Engineer at Honeycomb, joins Corey on Screaming in the Cloud to discuss how she fell into the world of observability by adopting Honeycomb. Brooke explains how observability was new to her in her former role, but she quickly found it to enable faster learning and even a form of self care for herself as a developer. Corey and Brooke discuss the differences of working at a large company where observability is a new idea, versus an observability company like Honeycomb. Brooke also reveals the importance of helping people reach a personal understanding of what observability can do for them when trying to introduce it to a company for the first time. About BrookeBrooke Sargent is a Software Engineer at Honeycomb, working on APIs and integrations in the developer ecosystem. She previously worked on IoT devices at Procter and Gamble in both engineering and engineering management roles, which is where she discovered an interest in observability and the impact it can have on engineering teams.Links Referenced: Honeycomb: https://www.honeycomb.io/ Twitter: https://twitter.com/codegirlbrooke TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted guest episode—which is another way of saying sponsored episode—is brought to us by our friends at Honeycomb. And today's guest is new to me. Brooke Sargent is a software engineer at Honeycomb. Welcome to the show, Brooke.Brooke: Hey, Corey, thanks so much for having me.Corey: So, you were part of I guess I would call it the new wave of Honeycomb employees, which is no slight to you, but I remember when Honeycomb was just getting launched right around the same time that I was starting my own company and I still think of it as basically a six-person company versus, you know, a couple of new people floating around. Yeah, turns out, last I checked, you were, what, north of 100 employees and doing an awful lot of really interesting stuff.Brooke: Yeah, we regularly have, I think, upwards of 100 in our all-hands meeting, so definitely growing in size. I started about a year ago and at that point, we had multiple new people joining pretty much every week. So yeah, a lot of new people.Corey: What was it that drove you to Honeycomb? Before this, you spent a bit of time over Procter and Gamble. You were an engineering manager and now you're going—you went from IC to management and now you're IC again. There's a school of thought that I vehemently disagree with, that that's a demotion. I think they are orthogonal skill sets to my mind, but I'm curious to hear your journey through your story.Brooke: Yeah, absolutely. So yeah, I worked at Procter and Gamble, which is a big Cincinnati company. That's where I live and I was there for around four years. And I worked in both engineering and engineering management roles there. I enjoy both types of roles.What really drove me to Honeycomb is, my time at Procter and Gamble, I spent probably about a year-and-a-half, really diving into observability and setting up an observability practice on the team that I was on, which was working on connected devices, connected toothbrushes, that sort of thing. So, I set up an observability practice there and I just saw so much benefit to the engineering team culture and the way that junior and apprentice engineers on the team were able to learn from it, that it really caught my attention. And Honeycomb is what we were using and I kind of just wanted to spend all of my time working on observability-type of stuff.Corey: When you say software engineer, my mind immediately shortcuts to a somewhat outdated definition of what that term means. It usually means application developer, to my mind, whereas I come from the world of operations, historically sysadmins, which it still is except now, with better titles, you get more money. But that's functionally what SRE and DevOps and all the rest of the terms still currently are, which is, if it plugs into the wall, congratulations. It's your problem now to go ahead and fix that thing immediately. Were you on the application development side of the fence? Were you focusing on the SRE side of the world or something else entirely?Brooke: Yeah, so I was writing Go code in that role at P&G, but also doing what I call it, like, AWS pipe-connecting, so a little bit of both writing application code but also definitely thinking about the architecture aspects and lining those up appropriately using a lot of AWS serverless and managed services. At Honeycomb, I definitely find myself—I'm on the APIs and partnerships team—I find myself definitely writing a lot more code and focusing a lot more on code because we have a separate platform team that is focusing on the AWS aspects.Corey: One thing that I find interesting is that it is odd, in many cases, to see, first, a strong focus on observability coming from the software engineer side of the world. And again, this might be a legacy of where I was spending a lot of my career, but it always felt like getting the application developers to instrument whatever it was that they were building felt in many ways like it was pulling teeth. And in many further cases, it seemed that you didn't really understand the value of having that visibility or that perspective into what's going on in your environment, until immediately after. You really wished you had that perspective into what was going on in your environment, but didn't. It's similar to, no one is as zealous about backups as someone who's just suffered a data loss. Same operating theory. What was it that you came from the software engineering side to give a toss about the idea of observability?Brooke: Yeah, so working on the IoT—I was working on, like, the cloud side of things, so in Internet of Things, you're keeping a mobile application, firmware, and cloud synced up. So, I was focused on the cloud aspect of that triangle. And we got pretty close to launching this greenfield IoT cloud that we were working on for P&G, like, we were probably a few months from the initial go-live date, as they like to call it, and we didn't have any observability. We were just kind of sending things to CloudWatch logs. And it was pretty painful to figure out when something went wrong, from, like, you know, hearing from a peer on a mobile app team or the firmware team that they sent us some data and they're not seeing it reflected in the cloud that is, like, syncing it up.Figuring out where that went wrong, just using CloudWatch logs was pretty difficult and syncing up the requests that they were talking about to the specific CloudWatch log that had the information that we needed, if we had even logged the right thing. And I was getting a little worried about the fact that people were going to be going into stores and buying these toothbrushes and we might not have visibility into what could be going wrong or even being able to be proactive about what is going wrong. So, then I started researching observability. I had seen people talking about it as a best practice thing that you should think about when you're building a system, but I just hadn't had the experience with it yet. So, I experimented with Honeycomb a bit and ended up really liking their approach to observability. It fit my mental model and made a lot of sense. And so, I went full-steam ahead with implementing it.Corey: I feel what you just said is very key: the idea of finding an observability solution that keys into the mental model that someone's operating with. I found that a lot of observability talk sailed right past me because it did not align with that, until someone said, “Oh yeah, and then here's events.” “Well, what do you mean by event?” It distills down to logs. And oh, if you start viewing everything as a log event, then yeah, that suddenly makes a lot of sense, and that made it click for me in a way that, honestly, is a little embarrassing that it didn't before then.But I come from a world before containers and immutable infrastructure and certainly before the black boxes that are managed serverless products, where I'm used to, oh, something's not working on this Linux box. Well, I have root, so let's go ahead and fix that and see what's going on. A lot of those tools don't work, either at scale or in ephemeral environments or in scenarios where you just don't have the access to do the environment. So, there's this idea that if you're trying to diagnose something that happened and the container that it happened on stopped existing 20 minutes ago, your telemetry game has got to be on point or you're just guessing at that point. That is something that I think I did myself a bit of a disservice by getting out of hands-on keyboard operations roles before that type of architecture really became widespread.Brooke: Yeah, that makes a lot of sense. On the team that I was on, we were using a lot of AWS Lambda and similarly, tracking things down could be a little bit challenging. And emitting telemetry data also has some quirks [laugh] with Lambda.Corey: There certainly is. It's also one of those areas that, on some level, being stubborn to adopt it works to your benefit. Because when Lambda first came out, it was a platform that was almost entirely identified by its constraints. And Amazon didn't do a terrific job, at least in the way that I tend to learn, of articulating what those constraints are. So, you learn by experimenting and smacking face first into a lot of those things.What the hell do you mean you can't write to the file? Oh, it's a read-only file system. [except slash tap 00:08:39]. What do you mean, it's only half a gigabyte? Oh, that's the constraint there. Well, what do you mean, it automatically stops after—I think back in that point it was five or ten minutes; it's 15 these days. But—Brooke: Right.Corey: —I guess it's their own creative approach to solving the halting problem from computer science classes, where after 15 minutes, your code will stop executing, whether you want it to or not. They're effectively evolving these things as we go and once you break your understanding in a few key ways, at least from where I was coming from, it made a lot more sense. But ugh, that was a rough couple of weeks for me.Brooke: Yeah [laugh]. Agreed.Corey: So, a topic that you have found personally inspiring is that observability empowers junior engineers in a bunch of ways. And I do want to get into that, but beforehand, I am curious as to the modern-day path for SREs because it doesn't feel to me like there is a good answer for, “What does a junior SRE look like?” Because the answer is, “Oh, they don't.” It goes back to the old sysadmin school of thought, which is that, oh, you basically learn by having experience. I've lost count a number of startups I've encountered where you have a bunch of early-20-something engineers but the SRE folks are all generally a decade into what they're what they've been doing because the number-one thing you want to hear from someone in that role is, “Oh, the last time I saw it, here's what it was.” What is the observability story these days for junior engineers?Brooke: So, with SRE I agr—like, that's a conversation that I've had a lot of times on different teams that I've been on, is just can a junior SRE exist? And I think that they can.Corey: I mean, they have to because otherwise, it's well, where does it SRE come from? Oh, they spring—Brooke: [laugh].Corey: —fully formed from the forehead of some God out of mythology. It doesn't usually work that way.Brooke: Right. But you definitely need a team that is ready to support a junior SRE. You need a robust team that is interested in teaching and mentoring. And not all teams are like that, so making sure that you have a team culture that is receptive to taking on a junior SRE is step number one. And then I think that the act of having an observability practice on a team is very empowering to somebody who is new to the industry.Myself, I came from a self-taught background, learning to code. I actually have a music degree; I didn't go to school for computer science. And when I finally found my way to observability, it made so many, kind of, light bulbs go off of just giving me more visuals to go from, “I think this is happening,” to, “I know this is happening.” And then when I started mentoring juniors and apprentices and putting that same observability data in front of them, I noticed them learning so much faster.Corey: I am curious in that you went from implementing a lot of these things and being in a management role of mentoring folks on observability concepts to working for an observability vendor, which is… I guess I would call Honeycomb the observability vendor. They were the first to really reframe a lot of how we considered what used to be called monitoring and now it's called observability, or as I think of it, hipster monitoring.Brooke: [laugh].Corey: But I am curious as to when you look at this, my business partner wrote a book for O'Reilly, Practical Monitoring, and he loved it so much that by the end of that book, he got out of the observability monitoring space entirely and came to work on AWS bills with me. Did you find that going to Honeycomb has changed your perspective on observability drastically?Brooke: I had definitely consumed a lot of Honeycomb's blog posts, like, that's one of the things that I had loved about the company is they put out a lot of interesting stuff, not just about observability but about operating healthy teams, and like you mentioned, like, a pendulum between engineering management and being an IC and just interesting concepts within our industry overall as, like, software engineers and SREs. So, I knew a lot of the thought leadership that the company put out, and that was very helpful. It was a big change going from an enterprise like Procter and Gamble to a startup observability company like Honeycomb, just—and also, going from a company that very much believes in in-person work to remote-first work at Honeycomb, now. So, there were a lot of, like, cultural changes, but I think I kind of knew what I was getting myself into as far as the perspective that the company takes on observability.Corey: That is always the big, somewhat awkward question because of the answer goes a certain way, it becomes a real embarrassment, but I'm confident enough, having worked with Honeycomb as a recurring sponsor and having helped out on the AWS bill side of the world since you were a reference client on both sides of that business, I want to be very clear that I don't think I'm throwing you under a bus on this one. But do you find that the reality, now that you've been there for a year, has matched the external advertising and the ethos of the story they tell about Honeycomb from the outside?Brooke: I definitely think it matches up. One thing that is just different about working inside of a company like Honeycomb versus working at a company that doesn't have any observability at all yet, is that there are a lot of abstraction layers in our codebase and things like that. So, me being a software engineer and writing code Honeycomb compared to P&G, I don't have to think about observability as much because everybody in the company is thinking about observability and had thought about it before I joined and had put in a lot of thought to how to make sure that we consistently have telemetry data that we need to solve problems versus I was thinking about this stuff on the daily at P&G.Corey: Something I've heard from former employees of a bunch of different observability companies has a recurring theme to it, and that it's hard to leave. Because when you're at an observability company, everything is built with an eye toward observability. And there's always the dogfooding story of, we instrument absolutely everything we have with everything that we sell the customers. Now, in practice, you leave and go to a different company, that is almost never going to be true, if for no other reason than based on simple economics. Turning on every facet of every observability tool that a given company sells becomes extraordinarily expensive and is an investment decision, so companies say yes to some, no to others. Do you think you're going to have that problem if and when you decide it's time to move on to your next role, assuming of course, that it's not at a competing observability company?Brooke: I'm sure there will be some challenges if I decide to move on from working for observability platforms in the future. The one that I think would be the most challenging is joining a team where people just don't understand the value of observability and don't want to invest, like, the time and effort into actually instrumenting their code, and don't see why they need to do it, versus just, like, they haven't gotten there yet or they haven't had enough people hired to do it just yet. But if people are actively, like, kind of against the idea of instrumenting your code, I think that would be really challenging to kind of shift to especially after, over the last two-and-a-half years or so, being so used to having this, like, extra sense when I'm debugging problems and dealing with outages.Corey: I will say, it was a little surreal the first time I wound up taking a look at Honeycomb's environment—because I do believe that cost and architecture are fundamentally the same thing when it comes to cloud—and you had clear lines of visibility into what was going on in your AWS bill by way of Honeycomb as a product. And that's awesome. I haven't seen anyone else do that yet and I don't know that it would necessarily work as well because, as you said, there, everyone's thinking about it through this same shared vision, whereas in a number of other companies, it flat out does not work that way. There are certain unknowns and questions. And from the outside, and when you first start down this path, it feels like a ridiculous thing to do, until you get to a point of seeing the payoff, and yeah, this makes an awful lot of sense.I don't know that it would, for example, work as a generic solution for us to roll out to our various clients and say, oh, let's instrument your environment with this and see what's going on because first, we don't have that level of ability to make change in customer environments. We are read-only for some very good reasons. And further, it also seems like it's a, “Step one: change your entire philosophy around these sorts of things so we can help you spend less on AWS,” seems like a bit of a tall order.Brooke: Yeah, agreed. And yeah, on previous teams that I've been on, I definitely—and I think it's fair, absolutely fair, that there were things where, especially using AWS serverless services, I was trying to get as much insight as possible into adding some of these services to our traces, like, AppSync was one example where I could not for the life of me figure out how to get AppSync API requests onto my Honeycomb trace. And I spent a lot of time trying to figure it out. And I had team members that would just be, like, you know, “Let's timebox this; let's not, like, sink all of our time into it.” And so, I think as observability evolves, hopefully, carving out those patterns continues to get easier so that engineers don't have to spend all of their time, kind of, carving out those patterns.Corey: It feels like that's the hard part, is the shift in perspective. Instrumenting a given tool into an environment is not the heavy lift compared to appreciating the value of it. Do you find that that was an easy thing for you to overcome, back when you were at Procter and Gamble, as far as people already have bought in, on some level, to observability from having seen it in some kind of scenarios where it absolutely save folks' bacon? Or was it the problem of, first you have to educate people about the painful problem that they have before they realize it is in fact, A, painful, and B, a problem, and then C, that you have something to sell them that will solve that? Because that pattern is a very hard sales motion to execute in most spaces. But you were doing it at it, from the customer side first.Brooke: Yeah. Yeah, doing it from the customer side, I was able to get buy-in on the team that I was on, and I should also say, like, the team that I was on was considered an innovation team. We were in a separate building from, like, the corporate building and things like that, which I'm sure played into some of those cultural aspects and dynamics. But trying to educate people outside of our team and trying to build an observability practice within this big enterprise company was definitely very challenging, and it was a lot of spending time sharing information and talking to people about their stack and what languages and tools that they're using and how this could help them. I think until people have had that, kind of, magical moment of using observability data to solve a problem for themselves, it's very hard, it can be very hard to really make them understand the value.Corey: That was is always my approach because it feels like observability is a significant and sizable investment in infrastructure alone, let alone mental overhead, the teams to manage these things, et cetera, et cetera. And until you have a challenge that observability can solve, it feels like it is pure cost, similar to backups, where it's just a whole bunch of expense for no benefit until suddenly, one day, you're very glad you had it. Now, the world is littered with stories that are very clear about what happens when you don't have backups. Most people have a personal story around that, but it feels like it's less straightforward to point at a visceral story where not having observability really hobbled someone or something.It feels like—because in the benefit of perfect hindsight, oh yeah, like a disk filled up and we didn't know about that. Like, “Ah, if we just had the right check, we would have caught that early on.” Yeah, coulda, woulda shoulda, but it was a cascading failure that wasn't picked up until seven levels downstream. Do you think that that's the situation these days or am I misunderstanding how people are starting to conceive about this stuff?Brooke: Yeah. I mean, I definitely have a couple of stories of even once I was on the journey to observability adoption—which I call it a journey because you don't just—kind of—snap your fingers and have observability—I started with one service, instrumenting that and just, like, gradually, over sprint's would instrument more services and pull more team members in to do that as well. But when we were in that process of instrumenting services, there was one service which was our auth service—which maybe should have been the first one that we instrumented—that a code change was made and it was erroring every time somebody tried to sign up in the app. And if we had observability instrumentation in place for that service, it wouldn't have taken us, like, the four or five hours to find the problem of the one line of code that we had changed; we would have been able to see more clearly what error was happening and what line of code it was happening on and probably fix it within an hour.And we had a similar issue with a Redshift database that we were running more on the metrics side of things. We were using it to send analytics data to other people in the company and that Redshift database just got maxed out at a certain point. The CPU utilization was at, like, 98% and people in the company were very upset and [laugh] having a very bad time querying their analytics data.Corey: It's a terrific sales pitch for Snowflake, to be very direct, because you hear that story kind of a lot.Brooke: Yeah, it was not a fun time. But at that point, we started sending Redshift metrics data over to Honeycomb as well, so that we could keep a better pulse on what exactly was happening with that database.Corey: So, here's sort of the acid test: people tend to build software when they're starting out greenfield, in ways that emphasize their perspective on the world. For example, when I'm building something new, doesn't matter if it's tiny or for just a one-off shitposting approach, and it touches anything involving AWS, first thing I do out of the gate is I wind up setting tags so that I can do cost allocation work on it; someday, I'm going to wonder how much this thing cost. That is, I guess my own level of brokenness.Brooke: [laugh].Corey: When you start building something at work from scratch, I guess this is part ‘you,' part ‘all of Honeycomb,' do you begin from that greenfield approach of Hello World of instrumenting it for observability, even if it's not explicitly an observability-focused workload? Or is it something that you wind up retrofitting with observability insights later, once it hits a certain point of viability?Brooke: Yeah. So, if I'm at the stage of just kind of trying things out locally on my laptop, kind of outside of, like, the central repo for the company, I might not do observability data because I'm just kind of learning and trying things out on my laptop. Once I pull it into our central repo, there is some observability data that I am going to get, just in the way that we kind of have our services set up. And as I'm going through writing code to do this whatever new feature I'm trying to do, I'm thinking about what things, when this breaks—not if it breaks; when it breaks [laugh]—am I going to want to know about in the future. And I'll add those things, kind of, on the fly just to make things easier on myself, and that's just kind of how my brain works at this point of thinking about my future self, which is, kind of like, the same definition of self-care. So, I think of observability as self-care for developers.But later on, when we're closer to actually launching a thing, I might take another pass at just, like, okay, let's once again take a look at the error paths and how this thing can break and make sure that we have enough information at those points of error to know what is happening within a trace view of this request.Corey: My two programming languages that I rely on the most are enthusiasm and brute force, and I understand this is not a traditional software engineering approach. But I've always found that having to get observability in involved a retrofit, on some level. And it always was frustrating to me just because it felt like it was so much effort in various ways that I've just always kicked myself: I should have done this early on. But I've been on the other side of that, and it's like, should I instrument this with good observability? No, that sounds like work. I want to see if this thing actually works at all, or not first.And I don't know what side of the fence is the correct one to be on, but I always find that I'm on the wrong one. Like, I don't know if it's, like, one of those, there's two approaches and neither one works. I do see in client environments where observability is always, always, always something that has to be retrofit into what it is that they're doing. Does it stay that way once companies get past a certain point? Does observability of adoption among teams just become something that is ingrained into them or do people have to consistently relearn that same lesson, in your experience?Brooke: I think it depends, kind of, on the size of your company. If you are a small company with a, you know, smaller engineering organization where it's not, I won't say easy, but easier to get kind of full team buy-in on points of view and decisions and things like that, it becomes more built-in. If you're in a really big company like the one that I came from, I think it is continuously, like, educating people and trying to show the value of, like, why we are doing this—coming back to that why—and like, the magical moment of, like, stories of problems that have been solved because of the instrumentation that was in place. So, I guess, like most things, it's an, ‘it depends.' But the larger that your company becomes, I think the harder it gets to keep everybody on the same page.Corey: I am curious, in that I tend to see the world through AWS bills, which is a sad, pathetic way to live that I don't recommend to basically anyone, but I do see the industry, or at least my client base, forming a bit of a bimodal distribution. On one side, you have companies like Honeycomb, including, you know, Honeycomb, where the majority of your AWS spend is driven by the application that is Honeycomb, you know, the SaaS thing you sell to people to solve their problems. The other side of the world are companies that look a lot more like Procter and Gamble, presumably, where—because I think of oh, what does Procter and Gamble do? And the answer is, a lot. They're basically the definition of conglomerate in some ways.So, you look at that, a bill at a big company like that and it might be hundreds of millions of dollars, but the largest individual workload is going to be a couple million at best. So, it feels very much like it's this incredibly diffuse selection of applications. And in those environments, you have to start thinking a lot more about centralization things you can do, for example, for savings plan purchases and whatnot, whereas at Honeycomb-like companies, you can start looking at, oh, well, you have this single application that's the lion's share of everything. We can go very deep into architecture and start looking at micro-optimizations here that will have a larger impact. Having been an engineer at both types of companies, do you find that there's a different internal philosophy, or is it that when you're working in a larger company on a specific project, that specific project becomes your entire professional universe?Brooke: Yeah, definitely at P&G, for the most part, IoT was kind of the center of my universe. But one philosophy that I noticed as being different—and I think this is from being an enterprise in a startup—is just the way that thinking about cost and architecture choices, kind of, happened. So, at P&G, like I said, we were using a lot of Lambda, and pretty much any chance we got, we used a serverless or managed offering from AWS. And I think a big part of that reasoning was because, like I said earlier, P&G is very interested in in-person work. So, everybody that we hired her to be located in Cincinnati.And it became hard to hire for people who had Go and Terraform experience because a lot of people in the Midwest are much more comfortable in .NET and Java; there's just a lot more jobs using those technologies. So, we had a lot of trouble hiring and would choose—because P&G had a lot of money to spend—to give AWS that money because we had trouble finding engineers to hire, whereas Honeycomb really does not seem to have trouble hiring engineers. They hire remote employees and lots of people are interested in working at Honeycomb and they also do not have the bank account [laugh] that Procter and Gamble has, so just thinking about cost and architecture is kind of a different beast. So, at Honeycomb, we are building a lot more services versus just always choosing a serverless or easy, like, AWS managed option to think about it less.Corey: Yeah, at some level, it's an unfair question, just because it comes down to, in the fullness of time, even Honeycomb turns into something that looks a lot more like Procter and Gamble. Because, okay, you have the Honeycomb application. That's great, but as the company continues to grow and offer different things to different styles of customers, you start seeing a diffusion where, yeah, everything stills observability focused, but I can see a future where it becomes a bunch of different subcomponents. You make acquisitions of other companies that wind up being treated as separate environments and the rest. And in the fullness of time, I can absolutely see that that is the path that a lot of companies go down.So, it might also just be that I'm looking at this through a perspective lens of… just company stage, as opposed to what the internal story of the company is. I mean, Procter and Gamble's, what, a century old give or take? Whereas Honeycomb is an ancient tech company, by which I mean it's over 18 months old.Brooke: Yeah, P&G was founded in 1837. So that's—Corey: Almost 200 years old. Wonderful.Brooke: —quite old [laugh]. Yeah [laugh].Corey: And for some reason, they did not choose to build their first technical backbone on top of AWS back then. I don't understand why, for the life of me.Brooke: [laugh]. Yeah, but totally agree on your point that the kind of difference of thinking about cost and architecture definitely comes from company's stage rather than necessarily the industry.Corey: I really want to thank you for taking the time out of your day to talk with me about what you're up to and how you view these things. If people want to learn more, what's the best place for them to find you?Brooke: Yeah, so I think the main place that I still sometimes am, is Twitter: @codegirlbrooke is my username. But I'm only there sometimes, now [laugh].Corey: I feel like that's a problem a lot of us are facing right now. Like, I'm more active on Bluesky these days, but it's still invite only and it feels like it's too much of a weird flex to wind up moving people to just yet. I'm hoping that changes soon, but we'll see how it plays. We'll, of course, put links to that in the [show notes 00:31:53]. I really want to thank you for taking the time out of your day to talk with me.Brooke: Yeah, thanks so much for chatting with me. It was a good time.If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Meet four-time HQ MIX Trophy winner for best Webcomic, Mário César and his new book Blessed Cure. Like Alison Bechdel's Fun Home, César's Blessed Cure uses comics to explore queer life. In this book, our hero, Acacio do Nascimento, knew he was a different boy from others. He would rather be playing with dolls than playing soccer. The hula hoop interested him more than cowboy pistols. Scared by the possibility of their son being a homosexual, Acacio's parents begin conversion therapy, starting when he was five years old, to make him a "normal boy" like the others.Mario Cesar is an awarded comic book's author from Brazil. He has published comics since 2006 and he was one of the first openly gay authors to address issues of sexual and gender diversity in Brazilian comics with the book Ciranda da Solidão (2013). He is also one of the creators and producers of POC CON, the first brazilian LGBTQ+ Comic Con.Buy Blessed CureHead to our Bookshop store or buy directly: https://bookshop.org/a/82376/9781908030542Connect with Mário CésarTwitter: @mas_que_marioInstagram: @mas_que_marioBecome an Associate Producer!Become an Associate Producer of our podcast through a $20/month sponsorship on Patreon! A professionally recognized credit, you can gain access to Associate Producer meetings to help guide our podcast into the future! Get started today: patreon.com/thisqueerbookCreditsHost/Founder: J.P. Der BoghossianExecutive Producer: Jim PoundsAssociate Producers: Archie Arnold, Natalie Cruz, Paul Kaefer, Nicole Olila, Joe Perazzo, Bill Shay, and Sean SmithPatreon Subscribers: Awen Briem, Stephen D., Thomas Michna, and Gary Nygaard. Ask for the Pride Special at the Spectacle Shoppe!Until August 1st, the Spectacle Shoppe is offering you $250 off as a Pride special. For locations, visit: https://spectacleshoppe.com Join Me In Supporting Lambda LiteraryAs a Lambda Literary Fellow, I hope you can donate to Lambda's Writers Retreat for Emerging LGBTQ Voices. They're raising $56k to ensure every fellow attend!You can donate to the scholarship fundraising campaign by visiting lambdaliterary.org/writers-retreat & clicking on SUPPORT EMERGING WRITERS or by texting LITVOICES to 44-321. Support the show

Mike Brevoort, Chief Product Officer at Gitpod, joins Corey on Screaming in the Cloud to discuss all the intricacies of remote development and how Gitpod is simplifying the process. Mike explains why he feels the infinite resources cloud provides can be overlooked when discussing remote versus local development environments, and how simplifying build abstractions is a fantastic goal, but that focusing on the tools you use in a build abstraction in the meantime can be valuable. Corey and Mike also dive into the security concerns that come with remote development, and Mike reveals the upcoming plans for Gitpod's local conference environment, CDE Universe. About MikeMike has a passion for empowering people to be creative and work together more effectively. He is the Chief Product Officer at Gitpod striving to remove the friction and drudgery from software development through Cloud Developer Environments. He spent the previous four years at Slack where he created Workflow Builder and “Platform 2.0” after his company Missions was acquired by Slack in 2018. Mike lives in Denver, Colorado and enjoys cycling, hiking and being outdoors.Links Referenced: Gitpod: https://www.gitpod.io/ CDE Universe: https://cdeuniverse.com/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It's easy to **BEEP** up on AWS. Especially when you're managing your cloud environment on your own!Mission Cloud un **BEEP**s your apps and servers. Whatever you need in AWS, we can do it. Head to missioncloud.com for the AWS expertise you need. Corey: Have you listened to the new season of Traceroute yet? Traceroute is a tech podcast that peels back the layers of the stack to tell the real, human stories about how the inner workings of our digital world affect our lives in ways you may have never thought of before. Listen and follow Traceroute on your favorite platform, or learn more about Traceroute at origins.dev. My thanks to them for sponsoring this ridiculous podcast. Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. I have had loud, angry, and admittedly at times uninformed opinions about so many things over the past few years, but something that predates that a lot is my impression on the idea of using remote systems for development work as opposed to doing local dev, and that extends to build and the rest. And my guest today here to argue with me about some of it—or agree; we'll find out—is Mike Brevoort, Chief Product Officer at Gitpod, which I will henceforth be mispronouncing as JIT-pod because that is the type of jerk I am. Mike, thank you for joining me.Mike: Thank you for insulting my company. I appreciate it.Corey: No, by all means, it's what we do here.Mike: [laugh].Corey: So, you clearly have opinions on the idea of remote versus local development that—I am using the word remote development; I know you folks like to use the word cloud, in place of remote, but I'm curious to figure out is, is that just the zeitgeist that has shifted? Do you have a belief that it should be in particular places, done in certain ways, et cetera? Where do your opinion on this start and stop?Mike: I think that—I mean, remote is accurate, an accurate description. I don't like to emphasize the word remote because I don't think it's important that it's remote or local. I think that the term cloud connotes different values around the elasticity of environments and the resources that are more than what you might have on your local machine versus a remote machine. It's not so much whether the one machine is local or remote as much of it is that there are infinite numbers of resources that you can develop across in the cloud. That's why we tend to prefer our cloud development environments.Corey: From my perspective, I've been spending too many years now living in basically hotels and airports. And when I was doing that, for a long time, the only computer I bring with me has been my iPad Pro. That used to be a little bit on the challenging side and these days, that's gotten capable enough where it's no longer interesting in isolation. But there's no local development environment that is worth basically anything on that. So, I've been SSHing into things and using VI as my development environment for many years.When I started off as a grumpy Unix sysadmin, there was something reassuring about the latest state of whatever it is I'm working on lives in a data center somewhere rather than on a laptop, I'm about to leave behind a coffee shop because I'm careless. So, there's a definite value and sense that I am doing something virtuous, historically. But it didn't occur to me till I started talking to people about this, just how contentious the idea was. People would love to ask all kinds of fun objections to this where it was, “Oh, well, what about when you're on a plane and need to do work?” It's, well, I spend an awful lot of time on planes and that is not a limiting factor in me writing the terrible nonsense that I will charitably called code, in my case. I just don't find that that idea holds up anywhere. The world has become so increasingly interconnected that that seems unlikely. But I do live in San Francisco, so here, every internet is generally pretty decent; not every place is. What are your thoughts?Mike: I agree. I mean, I think one thing is, I would just like not to think about it, whether I can or can't develop because I'm connected or not. And I think that we tend to be in a world where that is moreso the case. And I think a lot of times when you're not connected, you become reconnected soon, like if your connection is not reliable or if you're going in and out of connectivity issues. And when you're trying to work on a local laptop and you're connecting and disconnecting, it's not like we develop these days, and everything is just isolated on our local laptop, especially we talk about cloud a lot on this podcast and a lot of apps now go way beyond just I'm running a process on my machine and I'm connecting to data on my machine.There are local emulators you could use for some of these services, but most of them are inferior. And if you're using SQS or using any other, like, cloud-based service, you're usually, as a developer, connecting to some version of that and if you're disconnected anyway, you're not productive either. And so, I find that it's just like an irrelevant conversation in this new world. And that the way we've developed traditionally has not followed along with this view of I need to pile everything in on my laptop, to be able to develop and be productive has not, like, followed along with the trend that moved into the cloud.Corey: Right. The big problem for a long time has been, how do I make this Mac or Windows laptop look a lot like Linux EC2 instance? And there have been a bunch of challenges and incompatibility issues and the rest, and from my perspective, I like to develop in an environment that at least vaguely resembles the production environment it's going to run in, which in AWS's case, of course, comes down to expensive. Bu-dum-tss.Mike: Yeah, it's a really big challenge. It's been a challenge, right? When you've worked with coworkers that were on a Windows machine and you were on a Mac machine, and you had the one person on their Linux machine forever, and we all struggled with trying to mimic these development environments that were representative, ultimately, of what we would run in production. And if you're counting costs, we can count the cost of those cloud resources, we can count the cost of those laptops, but we also need to count the cost of the people who are using those laptops and how inefficient and how much churn they have, and how… I don't know, there was for years of my career, someone would show up every morning to the stand-up meeting and say, it's like, “Well, I wasted all afternoon yesterday trying to work out my, you know, issues with my development environment.” And it's, like, “I hope I get that sorted out later today and I hope someone can help me.”And so, I think cost is one thing. I think that there's a lot of inconsistencies that lead to a lot of inefficiencies and churn. And I think that, regardless of where you're developing, the more that you can make your environments more consistent and sound, not for you, but for your own team and have those be more representative of what you are running in production, the better.Corey: We should disambiguate here because I fear this is one of the areas where my use case tends to veer off into the trees, which is I tend to operate largely in isolation, from a development point of view. I build small, micro things that wind up doing one thing, poorly. And that is, like, what I do is a proof of concept, or to be funny, or to kick the tires on a new technology. I'll also run a bunch of random things I find off of JIF-ub—yes, that's how I pronounce GitHub. And that's great, but it also feels like I'm learning as a result, every stack, and every language, in every various version that it has, and very few of the cloud development environments that I've seen, really seems to cater to the idea that simultaneously, I want to have certain affordances in my shell environment set up the way that I want them, tab complete this particular suite of tools generically across the board, but then reset to that baseline and go in a bunch of different directions of, today, it's Python in this version and tomorrow, it's Node in this other version, and three, what is a Typescript anyway, and so on and so forth.It feels like it's either, in most cases, you either get this generic, one-size-fits-everyone in this company, for this project, approach, or it's, here's a very baseline untuned thing that does not have any of your dependencies installed. Start from scratch every time. And it's like, feels like there are two paths, and they both suck. Where are you folks at these days on that spectrum?Mike: Yeah, I think that, you know, one, if you do all of that development across all these different libraries and technology stacks and you're downloading all these repos from JIF-hub—I say it right—and you're experimenting, you tend to have a lot of just collision of things. Like if you're using Python, it's, like, really a pain to maintain isolation across projects and not have—like, your environment is, like, one big bucket of things on your laptop and it's very easy to get that into a state where things aren't working, and then you're struggling. There's no big reset on your laptop. I mean, there is but it takes—it's a full reset of everything that you have.And I think the thing that's interesting to me about cloud development environments is I could spin one of these up, I could trash it to all hell and just throw it away and get another one. And I could get another one of those at a base of which has been tuned for whatever project or technology I'm working on. So, I could take—you know, do the effort to pre-setup environments, one that is set up with all of my, like, Python tooling, and another one that's set up with all my, like, Go or Rust tooling, or our front-end development, even as a base repo for what I tend to do or might tend to experiment with. What we find is that, whether you're working alone or you're working with coworkers, that setting up a project and all the resources and the modules and the libraries and the dependencies that you have, like, someone has to do that work to wire that up together and the fact that you could just get an environment and get another one and another one, we use this analogy of, like, tissue boxes where, like, you should just be able to pull a new dev environment out of a tissue box and use it and throw it away and pull as many tissues out of the box as you want. And they should be, like, cheap and ephemeral because—and they shouldn't be long-lived because they shouldn't be able to drift.And whether you're working alone or you're working in a team, it's the same value. The fact that, like, I could pull on these out, I have it. I'm confident in it of what I got. Like for example, ideally, you would just start a dev environment, it's available instantly, and you're ready to code. You're in this project with—and maybe it's a project you've never developed on. Maybe it's an open-source project.This is where I think it really improves the sort of equitability of being able to develop, whether it's in open-source, whether it's inner-source in companies, being able to approach any project with a click of a button and get the same environment that the tech lead on the project who started it five years ago has, and then I don't need to worry about that and I get the same environment. And I think that's the value. And so, whether you're individual or you're on a team, you want to be able to experiment and thrash and do things and be able to throw it away and start over again, and not have to—like for example, maybe you're doing that on your machine and you're working on this thing and then you actually have to do some real work, and then now that you've done something that conflicts with the thing that you're working on and you're just kind of caught in this tangled mess, where it's like, you should just be able to leave that experiment there and just go work on the thing you need to work on. And why can't you have multiples of these things at any given time?Corey: Right. One of the things I loved about EC2 dev environments has been that I can just spin stuff up and okay, great, it's time for a new project. Spin up another one and turn it off when I'm done using it—which is the lie we always tell ourselves in cloud and get charged for things we forget to turn off. But then, okay, I need an Intel box one day. Done. Great, awesome. I don't have any of those lying around here anymore but clickety, clickety, and now I do.It's nice being able to have that flexibility, but it's also sometimes disconcerting when I'm trying to figure out what machine I was on when I was building things and the rest, and having unified stories around this becomes super helpful. I'm also finding that my overpowered desktop is far more cost-efficient when I need to compile something challenging, as opposed to finding a big, beefy, EC2 box for that thing as well. So, much of the time, what my remote system is doing is sitting there bored. Even when I'm developing on it, it doesn't take a lot of modern computer resources to basically handle a text editor. Unless it's Emacs, in which case, that's neither here nor there.Mike: [laugh]. I think that the thing that becomes costly, especially when using cloud development environments, is when you have to continue to run them even when you're not using them for the sake of convenience because you're not done with it, you're in the middle of doing some work and it still has to run or you forget to shut it off. If you are going to just spin up a really beefy EC2 instance for an hour to do that big compile and it costs you 78 cents. That's one thing. I mean, I guess that adds up over time and yes, if you've already bought that Mac Studio that's sitting under your desk, humming, it's going to be more cost-efficient to use that thing.But there's, like, an element of convenience here that, like, what if I haven't bought the Mac Studio, but I still need to do that big beefy compilation? And maybe it's not on a project I work on every single day; maybe it's the one that I'm just trying to help out with or just starting to contribute to. And so, I think that we need to get better about, and something that we're very focused on at JIT-pod, is—Gitpod—is—Corey: [laugh]. I'm going to get you in trouble at this rate.Mike: —[laugh]—is really to optimize that underlying runtime environment so that we can optimize the resources that you're using only when you're using it, but also provide a great user experience. Which is, for me, as someone who's responsible for the product at Gitpod, the thing I want to get to is that you never have to think about a machine. You're not thinking about this dev environment as something that lives somewhere, that you're paying for, that there's a meter spinning that if you forget it, that you're like, ah, it's going to cost me a lot of money, that I have to worry about ever losing it. And really, I just want to be able to get a new environment, have one, use it, come back to it when I need it, have it not cost me a lot of money, and be able to have five or ten of those at a time because I'm not as worried about what it's going to cost me. And I'm sure it'll cost something, but the convenience factor of being able to get one instantly and have it and not have to worry about it ultimately saves me a lot of time and aggravation and improves my ability to focus and get work done.And right now, we're still in this mode where we're still thinking about, is it on my laptop? Is it remote? Is it on this EC2 instance or that EC2 instance? Or is this thing started or stopped? And I think we need to move beyond that and be able to just think of these things as development environments that I use and need and they're there when I want to, when I need to work on them, and I don't have to tend to them like cattle.Corey: Speaking of tending large things in herds—I guess that's sort of for the most tortured analogy slash segway I've come up with recently—you folks have a conference coming up soon in San Francisco. What's the deal with that? And I'll point out, it's all on-site, locally, not in the cloud. So, hmm…Mike: Yeah, so we have a local conference environment, a local conference that we're hosting in San Francisco called CDE Universe on June 1st and 2nd, and we are assembling all the thought leaders in the industry who want to get together and talk about where not just cloud development is going, but really where development is going. And so, there's us, there's a lot of companies that have done this themselves. Like, before I joined Gitpod, I was at Slack for four years and I got to see the transition of a, sort of, remote development hosted on EC2 instances transition and how that really empowered our team of hundreds of engineers to be able to contribute and like work together better, more efficiently, to run this giant app that you can't run just alone on your laptop. And so, Slack is going to be there, they're going to be talking about their transition to cloud development. The Uber team is going to be there, there's going to be some other companies.So, Nathan who's building Zed, he was the one that originally built Adam at GitHub is now building Zed, which is a new IDE, is going to be there. And I can't mention all the speakers, but there's going to be a lot of people that are really looking at how do we drive forward development and development environments. And that experience can get a lot better. So, if you're interested in that, if you're going to be in San Francisco on June 1st and 2nd and want to talk to these people, learn from them, and help us drive this vision forward for just a better development experience, come hang out with us.Corey: I'm a big fan of collaborating with folks and figuring out what tricks and tips they've picked up along the way. And this is coming from the perspective of someone who acts as a solo developer in many cases. But it always drove me a little nuts when you see people spending weeks of their lives configuring their text editor—VIM in my case because I'm no better than these people; I am one of them—and getting it all setup and dialed in. It's, how much productivity you gaining versus how much time are you spending getting there?And then when all was said and done a few years ago, I found myself switching to VS Code for most of what I do, and—because it's great—and suddenly the world's shifting on its axis again. At some point, you want to get away from focusing on productivity on an individualized basis. Now, the rules change when you're talking about large teams where everyone needs a copy of this running locally or in their dev environment, wherever happens to be, and you're right, often the first two weeks of a new software engineering job are, you're now responsible for updating the onboarding docs because it's been ten minutes since the last time someone went through it. And oh, the versions bumped again of what we would have [unintelligible 00:16:44] brew install on a Mac and suddenly everything's broken. Yay. I don't miss those days.Mike: Yeah, the new, like, ARM-based Macs came out and then you were—now all of a sudden, all your builds are broken. We hear that a lot.Corey: Oh, what I love now is that, in many cases, I'm still in a process of, okay, I'm developing locally on an ARM-based Mac and I'm deploying it to a Graviton2-based Lambda or instance, but the CI/CD builder is going to run on Intel, so it's one of those, what is going on here? Like, there's a toolchain lag of round embracing ARM as an architecture. That's mostly been taken care of as things have evolved, but it's gotten pretty amusing at some point, just as quickly that baseline architecture has shifted for some workloads. And for some companies.Mike: Yeah, and things just seem to be getting more [laugh] and more complicated not less complicated, and so I think the more that we can—Corey: Oh, you noticed?Mike: Try to simplify build abstractions [laugh], you know, the better. But I think in those cases where, I think it's actually good for people to struggle with setting up their environment sometime, with caring about the tools that they use and their experience developing. I think there has to be some ROI with that. If it's like a chronic thing that you have to continue to try to fix and make better, it's one thing, but if you spend a whole day improving the tools that you use to make you a better developer later, I think there's a ton of value in that. I think we should care a lot about the tools we use.However, that's not something we want to do every day. I mean, ultimately, I know I don't build software for the sake of building software. I want to create something. I want to create some value, some change in the world. There's some product ultimately that I'm trying to build.And, you know, early on, I've done a lot of work in my career on, like, workflow-type builders and visual builders and I had this incorrect assumption somewhere along the way—and this came around, like, sort of the maker movement, when everybody was talking about everybody should learn how to code, and I made this assumption that everybody really wants to create; everybody wants to be a creator, and if given the opportunity, they will. And I think what I finally learned is that, actually most people don't like to create. A lot of people just want to be served; like, they just want to consume and they don't want the hassle of it. Some people do, if they have the opportunity and the skillsets, too, but it's also similar to, like, if I'm a professional developer, I need to get my work done. I'm not measured on how well my local tooling is set up; I'm sort of measured on my output and the impact that I have in the organization.I tend to think about, like, chefs. If I'm a chef and I work 60 hours in a restaurant, 70 hours in a restaurant, the last thing I want to do is come home and cook myself a meal. And most of the chefs I know actually don't have really nice kitchens at home. They, like, tend to, they want other people to cook for them. And so, I think, like, there's a place in professional setting where you just need to get the work done and you don't want to worry about all the meta things and the time that you could waste on it.And so, I feel like there's a happy medium there. I think it's good for people to care about the tools that they use the environment that they develop in, to really care for that and to curate it and make it better, but there's got to be some ROI and it's got to have value to you. You have to enjoy that. Otherwise, you know, what's the point of it in the first place?Corey: One thing that I used to think about was that if you're working in regulated industries, as I tended to a fair bit, there's something very nice about not having any of the data or IP or anything like that locally. Your laptop effectively just becomes a thin client to something that's already controlled by the existing security and compliance apparatus. That's very nice, where suddenly it's all someone steals my iPad, or I drop it into the bay, it's locked, it's encrypted. Cool, I go to the store, get myself a new one, restore a backup from iCloud, and I'm up and running again in a very short period of time as if nothing had ever changed. Whereas when I was doing a lot of local development and had bad hard drive issues in the earlier part of my career, well, there goes that month.Mike: Yeah, it's a really good point. I think that we're all walking around with these laptops with really sensitive IP on it and that those are in bars and restaurants. And maybe your drives are encrypted, but there's a lot of additional risks, including, you know, everything that is going over the network, whether I'm on a local coffee shop, and you know, the latest vulnerability that, an update I have to do on my Mac if I'm behind. And there's actually a lot of risk and having all that just sort of thrown to the wind and spread across the world and there's a lot of value in having that in a very safe place. And what we've even found that, at Gitpod now, like, the latest product we're working on is one that we called Gitpod Dedicated, which gives you the ability to run inside your own cloud perimeter. And we're doing that on AWS first, and so we can set up and manage an installation of Gitpod inside your own AWS account.And the reason that became important to us is that a lot of companies, a lot of our customers, treat their source code as their most sensitive intellectual property. And they won't allow it to leave their perimeter, like, they may run in AWS, but they have this concept of, sort of like, our perimeter and you're either inside of that and outside of it. And I think this speaks a little bit to a blog post that you wrote a few months ago about the lagging adoption of remote development environments. I think one of those aspects is, sort of, convenience and the user experience, but the other is that you can't use them very well with your stack and all the tools and resources that you need to use if they're not running, sort of, close within your perimeter. And so, you know, we're finding that companies have this need to be able to have greater control, and now with the, sort of, trends around, like, coding assistance and generative AI and it's even the perfect storm of not only am I like sending my source code from my editor out into some [LM 00:22:36], but I also have the risk of an LM that might be compromised, that's injecting code and I'm committing on my behalf that may be introducing vulnerabilities. And so, I think, like, getting that off to a secure space that is consistent and sound and can be monitored, to be kept up-to-date, I think it has the ability to, sort of, greatly increase a customer's security posture.Corey: While we're here kicking the beehive, for lack of a better term, your support for multiple editors in Gitpod the product, I assumed that most people would go with VS Code because I tend to see it everywhere, and I couldn't help but notice that neither VI nor Emacs is one of the options, the last time I checked. What are you seeing as far as popularity contests go? And that might be a dangerous question because I'm not suggesting you alienate many of the other vendors who are available, but in the world I live in, it's pretty clear where the zeitgeist of my subculture is going.Mike: Yeah, I mean, VS Code is definitely the most popular IDE. The majority of people that use Gitpod—and especially we have a, like, a pretty heavy free usage tier—uses it in the browser, just for the convenience of having that in the browser and having many environments in the browser. We tend to find more professional developers use VS Code desktop or the JetBrains suite of IDEs.Corey: Yeah, JetBrains I'm seeing a fair bit of in a bunch of different ways and I think that's actually most of what your other options are. I feel like people have either gone down the JetBrains path or they haven't and it seems like it's very, people who are into it are really into it and people who are not are just, never touch it.Mike: Yeah, and we want to provide the options for people to use the tools that they want to use and feel comfortable on. And we also want to provide a platform for the next generation of IDEs to be able to build on and support and to be able to support this concept of cloud or remote development more natively. So, like I mentioned, Nathan Sobo at Zed, I met up with him last week—I'm in Denver; he's in Boulder—and we were talking about this and he's interested in Zed working in the browser, and he's talked about this publicly. And for us, it's really interesting because, like, IDEs working in the browser is, like, a really great convenience. It's not the perfect way to work, necessarily, in all circumstances.There's some challenges with, like, all this tab sprawl and stuff, but it gives us the opportunity, if we can make Zed work really well in for Gitpod—or anybody else building an IDE—for that to work in the browser. Ultimately what we want is that if you want to use a terminal, we want to create a great experience for you for that. And so, we're working on this ability in Gitpod to be able to effectively, like, bring your own IDE, if you're building on that, and to be able to offer it and distribute on Gitpod, to be able to create a new developer tool and make it so that anybody in their Gitpod workspace can launch that as part of their workspace, part of their tool. And we want to see developer tools and IDEs flourish on top of this platform that is cloud development because we want to give people choice. Like, at Gitpod, we're not building our own IDE anymore.The team started to. They created Theia, which was one of the original cloud, sort of, web-based IDEs that now has been handed over to the Eclipse Foundation. But we moved to VS Code because we found that that's where the ecosystem were. That's where our users were, and our customers, and what they wanted to use. But we want to expand beyond that and give people the ability to choose, not only the options that are available today but the options that should be available in the future. And we think that choice is really important.Corey: When you see people kicking the tires on Gitpod for the first time, where does the bulk of their hesitancy come from? Like, what is it where—people, in my experience, don't love to embrace change. So, it's always this thing, “This thing sucks,” is sort of the default response to anything that requires them to change their philosophy on something. So okay, great. That is a thing that happens. We'll see what people say or do. But are they basing it on anything beyond just familiarity and comfort with the old way of doing things or are there certain areas that you're finding the new customers are having a hard time wrapping their head around?Mike: There's a couple of things. I think one thing is just habit. People have habits and preferences, which are really valuable because it's the way that they've learned to be successful in their careers and the way that they expect things. Sometimes people have these preferences that are fairly well ingrained that maybe are irrational or rational. And so, one thing is just people's force of habit.And then getting used to this idea that if it's not on my laptop, it means—like what you mentioned before, it's always what-ifs of, like, “What if I'm on a plane?” Or like, “What if I'm at the airport in a hurricane?” “What if I'm on a train with a spotty internet connection?” And so, there's all these sort of what-if situations. And once people get past that and they start actually using Gitpod and trying to set their projects up, the other limiting factor we have is just connectivity.And that's, like, connectivity to the other resources that you use to develop. So, whether that's, you know, package or module repositories or that some internal services or a database that might be running behind a firewall, it's like getting connectivity to those things. And that's where the dedicated deployment model that I talked about, running inside of your perimeter on our network, they have control over, kind of helps, and that's why we're trying to overcome that. Or if you're using our SaaS product, using something like Tailscale or a more modern VPN that way. But those are the two main things.It's like familiarity, this comfort for how to work, sort of, in this new world and not having this level of comfort of, like, it's running on this thing I can hold, as well as connectivity. And then there is some cost associated with people now paying for this infrastructure they didn't have to pay for before. And I think it's a, you know, it's a mistake to say that we're going to offset the cost of laptops. Like, that shouldn't be how you justify a cloud development environment. Like—Corey: Yeah, I feel like people are not requesting under-specced laptops much these days anymore.Mike: It's just like, I want to use a good laptop; I want to use a really nice laptop with good hardware and that shouldn't be the cost. The proposition shouldn't be, it's like, “Save a thousand dollars on every developer's laptop by moving this off to the cloud.” It's really the time savings. It's the focus. It's the, you know, removing all of that drift and creating these consistent environments that are more secure, and effectively, like, automating your development environment that's the same for everybody.But that's the—I think habits are the big thing. And there is, you know, I talked about a little bit that element of, like, we still have this concept of, like, I have this environment and I start it and it's there, and I pay for it while it's there and I have to clean it up or I have to make sure it stopped. I think that still exists and it creates a lot of sort of cognitive overhead of things that I have to manage that I didn't have to manage before. And I think that we have to—Gitpod needs to be better there and so does everybody else in the industry—about removing that completely. Like, there's one of the things that I really love that I learned from, like, Stewart Butterfield when I was at Slack was, he always brought up this concept called the convenience threshold.And it was just the idea that when a certain threshold of convenience is met, people's behavior suddenly changes. And as we thought about products and, like, the availability of features, that it really drove how we thought about even how to think about you know, adoption or, like, what is the threshold, what would it take? And, like, a good example of this is even, like, the way we just use credit cards now or debit cards to pay for things all the time, where we're used to carry cash. And in the beginning, when it was kind of novel that you could use a credit card to pay for things, like even pay for gas, you always had to have cash because you didn't know if it'd be accepted. And so, you still had to have cash, you still had to have it on hand, you still had to get it from the ATM, you still have to worry about, like, what if I get there and they don't accept my cards and how much money is it going to be, so I need to make sure I have enough of it.But the convenience of having this card where I don't have to carry cash is I don't have to worry about that anymore, as long as they have money in my bank account. And it wasn't until those cards were accepted more broadly that I could actually rely on having that card and not having the cash. It's similar when it comes to cloud development environments. It needs to be more convenient than my local development environment. It needs to be—it's kind of like early—I remember when laptops became more common, I was used to developing on a desktop, and people were like, nobody's ever going to develop on a laptop, it's not powerful enough, the battery runs out, I have to you know, when I close the lid, when you open the lid, it used to take, like, five minutes before, like, it would resume an unhibernate and stuff, and it was amazing where you could just close it and open it and get back to where you were.But like, that's the case where, like, laptops weren't convenient as desktops were because they were always plugged in, powered on, you can leave them and you can effectively just come back and sit down and pick up where you left off. And so, I think that this is another moment where we need to make these cloud development environments more convenient to be able to use and ultimately better. And part of that convenience is to make it so that you don't have to think about all these parts of them of whether they're running, not running, how much they cost, whether you're going to be there [unintelligible 00:31:35] or lose their data. Like, that should be the value of it that I don't have to think about any of that stuff.Corey: So, my last question for you is, when you take a look at people who have migrated to using Gitpod, specifically from the corporate perspective, what are their realizations after the fact—I mean, assuming they still take your phone calls because that's sort of feedback of a different sort—but what have they realized has worked well? What keeps them happy and coming back and taking your calls?Mike: Yeah, our customers could focus on their business instead of focusing on all the issues that they have with configuring development environments, everything that could go wrong. And so, a good example of this is a customer they have, Quizlet, Quizlet saw a 45-point increase in developer satisfaction and a 60% reduction in incidents, and the time that it takes to onboard new engineers went down to ten minutes. So, we have some customers that we talk to that come to us and say, “It takes us 20 days to onboard an engineer because of all the access they need and everything you need to set up and credentials and things, and now we could boil that down to a button click.” And that's the thing that we tend to hear from people is that, like, they just don't have to worry about this anymore and they tend to be able to focus on their business and what the developers are actually trying to do, which is build their product.And in Quizlet's example, it was really cool to see them mention in one of the recent OpenAI announcements around GPT4 and plugins is they were one of the early customers that built GPT4 plugins, or ChatGPT, and they mentioned that they were sharing a lot of Gitpod URLs around when we reached out to congratulate them. And the thing that was great about that, for us is, like, they were talking about their business and what they were developing and how they were being successful. And we'd rather see Gitpod in your development environment just sort of disappear into the background. We'd actually like to not hear from customers because it's just working so well from them. So, that's what we found is that customers are just able to get to this point where they could just focus on their business and focus on what they're trying to develop and focus on making their customers successful and not have to worry about infrastructure for development.Corey: I think that really says it all. On some level, when you have customers who are happy with what's happening and how they're approaching this, that really is the best marketing story I can think of because you can say anything you want about it, but when customers will go out and say, “Yeah, this has made our lives better; please keep doing what you're doing,” it counts for a lot.Mike: Yeah, I agree. And that's what we're trying to do. You know, we're not trying to win, sort of, a tab versus spaces debate here around local or cloud or—I actually just want to enable customers to be able to do their work of their business and develop software better. We want to try to provide a method and a platform that's extensible and customizable and gives them all the power they need to be able to just be ready to code, to get to work as soon as they can.Corey: I really want to thank you for being so generous with your time. If people want to learn more, where's the best place for them to find you, other than at your conference in San Francisco in a few weeks?Mike: [laugh]. Yeah, thank you. I really appreciate the banter back and forth. And I hope to see you there at our conference. You should come. Consider this an invite for June 1st and 2nd in San Francisco at CDE Universe.Corey: Of course. And we will put links to this in the [show notes 00:34:53]. Thank you so much for being so generous with your time. I appreciate it.Mike: Thanks, Corey. That was really fun.Corey: Mike Brevoort, Chief Product Officer at Gitpod. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment detailing exactly why cloud development is not the future, but then lose your content halfway through because your hard drive crashed.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

On this episode of Remote Ruby, the guys discuss various topics relating to hosting options, web frameworks, open source projects, and give us a recap on RailsConf 2023.  They dive into the pros and cons of serverless architectures like Lambda, Jason's experience with Roda, their interests in front-end technologies and JavaScript integration in Rails, and Andrew tells us about regex for playground. We'll hear their thoughts on RailsConf, their favorite talks, Chris's workshop, things that could have been better, and the importance of community contributions, transparency, and the need for clearer communication. Also, if you missed this RailsConf, they mention some other conferences coming up, so hit download to hear more![00:00:10] Chris brings up the blog post on Amazon's AWS blog which sparks a discussion about the effectiveness of serverless architectures like Lambda. [00:02:02] The conversation shifts to Jason telling us his experience with building a microservice using Roda. Then he tells us the benefits of Roda and compared it to Sinatra, and now Andrew wants to upgrade his Sinatra app to Roda since Jason had such a positive experience.[00:05:48] Cloudflare Workers, Puppeteer, Rust and JavaScript are discussed. [00:09:06] Chris shares his thoughts on RailsConf, mentioning attendance was smaller than expected. The guys also bring up that there was no hallway track and the spread out nature of the event, which made it less conducive to casual networking and impromptu conversations. Chris enjoyed the keynotes and attending a talk by Jordan Burke on hosting with Hatchbox, Fly , and Render. [00:12:10] There's a conversation on the need for more direction and talks on front-end technologies and JavaScript integration in Rails, and where to go if you want to learn more about these topics and contribute to the community. [00:14:26] Chris shares his takeaway from RailsConf, mentioning his interest in reading Rails commits daily to stay up to date with the community's progress. He also talks about his favorite part of the conference was an encounter with a Lightning Talk presenter who worked on the same project he did 13 years ago.  [00:17:16] Jumpstart Pro has been updated to Rails 7.1 and we hear the changes, and the conversation shifts to regex and a tool Andrew finds useful called “iHateRegex” and “regex for playground” that helps visualize regular expressions. [00:21:19] At RailsConf, Chris gave his first ever workshop with Colin Loretz. The talk focused on Webhooks and their handling in Rails and Chris made a screencast of the workshop and integrated the code into Jumpstart Pro.[00:26:06] Chris and Andrew talk about needing more scholars and promotions for the guide program at RailsConf. Also, they liked how there was a huge emphasis on Junior developers this year.  [00:29:03] Ruby Central is talked about and how more clarity regarding how community contributions are used, and they mention the change in leadership within Ruby Central and the impact it has had on the community. [00:38:24] The guys talk about all the upcoming conferences, including RailsConf and RubyConf. and Andrew shares his experience with social anxiety during the conference.[00:43:25] Chris mentions a hearing a rumor about Rails 7.1 shipping very soon, and Andrew tells us Jason dunked on him at RailsConf in front of everybody. [00:46:49] We end with the guys expressing their gratitude to the organizers and sponsors of RailsConf and encourage juniors to attend conferences to find job opportunities. Panelists:Jason CharnesChris OliverAndrew MasonSponsor:HoneybadgerLinks:Jason Charnes TwitterChris Oliver TwitterAndrew Mason TwitterRuby Conferences 2023Even Amazon can't make sense of serverless or microservices by David Heinemeier HanssonRodaCloudflare WorkersPuppeteerRustThis Week in Railsregex for playgroundHow to Process Inbound Webhooks (RailsConf 2023)-GoRailsRuby CentralRuby Radar TwitterRuby for All Podcast

Our guest today is writer, actor, and teacher Jonathan Fried. Jonathan shares with us how the play The Glass Menagerie by Tennessee Williams saved his life. What's extra special about this for Jonathan is that not only did this play have life-saving features for him as a teenager, he went on to star in it in a production with Olympia Dukakis!The Glass Menagerie follows the lives of the dysfunctional Wingfield family, son Tom, mother Amanda, daughter Laura. Tom longs to escape from his stifling home, wanting to be a poet and escape the realities of working in a shoe warehouse. While he "goes to the movies" every night, his mother struggles to find a husband for Laura who lives with a disability.Buy The Glass MenagerieVisit our Bookshop or buy it directly right now: https://bookshop.org/a/82376/9780811214049Become an Associate Producer!Become an Associate Producer of our podcast through a $20/month sponsorship on Patreon! A professionally recognized credit, you can gain access to Associate Producer meetings to help guide our podcast into the future! Get started today: patreon.com/thisqueerbookCreditsHost/Founder: J.P. Der BoghossianExecutive Producer: Jim PoundsAssociate Producers: Archie Arnold, Natalie Cruz, Paul Kaefer, Nicole Olila, Joe Perazzo, Bill Shay, and Sean SmithPatreon Subscribers: Awen Briem, Stephen D., Thomas Michna, and Gary Nygaard.Join us May 18th!Ask for the Pride Special at the Spectacle Shoppe!Until August 1st, the Spectacle Shoppe is offering you $250 off as a Pride special. For locations, visit: https://spectacleshoppe.com Join Me In Supporting Lambda LiteraryAs a Lambda Literary Fellow, I hope you can donate to Lambda's Writers Retreat for Emerging LGBTQ Voices. They're raising $56k to ensure every fellow attend!You can donate to the scholarship fundraising campaign by visiting lambdaliterary.org/writers-retreat & clicking on SUPPORT EMERGING WRITERS or by texting LITVOICES to 44-321. Support the show

David Colebatch, CEO at Tidal.cloud, joins Corey on Screaming in the Cloud to discuss how Tidal is demystifying cloud migration strategy. David and Corey discuss the pros and cons of a hybrid cloud migration strategy, and David reveals the approach that Tidal takes to ensure they're setting their customers up for success. David also discusses the human element to cloud migration initiatives, and how to overcome roadblocks when handling the people side of migrations. Corey and David also expand on all the capabilities cloud migration unlocks, and David explains how that translates to a distributed product team approach.About DavidDavid is the CEO & Founder of Tidal.  Tidal is empowering businesses to transform from traditional on-premises IT-run organizations to lean-agile-cloud powered machines.Links Referenced: Tidal.cloud: https://tidal.cloud Twitter: https://twitter.com/dcolebatch LinkedIn: https://www.linkedin.com/in/davidcolebatch/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey:  LANs of the late 90's and early 2000's were a magical place to learn about computers, hang out with your friends, and do cool stuff like share files, run websites & game servers, and occasionally bring the whole thing down with some ill-conceived software or network configuration. That's not how things are done anymore, but what if we could have a 90's style LAN experience along with the best parts of the 21st century internet? (Most of which are very hard to find these days.) Tailscale thinks we can, and I'm inclined to agree. With Tailscale I can use trusted identity providers like Google, or Okta, or GitHub to authenticate users, and automatically generate & rotate keys to authenticate devices I've added to my network. I can also share access to those devices with friends and teammates, or tag devices to give my team broader access. And that's the magic of it, your data is protected by the simple yet powerful social dynamics of small groups that you trust.Try now - it's free forever for personal use. I've been using it for almost two years personally, and am moderately annoyed that they haven't attempted to charge me for what's become an essential-to-my-workflow service.Corey: Have you listened to the new season of Traceroute yet? Traceroute is a tech podcast that peels back the layers of the stack to tell the real, human stories about how the inner workings of our digital world affect our lives in ways you may have never thought of before. Listen and follow Traceroute on your favorite platform, or learn more about Traceroute at origins.dev. My thanks to them for sponsoring this ridiculous podcast. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while at The Duckbill Group, I like to branch out and try something a little bit different before getting smashed vocally, right back into the box I find myself in for a variety of excellent reasons. One of these areas has been for a while, the idea of working with migrations on getting folks into cloud. There's a lot of cost impact to it, but there's also a lot of things that I generally consider to be unpleasant nonsense with which to deal. My guest today sort of takes a different philosophy to this. David Colebatch is the CEO and founder of Tidal.cloud. David, thank you for joining me.David: Oh, thanks for having me, Corey.Corey: Now, cloud migrations tend to be something that is, I want to say contentious, and for good reason. You have all the cloud providers who are ranting that cloud is the way and the light, as if they've just found religion, and yeah, the fact that it basically turns into a money-printing machine for them has nothing to do with their newfound advocacy for this approach. Now, I do understand that we all have positions that we come from that shape our perspective. You do run and did found a cloud migration company. What's your take on it? Is this as big as the cloud providers say it is, is it overhyped, or is it underhyped?David: I think it's probably in the middle of this stage of the hype cycle. But the reason that that Tidal exists and why I founded it was that many customers were approaching cloud just for cloud's sake, you know, and they were looking at cloud as a place to park VMs. And our philosophy as software engineers at Tidal is that customers were missing out on all the new capabilities that cloud provided, you know, cloud is a new paradigm in compute. And so, our take on it is the customer should not look at cloud as a place to migrate to, but rather as a place to transform to and embrace all the new capabilities that are on offer.Corey: I've been saying for a while that if you sit there and run a total cost analysis for going down the path of a cloud migration, you will not save money in the short term, call it five years or whatnot. So, if you're migrating to the cloud specifically to save money, in the common case, it should be for a capability story, not because it's going to save you money off of what you're currently doing in the data center. Agree, disagree, or it's complicated?David: It's complicated, but you're right in one case: you need to work backwards from the outcomes, I think that much is pretty simple and clear, but many teams overlook that. And again, when you look at cloud for the sake of cloud, you generally do overlook that. But when we work with customers and they log into to our platform, what we find is that they're often articulating their intent as I want to improve business agility, I want to improve staff productivity, and it's less about just moving workloads to the cloud. Anyone can run a VM somewhere. And so, I think, when we work backwards from what the customer is trying to achieve and we look at TCO holistically, not just about how much a computer costs to run and operate in a colo facility, look at it holistically from a staff productivity perspective as well, then the business case for cloud becomes very profound.Corey: I've been saying for a while that I can make a good-faith Total Cost of Ownership analysis—or TCO analysis—in either direction, so tell me what outcome you want and I can come up with a very good-faith effort answer that gives you what you want. I don't think I've seen too many TCO analyses, especially around cloud migrations, that were not justification exercises. They were very rarely open questions. It was, we've decided what we want to do. Now, let's build a business case to do that thing. Agree, disagree?David: [laugh]. Agree. I've seen that. Yeah, we again, like to understand the true picture of total cost of ownership on-premises first, and many customers, depending on who you're engaging with, but on the IT side, might actually shield a few of those costs or they might just not know them. And I'm talking about things like in the facilities, insurance costs, utility bills, and things like that, that might not bubble up.We need to get all those cards on the table in order to conduct a full TCO analysis. And then in the cloud side, we need to look at multiple scenarios per workload. So, we want to understand that lift-and-shift base case that many people come from, but also that transformative migration case which says, I might be running in a server-ful architecture today on-premises, but based on the source code and database analysis that we've done, we can see an easy lift to think like Lambda and serverless frameworks on the cloud. And so, when you take that transformative approach, you may spend some time upfront doing that transformation, or if it's tight fit, it might be really easy; it might actually be faster than reverse-engineering firewall rules and doing a lift-and-shift. And in that case, you can save up to 97% in annual OPEX, which is a huge savings, of course.Corey: You said the magic words, lift-and-shift, which means all right, the gloves come off. Let's have this conversation.David: Oh yeah.Corey: I work on AWS bills for a living. Cloud cost and architecture are fundamentally the same thing, and when I start looking at a company's monthly bill, I can start to see the architectural patterns emerge with no further information than what's shown in the exploded bill view, at least at a high level. It starts to be indicative of different things. And you can generally tell, on some level, when companies have come from a data center environment or at least a data center mentality, in what they've built. And I've talked to a number of companies where they have effectively completely lifted their data center into the cloud and the only real change that they have gotten in terms of value for it has been that machines are going down a lot less because the hard drive failed and they were really bad at replacing hard drives.Now, for companies in that position who have that challenge, yeah, the value is there and it's apparent because I promise, whoever you are, the cloud providers are better at replacing failed hard drives than you are, full stop. And if that's the value proposition you want, great, but it also feels like that is just scratching the surface of what the benefit of cloud providers can be.David: Absolutely. I mean, we look at cloud as a way to unlock new ways of working and it's totally aligned with the new distributed product team approach that many enterprises are pursuing. You know, the rise of Agile and DevOps has sort of facilitated this movement away from single choke points of IT service delivery, like we used to with ITIL, into much more modern ways of working. And so, I imagine when you're looking at those cloud bills, you might see a whole host of workloads centered into one or two accounts, like they've just replicated a data center into one or two accounts and lifted-and-shifted a bunch of EC2 to it. And yeah, that is not the most ideal architectural pattern to follow in the cloud. If you're working backwards from, “I want to improve staff productivity; I want to improve business agility,” you need to do things like limit your blast radius and have a multi-account strategy that supports that.Corey: We've seen this as well and born-in-the-cloud companies, too, because for a long time, that was AWS's guidance of put everything in a single AWS account. The end. And then just, you know, get good with IAM issues. Like, “Well okay, I found that developer environments impacted production.” Then, “Sounds like a skill issue.”Great, but then you also have things that cannot be allocated, like service quotas. When you have something in development run amok and exhaust service quotas for number of EC2 get instance info requests, suddenly, load balancers don't anymore and auto-scaling is kind of aspirational when everything explodes on you. It's the right path, but very often, people got there through following the best advice that AWS offers. I am in the middle of a migration myself from the quote-unquote, “Legacy” AWS account, I built a bunch of stuff in 2016 into its own dedicated account and honestly, it's about as challenging as some data center moves that I've done historically.David: Oh, absolutely. I mean, the cobwebs build up over time and you have a lot of dependencies on services, you completely forget about.Corey: “How do I move this S3 bucket to another account?” “That's the neat part. You don't.”David: [laugh]. We shouldn't just limit that to AWS. I mean, the other cloud providers have similar issues to deal with through their older cloud adoption frameworks which are now playing out. And some of those guidance points were due to technology limitations in the underlying platform, too, and so you know, at the time, that was the best way to go to cloud. But as I think customers have demanded more agility and more control over their blast radiuses and enabling self-service teams, this has forced everyone to sort of come along and embrace this multi-account strategy. Where the challenge is, with a lot of our enterprise clients, and especially in the public—Corey: Embrace it or you'll be made to embrace it.David: Yeah [laugh]. We see with both our enterprise accounts that were early adopters, they certainly have that issue with too much concentration on one or two accounts, but public sector accounts as well, which we're seeing a lot of momentum in, they come from a place where they're heavily regulated and follow heavy architectural standards which dictate some of these things. And so, in order for those clients to be successful in the cloud, they have to have real leadership and real champions that are able to, sort of, forge through some of those issues and break outside of the mold in order to demonstrate success.Corey: On some level, when I see a lift that failed to shift, it's an intentional choice in some cases where the company has decided to improve their data center environment at the cost of their cloud environment. And it feels, on some level, like it's a transitional step, but then it's almost a question that I always have is, was this the grand plan? So, I guess my question for you is, when you see a company that has some workloads in a data center and some living in the cloud provider in what most people call hybrid, is that outcome intentional or is it accidental, where midway through, they realize that some workloads are super hard to migrate? They have a mainframe and there is no AWS/400 available for their use, so they're going to give up halfway, declare victory, and yep we're hybrid now. How did they get there?David: I think it's intentional, quite often that they see hybrid cloud as a stepping stone to going full cloud. And this just comes down to project scoping and governance, too. So, many leaders will draw a ring around the workloads that are easy to migrate and they'll claim success at the end of that and move on to another job quite often. But the visionary leaders will actually chart a path to course that has a hundred percent adoption, full data center closure, off the mainframe, off AS/400, you know, refactored usually, but they'll chart that course at a rate of change that the organization can accept. Because, you know, cloud being a new paradigm, cloud requiring new ways of working, they can't just ram that kind of change through in their enterprise in one or two years; they really need to make sure that it's being absorbed and adopted and embraced by the teams and not alienating the whole company as they go through. And so, I do see it as intentional, but that stepping stone that many companies take is also an okay thing in my mind.Corey: And to be clear, I should bound what I'm saying from the perspective that I'm talking about this from a platonic ideal perspective. I am not suggesting that, “Oh, this thing that you built at your company is crappy,” I mean, any more so than anything else is. I've never yet seen any infrastructure that the people running it would step back and say, “This is amazing and perfect.” Everyone thinks it's a burning dumpster fire of sadness and regret and I'm not entirely sure that they're wrong.I mean, designing an architecture—cloud or otherwise—on a whiteboard is relatively straightforward, for a junior employee, even. The problem is most people don't get to start from scratch and build that thing. There's existing stuff that needs to be migrated in and most of us don't get the luxury of taking two years of downtime for that service while we wind up rebuilding it from scratch. So, it's one of those how do you rebuild a car without taking it off the highway to do it type of questions.David: Well, you want to have a phased migration approach, quite often. Your business can't stop and start because you're doing a migration, so you want to build momentum with the early adopters that are easy to migrate and don't require big interruptions to business. And then for those mission-critical workloads that do need to migrate—and you mentioned mainframe and AS/400 before—they might be areas where you introduce, like, a strangler fig pattern, you know, draw a ring around it, start replicating some services into cloud, and then phase that migration over a year or two, depending on your timeline and scale. And so, we're very much pragmatic in this business that we want to make sure we're doing everything for the right reasons, for the business-led reasons, and fitting in migrations around business objectives and strategies is super critical to success.Corey: What I'm curious about is when we talk about migrations, in fact, when I invited you on the show, and it was like, well, Tidal migrations—one thing I love about calling it that for the domain, in some cases, as well as other things is, “Huh, says right in the tin what it is. Awesome.” But it's migrations, which I assumed to be, you know, from data centers into cloud. That's great. But then you've got the question of, is that what your work looks like? Is it migrations in the other direction? Is cloud repatriation a thing that people are doing, and no one bothered to actually ever bother to demonstrate that to me? Is cloud to cloud? What are you migrating from and to?David: Well, that's great. And we actually dropped migrations from the name.Corey: Oh, my apologies. Events, once again, outpace me.David: Tidal.cloud is our URL and essentially, Corey, the business of migration is something that's only becoming increasingly frequent. Customers are not just migrating from on-premises data centers to cloud, they're also migrating in between their cloud accounts like you are, but also from one cloud provider to another. And our business hypothesis here Tidal is that that innovation cycle is continuing to shrink, and so whereas when I was in the data center automation business, we used to have a 10 and 15-year investment cycle, now customers have embraced continuous delivery of their applications and so there's this huge shift of investment horizons, bringing it down to an almost an annual event for many of the applications that we touch.Corey: You are in fact correct. Tidal.cloud does have a banner at the top that says, “Tidal Migrations is now Tidal.” Yep, you're correct, not that I'm here to like incorrect you on the name of your own company, for God's sake. That's a new level of mansplaining I dare not delve into.But it does say, “Migration made modern,” right at the top, which is great because there's a sense that I've always had that lift-and-shift is poo-pooed as a bad approach to migrating, but I've done it other ways and it becomes disastrous. I've always liked the approach of take something in a data center, migrated into cloud, in the process, changing as few things as possible, and then just get it stable and working there, and step two becomes the transformation because if you try and transform while it moves, yeah, that gets you a little closer to outcome in theory, but when things don't work right—and their computers; let's not kid ourselves, nothing works right—it's a question now of was it my changes? Is it the cloud environment? Is there an unknown dependency that assumes things in the data center that are not true in cloud? It becomes very hard to track down the why of these things.David: There's no one-size-fits-all for migration. It's why we have the seven-hour assessment capabilities. You know, if one application, like you've just talked about, that one application might be better to lift and shift than modernize, there might be real business reasons for doing that. But what we've seen over the years is the customers generally have one migration budget. Now, IT gets one migration budget and they get to end a job in a lift-and-shift scenario and the business says, “Well, what changed? Nothing, my apps still run the same, I don't notice any new capabilities.” And IT then says, “Yeah, yeah. Now, we need the modernization budget to finish.” And they said, “No, no, no. We've just given you a bunch of money. You're not getting any more.”And so, that's what quite often the migrate as a lift-and-shift kind of stalls and you see an exodus of talent out of those organizations, people leave to go on to the next migration project elsewhere and that organization really didn't embrace any of the cloud-native changes that were required. We'd like to really say that—and you saw this on our header—that migrations made modern, we'd like to dispel the myth that you can either migrate or modernize. It's really not an either/or. There's a full spectrum of our methods, like replatform, and refactor, rehosting, in the middle there. And when we work backwards from customers, we want to understand their core objectives for going to cloud, their intent, their, “Why cloud?”We want to understand how it aligns on the cloud value framework, so business agility gains, staff productivity gains, total cost of ownership is important, of course. And then for each of their application workloads, choose the right 6R based on those business outcomes. And it can seem like a complicated or comprehensive problem, but if you automate it like we do, you can get very consistent results very quickly. And that's really the accelerant that we give customers to accelerate their migration to cloud.Corey: One thing that I've noticed—and maybe this makes me cynical—but when I see companies doing lift-and-shift, often they will neglect to do the shift portion of it. Because there's a compelling reason to do a migration to get out of a data center and into a cloud, and often that is a data center contract expiry coming up. But companies are very rarely going to invest the time, energy, and money—which all become the same thing, effectively, at company scale—in refactoring existing applications if they're not already broken.I see that all the time in my work, I don't make recommendations to folks very often have the form, “Oh, just migrate this entire application to serverless and you'll save 80% or more on it.” And it's, “That's great, but that's 18 months' worth of work and it doesn't actually get us closer to our business milestones, so yeah, we're not going to do that.” Cost directly is very rarely a compelling reason to make a migration, but when you're rebuilding something for business purposes, factoring cost concerns into it seems to be a much better way to gain adoption and traction of those ideals.David: Yeah, yeah. Counterpoint on that, when we look at a portfolio of applications, like, hundreds or thousands of applications in an enterprise and we do this type of analysis on them with the customers, what we've learned is that they may refactor and replatform ten, 20% of their workloads, they may rehost 40%, and they'll often turn off the rest, retire them, not migrate them. And many of our enterprise customers that we've spoken to have gone through rationalizations as they've gone to cloud and saved, you know, 59%, just turned off that 59% of an infrastructure, and the apps that they do end up refactoring and modernizing are the ones where either there's a very easy path for them, like, the code is super compatible and written in a way that's fitting with Lambda and so they've done that, or they've got, like you said, business needs coming up. So, the business is already investigating making some changes to the application, they already want to embrace CI/CD pipelines where they haven't today. And for those applications, what we see teams doing is actually building new in the cloud and then managing that as an application migration, like, cutting over that.But in the scheme of an entire portfolio of hundreds or thousands of applications that might be 5, 10, 20% of the portfolio. It won't be all of them. And that's what we say, there's a full spectrum of migration methods and we want to make sure we apply the right ones to each workload.Corey: Yeah, I want to be clear that there are different personas. I find that most of my customers tend to fall into two buckets. The first is that you have the born-in-the-cloud SaaS companies, and that's the world I come from, where you have basically one workload that's 80% of your application spend, your revenue, et cetera. Like, they are not a customer, but take Datadog as an example. Like, the Datadog monitoring application suite would be a good example of this, and then you have a bunch of longtail stuff.Conversely, you've got a large enterprise that might be spending $100 million or so every year, but their largest single application is a couple million bucks because it just has thousands upon thousands of them. And at that point, it becomes much more of a central IT planning problem. In one of those use cases, spending significant effort refactoring and rebuilding things, from an optimization perspective, can pay dividends. In other cases, it tends not to work in quite the same way, just because the economies of scale aren't there. Do you find that most of your customers fall into one of those two buckets? Do you take a different view of the world? How do you see the market?David: Same view, we do. Enterprise customers are generally the areas that we find the most fit with, the ISVs, you know, that have one or two primary applications. Born in the cloud, they don't need to do portfolio assessments. And with the enterprise customers, the central IT bit used to be a blocker and impediment for cloud. We're increasingly seeing more interest from central IT who is trying to lead their organization to cloud, which is great, that's a great sign.But in the past, it had been more of a business-led conversation where one business unit within an enterprise wants to branch away from central IT, and so they take it upon themselves to do an application assessment, they take it upon themselves to get their own cloud accounts, you know, a shadow IT move, in a way. And that had a lot of success because the business would always tie it back to business outcomes that they were trying to achieve. Now, into IT, doing mass migration, mass portfolio assessment, this does require them to engage deeply with the business areas and sometimes we're seeing that happening for the very first time. There's no longer IT at the end of a chain, but rather it's a joint partnership as they go to cloud, which is really cool to see.Corey: When I go to Tidal.cloud, you have a gif—yes, that's how it's pronounced, I'm not going to take debates on that matter—but you have a gif at the top of your site a showing a command line tool that runs an analyze command on an application. What are you looking at to establish an application or workload's suitability for migration? Because I have opinions on this, but you have, you know, a business around this and I'm not going to assume that my strongly-held opinions informed by several weeks of work are going to trump, you know, the thing that your entire company is built around.David: Thanks, Corey. Yeah, you're looking at our command-line utilities there. It's an accompanying part of our product suite. We have a web application and the command-line utilities are what customers use behind their firewall to analyze their applications. The data points that we look at are infrastructure, as you can imagine, you might plug into VMware and discover VMs that are running, we'll look for non-x86 workloads on the network.So, infrastructure is sort of bread and butter; everyone does that. Where Tidal differentiates is going up the stack, analyzing source code, analyzing database technologies, and looking at the schema usage within your on-premises database, for example, which features and functionality are using, and then how that fits to more cloud-native database offerings. And then we'll look at the technology age as well. And when you combine all of those technology factors together, we sort of form a view of what the migration difficulty to cloud will be on various migration outcomes, be it rehost, replatform, or refactor.The other thing that we add there is on the business side and the business intent. So, we want to understand from leadership what their intent is with cloud, and there's some levers they pull in the Tidal platform there. But then we also want to understand from each application owner how they think about their applications, what the value of those applications are to them and what their forward-looking plans are. We capture all these things in our tool, we then run it through our recommendation engine, and that's how we come up with a bespoke migration plan per client.Corey: One of the challenges I have in the cost arena around a lot of these tools that oh, we're going to look at your various infrastructure-as-code situation and see what that's going to cost you for a given change. It's like, sure, that that's not hard from a baseline of I want to spin up ten more EC2 instances. Yes, that is the tricky part of cloud economics known as basic arithmetic. The problem where I see is that okay, and then they're going to run Kubernetes, which has no sense of zone affinity, so it's going to wind up putting nondeterministic amounts of traffic across a AZ boundary and that's going to spike data transfer in some use cases, but none of these tools have any conception as to what those workloads look like. Now, that's a purely cost perspective, but that does have architectural approaches. Do you factor things like that in when you move up the stack?David: Absolutely. And really understanding on a Tidal inventory basis, understanding what the intent is of each of those workloads really does help you, from a cloud economics basics, to work out how much is reasonable in terms of cloud costs. So, for example, in Tidal, if you're doing app assessment, you're capturing any revenue to business that it generates, any staff productivity that it creates. And so, you've got the income side of that application workload. When you map that to on-premises costs and then later to cloud costs, your FinOps job becomes a lot easier because now you have the business context of those workloads too.Corey: So, one of the things that I have found is that you can judge the actual success of a project by how many people who work at the company claimed credit for it on LinkedIn, whereas conversely, when things don't work out super well, it's sort of a crickets moment. I'm curious as to your perspective on whether there is such a thing as a migration failure, or is it simply a, “Oh, we're going to iterate on this in a new direction. We've replaced a failing part, which turned out, from our perspective, to be our CIO, but we have a new one who's going to move us into cloud in the proper time and space.” We go through more of those things than some people do underwear. My God. But is there such a thing as a failed cloud migration?David: There absolutely is. And I get your point that success has many fathers. You know, when clients have brought us in for that success party at the end, you don't recognize everybody there. But you know, failure can be, you know, you've missed on time, scope, or budget, and by those measures, I think 76% of IT projects were failing in 2018, when we ran those numbers.So absolutely, by those metrics, there are failed cloud migrations. What tends to happen is people claim success on the workloads that did migrate. They may then kick it out into a new project scope, the organizational change bit. So, we've had many customers who viewed the cloud migration as a lift-and-shift exercise and failed to execute on the organizational change and then months later realized, oh, that is important in order for my day two operations to really hum, and so then have embarked on that under a separate initiative. So, there's certainly a lot of rescoping that goes on with these things.And what we like to make sure we're teaching people—and we do this for free—is those lessons learned and pitfalls with cloud early on because we don't want to see all those headlines of failed projects under that; we want to make sure that customers are armed with here are the things you should consider to execute on as you go to cloud.Corey: Do you ever run an analysis on a workload when a customer is asking, “So, how should we go about migrating this?” And your answer is, “You should absolutely not?”David: Well, all applications can go to cloud, it's just a matter of how much elbow grease you want to put into it. And so, the absolutely not call comes from when that app doesn't provide any utility to the business or maybe it has a useful life of six more months and the data center is going to be alive for seven. So, that's when those types of judgment calls come in. Other times we've seen, you know, there's already a replacement initiative underway by the business. IT wasn't aware of it, but through our process and methodology, they engaged with the business for the first time and learned about it. And so, that helps them to avoid needing to migrate workloads because the business is already moving to Salesforce, for example.Corey: I imagine you're also relatively used to the sinking realization that customers often have when they're used to data center thinking and you ask them a question, like, “How many gigabytes a month does your application server send back and forth to your database server?” And their response, very reasonably, is, “Why on earth would I know the answer to that quest—oh, God. You mean, that's how it bills?” It's the sense of everything is different in cloud, sometimes, subtly, sometimes massively. But it's a different way of thinking.So, I guess my last real big question for you on this is, moving technology is relatively straightforward but migrating people is very challenging. How do you find that the people and the processes that have grown up in data center environments with people whose identities are inextricably linked the technology they work on, being faced with the idea of it is now time to pick up and move these things into an environment where things that were incredibly valuable guardrails in a data center environment no longer serve you well?David: Yeah. The people side of cloud migration is the more challenging part. It's actually one of the reasons we introduced a service offering around people change management. The general strategy is sort of the Kotter change process of creating that guiding coalition, the people who want to do something different, get them outside of IT, reporting out to the executives directly, so they're unencumbered by the traditional processes. And once they start to demonstrate some success of a new way of working, a new paradigm, you kind of sell that back into the organization in order to drive that change.It's getting a lot easier to position that organizational change aspects with customers. There's enough horror stories out there of people that did not take that approach. And quite rightly. I mean, it's tough to imagine, as a customer, like, if I'm applying my legacy processes to cloud migration, why would I expect to get anything but a legacy result? You know, and most of the customers that we talk to that are going to cloud want a transformational outcome, they want more business agility and greater staff productivity, and so they need to recognize that that doesn't come without change to people and change the organization. It doesn't mean you have to change the people out individually, but skilling the way we work, those types of things, are really important to invest in and I'd say even more so than the technology aspects of any cloud migration.Corey: David, I really want to thank you for taking the time to talk to me about something that is, I'd say near and dear to my heart, except I'm trying desperately not to deal with it more than I absolutely have to. If people want to learn more, where's the best place for them to find you?David: Sure. I mean, tidalcloud.com is our website. I'm also on Twitter @dcolebatch. I like to tweet there a little bit, increasingly these days. I'm not on Bluesky yet, though, so I won't see you there. And also on LinkedIn, of course.Corey: And we will, of course, put links to that in the [show notes 00:29:57]. Thank you so much for your time. I really appreciate it.David: Thanks, Corey. Great to be here.Corey: David Colebatch, CEO and founder of Tidal.cloud. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you will then struggle to migrate to a different podcast platform of your choice.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

In this episode, I spoke with Eduard Bargues, who is a Principal Engineer at Ohpen, a cloud-native open banking platform.We talked about Ohpen's migration from EC2 to a mix of Fargate and Lambda, and along the way, we touched on many topics:migration patternsthe "serverless-first" mindsethow to choose when to use Lambda vs Fargatewhy monolith functions are painful and should be avoidedrecommendations and pitfallswhy you should build a DX teamWe also talked about how being cloud-agnostic makes you use the cloud in a way that is inefficient and creates many layers of unnecessary abstraction layers in your architecture, which Eduard calls "legacy lock-in"!We talked about their event-driven architecture and how they are using an in-house Event Broker to add FIFO support (which EventBridge doesn't support) for some subscribers. It's similar to what Luc van Donkersgoed talked about in episode 68. We talked about their EventBridge topology and how they manage over 200 AWS accounts, and finally, what are the biggest shortcomings with EventBridge right now.Links from the episode:Job openings at OhpenEpisode 68 with PostNLEpisode 73 with NNLuc van Donkersgoed's Twitter and LinkedInThe decoupled invocation patternFor more stories about real-world use of serverless technologies, please follow me on Twitter as @theburningmonk and subscribe to this podcast.Want to step up your AWS game and learn how to build production-ready serverless applications? Check out my upcoming workshops and I will teach you everything I know.Opening theme song:Cheery Monday by Kevin MacLeodLink: https://incompetech.filmmusic.io/song/3495-cheery-mondayLicense: http://creativecommons.org/licenses/by/4.0

Everett Berry, Growth and Open Source at Vantage, joins Corey at Screaming in the Cloud to discuss the complex world of cloud costs. Everett describes how Vantage takes a broad approach to understanding and cutting cloud costs across a number of different providers, and reveals which providers he feels generate large costs quickly. Everett also explains some of his best practices for cutting costs on cloud providers, and explores what he feels the impact of AI will be on cloud providers. Corey and Everett also discuss the pros and cons of AWS savings plans, why AWS can't be counted out when it comes to AI, and why there seems to be such a delay in upgrading instances despite the cost savings. About EverettEverett is the maintainer of ec2instances.info at Vantage. He also writes about cloud infrastructure and analyzes cloud spend. Prior to Vantage Everett was a developer advocate at Arctype, a collaborative SQL client acquired by ClickHouse. Before that, Everett was cofounder and CTO of Perceive, a computer vision company. In his spare time he enjoys playing golf, reading sci-fi, and scrolling Twitter.Links Referenced: Vantage: https://www.vantage.sh/ Vantage Cloud Cost Report: https://www.vantage.sh/cloud-cost-report Everett Berry Twitter: https://twitter.com/retttx Vantage Twitter: https://twitter.com/JoinVantage TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey:  LANs of the late 90's and early 2000's were a magical place to learn about computers, hang out with your friends, and do cool stuff like share files, run websites & game servers, and occasionally bring the whole thing down with some ill-conceived software or network configuration. That's not how things are done anymore, but what if we could have a 90's style LAN experience along with the best parts of the 21st century internet? (Most of which are very hard to find these days.) Tailscale thinks we can, and I'm inclined to agree. With Tailscale I can use trusted identity providers like Google, or Okta, or GitHub to authenticate users, and automatically generate & rotate keys to authenticate devices I've added to my network. I can also share access to those devices with friends and teammates, or tag devices to give my team broader access. And that's the magic of it, your data is protected by the simple yet powerful social dynamics of small groups that you trust.Try now - it's free forever for personal use. I've been using it for almost two years personally, and am moderately annoyed that they haven't attempted to charge me for what's become an essential-to-my-workflow service.Corey: Have you listened to the new season of Traceroute yet? Traceroute is a tech podcast that peels back the layers of the stack to tell the real, human stories about how the inner workings of our digital world affect our lives in ways you may have never thought of before. Listen and follow Traceroute on your favorite platform, or learn more about Traceroute at origins.dev. My thanks to them for sponsoring this ridiculous podcast. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This seems like an opportune moment to take a step back and look at the overall trend in cloud—specifically AWS—spending. And who better to do that than this week, my guest is Everett Berry who is growth in open-source over at Vantage. And they've just released the Vantage Cloud Cost Report for Q1 of 2023. Everett, thank you for joining me.Everett: Thanks for having me, Corey.Corey: I enjoy playing slap and tickle with AWS bills because I am broken in exactly that kind of way where this is the thing I'm going to do with my time and energy and career. It's rare to find people who are, I guess, similarly afflicted. So, it's great to wind up talking to you, first off.Everett: Yeah, great to be with you as well. Last Week in AWS and in particular, your Twitter account, are things that we follow religiously at Vantage.Corey: Uh-oh [laugh]. So, I want to be clear because I'm sure someone's thinking it out there, that, wait, Vantage does cloud cost optimization as a service? Isn't that what I do? Aren't we competitors? And the answer that I have to that is not by any definition that I've ever seen that was even halfway sensible.If SaaS could do the kind of bespoke consulting engagements that I do, we would not sell bespoke consulting engagements because it's easier to click button: receive software. And I also will point out that we tend to work once customers are at a certain point at scale that in many cases is a bit prohibitive for folks who are just now trying to understand what the heck's going on the first time finance has some very pointed questions about the AWS bill. That's how I see it from my perspective, anyway. Agree? Disagree?Everett: Yeah, I agree with that. I think the product solution, the system of record that companies need when they're dealing with Cloud costs ends up being a different service than the one that you guys provide. And I think actually the to work in concert very well, where you establish a cloud cost optimization practice, and then you keep it in place via software and via sort of the various reporting tools that the Vantage provide. So, I completely agree with you. In fact, in the hundreds of customers and deals that Vantage has worked on, I don't think we have ever come up against Duckbill Group. So, that tells you everything you need to know in that regard.Corey: Yeah. And what's interesting about this is that you have a different scale of visibility into the environment. We wind up dealing with a certain profile, or a couple of profiles, in our customer base. We work with dozens of companies a year; you work with hundreds. And that's bigger numbers, of course, but also in many cases at different segments of the industry.I also am somewhat fond of saying that Vantage is more focused on going broad in ways where we tend to focus on going exclusively deep. We do AWS; the end. You folks do a number of different cloud providers, you do Datadog cost visibility. I've lost track of all the different services that you wind up tracking costs for.Everett: Yeah, that's right. We just launched our 11th provider, which was OpenAI and for the first time in this report, we're actually breaking out data among the different clouds and we're comparing services across AWS, Google, and Azure. And I think it's a bit of a milestone for us because we started on AWS, where I think the cost problem is the most acute, if you will, and we've hit a point now across Azure and Google where we actually have enough data to say some interesting things about how those clouds work. But in general, we have this term, single pane of glass, which is the idea that you use 5, 6, 7 services, and you want to bundle all those costs into one report.Corey: Yeah. And that is something that we see in many cases where customers are taking a more holistic look at things. But, on some level, when people ask me, “Oh, do you focus on Google bills, too,” or Azure bills in the early days, it was, “Well, not yet. Let's take a look.” And what I was seeing was, they're spending, you know, millions or hundreds of millions, in some cases, on AWS, and oh, yeah, here's, like, a $300,000 thing we're running over on GCP is a proof-of-concept or some bizdev thing. And it's… yeah, why don't we focus on the big numbers first? The true secret of cloud economics is, you know, big numbers first rather than alphabetical, but don't tell anyone I told you that.Everett: It's pretty interesting you say that because, you know, in this graph where we break down costs across providers, you can really see that effect on Google and Azure. So, for example, the number three spending category on Google is BigQuery and I think many people would say BigQuery is kind of the jewel of the Google Cloud empire. Similarly for Azure, we actually found Databricks showing up as a top-ten service. Compare that to AWS where you just see a very routine, you know, compute, database, storage, monitoring, bandwidth, down the line. AWS still is the king of costs, if you will, in terms of, like, just running classic compute workloads. And the other services are a little bit more bespoke, which has been something interesting to see play out in our data.Corey: One thing that I've heard that's fascinating to me is that I've now heard from multiple Fortune 500 companies where the Datadog bill is now a board-level concern, given the size and scale of it. And for fun, once I modeled out all the instance-based pricing models that they have for the suite of services they offer, and at the time was three or $400 a month, per instance to run everything that they've got, which, you know, when you look at the instances that I have, costing, you know, 15, 20 bucks a month, in some cases, hmm, seems a little out of whack. And I can absolutely see that turning into an unbounded growth problem in kind of the same way. I just… I don't need to conquer the world. I'm not VC-backed. I am perfectly content at the scale that I'm at—Everett: [laugh].Corey: —with the focus on the problems that I'm focused on.Everett: Yeah, Datadog has been fascinating. It's been one of our fastest-growing providers of sort of the ‘others' category that we've launched. And I think the thing with Datadog that is interesting is you have this phrase cloud costs are all about cloud architecture and I think that's more true on Datadog than a lot of other services because if you have a model where you have, you know, thousands of hosts, and then you add-on one of Datadogs 20 services, which charges per host, suddenly your cloud bill has grown exponentially compared to probably the thing that you were after. And a similar thing happens—actually, my favorite Datadog cost recommendation is, when you have multiple endpoints, and you have sort of multiple query parameters for those endpoints, you end up in this cardinality situation where suddenly Datadog is tracking, again, like, exponentially increasing number of data points, which it's then charging to you on a usage-based model. And so, Datadog is great partners with AWS and I think it's no surprise because the two of them actually sort of go hand-in-hand in terms of the way that they… I don't want to say take ad—Corey: Extract revenue?Everett: Yeah, extract revenue. That's a good term. And, you know, you might say a similar thing about Snowflake, possibly, and the way that they do things. Like oh, the, you know, warehouse has to be on for one minute, minimum, no matter how long the query runs, and various architectural decisions that these folks make that if you were building a cost-optimized version of the service, you would probably go in the other direction.Corey: One thing that I'm also seeing, too, is that I can look at the AWS bill—and just billing data alone—and then say, “Okay, you're using Datadog, aren't you?” Like, “How did you know that?” Like, well, first, most people are secondly, CloudWatch is your number two largest service spend right now. And it's the downstream effect of hammering all the endpoints with all of the systems. And is that data you're actually using? Probably not, in some cases. It's, everyone turns on all the Datadog integrations the first time and then goes back and resets and never does it again.Everett: Yeah, I think we have this set of advice that we give Datadog folks and a lot of it is just, like, turn down the ingestion volume on your logs. Most likely, logs from 30 days ago that are correlated with some new services that you spun up—like you just talked about—are potentially not relevant anymore, for the kind of day-to-day cadence that you want to get into with your cloud spending. So yeah, I mean, I imagine when you're talking to customers, they're bringing up sort of like this interesting distinction where you may end up in a meeting room with the actual engineering team looking at the actual YAML configuration of the Datadog script, just to get a sense of like, well, what are the buttons I can press here? And so, that's… yeah, I mean, that's one reason cloud costs are a pretty interesting world is, on the surface level, you may end up buying some RIs or savings plans, but then when you really get into saving money, you end up actually changing the knobs on the services that you're talking about.Corey: That's always a fun thing when we talk to people in our sales process. It's been sord—“Are you just going to come in and tell us to buy savings plans or reserved instances?” Because the answer to that used to be, “No, that's ridiculous. That's not what we do.” But then we get into environments and find they haven't bought any of those things in 18 months.Everett: [laugh].Corey: —and it's well… okay, that's step two. Step one is what are you using you shouldn't be? Like, basically measure first then cut as opposed to going the other direction and then having to back your way into stuff. Doesn't go well.Everett: Yeah. One of the things that you were discussing last year that I thought was pretty interesting was the gp3 volumes that are now available for RDS and how those volumes, while they offer a nice discount and a nice bump in price-to-performance on EC2, actually don't offer any of that on RDS except for specific workloads. And so, I think that's the kind of thing where, as you're working with folks, as Vantage is working with people, the discussion ends up in these sort of nuanced niche areas, and that's why I think, like, these reports, hopefully, are helping people get a sense of, like, well, what's normal in my architecture or where am I sort of out of bounds? Oh, the fact that I'm spending most of my bill on NAT gateways and bandwidth egress? Well, that's not normal. That would be something that would be not typical of what your normal AWS user is doing.Corey: Right. There's always a question of, “Am I normal?” is one of the first things people love to ask. And it comes in different forms. But it's benchmarking. It's, okay, how much should it cost us to service a thousand monthly active users? It's like, there's no good way to say that across the board for everyone.Everett: Yeah. I like the model of getting into the actual unit costs. I have this sort of vision in my head of, you know, if I'm Uber and I'm reporting metrics to the public stock market, I'm actually reporting a cost to serve a rider, a cost to deliver an Uber Eats meal, in terms of my cloud spend. And that sort of data is just ridiculously hard to get to today. I think it's what we're working towards with Vantage and I think it's something that with these Cloud Cost Reports, we're hoping to get into over time, where we're actually helping companies think about well, okay, within my cloud spend, it's not just what I'm spending on these different services, there's also an idea of how much of my cost to deliver my service should be realized by my cloud spending.Corey: And then people have the uncomfortable realization that wait, my bill is less a function of number of customers I have but more the number of engineers I've hired. What's going on with that?Everett: [laugh]. Yeah, it is interesting to me just how many people end up being involved in this problem at the company. But to your earlier point, the cloud spending discussion has really ramped up over the past year. And I think, hopefully, we are going to be able to converge on a place where we are realizing the promise of the cloud, if you will, which is that it's actually cheaper. And I think what these reports show so far is, like, we've still got a long ways to go for that.Corey: One thing that I think is opportune about the timing of this recording is that as of last week, Amazon wound up announcing their earnings. And Andy Jassy has started getting on the earnings calls, which is how you know it's bad because the CEO of Amazon never deigned to show up on those things before. And he said that a lot of AWS employees are focused and spending their time on helping customers lower their AWS bills. And I'm listening to this going, “Oh, they must be talking to different customers than the ones that I'm talking to.” Are you seeing a lot of Amazonian involvement in reducing AWS bills? Because I'm not and I'm wondering where these people are hiding.Everett: So, we do see one thing, which is reps pushing savings plans on customers, which in general, is great. It's kind of good for everybody, it locks people into longer-term spend on Amazon, it gets them a lower rate, savings plans have some interesting functionality where they can be automatically applied to the area where they offer the most discount. And so, those things are all positive. I will say with Vantage, we're a cloud cost optimization company, of course, and so when folks talk to us, they often already have talked to their AWS rep. And the classic scenario is, that the rep passes over a large spreadsheet of options and ways to reduce costs, but for the company, that spreadsheet may end up being quite a ways away from the point where they actually realize cost savings.And ultimately, the people that are working on cloud cost optimization for Amazon are account reps who are comped by how much cloud spending their accounts are using on Amazon. And so, at the end of the day, some of the, I would say, most hard-hitting optimizations that you work on that we work on, end up hitting areas where they do actually reduce the bill which ends up being not in the account manager's favor. And so, it's a real chicken-and-egg game, except for savings plans is one area where I think everybody can kind of work together.Corey: I have found that… in fairness, there is some defense for Amazon in this but their cost-cutting approach has been rightsizing instances, buy some savings plans, and we are completely out of ideas. Wait, can you switch to Graviton and/or move to serverless? And I used to make fun of them for this but honestly that is some of the only advice that works across the board, irrespective in most cases, of what a customer is doing. Everything else is nuanced and it depends.That's why in some cases, I find that I'm advising customers to spend more money on certain things. Like, the reason that I don't charge percentage of savings in part is because otherwise I'm incentivized to say things like, “Backups? What are you, some kind of coward? Get rid of them.” And that doesn't seem like it's going to be in the customer's interest every time. And as soon as you start down that path, it starts getting a little weird.But people have asked me, what if my customers reach out to their account teams instead of talking to us? And it's, we do bespoke consulting engagements; I do not believe that we have ever had a client who did not first reach out to their account team. If the account teams were capable of doing this at the level that worked for customers, I would have to be doing something else with my business. It is not something that we are seeing hit customers in a way that is effective, and certainly not at scale. You said—as you were right on this—that there's an element here of account managers doing this stuff, there's an [unintelligible 00:15:54] incentive issue in part, but it's also, quality is extraordinarily uneven when it comes to these things because it is its own niche and a lot of people focus in different areas in different ways.Everett: Yeah. And to the areas that you brought up in terms of general advice that's given, we actually have some data on this in this report. In particular Graviton, this is something we've been tracking the whole time we've been doing these reports, which is the past three quarters and we actually are seeing Graviton adoption start to increase more rapidly than it was before. And so, for this last quarter Q1, we're seeing 5% of our costs that we're measuring on EC2 coming from Graviton, which is up from, I want to say 2% the previous quarter, and, like, less than 1% the quarter before. The previous quarter, we also reported that Lambda costs are now majority on ARM among the Vantage customer base.And that one makes some sense to me just because in most cases with Lambda, it's a flip of a switch. And then to your archival point on backups, this is something that we report in this one is that intelligent tiering, which we saw, like, really make an impact for folks towards the end of last year, the numbers for that were flat quarter over quarter. And so, what I mean by that is, we reported that I think, like, two-thirds of our S3 costs are still in the standard storage tier, which is the most expensive tier. And folks have enabled S3 intelligent tiering, which moves your data to progressively cheaper tiers, but we haven't seen that increase this quarter. So, it's the same number as it was last quarter.And I think speaks to what you're talking about with a ceiling on some cost optimization techniques, where it's like, you're not just going to get rid of all your backups; you're not just going to get rid of your, you know, Amazon WorkSpaces archived desktop snapshots that you need for some HIPAA compliance reason. Those things have an upper limit and so that's where, when the AWS rep comes in, it's like, as they go through the list of top spending categories, the recommendations they can give start to provide diminishing returns.Corey: I also think this is sort of a law of large numbers issue. When you start seeing a drop off in the growth rate of large cloud providers, like, there's a problem, in that there are only so many exabyte scale workloads that can be moved inside of a given quarter into the cloud. You're not going to see the same unbounded infinite growth that you would expect mathematically. And people lose their minds when they start to see those things pointed out, but the blame that oh, that's caused by cost optimization efforts, with respect, bullshit it is. I have seen customers devote significant efforts to reducing their AWS bills and it takes massive amounts of work and even then they don't always succeed in getting there.It gets better, but they still wind up a year later, having spent more on a month-by-month basis than they did when they started. Sure they understand it better and it's organic growth that's driving it and they've solved the low hanging fruit problem, but there is a challenge in acting as a boundary for what is, in effect, an unbounded growth problem.Everett: Yeah. And speaking to growth, I thought Microsoft had the most interesting take on where things could happen next quarter, and that, of course, is AI. And so, they attributed, I think it was, 1% of their guidance regarding 26 or 27% growth for Q2 Cloud revenue and it attributed 1% of that to AI. And I think Amazon is really trying to be in the room for those discussions when a large enterprise is talking about AI workloads because it's one of the few remaining cloud workloads that if it's not in the cloud already, is generating potentially massive amounts of growth for these guys.And so, I'm not really sure if I believe the 1% number. I think Microsoft may be having some fun with the fact that, of course, OpenAI is paying them for acting as a cloud provider for ChatGPT and further API, but I do think that AWS, although they were maybe a little slow to the game, they did, to their credit, launch a number of AI services that I'm excited to see if that contributes to the cost that we're measuring next quarter. We did measure, for the first time, a sudden increase on those new [Inf1 00:20:17] EC2 instances, which are optimized for machine learning. And I think if AWS can have success moving customers to those the way they have with Graviton, then that's going to be a very healthy area of growth for them.Corey: I'll also say that it's pretty clear to me that Amazon does not know what it's doing in its world of machine-learning-powered services. I use Azure for the [unintelligible 00:20:44] clients I built originally for Twitter, then for Mastodon—I'm sure Bluesky is coming—but the problem that I'm seeing there is across the board, start to finish, that there is no cohesive story from the AWS side of here's a picture tell me what's in it and if it's words, describe it to me. That's a single API call when we go to Azure. And the more that Amazon talks about something, I find, the less effective they're being in that space. And they will not stop talking about machine learning. Yes, they have instances that are powered by GPUs; that's awesome. But they're an infrastructure provider and moving up the stack is not in their DNA. But that's where all the interest and excitement and discussion is going to be increasingly in the AI space. Good luck.Everett: I think it might be something similar to what you've talked about before with all the options to run containers on AWS. I think they today have a bit of a grab bag of services and they may actually be looking forward to the fact that they're these truly foundational models which let you do a number of tasks, and so they may not need to rely so much on you know, Amazon Polly and Amazon Rekognition and sort of these task-specific services, which to date, I'm not really sure of the takeoff rates on those. We have this cloud costs leaderboard and I don't think you would find them in the top 50 of AWS services. But we'll see what happens with that.AWS I think, ends up being surprisingly good at sticking with it. I think our view is that they probably have the most customer spend on Kubernetes of any major cloud, even though you might say Google at first had the lead on Kubernetes and maybe should have done more with GKE. But to date, I would kind of agree with your take on AI services and I think Azure is… it's Azure's to lose for the moment.Corey: I would agree. I think the future of the cloud is largely Azure's to lose and it has been for a while, just because they get user experience, they get how to talk to enterprises. I just… I wish they would get security a little bit more effectively, and if failing that, communicating with their customers about security more effectively. But it's hard for a leopard to change its spots. Microsoft though has demonstrated an ability to change their nature multiple times, in ways that I would have bet were impossible. So, I just want to see them do it again. It's about time.Everett: Yeah, it's been interesting building on Azure for the past year or so. I wrote a post recently about, kind of, accessing billing data across the different providers and it's interesting in that every cloud provider is unique in the way that it simply provides an external endpoint for downloading your billing data, but Azure is probably one of the easiest integrations; it's just a REST API. However, behind that REST API are, like, years and years of different ways to pay Microsoft: are you on a pay-as-you-go plan, are you on an Azure enterprise plan? So, there's all this sort of organizational complexity hidden behind Azure and I think sometimes it rears its ugly head in a way that stringing together services on Amazon may not, even if that's still a bear in and of itself, if you will.Corey: Any other surprises that you found in the Cloud Cost Report? I mean, looking through it, it seems directionally aligned with what I see in my environments with customers. Like for example, you're not going to see Kubernetes showing up as a line item on any of these things just because—Everett: Yeah.Corey: That is indistinguishable from a billing perspective when we're looking at EC2 spend versus control plane spend. I don't tend to [find 00:24:04] too much that's shocking me. My numbers are of course, different percentage-wise, but surprise, surprise, different companies doing different things doing different percentages, I'm sure only AWS knows for sure.Everett: Yeah, I think the biggest surprise was just the—and, this could very well just be kind of measurement method, but I really expected to see AI services driving more costs, whether it was GPU instances, or AI-specific services—which we actually didn't report on at all, just because they weren't material—or just any indication that AI was a real driver of cloud spending. But I think what you see instead is sort of the same old folks at the top, and if you look at the breakdown of services across providers, that's, you know, compute, database, storage, bandwidth, monitoring. And if you look at our percentage of AI costs as a percentage of EC2 costs, it's relatively flat, quarter over quarter. So, I would have thought that would have shown up in some way in our data and we really didn't see it.Corey: It feels like there's a law of large numbers things. Everyone's talking about it. It's very hype right now—Everett: Yeah.Corey: But it's also—you talk to these companies, like, “Okay, we have four exabytes of data that we're storing and we have a couple 100,000 instances at any given point in time, so yeah, we're going to start spending $100,000 a month on our AI adventures and experiments.” It's like, that's just noise and froth in the bill, comparatively.Everett: Exactly, yeah. And so, that's why I think Microsoft's thought about AI driving a lot of growth in the coming quarters is, we'll see how that plays out, basically. The one other thing I would point to is—and this is probably not surprising, maybe, for you having been in the infrastructure world and seeing a lot of this, but for me, just seeing the length of time it takes companies to upgrade their instance cycles. We're clocking in at almost three years since the C6 series instances have been released and for just now seeing C6 and R6 start to edge above 10% of our compute usage. I actually wonder if that's just the stranglehold that Intel has on cloud computing workloads because it was only last year around re:Invent that the C6in and the Intel version of the C6 series instances had been released. So, I do think in general, there's supposed to be a price-to-performance benefit of upgrading your instances, and so sometimes it surprises me to see how long it takes companies to get around to doing that.Corey: Generation 6 to 7 is also 6% more expensive in my sampling.Everett: Right. That's right. I think Amazon has some work to do to actually make that price-to-performance argument, sort of the way that we were discussing with gp2 versus gp3 volumes. But yeah, I mean, other than that, I think, in general, my view is that we're past the worst of it, if you will, for cloud spending. Q4 was sort of a real letdown, I think, in terms of the data we had and the earnings that these cloud providers had and I think Q1 is actually everyone looking forward to perhaps what we call out at the beginning of the report, which is a return to normal spend patterns across the cloud.Corey: I think that it's going to be an interesting case. One thing that I'm seeing that might very well explain some of the reluctance to upgrade EC2 instances has been that a lot of those EC2 instances are databases. And once those things are up and running and working, people are hesitant to do too much with them. One of the [unintelligible 00:27:29] roads that I've seen of their savings plan approach is that you can migrate EC2 spend to Fargate to Lambda—and that's great—but not RDS. You're effectively leaving a giant pile of money on the table if you've made a three-year purchase commitment on these things. So, all right, we're not going to be in any rush to migrate to those things, which I think is AWS getting in its own way.Everett: That's exactly right. When we encounter customers that have a large amount of database spend, the most cost-effective option is almost always basically bare-metal EC2 even with the overhead of managing the backup-restore scalability of those things. So, in some ways, that's a good thing because it means that you can then take advantage of the, kind of, heavy committed use options on EC2, but of course, in other ways, it's a bit of a letdown because, in the ideal case, RDS would scale with the level of workloads and the economics would make more sense, but it seems that is really not the case.Corey: I really want to thank you for taking the time to come on the show and talk to me. I'll include a link in the [show notes 00:28:37] to the Cost Report. One thing I appreciate is the fact that it doesn't have one of those gates in front of it of, your email address, and what country you're in, and how can our salespeople best bother you. It's just, here's a link to the PDF. The end. So, thanks for that; it's appreciated. Where else can people go to find you?Everett: So, I'm on Twitter talking about cloud infrastructure and AI. I'm at@retttx, that's R-E-T-T-T-X. And then of course, Vantage also did quick hot-takes on this report with a series of graphs and explainers in a Twitter thread and that's @JoinVantage.Corey: And we will, of course, put links to that in the [show notes 00:29:15]. Thank you so much for your time. I appreciate it.Everett: Thanks, Corey. Great to chat.Corey: Everett Berry, growth in open-source at Vantage. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment that will increase its vitriol generation over generation, by approximately 6%.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Thanks for tuning in to RealAg on the Weekend! On this episode, host Shaun Haney is joined by Kody Blois, chair of standing committee on agriculture to discuss recommendations for supporting Canadian agriculture. As well as Carmen Prang, Saskatchewan Wheat to talk about seed testing and an update on the use of Lambda-cy products. We... Read More

Liz Rice, Chief Open Source Officer at Isovalent, joins Corey on Screaming in the Cloud to discuss the release of her newest book, Learning eBPF, and the exciting possibilities that come with eBPF technology. Liz explains what got her so excited about eBPF technology, and what it was like to write a book while also holding a full-time job. Corey and Liz also explore the learning curve that comes with kernel programming, and Liz illustrates why it's so important to be able to explain complex technologies in simple terminology. About LizLiz Rice is Chief Open Source Officer with eBPF specialists Isovalent, creators of the Cilium cloud native networking, security and observability project. She sits on the CNCF Governing Board, and on the Board of OpenUK. She was Chair of the CNCF's Technical Oversight Committee in 2019-2022, and Co-Chair of KubeCon + CloudNativeCon in 2018. She is also the author of Container Security, and Learning eBPF, both published by O'Reilly.She has a wealth of software development, team, and product management experience from working on network protocols and distributed systems, and in digital technology sectors such as VOD, music, and VoIP. When not writing code, or talking about it, Liz loves riding bikes in places with better weather than her native London, competing in virtual races on Zwift, and making music under the pseudonym Insider Nine.Links Referenced: Isovalent: https://isovalent.com/ Learning eBPF: https://www.amazon.com/Learning-eBPF-Programming-Observability-Networking/dp/1098135121 Container Security: https://www.amazon.com/Container-Security-Fundamental-Containerized-Applications/dp/1492056707/ GitHub for Learning eBPF: https://github.com/lizRice/learning-eBPF TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Our returning guest today is Liz Rice, who remains the Chief Open Source Officer with Isovalent. But Liz, thank you for returning, suspiciously closely timed to when you have a book coming out. Welcome back.Liz: [laugh]. Thanks so much for having me. Yeah, I've just—I've only had the physical copy of the book in my hands for less than a week. It's called Learning eBPF. I mean, obviously, I'm very excited.Corey: It's an O'Reilly book; it has some form of honeybee on the front of it as best I can tell.Liz: Yeah, I was really pleased about that. Because eBPF has a bee as its logo, so getting a [early 00:01:17] honeybee as the O'Reilly animal on the front cover of the book was pretty pleasing, yeah.Corey: Now, this is your second O'Reilly book, is it not?Liz: It's my second full book. So, I'd previously written a book on Container Security. And I've done a few short reports for them as well. But this is the second, you know, full-on, you can buy it on Amazon kind of book, yeah.Corey: My business partner wrote Practical Monitoring for O'Reilly and that was such an experience that he got entirely out of observability as a field and ran running to AWS bills as a result. So, my question for you is, why would anyone do that more than once?Liz: [laugh]. I really like explaining things. And I had a really good reaction to the Container Security book. I think already, by the time I was writing that book, I was kind of interested in eBPF. And we should probably talk about what that is, but I'll come to that in a moment.Yeah, so I've been really interested in eBPF, for quite a while and I wanted to be able to do the same thing in terms of explaining it to people. A book gives you a lot more opportunity to go into more detail and show people examples and get them kind of hands-on than you can do in their, you know, 40-minute conference talk. So, I wanted to do that. I will say I have written myself a note to never do a full-size book while I have a full-time job because it's a lot [laugh].Corey: You do have a full-time job and then some. As we mentioned, you're the Chief Open Source Officer over at Isovalent, you are on the CNCF governing board, you're on the board of OpenUK, and you've done a lot of other stuff in the open-source community as well. So, I have to ask, taking all of that together, are you just allergic to things that make money? I mean, writing the book as well on top of that. I'm told you never do it for the money piece; it's always about the love of it. But it seems like, on some level, you're taking it to an almost ludicrous level.Liz: Yeah, I mean, I do get paid for my day job. So, there is that [laugh]. But so, yeah—Corey: I feel like that's the only way to really write a book is, in turn, to wind up only to just do it for—what someone else is paying you to for doing it, viewing it as a marketing exercise. It pays dividends, but those dividends don't, in my experience from what I've heard from everyone say, pay off as of royalties on book payments.Liz: Yeah, I mean, it's certainly, you know, not a bad thing to have that income stream, but it certainly wouldn't make you—you know, I'm not going to retire tomorrow on the royalty stream unless this podcast has loads and loads of people to buy the book [laugh].Corey: Exactly. And I'm always a fan of having such [unintelligible 00:03:58]. I will order it while we're on the call right now having this conversation because I believe in supporting the things that we want to see more of in the world. So, explain to me a little bit about what it is. Whatever you talking about learning X in a title, I find that that's often going to be much more approachable than arcane nonsense deep-dive things.One of the O'Reilly books that changed my understanding was Linux Kernel Internals, or Understanding the Linux Kernel. Understanding was kind of a heavy lift at that point because it got very deep very quickly, but I absolutely came away understanding what was going on a lot more effectively, even though I was so slow I needed a tow rope on some of it. When you have a book that started with learning, though, I imagined it assumes starting at zero with, “What's eBPF?” Is that directionally correct, or does it assume that you know a lot of things you don't?Liz: Yeah, that's absolutely right. I mean, I think eBPF is one of these technologies that is starting to be, particularly in the cloud-native world, you know, it comes up; it's quite a hot technology. What it actually is, so it's an acronym, right? EBPF. That acronym is almost meaningless now.So, it stands for extended Berkeley Packet Filter. But I feel like it does so much more than filtering, we might as well forget that altogether. And it's just become a term, a name in its own right if you like. And what it really does is it lets you run custom programs in the kernel so you can change the way that the kernel behaves, dynamically. And that is… it's a superpower. It's enabled all sorts of really cool things that we can do with that superpower.Corey: I just pre-ordered it as a paperback on Amazon and it shows me that it is now number one new release in Linux Networking and Systems Administration, so you're welcome. I'm sure it was me that put it over the top.Liz: Wonderful. Thank you very much. Yeah [laugh].Corey: Of course, of course. Writing a book is one of those things that I've always wanted to do, but never had the patience to sit there and do it or I thought I wasn't prolific enough, but over the holidays, this past year, my wife and business partner and a few friends all chipped in to have all of the tweets that I'd sent bound into a series of leather volumes. Apparently, I've tweeted over a million words. And… yeah, oh, so I have to write a book 280 characters at a time, mostly from my phone. I should tweet less was really the takeaway that I took from a lot of that.But that wasn't edited, that wasn't with an overall theme or a narrative flow the way that an actual book is. It just feels like a term paper on steroids. And I hated term papers. Love reading; not one to write it.Liz: I don't know whether this should make it into the podcast, but it reminded me of something that happened to my brother-in-law, who's an artist. And he put a piece of video on YouTube. And for unknowable reasons if you mistyped YouTube, and you spelt it, U-T-U-B-E, the page that you would end up at from Google search was a YouTube video and it was in fact, my brother-in-law's video. And people weren't expecting to see this kind of art movie about matches burning. And he just had the worst comment—like, people were so mean in the comments. And he had millions of views because people were hitting this page by accident, and he ended up—Corey: And he made the cardinal sin of never read the comments. Never break that rule. As soon as you do that, it doesn't go well. I do read the comments on various podcast platforms on this show because I always tell people to insulted all they want, just make sure you leave a five-star review.Liz: Well, he ended up publishing a book with these comments, like, one comment per page, and most of them are not safe for public consumption comments, and he just called it Feedback. It was quite something [laugh].Corey: On some level, it feels like O'Reilly books are a little insulated from the general population when it comes to terrible nonsense comments, just because they tend to be a little bit more expensive than the typical novel you'll see in an airport bookstore, and again, even though it is approachable, Learning eBPF isn't exactly the sort of title that gets people to think that, “Ooh, this is going to be a heck of a thriller slash page-turner with a plot.” “Well, I found the protagonist unrelatable,” is not sort of the thing you're going to wind up seeing in the comments because people thought it was going to be something different.Liz: I know. One day, I'm going to have to write a technical book that is also a murder mystery. I think that would be, you know, quite an achievement. But yeah, I mean, it's definitely aimed at people who have already come across the term, want to know more, and particularly if you're the kind of person who doesn't want to just have a hand-wavy explanation that involves boxes and diagrams, but if, like me, you kind of want to feel the code, and you want to see how things work and you want to work through examples, then that's the kind of person who might—I hope—enjoy working through the book and end up with a possible mental model of how eBPF works, even though it's essentially kernel programming.Corey: So, I keep seeing eBPF in an increasing number of areas, a bunch of observability tools, a bunch of security tools all tend to tie into it. And I've seen people do interesting things as far as cost analysis with it. The problem that I run into is that I'm not able to wind up deploying it universally, just because when I'm going into a client engagement, I am there in a purely advisory sense, given that I'm biasing these days for both SaaS companies and large banks, that latter category is likely going to have some problems if I say, “Oh, just take this thing and go ahead and deploy it to your entire fleet.” If they don't have a problem with that, I have a problem with their entire business security posture. So, I don't get to be particularly prescriptive as far as what to do with it.But if I were running my own environment, it is pretty clear by now that I would have explored this in some significant depth. Do you find that it tends to be something that is used primarily in microservices environments? Does it effectively require Kubernetes to become useful on day one? What is the onboard path where people would sit back and say, “Ah, this problem I'm having, eBPF sounds like the solution.”Liz: So, when we write tools that are typically going to be some sort of infrastructure, observability, security, networking tools, if we're writing them using eBPF, we're instrumenting the kernel. And the kernel gets involved every time our application wants to do anything interesting because whenever it wants to read or write to a file, or send receive network messages, or write something to the screen, or allocate memory, or all of these things, the kernel has to be involved. And we can use eBPF to instrument those events and do interesting things. And the kernel doesn't care whether those processes are running in containers, under Kubernetes, just running directly on the host; all of those things are visible to eBPF.So, in one sense, doesn't matter. But one of the reasons why I think we're seeing eBPF-based tools really take off in cloud-native is that you can, by applying some programming, you can link events that happened in the kernel to specific containers in specific pods in whatever namespace and, you know, get the relationship between an event and the Kubernetes objects that are involved in that event. And then that enables a whole lot of really interesting observability or security tools and it enables us to understand how network packets are flowing between different Kubernetes objects and so on. So, it's really having this vantage point in the kernel where we can see everything and we didn't have to change those applications in any way to be able to use eBPF to instrument them.Corey: When I see the stories about eBPF, it seems like it's focused primarily on networking and flow control. That's where I'm seeing it from a security standpoint, that's where I'm seeing it from cost allocation aspect. Because, frankly, out of the box, from a cloud provider's perspective, Kubernetes looks like a single-tenant application with a really weird behavioral pattern, and some of that crosstalk gets very expensive. Is there a better way than either using eBPF and/or VPC flow logs to figure out what's talking to what in the Kubernetes ecosystem, or is BPF really your first port of call?Liz: So, I'm coming from a position of perspective of working for the company that created the Cilium networking project. And one of the reasons why I think Cilium is really powerful is because it has this visibility—it's got a component called Hubble—that allows you to see exactly how packets are flowing between these different Kubernetes identities. So, in a Kubernetes environment, there's not a lot of point having network flows that talk about IP addresses and ports when what you really want to know is, what's the Kubernetes namespace, what's the application? Defining things in terms of IP addresses makes no sense when they're just being refreshed and renewed every time you change pods. So yeah, Kubernetes changes the requirements on networking visibility and on firewalling as well, on network policy, and that, I think, is you don't have to use eBPF to create those tools, but eBPF is a really powerful and efficient platform for implementing those tools, as we see in Cilium.Corey: The only competitor I found to it that gives a reasonable explanation of why random things are transferring multiple petabytes between each other in the middle of the night has been oral tradition, where I'm talking to people who've been around there for a while. It's, “So, I'm seeing this weird traffic pattern at these times a day. Any idea what that might be?” And someone will usually perk up and say, “Oh, is it—” whatever job that they're doing. Great. That gives me a direction to go in.But especially in this era of layoffs and as environments exist for longer and longer, you have to turn into a bit of a data center archaeologist. That remains insufficient, on some level. And some level, I'm annoyed with trying to understand or needing to use tooling like this that is honestly this powerful and this customizable, and yes, on some level, this complex in order to get access to that information in a meaningful sense. But on the other, I'm glad that that option is at least there for a lot of workloads.Liz: Yeah. I think, you know, that speaks to the power of this new generation of tooling. And the same kind of applies to security forensics, as well, where you might have an enormous stream of events, but unless you can tie those events back to specific Kubernetes identities, which you can use eBPF-based tooling to do, then how do you—the forensics job of tying back where did that event come from, what was the container that was compromised, it becomes really, really difficult. And eBPF tools—like Cilium has a sub-project called Tetragon that is really good at this kind of tying events back to the Kubernetes pod or whether we want to know what node it was running on what namespace or whatever. That's really useful forensic information.Corey: Talk to me a little bit about how broadly applicable it is. Because from my understanding from our last conversation, when you were on the show a year or so ago, if memory serves, one of the powerful aspects of it was very similar to what I've seen some of Brendan Gregg's nonsense doing in his kind of various talks where you can effectively write custom programming on the fly and it'll tell you exactly what it is that you need. Is this something that can be instrument once and then effectively use it for basically anything, [OTEL 00:16:11]-style, or instead, does it need to be effectively custom configured every time you want to get a different aspect of information out of it?Liz: It can be both of those things.Corey: “It depends.” My least favorite but probably the most accurate answer to hear.Liz: [laugh]. But I think Brendan did a really great—he's done many talks talking about how powerful BPF is and built lots of specific tools, but then he's also been involved with Bpftrace, which is kind of like a language for—a high-level language for saying what it is that you want BPF to trace out for you. So, a little bit like, I don't know, awk but for events, you know? It's a scripting language. So, you can have this flexibility.And with something like Bpftrace, you don't have to get into the weeds yourself and do kernel programming, you know, in eBPF programs. But also there's gainful employment to be had for people who are interested in that eBPF kernel programming because, you know, I think there's just going to be a whole range of more tools to come, you know>? I think we're, you know, we're seeing some really powerful tools with Cilium and Pixie and [Parker 00:17:27] and Kepler and many other tools and projects that are using eBPF. But I think there's also a whole load of more to come as people think about different ways they can apply eBPF and instrument different parts of an overall system.Corey: We're doing this over audio only, but behind me on my wall is one of my least favorite gifts ever to have been received by anyone. Mike, my business partner, got me a thousand-piece puzzle of the Kubernetes container landscape where—Liz: [laugh].Corey: This diagram is psychotic and awful and it looks like a joke, except it's not. And building that puzzle was maddening—obviously—but beyond that, it was a real primer in just how vast the entire container slash Kubernetes slash CNCF landscape really is. So, looking at this, I found that the only reaction that was appropriate was a sense of overwhelmed awe slash frustration, I guess. It's one of those areas where I spend a lot of time focusing on drinking from the AWS firehose because they have a lot of products and services because their product strategy is apparently, “Yes,” and they're updating these things in a pretty consistent cadence. Mostly. And even that feels like it's multiple full-time jobs shoved into one.There are hundreds of companies behind these things and all of them are in areas that are incredibly complex and difficult to go diving into. EBPF is incredibly powerful, I would say ridiculously so, but it's also fiendishly complex, at least shoulder-surfing behind people who know what they're doing with it has been breathtaking, on some level. How do people find themselves in a situation where doing a BPF deep dive make sense for them?Liz: Oh, that's a great question. So, first of all, I'm thinking is there an AWS Jigsaw as well, like the CNCF landscape Jigsaw? There should be. And how many pieces would it have? [It would be very cool 00:19:28].Corey: No, because I think the CNCF at one point hired a graphic designer and it's unclear that AWS has done such a thing because their icons for services are, to be generous here, not great. People have flashcards that they've built for is what services does logo represent? Haven't a clue, in almost every case because I don't care in almost every case. But yeah, I've toyed with the idea of doing it. It's just not something that I'd ever want to have my name attached to it, unfortunately. But yeah, I want someone to do it and someone else to build it.Liz: Yes. Yeah, it would need to refresh every, like, five minutes, though, as they roll out a new service.Corey: Right. Because given that it appears from the outside to be impenetrable, it's similar to learning VI in some cases, where oh, yeah, it's easy to get started with to do this trivial thing. Now, step two, draw the rest of the freaking owl. Same problem there. It feels off-putting just from a perspective of you must be at least this smart to proceed. How do you find people coming to it?Liz: Yeah, there is some truth in that, in that beyond kind of Hello World, you quite quickly start having to do things with kernel data structures. And as soon as you're looking at kernel data structures, you have to sort of understand, you know, more about the kernel. And if you change things, you need to understand the implications of those changes. So, yeah, you can rapidly say that eBPF programming is kernel programming, so why would anybody want to do it? The reason why I do it myself is not because I'm a kernel programmer; it's because I wanted to really understand how this is working and build up a mental model of what's happening when I attach a program to an event. And what kinds of things can I do with that program?And that's the sort of exploration that I think I'm trying to encourage people to do with the book. But yes, there is going to be at some point, a pretty steep learning curve that's kernel-related but you don't necessarily need to know everything in order to really have a decent understanding of what eBPF is, and how you might, for example—you might be interested to see what BPF programs are running on your existing system and learn why and what they might be doing and where they're attached and what use could that be.Corey: Falling down that, looking at the process table once upon a time was a heck of an education, one week when I didn't have a lot to do and I didn't like my job in those days, where, “Oh, what is this Avahi daemon that constantly running? MDNS forwarding? Who would need that?” And sure enough, that tickled something in the back of my mind when I wound up building out my networking box here on top of BSD, and oh, yeah, I want to make sure that I can still have discovery work from the IoT subnet over to whatever it is that my normal devices live. Ah, that's what that thing always running for. Great for that one use case. Almost never needed in other cases, but awesome. Like, you fire up a Raspberry Pi. It's, “Why are all these things running when I'm just want to have an embedded device that does exactly one thing well?” Ugh. Computers have gotten complicated.Liz: I know. It's like when you get those pop-ups on—well certainly on Mac, and you get pop-ups occasionally, let's say there's such and such a daemon wants extra permissions, and you think I'm not hitting that yes button until I understand what that daemon is. And it turns out, it's related, something completely innocuous that you've actually paid for, but just under a different name. Very annoying. So, if you have some kind of instrumentation like tracing or logging or security tooling that you want to apply to all of your containers, one of the things you can use is a sidecar container approach. And in Kubernetes, that means you inject the sidecar into every single pod. And—Corey: Yes. Of course, the answer to any Kubernetes problem appears to be have you tried running additional containers?Liz: Well, right. And there are challenges that can come from that. And one of the reasons why you have to do that is because if you want a tool that has visibility over that container that's inside the pod, well, your instrumentation has to also be inside the pod so that it has visibility because your pod is, by design, isolated from the host it's running on. But with eBPF, well eBPF is in the kernel and there's only one kernel, however many containers were running. So, there is no kind of isolation between the host and the containers at the kernel level.So, that means if we can instrument the kernel, we don't have to have a separate instance in every single pod. And that's really great for all sorts of resource usage, it means you don't have to worry about how you get those sidecars into those pods in the first place, you know that every pod is going to be instrumented if it's instrumented in the kernel. And then for service mesh, service mesh usually uses a sidecar as a Layer 7 Proxy injected into every pod. And that actually makes for a pretty convoluted networking path for a packet to sort of go from the application, through the proxy, out to the host, back into another pod, through another proxy, into the application.What we can do with eBPF, we still need a proxy running in userspace, but we don't need to have one in every single pod because we can connect the networking namespaces much more efficiently. So, that was essentially the basis for sidecarless service mesh, which we did in Cilium, Istio, and now we're using a similar sort of approach with Ambient Mesh. So that, again, you know, avoiding having the overhead of a sidecar in every pod. So that, you know, seems to be the way forward for service mesh as well as other types of instrumentation: avoiding sidecars.Corey: On some level, avoiding things that are Kubernetes staples seems to be a best practice in a bunch of different directions. It feels like it's an area where you start to get aligned with the idea of service meesh—yes, that's how I pluralize the term service mesh and if people have a problem with that, please, it's imperative you've not send me letters about it—but this idea of discovering where things are in a variety of ways within a cluster, where things can talk to each other, when nothing is deterministically placed, it feels like it is screaming out for something like this.Liz: And when you think about it, Kubernetes does sort of already have that at the level of a service, you know? Services are discoverable through native Kubernetes. There's a bunch of other capabilities that we tend to associate with service mesh like observability or encrypted traffic or retries, that kind of thing. But one of the things that we're doing with Cilium, in general, is to say, but a lot of this is just a feature of the networking, the underlying networking capability. So, for example, we've got next generation mutual authentication approach, which is using SPIFFE IDs between an application pod and another application pod. So, it's like the equivalent of mTLS.But the certificates are actually being passed into the kernel and the encryption is happening at the kernel level. And it's a really neat way of saying we don't need… we don't need to have a sidecar proxy in every pod in order to terminate those TLS connections on behalf of the application. We can have the kernel do it for us and that's really cool.Corey: Yeah, at some level, I find that it still feels weird—because I'm old—to have this idea of one shared kernel running a bunch of different containers. I got past that just by not requiring that [unintelligible 00:27:32] workloads need to run isolated having containers run on the same physical host. I found that, for example, running some stuff, even in my home environment for IoT stuff, things that I don't particularly trust run inside of KVM on top of something as opposed to just running it as a container on a cluster. Almost certainly stupendous overkill for what I'm dealing with, but it's a good practice to be in to start thinking about this. To my understanding, this is part of what AWS's Firecracker project starts to address a bit more effectively: fast provisioning, but still being able to use different primitives as far as isolation boundaries go. But, on some level, it's nice to not have to think about this stuff, but that's dangerous.Liz: [laugh]. Yeah, exactly. Firecracker is really nice way of saying, “Actually, we're going to spin up a whole VM,” but we don't ne—when I say ‘whole VM,' we don't need all of the things that you normally get in a VM. We can get rid of a ton of things and just have the essentials for running that Lambda or container service, and it becomes a really nice lightweight solution. But yes, that will have its own kernel, so unlike, you know, running multiple kernels on the same VM where—sorry, running multiple containers on the same virtual machine where they would all be sharing one kernel, with Firecracker you'll get a kernel per instance of Firecracker.Corey: The last question I have for you before we wind up wrapping up this episode harkens back to something you said a little bit earlier. This stuff is incredibly technically nuanced and deep. You clearly have a thorough understanding of it, but you also have what I think many people do not realize is an orthogonal skill of being able to articulate and explain those complex concepts simply an approachably, in ways that make people understand what it is you're talking about, but also don't feel like they're being spoken to in a way that's highly condescending, which is another failure mode. I think it is not particularly well understood, particularly in the engineering community, that there are—these are different skill sets that do not necessarily align congruently. Is this something you've always known or is this something you've figured out as you've evolved your career that, oh I have a certain flair for this?Liz: Yeah, I definitely didn't always know it. And I started to realize it based on feedback that people have given me about talks and articles I'd written. I think I've always felt that when people use jargon or they use complicated language or they, kind of, make assumptions about how things are, it quite often speaks to them not having a full understanding of what's happening. If I want to explain something to myself, I'm going to use straightforward language to explain it to myself [laugh] so I can hold it in my head. And I think people appreciate that.And you can get really—you know, you can get quite in-depth into something if you just start, step by step, build it up, explain everything as you go along the way. And yeah, I think people do appreciate that. And I think people, if they get lost in jargon, it doesn't help anybody. And yeah, I very much appreciate it when people say that, you know, they saw a talk or they read something I wrote and it meant that they finally grokked whatever that concept was that that I was trying to explain. I will say at the weekend, I asked ChatGPT to explain DNS in the style of Liz Rice, and it started off, it was basically, “Hello there. I'm Liz Rice and I'm here to explain DNS in very simple terms.” I thought, “Okay.” [laugh].Corey: Every time I think I've understood DNS, there's another level to it.Liz: I'm pretty sure there is a lot about DNS that I don't understand, yeah. So, you know, there's always more to learn out there.Corey: There's certainly is. I really want to thank you for taking time to speak with me today about what you're up to. Where's the best place for people to find you to learn more? And of course, to buy the book.Liz: Yeah, so I am Liz Rice pretty much everywhere, all over the internet. There is a GitHub repo that accompanies the books that you can find that on GitHub: lizRice/learning-eBPF. So, that's a good place to find some of the example code, and it will obviously link to where you can download the book or buy it because you can pay for it; you can also download it from Isovalent for the price of your contact details. So, there are lots of options.Corey: Excellent. And we will, of course, put links to that in the [show notes 00:32:08]. Thank you so much for your time. It's always great to talk to you.Liz: It's always a pleasure, so thanks very much for having me, Corey.Corey: Liz Rice, Chief Open Source Officer at Isovalent. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you have somehow discovered this episode by googling for knitting projects.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Ya disponible el podcast.Para descargarlo pincha aquí.Programa: 16x13Fecha de emisión: 28/04/23Duración: 2h15'17''Peso: 185,8MBEsta vez os traemos uno de los juegos narrativos más relevantes de 2022 con el análisis del título independiente, Norco. En el Debate hemos reflexionado sobre el futuro de la saga Resident Evil.Esperamos que hayáis disfrutado del programa. Como siempre, volveremos con muchos más videojuegos y nuestro toque característico.¡No os lo perdáis!

Welcome to the newest episode of The Cloud Pod podcast! Justin, Ryan and Matthew are your hosts this week as we discuss all the latest news and announcements in the world of the cloud and AI. Do people really love Matt's Azure know-how? Can Google make Bard fit into literally everything they make? What's the latest with Azure AI and their space collaborations? Let's find out! Titles we almost went with this week: Clouds in Space, Fictional Realms of Oracles, Oh My.  The cloudpod streams lambda to the cloud A big thanks to this week's sponsor:  Foghorn Consulting, provides top-notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you have trouble hiring?  Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week.

Jean Yang, CEO of Akita Software, joins Corey on Screaming in the Cloud to discuss how she went from academia to tech founder, and what her company is doing to improve monitoring and observability. Jean explains why Akita is different from other observability & monitoring solutions, and how it bridges the gap from what people know they should be doing and what they actually do in practice. Corey and Jean explore why the monitoring and observability space has been so broken, and why it's important for people to see monitoring as a chore and not a hobby. Jean also reveals how she took a leap from being an academic professor to founding a tech start-up. About JeanJean Yang is the founder and CEO of Akita Software, providing the fastest time-to-value for API monitoring. Jean was previously a tenure-track professor in Computer Science at Carnegie Mellon University.Links Referenced: Akita Software: https://www.akitasoftware.com/ Aki the dog chatbot: https://www.akitasoftware.com/blog-posts/we-built-an-exceedingly-polite-ai-dog-that-answers-questions-about-your-apis Twitter: https://twitter.com/jeanqasaur TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is someone whose company has… well, let's just say that it has piqued my interest. Jean Yang is the CEO of Akita Software and not only is it named after a breed of dog, which frankly, Amazon service namers could take a lot of lessons from, but it also tends to approach observability slash monitoring from a perspective of solving the problem rather than preaching a new orthodoxy. Jean, thank you for joining me.Jean: Thank you for having me. Very excited.Corey: In the world that we tend to operate in, there are so many different observability tools, and as best I can determine observability is hipster monitoring. Well, if we call it monitoring, we can't charge you quite as much money for it. And whenever you go into any environment of significant scale, we pretty quickly discover that, “What monitoring tool are you using?” The answer is, “Here are the 15 that we use.” Then you talk to other monitoring and observability companies and ask them which ones of those they've replace, and the answer becomes, “We're number 16.” Which is less compelling of a pitch than you might expect. What does Akita do? Where do you folks start and stop?Jean: We want to be—at Akita—your first stop for monitoring and we want to be all of the monitoring, you need up to a certain level. And here's the motivation. So, we've talked with hundreds, if not thousands, of software teams over the last few years and what we found is there is such a gap between best practice, what people think everybody else is doing, what people are talking about at conferences, and what's actually happening in software teams. And so, what software teams have told me over and over again, is, hey, we either don't actually use very many tools at all, or we use 15 tools in name, but it's you know, one [laugh] one person on the team set this one up, it's monitoring one of our endpoints, we don't even know which one sometimes. Who knows what the thresholds are really supposed to be. We got too many alerts one day, we turned it off.But there's very much a gap between what people are saying they're supposed to do, what people in their heads say they're going to do next quarter or the quarter after that and what's really happening in practice. And what we saw was teams are falling more and more into monitoring debt. And so effectively, their customers are becoming their monitoring and it's getting harder to catch up. And so, what Akita does is we're the fastest, easiest way for teams to quickly see what endpoints you have in your system—so that's API endpoints—what's slow and what's throwing errors. And you might wonder, okay, wait, wait, wait, Jean. Monitoring is usually about, like, logs, metrics, and traces. I'm not used to hearing about API—like, what do APIs have to do with any of it?And my view is, look, we want the most simple form of what might be wrong with your system, we want a developer to be able to get started without having to change any code, make any annotations, drop in any libraries. APIs are something you can watch from the outside of a system. And when it comes to which alerts actually matter, where do you want errors to be alerts, where do you want thresholds to really matter, my view is, look, the places where your system interfaces with another system are probably where you want to start if you've really gotten nothing. And so, Akita view is, we're going to start from the outside in on this monitoring. We're turning a lot of the views on monitoring and observability on its head and we just want to be the tool that you reach for if you've got nothing, it's middle of the night, you have alerts on some endpoint, and you don't want to spend a few hours or weeks setting up some other tool. And we also want to be able to grow with you up until you need that power tool that many of the existing solutions out there are today.Corey: It feels like monitoring is very often one of those reactive things. I come from the infrastructure world, so you start off with, “What do you use for monitoring?” “Oh, we wait till the help desk calls us and users are reporting a problem.” Okay, that gets you somewhere. And then it becomes oh, well, what was wrong that time? The drive filled up. Okay, so we're going to build checks in that tell us when the drives are filling up.And you wind up trying to enumerate all of the different badness. And as a result, if you leave that to its logical conclusion, one of the stories that I heard out of MySpace once upon a time—which dates me somewhat—is that you would have a shift, so there were three shifts working around the clock, and each one would open about 5000 tickets, give or take, for the monitoring alerts that wound up firing off throughout their infrastructure. At that point, it's almost, why bother? Because no one is going to be around to triage these things; no one is going to see any of the signal buried and all of that noise. When you talk about doing this for an API perspective, are you running synthetics against those APIs? Are you shimming them in order to see what's passing through them? What's the implementation side look like?Jean: Yeah, that's a great question. So, we're using a technology called BPF, Berkeley Packet Filter. The more trendy, buzzy term is EBPF—Corey: The EBPF. Oh yes.Jean: Yeah, Extended Berkeley Packet Filter. But here's the secret, we only use the BPF part. It's actually a little easier for users to install. The E part is, you know, fancy and often finicky. But um—Corey: SEBPF then: Shortened Extended BPF. Why not?Jean: [laugh]. Yeah. And what BPF allows us to do is passively watch traffic from the outside of a system. So, think of it as you're sending API calls across the network. We're just watching that network. We're not in the path of that traffic. So, we're not intercepting the traffic in any way, we're not creating any additional overhead for the traffic, we're not slowing it down in any way. We're just sitting on the side, we're watching all of it, and then we're taking that and shipping an obfuscated version off to our cloud, and then we're giving you analytics on that.Corey: One of the things that strikes me as being… I guess, a common trope is there are a bunch of observability solutions out there that offer this sort of insight into what's going on within an environment, but it's, “Step one: instrument with some SDK or some agent across everything. Do an entire deploy across your fleet.” Which yeah, people are not generally going to be in a hurry to sign up for. And further, you also said a minute ago that the idea being that someone could start using this in the middle of the night in the middle of an outage, which tells me that it's not, “Step one: get the infrastructure sparkling. Step two: do a global deploy to everything.” How do you go about doing that? What is the level of embeddedness into the environment?Jean: Yeah, that's a great question. So, the reason we chose BPF is I wanted a completely black-box solution. So, no SDKs, no code annotations. I wanted people to be able to change a config file and have our solution apply to anything that's on the system. So, you could add routes, you could do all kinds of things. I wanted there to be no additional work on the part of the developer when that happened.And so, we're not the only solution that uses BPF or EBPF. There's many other solutions that say, “Hey, just drop us in. We'll let you do anything you want.” The big difference is what happens with the traffic once it gets processed. So, what EBPF or BPF gives you is it watches everything about your system. And so, you can imagine that's a lot of different events. That's a lot of things.If you're trying to fix an incident in the middle of the night and someone just dumps on you 1000 pages of logs, like, what are you going to do with that? And so, our view is, the more interesting and important and valuable thing to do here is not make it so that you just have the ability to watch everything about your system but to make it so that developers don't have to sift through thousands of events just to figure out what went wrong. So, we've spent years building algorithms to automatically analyze these API events to figure out, first of all, what are your endpoints? Because it's one thing to turn on something like Wireshark and just say, okay, here are the thousand API calls, I saw—ten thousand—but it's another thing to say, “Hey, 500 of those were actually the same endpoint and 300 of those had errors.” That's quite a hard problem.And before us, it turns out that there was no other solution that even did that to the level of being able to compile together, “Here are all the slow calls to an endpoint,” or, “Here are all of the erroneous calls to an endpoint.” That was blood, sweat, and tears of developers in the night before. And so, that's the first major thing we do. And then metrics on top of that. So, today we have what's slow, what's throwing errors. People have asked us for other things like show me what happened after I deployed. Show me what's going on this week versus last week. But now that we have this data set, you can imagine there's all kinds of questions we can now start answering much more quickly on top of it.Corey: One thing that strikes me about your site is that when I go to akitasoftware.com, you've got a shout-out section at the top. And because I've been doing this long enough where I find that, yeah, you work at a company; you're going to say all kinds of wonderful, amazing aspirational things about it, and basically because I have deep-seated personality disorders, I will make fun of those things as my default reflexive reaction. But something that AWS, for example, does very well is when they announce something ridiculous on stage at re:Invent, I make fun of it, as is normal, but then they have a customer come up and say, “And here's the expensive, painful problem that they solved for us.”And that's where I shut up and start listening. Because it's a very different story to get someone else, who is presumably not being paid, to get on stage and say, “Yeah, this solved a sophisticated, painful problem.” Your shout-outs page has not just a laundry list of people saying great things about it, but there are former folks who have been on the show here, people I know and trust: Scott Johnson over at Docker, Gergely Orosz over at The Pragmatic Engineer, and other folks who have been luminaries in the space for a while. These are not the sort of people that are going to say, “Oh, sure. Why not? Oh, you're going to send me a $50 gift card in a Twitter DM? Sure I'll say nice things,” like it's one of those respond to a viral tweet spamming something nonsense. These are people who have gravitas. It's clear that there's something you're building that is resonating.Jean: Yeah. And for that, they found us. Everyone that I've tried to bribe to say good things about us actually [laugh] refused.Corey: Oh, yeah. As it turns out that it's one of those things where people are more expensive than you might think. It's like, “What, you want me to sell my credibility down the road?” Doesn't work super well. But there's something like the unsolicited testimonials that come out of, this is amazing, once people start kicking the tires on it.You're currently in open beta. So, I guess my big question for you is, whenever you see a product that says, “Oh, yeah, we solve everything cloud, on-prem, on physical instances, on virtual machines, on Docker, on serverless, everything across the board. It's awesome.” I have some skepticism on that. What is your ideal application architecture that Akita works best on? And what sort of things are you a complete nonstarter for?Jean: Yeah, I'll start with a couple of things we work well on. So, container platforms. We work relatively well. So, that's your Fargate, that's your Azure Web Apps. But that, you know, things running, we call them container platforms. Kubernetes is also something that a lot of our users have picked us up and had success with us on. I will say our Kubernetes deploy is not as smooth as we would like. We say, you know, you can install us—Corey: Well, that is Kubernetes, yes.Jean: [laugh]. Yeah.Corey: Nothing in Kubernetes is as smooth as we would like.Jean: Yeah, so we're actually rolling out Kubernetes injection support in the next couple of weeks. So, those are the two that people have had the most success on. If you're running on bare metal or on a VM, we work, but I will say that you have to know your way around a little bit to get that to work. What we don't work on is any Platform as a Service. So, like, a Heroku, a Lambda, a Render at the moment. So those, we haven't found a way to passively listen to the network traffic in a good way right now.And we also work best for unencrypted HTTP REST traffic. So, if you have encrypted traffic, it's not a non-starter, but you need to fall into a couple of categories. You either need to be using Kubernetes, you can run Akita as a sidecar, or you're using Nginx. And so, that's something we're still expanding support on. And we do not support GraphQL or GRPC at the moment.Corey: That's okay. Neither do I. It does seem these days that unencrypted HTTP API calls are increasingly becoming something of a relic, where folks are treating those as anti-patterns to be stamped out ruthlessly. Are you still seeing significant deployments of unencrypted APIs?Jean: Yeah. [laugh]. So, Corey—Corey: That is the reality, yes.Jean: That's a really good question, Corey, because in the beginning, we weren't sure what we wanted to focus on. And I'm not saying the whole deployment is unencrypted HTTP, but there is a place to install Akita to watch where it's unencrypted HTTP. And so, this is what I mean by if you have encrypted traffic, but you can install Akita as a Kubernetes sidecar, we can still watch that. But there was a big question when we started: should this be GraphQL, GRPC, or should it be REST? And I read the “State of the API Report” from Postman for you know, five years, and I still keep up with it.And every year, it seemed that not only was REST, remaining dominant, it was actually growing. So, [laugh] this was shocking to me as well because people said, well, “We have this more structured stuff, now. There's GRPC, there's GraphQL.” But it seems that for the added complexity, people weren't necessarily seeing the value and so, REST continues to dominate. And I've actually even seen a decline in GraphQL since we first started doing this. So, I'm fully on board the REST wagon. And in terms of encrypted versus unencrypted, I would also like to see more encryption as well. That's why we're working on burning down the long tail of support for that.Corey: Yeah, it's one of those challenges. Whenever you're deploying something relatively new, there's this idea that it should be forward-looking and you, on some level, want to modernize your architecture and infrastructure to keep up with it. An AWS integration story I see that's like that these days is, “Oh, yeah, generate an IAM credential set and just upload those into our system.” Yeah, the modern way of doing that is role assumption: to find a role and here's how to configure it so that it can do what we need to do. So, whenever you start seeing things that are, “Oh, yeah, just turn the security clock back in time a little bit,” that's always a little bit of an eyebrow raise.I can also definitely empathize with the joys of dealing with anything that even touches networking in a Lambda context. Building the Lambda extension for Tailscale was one of the last big dives I made into that area and I still have nightmares as a result. It does a lot of interesting things right up until you step off the golden path. And then suddenly, everything becomes yaks all the way down, in desperate need of shaving.Jean: Yeah, Lambda does something we want to handle on our roadmap, but I… believe we need a bigger team before [laugh] we are ready to tackle that.Corey: Yeah, we're going to need a bigger boat is very often [laugh] the story people have when they start looking at entire new architectural paradigms. So, you end up talking about working in containerized environments. Do you find that most of your deployments are living in cloud environments, in private data centers, some people call them private cloud. Where does the bulk of your user applications tend to live these days?Jean: The bulk of our user applications are in the cloud. So, we're targeting small to medium businesses to start. The reason being, we want to give our users a magical deployment experience. So, right now, a lot of our users are deploying in under 30 minutes. That's in no small part due to automations that we've built.And so, we initially made the strategic decision to focus on places where we get the most visibility. And so—where one, we get the most visibility, and two, we are ready for that level of scale. So, we found that, you know, for a large business, we've run inside some of their production environments and there are API calls that we don't yet handle well or it's just such a large number of calls, we're not doing the inference as well and our algorithms don't work as well. And so, we've made the decision to start small, build our way up, and start in places where we can just aggressively iterate because we can see everything that's going on. And so, we've stayed away, for instance, from any on-prem deployments for that reason because then we can't see everything that's going on. And so, smaller companies that are okay with us watching pretty much everything they're doing has been where we started. And now we're moving up into the medium-sized businesses.Corey: The challenge that I guess I'm still trying to wrap my head around is, I think that it takes someone with a particularly rosy set of glasses on to look at the current state of monitoring and observability and say that it's not profoundly broken in a whole bunch of ways. Now, where it all falls apart, Tower of Babelesque, is that there doesn't seem to be consensus on where exactly it's broken. Where do you see, I guess, this coming apart at the seams?Jean: I agree, it's broken. And so, if I tap into my background, which is I was a programming languages person in my very recently, previous life, programming languages people like to say the problem and the solution is all lies in abstraction. And so, computing is all about building abstractions on top of what you have now so that you don't have to deal with so many details and you got to think at a higher level; you're free of the shackles of so many low-level details. What I see is that today, monitoring and observability is a sort of abstraction nightmare. People have just taken it as gospel that you need to live at the lowest level of abstraction possible the same way that people truly believe that assembly code was the way everybody was going to program forevermore back, you know, 50 years ago.So today, what's happening is that when people think monitoring, they think logs, not what's wrong with my system, what do I need to pay attention to? They think, “I have to log everything, I have to consume all those logs, we're just operating at the level of logs.” And that's not wrong because there haven't been any tools that have given people any help above the level of logs. Although that's not entirely correct, you know? There's also events and there's also traces, but I wouldn't say that's actually lifting the level of [laugh] abstraction very much either.And so, people today are thinking about monitoring and observability as this full control, like, I'm driving my, like, race car, completely manual transmission, I want to feel everything. And not everyone wants to or needs to do that to get to where they need to go. And so, my question is, how far are can we lift the level of abstraction for monitoring and observability? I don't believe that other people are really asking this question because most of the other players in the space, they're asking what else can we monitor? Where else can we monitor it? How much faster can we do it? Or how much more detail can we give the people who really want the power tools?But the people entering the buyer's market with needs, they're not people—you don't have, like, you know, hordes of people who need more powerful tools. You have people who don't know about the systems are dealing with and they want easier. They want to figure out if there's anything wrong with our system so they can get off work and do other things with their lives.Corey: That, I think, is probably the thing that gets overlooked the most. It's people don't tend to log into their monitoring systems very often. They don't want to. When they do, it's always out of hours, middle of the night, and they're confronted with a whole bunch of upsell dialogs of, “Hey, it's been a while. You want to go on a tour of the new interface?”Meanwhile, anything with half a brain can see there's a giant spike on the graph or telemetry stop coming in.Jean: Yeah.Corey: It's way outside of normal business hours where this person is and maybe they're not going to be in the best mood to engage with your brand.Jean: Yeah. Right now, I think a lot of the problem is, you're either working with monitoring because you're desperate, you're in the middle of an active incident, or you're a monitoring fanatic. And there isn't a lot in between. So, there's a tweet that someone in my network tweeted me that I really liked which is, “Monitoring should be a chore, not a hobby.” And right now, it's either a hobby or an urgent necessity [laugh].And when it gets to the point—so you know, if we think about doing dishes this way, it would be as if, like, only, like, the dish fanatics did dishes, or, like, you will just have piles of dishes, like, all over the place and raccoons and no dishes left, and then you're, like, “Ah, time to do a thing.” But there should be something in between where there's a defined set of things that people can do on a regular basis to keep up with what they're doing. It should be accessible to everyone on the team, not just a couple of people who are true fanatics. No offense to the people out there, I love you guys, you're the ones who are really helping us build our tool the most, but you know, there's got to be a world in which more people are able to do the things you do.Corey: That's part of the challenge is bringing a lot of the fire down from Mount Olympus to the rest of humanity, where at some level, Prometheus was a great name from that—Jean: Yep [laugh].Corey: Just from that perspective because you basically need to be at that level of insight. I think Kubernetes suffers from the same overall problem where it is not reasonably responsible to run a Kubernetes production cluster without some people who really know what's going on. That's rapidly changing, which is for the better, because most companies are not going to be able to afford a multimillion-dollar team of operators who know the ins and outs of these incredibly complex systems. It has to become more accessible and simpler. And we have an entire near century at this point of watching abstractions get more and more and more complex and then collapsing down in this particular field. And I think that we're overdue for that correction in a lot of the modern infrastructure, tooling, and approaches that we take.Jean: I agree. It hasn't happened yet in monitoring and observability. It's happened in coding, it's happened in infrastructure, it's happened in APIs, but all of that has made it so that it's easier to get into monitoring debt. And it just hasn't happened yet for anything that's more reactive and more about understanding what the system is that you have.Corey: You mentioned specifically that your background was in programming languages. That's understating it slightly. You were a tenure-track professor of computer science at Carnegie Mellon before entering industry. How tied to what your area of academic speciality was, is what you're now at Akita?Jean: That's a great question and there are two answers to that. The first is very not tied. If it were tied, I would have stayed in my very cushy, highly [laugh] competitive job that I worked for years to get, to do stuff there. And so like, what we're doing now is comes out of thousands of conversations with developers and desire to build on the ground tools that I'm—there's some technically interesting parts to it, for sure. I think that our technical innovation is our moat, but is it at the level of publishable papers? Publishable papers are a very narrow thing; I wouldn't be able to say yes to that question.On the other hand, everything that I was trained to do was about identifying a problem and coming up with an out-of-the-box solution for it. And especially in programming languages research, it's really about abstractions. It's really about, you know, taking a set of patterns that you see of problems people have, coming up with the right abstractions to solve that problem, evaluating your solution, and then, you know, prototyping that out and building on top of it. And so, in that case, you know, we identified, hey, people have a huge gap when it comes to monitoring and observability. I framed it as an abstraction problem, how can we lift it up?We saw APIs as this is a great level to build a new level of solution. And our solution, it's innovative, but it also solves the problem. And to me, that's the most important thing. Our solution didn't need to be innovative. If you're operating in an academic setting, it's really about… producing a new idea. It doesn't actually [laugh]—I like to believe that all endeavors really have one main goal, and in academia, the main goal is producing something new. And to me, building a product is about solving a problem and our main endeavor was really to solve a real problem here.Corey: I think that it is, in many cases, useful when we start seeing a lot of, I guess, overflow back and forth between academia and industry, in both directions. I think that it is doing academia a disservice when you start looking at it purely as pure theory, and oh yeah, they don't deal with any of the vocational stuff. Conversely, I think the idea that industry doesn't have anything to learn from academia is dramatically misunderstanding the way the world works. The idea of watching some of that ebb and flow and crossover between them is neat to see.Jean: Yeah, I agree. I think there's a lot of academics I super respect and admire who have done great things that are useful in industry. And it's really about, I think, what you want your main goal to be at the time. Is it, do you want to be optimizing for new ideas or contributing, like, a full solution to a problem at the time? But it's there's a lot of overlap in the skills you need.Corey: One last topic I'd like to dive into before we call it an episode is that there's an awful lot of hype around a variety of different things. And right now in this moment, AI seems to be one of those areas that is getting an awful lot of attention. It's clear too there's something of value there—unlike blockchain, which has struggled to identify anything that was not fraud as a value proposition for the last decade-and-a-half—but it's clear that AI is offering value already. You have recently, as of this recording, released an AI chatbot, which, okay, great. But what piques my interest is one, it's a dog, which… germane to my interest, by all means, and two, it is marketed as, and I quote, “Exceedingly polite.”Jean: [laugh].Corey: Manners are important. Tell me about this pupper.Jean: Yeah, this dog came really out of four or five days of one of our engineers experimenting with ChatGPT. So, for a little bit of background, I'll just say that I have been excited about the this latest wave of AI since the beginning. So, I think at the very beginning, a lot of dev tools people were skeptical of GitHub Copilot; there was a lot of controversy around GitHub Copilot. I was very early. And I think all the Copilot people retweeted me because I was just their earlies—like, one of their earliest fans. I was like, “This is the coolest thing I've seen.”I've actually spent the decade before making fun of AI-based [laugh] programming. But there were two things about GitHub Copilot that made my jaw drop. And that's related to your question. So, for a little bit of background, I did my PhD in a group focused on program synthesis. So, it was really about, how can we automatically generate programs from a variety of means? From constraints—Corey: Like copying and pasting off a Stack Overflow, or—Jean: Well, the—I mean, that actually one of the projects that my group was literally applying machine-learning to terabytes of other example programs to generate new programs. So, it was very similar to GitHub Copilot before GitHub Copilot. It was synthesizing API calls from analyzing terabytes of other API calls. And the thing that I had always been uncomfortable with these machine-learning approaches in my group was, they were in the compiler loop. So, it was, you know, you wrote some code, the compiler did some AI, and then it spit back out some code that, you know, like you just ran.And so, that never sat well with me. I always said, “Well, I don't really see how this is going to be practical,” because people can't just run random code that you basically got off the internet. And so, what really excited me about GitHub Copilot was the fact that it was in the editor loop. I was like, “Oh, my God.”Corey: It had the context. It was right there. You didn't have to go tabbing to something else.Jean: Exactly.Corey: Oh, yeah. I'm in the same boat. I think it is basically—I've seen the future unfolding before my eyes.Jean: Yeah. Was the autocomplete thing. And to me, that was the missing piece. Because in your editor, you always read your code before you go off and—you know, like, you read your code, whoever code reviews your code reads your code. There's always at least, you know, two pairs of eyes, at least theoretically, reading your code.So, that was one thing that was jaw-dropping to me. That was the revelation of Copilot. And then the other thing was that it was marketed not as, “We write your code for you,” but the whole Copilot marketing was that, you know, it kind of helps you with boilerplate. And to me, I had been obsessed with this idea of how can you help developers write less boilerplate for years. And so, this AI-supported boilerplate copiloting was very exciting to me.And I saw that is very much the beginning of a new era, where, yes, there's tons of data on how we should be programming. I mean, all of Akita is based on the fact that we should be mining all the data we have about how your system and your code is operating to help you do stuff better. And so, to me, you know, Copilot is very much in that same philosophy. But our AI chatbot is, you know, just a next step along this progression. Because for us, you know, we collect all this data about your API behavior; we have been using non-AI methods to analyze this data and show it to you.And what ChatGPT allowed us to do in less than a week was analyze this data using very powerful large-language models and I have this conversational interface that both gives you the opportunity to check over and follow up on the question so that what you're spitting out—so what we're spitting out as Aki the dog doesn't have to be a hundred percent correct. But to me, the fact that Aki is exceedingly polite and kind of goofy—he, you know, randomly woofs and says a lot of things about how he's a dog—it's the right level of seriousness so that it's not messaging, hey, this is the end all, be all, the way, you know, the compiler loop never sat well with me because I just felt deeply uncomfortable that an AI was having that level of authority in a system, but a friendly dog that shows up and tells you some things that you can ask some additional questions to, no one's going to take him that seriously. But if he says something useful, you're going to listen. And so, I was really excited about the way this was set up. Because I mean, I believe that AI should be a collaborator and it should be a collaborator that you never take with full authority. And so, the chat and the politeness covered those two parts for me both.Corey: Yeah, on some level, I can't shake the feeling that it's still very early days there for Chat-Gipity—yes, that's how I pronounce it—and it's brethren as far as redefining, on some level, what's possible. I think that it's in many cases being overhyped, but it's solving an awful lot of the… the boilerplate, the stuff that is challenging. A question I have, though, is that, as a former professor, a concern that I have is when students are using this, it's less to do with the fact that they're not—they're taking shortcuts that weren't available to me and wanting to make them suffer, but rather, it's, on some level, if you use it to write your English papers, for example. Okay, great, it gets the boring essay you don't want to write out of the way, but the reason you write those things is it teaches you to form a story, to tell a narrative, to structure an argument, and I think that letting the computer do those things, on some level, has the potential to weaken us across the board. Where do you stand on it, given that you see both sides of that particular snake?Jean: So, here's a devil's advocate sort of response to it, is that maybe the writing [laugh] was never the important part. And it's, as you say, telling the story was the important part. And so, what better way to distill that out than the prompt engineering piece of it? Because if you knew that you could always get someone to flesh out your story for you, then it really comes down to, you know, I want to tell a story with these five main points. And in some way, you could see this as a playing field leveler.You know, I think that as a—English is actually not my first language. I spent a lot of time editing my parents writing for their work when I was a kid. And something I always felt really strongly about was not discriminating against people because they can't form sentences or they don't have the right idioms. And I actually spent a lot of time proofreading my friends' emails when I was in grad school for the non-native English speakers. And so, one way you could see this as, look, people who are not insiders now are on the same playing field. They just have to be clear thinkers.Corey: That is a fascinating take. I think I'm going to have to—I'm going to have to ruminate on that one. I really want to thank you for taking the time to speak with me today about what you're up to. If people want to learn more, where's the best place for them to find you?Jean: Well, I'm always on Twitter, still [laugh]. I'm @jeanqasaur—J-E-A-N-Q-A-S-A-U-R. And there's a chat dialog on akitasoftware.com. I [laugh] personally oversee a lot of that chat, so if you ever want to find me, that is a place, you know, where all messages will get back to me somehow.Corey: And we will, of course, put a link to that into the [show notes 00:35:01]. Thank you so much for your time. I appreciate it.Jean: Thank you, Corey.Corey: Jean Yang, CEO at Akita Software. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry insulting comment that you will then, of course, proceed to copy to the other 17 podcast tools that you use, just like you do your observability monitoring suite.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Darko Mesaroš, Senior Developer Advocate at AWS, joins Corey on Screaming in the Cloud to discuss all the weird and wonderful things that can be done with old hardware, as well as the necessary skills for being a successful Developer Advocate. Darko walks through how he managed to deploy Kubernetes on a computer from 1986, as well as the trade-offs we've made in computer technology as hardware has progressed. Corey and Darko also explore the forgotten art of optimizing when you're developing, and how it can help to cut costs. Darko also shares what he feels is the key skill every Developer Advocate needs to have, and walks through how he has structured his presentations to ensure he is captivating and delivering value to his audience.About DarkoDarko is a Senior Developer Advocate based in Seattle, WA. His goal is to share his passion and technological know-how with Engineers, Developers, Builders, and tech enthusiasts across the world. If it can be automated, Darko will definitely try to do so. Most of his focus is towards DevOps and Management Tools, where automation, pipelines, and efficient developer tools is the name of the game – click less and code more so you do not repeat yourself ! Darko also collects a lot of old technology and tries to make it do what it should not. Like deploy AWS infrastructure through a Commodore 64.Links Referenced: AWS: https://aws.amazon.com/ Blog post RE deploying Kubernetes on a TRS-80: https://www.buildon.aws/posts/i-deployed-kubernetes-with-a-1986-tandy-102-portable-computer AWS Twitch: https://twitch.tv/aws Twitter: https://twitter.com/darkosubotica Mastodon: https://hachyderm.io/@darkosubotica TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Chronosphere. When it costs more money and time to observe your environment than it does to build it, there's a problem. With Chronosphere, you can shape and transform observability data based on need, context and utility. Learn how to only store the useful data you need to see in order to reduce costs and improve performance at chronosphere.io/corey-quinn. That's chronosphere.io/corey-quinn. And my thanks to them for sponsor ing my ridiculous nonsense. Corey: Do you wish your developers had less permanent access to AWS? Has the complexity of Amazon's reference architecture for temporary elevated access caused you to sob uncontrollably? With Sym, you can protect your cloud infrastructure with customizable, just-in-time access workflows that can be setup in minutes. By automating the access request lifecycle, Sym helps you reduce the scope of default access while keeping your developers moving quickly. Say goodbye to your cloud access woes with Sym. Go to symops.com/corey to learn more. That's S-Y-M-O-P-S.com/coreyCorey: Welcome to Screaming in the Cloud. I'm Cloud Economist Corey Quinn and my guest today is almost as bizarre as I am, in a somewhat similar direction. Darko Mesaroš is a Senior Developer Advocate at AWS. And instead of following my path of inappropriately using things as databases that weren't designed to be used that way, he instead uses the latest of technology with the earliest of computers. Darko, thank you for joining me.Darko: Thank you so much, Corey. First of all, you know, you tell me, Darko is a senior developer advocate. No, Corey. I'm a system administrator by heart. I happen to be a developer advocate these days, but I was born in the cold, cold racks of a data center. I maintain systems, I've installed packages on Linux systems. I even set up Solaris Zones a long time ago. So yeah, but I happen to yell into the camera these days, [laugh] so thank you for having me here.Corey: No, no, it goes well. You started my career as a sysadmin. And honestly, my opinion, if you asked me—which no one does, but I share it anyway—is that the difference between an SRE and a sysadmin is about a 40% salary bump.Darko: Exactly.Corey: That's about it. It is effectively the same job. The tools are different, the approach we take is different, but the fundamental mandate of ‘keep the site up' has not materially changed.Darko: It has not. I don't know, like, what the modern SRS do, but like, I used to also semi-maintain AC units. Like, you have to walk around with a screwdriver nonetheless, so sometimes, besides just installing the freshest packages on your Red Hat 4 system, you have to also change the filters in the AC. So, not sure if that belongs into the SRE manifesto these days.Corey: Well, the reason that I wound up inviting you onto the show was a recent blog post you put up where you were able to deploy Kubernetes from the best computer from 1986, which is the TRS-80, or the Trash-80. For the record, the worst computer from 1986 was—and remains—IBM Cloud. But that's neither here nor there.What does it mean to deploy Kubernetes because, to be direct, the way that I tend to deploy anything these days, if you know, I'm sensible and being grown up about it, is a Git push and then the automation takes it away from there. I get the sense, you went a little bit deep.Darko: So, when it comes to deploying stuff from an old computer, like, you know, you kind of said the right thing here, like, I have the best computer from 1986. Actually, it's a portable version of the best computer from 1986; it's a TRS-80 Model 102. It's a portable, basically a little computer intended for journalists and people on the go to write stuff and send emails or whatever it was back in those days. And I deployed Kubernetes through that system. Now, of course, I cheated a bit because the way I did it is I just used it as a glorified terminal.I just hooked up the RS 232, the wonderful serial connection, to a Raspberry Pi somewhere out there and it just showed the stuff from a Raspberry Pi onto the TRS-80. So, the TRS-80 didn't actually know how to run kubectl—or ‘kube cuddle,' what they call it—it just asked somebody else to do it. But that's kind of the magic of it.Corey: You could have done a Lambda deployment then just as easily.Darko: Absolutely. Like that's the magic of, like, these old hunks of junks is that when you get down to it, they still do things with numbers and transmit electrical signals through some wires somewhere out there. So, if you're capable enough, if you are savvy, or if you just have a lot of time, you can take any old computer and have it do modern things, especially now. Like, and I will say 15 years ago, we could have not done anything like this because 15 years ago, a lot of the stuff at least that I was involved with, which was Microsoft products, were click only. I couldn't, for the love of me, deploy a bunch of stuff on an Active Directory domain by using a command line. PowerShell was not a thing back then. You could use VB Script, but sort of.Corey: Couldn't you wind up using something that would effect, like, Selenium or whatnot that winds up emulating a user session and moving the mouse to certain coordinates and clicking and then waiting some arbitrary time and clicking somewhere else?Darko: Yes.Corey: Which sounds like the absolute worst version of automation ever. That's like, “I deployed Kubernetes using a typewriter.” “Well, how the hell did you do that?” “Oh, I use the typewriter to hit the enter key. Problem solved.” But I don't think that counts.Darko: Well, yeah, so actually even back then, like, just thinking of, like, a 10, 12-year step back to my career, I automated stuff on Windows systems—like Windows 2000, and Windows 2003 systems—by a tool called AutoIt. It would literally emulate clicks of a mouse on a specific location on the screen. So, you were just really hoping that window pops up at the same place all the time. Otherwise, your automation doesn't work. So yeah, it was kind of like that.And so, if you look at it that way, I could take my Trash-80, I could write an AutoIt script with specific coordinates, and I could deploy Windows things. So actually, yeah, you can deploy anything with these days, with an old computer.Corey: I think that we've lost something in the world of computers. If I, like, throw a computer at you these days, you're going to be pretty annoyed with me. Those things are expensive, it'll probably break, et cetera. If I throw a computer from this era at you, your family is taking bereavement leave. Like, those things where—there would be no second hit.These things were beefy. They were a sense of solidity to them. The keyboards were phenomenal. We've been chasing that high ever since. And, yeah, they were obnoxiously heavy and the battery life was 20 seconds, but it was still something that—you felt like it is computer time. And now, all these things have faded into the background. I am not protesting the march of progress, particularly in this particular respect, but I do miss the sense of having keyboards didn't weren't overwhelmingly flimsy plastic.Darko: I think it's just a fact of, like, we have computers as commodities these days. Back then computers were workstations, computers were something you would buy to perform a specific tasks. Today, computer is anything from watching Twitch to going on Twitter, complaining about Twitter, to deploying Kubernetes, right? So, they have become such commodities such… I don't want to call them single-use items, but they're more becoming single-use items as time progresses because they're just not repairable anymore. Like, if you give me a computer that's five years old, I don't know what to do with it. I probably cannot fix it if it's broken. But if you give me a computer that's 35 years old, I bet you can fix it no matter what happened.Corey: And the sheer compute changes have come so fast and furious, it's easy to lose sight of them, especially with branding being more or less the same. But I saved up and took a additional loan out when I graduated high school to spend three grand on a Dell Inspiron laptop, this big beefy thing. And for fun, I checked the specs recently, and yep, that's a Raspberry Pi these days; they're $30, and it's not going to work super well to browse the web because it's underpowered. And I'm sitting here realizing wait a minute, even with a modern computer—forget the Raspberry Pi for a second—I'm sitting here and I'm pulling up web pages or opening Slack, or God forbid, Slack and Chrome simultaneously, and the fan spins up and it sounds incredibly anemic. And it's, these things are magical supercomputers from the future. Why are they churning this hard to show me a funny picture of a cat? What's going on here?Darko: So, my theory on this is… because we can. We can argue about this, but we currently—Corey: Oh, I think you're right.Darko: We have unlimited compute capacity in the world. Like, you can come up with an idea, you're probably going to find a supercomputer out there, you're probably going to find a cloud vendor out there that's going to give you all of the resources you need to perform this massive computation. So, we didn't really think about optimization as much as we used to do in the past. So, that's it: we can. Chrome doesn't care. You have 32 gigs of RAM, Corey. It doesn't care that it takes 28 gigs of that because you have—Corey: I have 128 gigs on this thing. I bought the Mac studio and maxed it out. I gave it the hostname of us-shitpost-1 and we run with it.Darko: [laugh]. There you go. But like, I did some fiddling around, like, recently with—and again, this is just the torture myself—I did some 6502 Assembly for the Atari 2600. 6502 is a CPU that's been used in many things, including the Commodore 64, the NES, and even a whole lot of Apple IIs, and whatnot. So, when you go down to the level of a computer that has 1.19 megahertz and it has only 128 bytes of RAM, you start to think about, okay, I can move these two numbers in memory in the following two ways: “Way number one will require four CPU cycles. Way number two will require seven CPU cycles. I'll go with way number one because it will save me three CPU cycles.”Corey: Oh, yeah. You take a look at some of the most advanced computer engineering out there and it's for embedded devices where—Darko: Yeah.Corey: You need to wind up building code to run in some very tight constraints, and that breeds creativity. And I remember those days. These days, it's well my computer is super-overpowered, what's it matter? In fact, when I go in and I look at customers' AWS bills, very often I'll start doing some digging, and sure enough, EC2 is always the number one expense—we accept that—but we take a look at the breakdown and invariably, there's one instance family and size that is the overwhelming majority, in most cases. You often a—I don't know—a c5.2xl or something or whatever it happens to be.Great. Why is that? And the answer—[unintelligible 00:10:17] to make sense is, “Well, we just started with that size and it seemed to work so we kept using it as our default.” When I'm building things, because I'm cheap, I take one of the smallest instances I possibly can—it used to be one of the Nanos and I'm sorry, half a gig or a gig of RAM is no longer really sufficient when I'm trying to build almost anything. Thanks, JavaScript. So okay, I've gone up a little bit.But at that point, when I need to do something that requires something beefier, well, I can provision those resources, but I don't have it as a default. That forces me to at least in the back of my mind, have a little bit of a sense of I should be parsimonious with what it is that I'm provisioning out there, which is apparently anathema to every data scientist I've ever met, but here we are.Darko: I mean, that's the thing, like, because we're so used to just having those resources, we don't really care about optimizations. Like, I'm not advocating that you all should go and just do assembly language. You should never do that, like, unless you're building embedded systems or you're working for something—Corey: If you need to use that level of programming, you know.Darko: Exactly.Corey: You already know and nothing you are going to talk about here is going to impact what people in that position are doing. Mostly you need to know assembly language because that's a weeder class and a lot of comp-sci programs and if you don't pass it, you don't graduate. That's the only reason to really know assembly language most of the time.Darko: But you know, like, it's also a thing, like, as a developer, right, think about the person using your thing, right? And they may have the 128 gig us—what is it you called it? Us-shitpost-1, right—that kind of power, kind of, the latest and greatest M2 Max Ultra Apple computer that just does all of the stuff. You may have a big ‘ol double Xeon workstation that does a thing.Or you just may have a Chromebook. Think about us with Chromebooks. Like, can I run your website properly? Did you really need all of those animations? Can you think about reducing the amount of animations depending on screen size? So, there's a lot of things that we need to kind of think about. Like, it goes back to the thing where ‘it works on my machine.' Oh, of course it works on your machine. You spent thousands of dollars on your machine. It's the best machine in the world. Of course, it runs smoothly.Corey: Wait 20 minutes and they'll release a new one, and now, “Who sold me this ancient piece of crap?” Honestly, the most depressing thing is watching an Apple Keynote because I love my computer until I watch the Apple Keynote and it's like, oh, like, “Look at this amazing keyboard,” and the keyboard I had was fine. It's like, “Who sold me this rickety piece of garbage?” And then we saw how the Apple butterfly keyboard worked out for everyone and who built that rickety piece of garbage. Let's go back again. And here we are.Darko: Exactly. So, that's kind of the thing, right? You know, like, your computer is the best. And if you develop for it, is great, but you always have to think other people who use it. Hence, containers are great to fix one part of that problem, but not all of the problems. So, there's a bunch of stuff you can do.And I think, like, for all of the developers out there, it's great what you're doing, you're building us so many tools, but always that take a step back and optimize stuff. Optimize, both for the end-user by the amount of JavaScript you're going to throw at me, and also for the back-end, think about if you have to run your web server on a Pentium III server, could you do it? And if you could, how bad would it be? And you don't have to run it on a Pentium III, but like, try to think about what's the bottom 5% of the capacity you need? So yeah, it's just—you'll save money. That's it. You'll save money, ultimately.Corey: So, I have to ask, what you do day to day is you're a senior developer advocate, which is, hmm, some words, yes. You spend a lot of your free time and public time talking about running ancient computers, but you also talk to customers who are looking forward, not back. How do you reconcile the two?Darko: So, I like to mix the two. There's a whole reason why I like old computers. Like, I grew up in Serbia. Like, when I was young in the '90s, I didn't have any of these computers. Like, I could only see, like, what was like a Macintosh from 1997 on TV and I would just drool. Like, I wouldn't even come close to thinking about getting that, let alone something better.So, I kind of missed all of that part. But now that I started collecting all of those old computers and just everything from the '80s and '90s, I've actually realized, well, these things are not that different from something else. So, I like to always make comparisons between, like, an old system. What does it actually do? How does it compare to a new system?So, I love to mix and match in my presentations. I like to mix it, mix and match in my videos. You saw my blog posts on deploying stuff. So, I think it's just a fun way to kind of create a little contrast. I do think we should still be moving forward. I do think that technology is getting better and better and it's going to help people do so much more things faster, hopefully cheaper, and hopefully better.So, I do think that we should definitely keep on moving forward. But I always have this nostalgic feeling about, like, old things and… sometimes I don't know why, but I miss the world without the internet. And I think that without the internet, I think I miss the world with dial-up internet. Because back then you would go on the internet for a purpose. You have to do a thing, you have to wait for a while, you have to make sure nobody's on the phone. And then—Corey: God forbid you dial into a long-distance call. And you have to figure out which town and which number would be long distance versus not, at least where I grew up, and your parents would lose their freaking minds because that was an $8 phone call, which you know, back in the '80s and early '90s was significant. And yeah, great. Now, I still think is a great prank opportunity to teach kids are something that it costs more to access websites that are far away, which I guess in theory, it kind of does, but not to the end-user. I digress.Darko: I have a story about this, and I'm going to take a little sidestep. But long-distance phone calls. Like in the '80s, the World Wide Web was not yet a thing. Like, the www, the websites all, just the general purpose internet was not yet a thing. We had things called BBSes, or Bulletin Board Systems. That was the extreme version of a dial-up system.You don't dial into the internet; you dial into a website. Imagine if you have a sole intent of visiting only one website and the cost of visiting such a website would depend on where that website currently is. If the website is in Germany and you're calling from Serbia, it's going to cost you a lot of money because you're calling internationally. I had a friend back then. The best software you can get were from American BBSes, but calling America from Serbia back then would have been prohibitively expensive, like, just insanely expensive.So, what this friend used to do, he figured out if he would be connected to a BBS six hours a day, it would actually reset the counter of his phone bill. It would loop through a mechanical counter from whatever number, it would loop back again to that number. So, it would take around six and some hours to complete the loop the entire phone counting metric—whatever they use back in the '80s—to kind of charge your bill, so it's effectively cost him zero money back then. So yeah, it was more expensive, kids, back then to call websites, the further away the websites were.[midroll 00:17:11]Corey: So, developer advocates do a lot of things. And I think it is unfair, but also true that people tend to shorthand those of those things do getting on stage and giving conference talks because that at least is the visible part of it. People see that and it's viscerally is understood that that takes work and a bit of courage for those who are not deep into public speaking and those who are, know it takes a lot of courage. And whereas writing a blog post, “Well, I have a keyboard and say dumb things on the internet all the time. I don't see why that's hard.” So, there's a definite perception story there. What's your take on giving technical presentations?Darko: So, yeah. Just as you said, like, I think being a DA, even in my head was always represented, like, oh, you're just on stage, you're traveling, you're doing presentations, you're doing all those things. But it's actually quite a lot more than that, right? We do a lot more. But still, we are the developer advocate. We are the front-facing thing towards you, the wonderful developers listening to this.And we tend to be on stage, we tend to do podcasts with wonderful internet personalities, we tend to do live streams, we tend to do videos. And I think one of the key skills that a DA needs to have—a Developer Advocate needs to have—is presentations, right? You need to be able to present a technical message in the best possible way. Now, being a good technical presenter doesn't mean you're funny, doesn't mean you're entertaining, that doesn't have to be a thing. You just need to take a complex technical message and deliver it in the best way possible so that everybody who has just given you their time, can get it fully.And this means—well, it means a lot of things, but it means taking this complicated topic, distilling it down so it can be digested within 30 to 45 minutes and it also needs to be… it needs to be interesting. Like, we can talk about the most interesting topic, but if I don't make it interesting, you're just going to walk out. So, I also lead, like, a coaching class within internally, like, to teach people how to speak better and I'm working with, like, really good speakers there, but a lot of the stuff I say applies to no matter if you're a top-level speaker, or if you're, like, just beginning out. And my challenge to all of you speakers out there, like, anybody who's listening to this and it has a plan to deliver a video, a keynote, a live stream or speak at a summit somewhere, is get outside of that box. Get outside of that PowerPoint box.I'm not saying PowerPoint is bad. I think PowerPoint is a wonderful tool, but I'm just saying you don't have to present in the way everybody else presents. The more memorable your presentation is, the more outside of that box it is, the more people will remember it. Again, you don't have to be funny. You don't have to be entertaining. You just have to take thing you are really passionate about and deliver it to us in the best possible way. What that best possible way is, well, it really depends. Like a lot of things, there is no concrete answer to this thing.Corey: One of the hard parts I found is that people will see a certain technical presenter that they like and want to emulate and they'll start trying to do what they do. And that works to a point. Like, “Well, I really enjoy how that presenter doesn't read their slides.” Yeah, that's a good thing to pick up. But past a certain point, other people's material starts to fit as well as other people's shoes and you've got to find your own path.My path has always been getting people's attention first via humor, but it's certainly not the only way. In many contexts, it's not even the most effective way. It works for me in the context in which I use it, but I assure you that when I'm presenting to clients, I don't start off with slapstick comedy. Usually. There are a couple of noteworthy exceptions because clients expect that for me, in some cases.Darko: I think one of the important things is that emulating somebody is okay, as you said, to an extent, like, just trying to figure out what the good things are, but good, very objectively good things. Never try to be funny if you're not funny. That's the thing where you can try comedy, but it's very difficult to—it's very difficult to do comedy if you're not that good at it. And I know that's very much a given, but a lot of people try to be funny when they're obviously not funny. And that's okay. You don't have to be funny.So, there are many of ways to get people's attentions, by again, just throwing a joke. What I did once on stage, I threw a bottle at the floor. I was just—I said, I said a thing and threw a bottle at the floor. Everybody started paying attention all of a sudden at me. I don't know why. So, it's going to be that. It can be something—it can be be a shocking statement. When I say shocking, I mean, something, well, not bad, but something that's potentially controversial. Like, for example, emacs is better than vim. I don't know, maybe—Corey: “Serverless is terrible.”Darko: Serverl—yeah.Corey: Like, it doesn't matter. It depends on the audience.Darko: It depends on the audience.Corey: “The cloud is a scam.” I gave a talk once called, “The Cloud is A Scam,” and it certainly got people's attention.Darko: Absolutely. So, breaking up the normal flow because as a participant of a show, of a presentation, you go there you expect, look, I'm going to sit down, Corey's going to come on stage and Corey says, “Hi, my name is Corey Quinn. I'm the CEO of The Duckbill Group. This is what I do. And welcome to my talk about blah.”Corey: Tactically, my business partner, Mike, is the CEO. I don't want to I don't want to step too close to that fire, let's be clear.Darko: Oh, okay [laugh]. Okay. Then, “Today's agenda is this. And slide one, slide two, slide three.” And that the expectation of the audience. And the audience comes in in this very autopilot way, like, “Okay, I'm just going to sit there and just nod my head as Corey speaks.”But then if Corey does a weird thing and Corey comes out in a bathtub. Just the bathtub and Corey. And Corey starts talking about how bathtubs are amazing, it's the best place to relax. “Oh, by the way, managing costs in the cloud is so easy, I can do it from a bathtub.” Right? All of a sudden, whoa [laugh], wait a second, this is something that's interesting. And then you can go through your rest of your conversation. But you just made a little—you ticked the box in our head, like, “Oh, this is something weird. This is different. I don't know what to expect anymore,” and people start paying more attention.Corey: “So, if you're managing AWS costs from your bathtub, what kind of computer do you use?” “In my case, a toaster.”Darko: [laugh]. Yes. But ultimately, like, some of those things are very good and they just kind of—they make you as a presenter, unpredictable, and that's a good thing. Because people will just want to sit on the edge of the seat and, like, listen to what you say because, I don't know what, maybe he throws that toaster in, right? I don't know. So, it is like that.And one of the things that you'll notice, Corey, especially if you see people who are more presenting for a longer time, like, they've been very common on events and people know them by name and their face, then that turns into, like, not just presenting but somebody comes, literally not because of the topic, but because they want to hear Corey talk about a thing. You can go there and talk about unicorns and cats, people will still come and listen to that because it's Corey Quinn. And that's where you, by getting outside of that box, getting outside of that ‘this is how we present things at company X,' this is what you get in the long run. People will know who you are people will know, what not to expect from your presentations, and they will ultimately be coming to your presentations to enjoy whatever you want to talk about.Corey: That is the dream. I really want to thank you for taking the time to talk so much about how you view the world and the state of ancient and modern technologies and the like. If people want to learn more, where's the best place for them to find you?Darko: The best way to find me is on twitch.tv/aws these days. So, you will find me live streaming twice a week there. You will find me on Twitter at @darkosubotica, which is my Twitter handle. You will find me at the same handle on Mastodon. And just search for my name Darko Mesaroš, I'm sure I'll pop up on MySpace as well or whatever. So, I'll post a lot of cloud-related things. I posted a lot of old computer-related things, so if you want to see me deploy Kubernetes through an Atari 2600, click that subscribe button or follow or whatever.Corey: And we will, of course, include a link to this in the show notes. Thank you so much for being so generous with your time. I appreciate it.Darko: Thank you so much, Corey, for having me.Corey: Darko Mesaroš, senior developer advocate at AWS, Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry and insulting comment that you compose and submit from your IBM Selectric typewriter.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.