Podcasts about DevOps

Open Source and other source available projects have been a huge driver of progress in our industry, but building and maintaining an open source project is about a lot more than just writing the initial code and putting together a good README. On this episode of the maintenance mini-series, we'll be discussing open source and the maintenance required to keep it going.

Eric and Brandon talk about RHEL 8.5 and RHEL 9 Beta and talk about some interesting tools for managing your datacenter and IT assets. Destination Linux Network (https://destinationlinux.network) Sudo Show Website (https://sudo.show) Sponsor: Bitwarden (https://bitwarden.com/dln) Sponsor: Digital Ocean (https://do.co/dln) Sudo Show Swag (https://sudo.show/swag) Contact Us: DLN Discourse (https://sudo.show/discuss) Email Us! (mailto:contact@sudo.show) Sudo Matrix Room (https://sudo.show/matrix) Follow our Hosts: Brandon's Website (https://open-tech.net) Brandon's GitHub (https://github.com/dbrandonjohnson) Brandon's Gitlab (https://gitlab.com/sudoshow) Eric's Website (https://itguyeric.com) Red Hat Streaming (https://www.redhat.com/en/livestreaming) Asset Management https://snipeitapp.com/ CMDB (Content Management Database) https://www.cmdbuild.org OS Ticket (can be morphed into it added it because it is a widely used solution) https://osticket.org IPAM/DCIM (IP Address Management and Datacenter Infrastructure Management) https://github.com/netbox-community/netbox Chapters 00:00 Intro 00:42 Welcome 02:00 Sponsor: Bitwarden 03:02 Sponsor: Digital Ocean 04:10 RHEL 8.5 and 9 Beta 21:18 Snipit Asset Management 33:56 CMDB Build 36:11 OSTicket 37:44 Netbox 46:55 Call to Action 48:20 Wrap Up

About TimTim's tech career spans over 20 years through various sectors. Tim's initial journey into tech started as a US Marine. Later, he left government contracting for the private sector, working both in large corporate environments and in small startups. While working in the private sector, he honed his skills in systems administration and operations for large Unix-based datastores. Today, Tim leverages his years in operations, DevOps, and Site Reliability Engineering to advise and consult with clients in his current role. Tim is also a father of five children, as well as a competitive Brazilian Jiu-Jitsu practitioner. Currently, he is the reigning American National and 3-time Pan American Brazilian Jiu-Jitsu champion in his division.TranscriptCorey: Hello, and welcome to Screaming in the Cloud with your host, Chief cloud economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: Welcome to Screaming in the Cloud. I am Cloud Economist Corey Quinn joined by Principal Cloud Economist here at The Duckbill Group Tim Banks. Tim, how are you?Tim: I'm doing great, Corey. How about yourself?Corey: I am tickled pink that we are able to record this not for the usual reasons you would expect, but because of the glorious pun in calling this our Banksgiving episode. I have a hard and fast rule of, I don't play pun games or make jokes about people's names because that can be an incredibly offensive thing. “And oh, you're making jokes about my name? I've never heard that one before.” It's not that I can't do it—I play games with language all the time—but it makes people feel crappy. So, when you suggested this out of the blue, it was yes, we're doing it. But I want to be clear, I did not inflict this on you. This is your own choice; arguably a poor one. We're going to find out.Tim: 1000% my idea.Corey: So, this is your show. It's a holiday week. So, what do you want to do with our Banksgiving episode?Tim: I want to give thanks for the folks who don't normally get acknowledged through the year. Like you know, we do a lot of thanking the rock stars, we do a lot of thanking the big names, right, we also do a lot of, you know, some snarky jabs at some folks. Deservingly—not folks, but groups and stuff like that; some folks deserve it, and we won't be giving them thanks—but some orgs and some groups and stuff like that. And I do think with that all said, we should acknowledge and thank the folks that we normally don't get to, folks who've done some great contributions this year, folks who have helped us, helped the industry, and help services that go unsung, I think a great one that you brought up, it's not the engineers, right? It's the people that make sure we get paid. Because I don't work for charity. And I don't know about you, Corey. I haven't seen the books yet, but I'm pretty sure none of us here do and so how do we get paid? Like I don't know.Corey: Oh, sure you have. We had a show on a somewhat simplified P&L during the all hands meeting because, you know, transparency matters. But you're right, those are numbers there and none of that is what we could have charged but didn't because we decided to do more volunteer work for AWS. If we were going to go down that path, we would just be Community Heroes and be done with it.Tim: That's true. But you know, it's like, I do my thing and then, you know, I get a paycheck every now and then. And so, as far as I know, I think most of that happens because of Dan.Corey: Dan is a perfect example. He's been a guest on this show, I don't know it has as aired at the time that this goes out because I don't have to think about that, which is kind of the point. Dan's our CFO and makes sure that a lot of the financial trains keep running on time. But let's also be clear, the fact that I can make predictions about what the business is going to be doing by a metric other than how much cash is in the bank account at this very moment really freed up some opportunity for us. It turned into adult supervision for folks who, when I started this place and then Mike joined, and it was very much not an area that either one of us was super familiar with. Which is odd given what we do here, but we learned quickly.The understanding not just how these things work—which we had an academic understanding of—but why it mattered and how that applies to real life. Finance is one of those great organizations that doesn't get a lot of attention or respect outside of finance itself. Because it's, “Oh, well they just control the money. How hard could it be?” Really, really hard.Tim: It really is. And when we dig into some of these things and some of the math that goes and some of what the concerns are that, you know, a lot of engineers don't really have a good grasp on, and it's eye opening to understand some of the concerns. At least some of the concerns at least from an engineering aspect. And I really don't give much consideration day to day about the things that go on behind the scenes to make sure that I get paid.But you look at this throughout the industry, like, how many of the folks that we work with, how many folks out there doing this great work for the industry, do they know who their payroll person is? Do they know who their accountant team is? Do they know who their CFO or the other people out there that are doing the work and making sure the lights stay on, that people get paid and all the other things that happen, right? You know, people take that for granted. And it's a huge work and those people really don't get the appreciation that I think they deserve. And I think it's about time we did that.Corey: It's often surprising to me how many people that I encounter, once they learn that there are 12 employees here, automatically assume that it's you, me, and maybe occasionally Mike doing all the work, and the other nine people just sort of sit here and clap when I tell a funny joke, and… well, yes, that is, of course, a job duty, but that's not the entire purpose of why people are here.Natalie in marketing is a great example. “Well, Corey, I thought you did the marketing. You go and post on Twitter and that's where business comes from.” Well, kind of. But let's be clear, when I do that, and people go to the website to figure out what the hell I'm talking about.Well, that website has words on it. I didn't put those words on that site. It directs people to contact us forms, and there are automations behind that that make sure they go to the proper place because back before I started this place and I was independent, people would email me asking for help with their bill and I would just never respond to them. It's the baseline adult supervision level of competence that I keep aspiring to. We have a sales team that does fantastic work.And that often is one of those things that'll get engineering hackles up, but they're not out there cold-calling people to bug them about AWS bills. It's when someone reaches out saying we have a problem with our AWS spend, can you help us? The answer is invariably, “Let's talk about that.” It's a consultative discussion about why do you care about the bill, what does success look like, how do you know this will be a success, et cetera, et cetera, et cetera, that make sure that we're aimed at the right part of the problem. That's incredibly challenging work and I am grateful beyond words, I don't have to be involved with the day-in, day-out of any of those things.Tim: I think even beyond just that handling, like, the contracts and the NDAs, and the various assets that have to be exchanged just to get us virtually on site, I've [unintelligible 00:06:46] a couple of these things, I'm glad it's not my job. It is, for me, overwhelmingly difficult for me to really get a grasp and all that kind of stuff. And I am grateful that we do have a staff that does that. You've heard me, you see me, you know, kind of like, sales need to do better, and a lot of times I do but I do want to make sure we are appreciating them for the work that they do to make sure that we have work to do. Their contribution cannot be underestimated.Corey: And I think that's something that we could all be a little more thankful for in the industry. And I see this on Twitter sometimes, and it's probably my least favorite genre of tweet, where someone will wind up screenshotting some naive recruiter outreach to them, and just start basically putting the poor person on blast. I assure you, I occasionally get notices like that. The most recent example of that was, I got an email to my work email address from an associate account exec at AWS asking what projects I have going on, how my work in the cloud is going, and I can talk to them about if I want to help with cost optimization of my AWS spend and the rest. And at first, it's one of those, I could ruin this person's entire month, but I don't want to be that person.And I did a little LinkedIn stalking and it turns out, this looks like this person's first job that they've been in for three months. And I've worked in jobs like that very early in my career; it is a numbers game. When you're trying to reach out to 1000 people a month or whatnot, you aren't sitting there googling what every one of them is, does, et cetera. It's something that I've learned, that is annoying, sure. But I'm in an incredibly privileged position here and dunking on someone who's doing what they are told by an existing sales apparatus and crapping on them is not fair.That is not the same thing as these passive-aggressive [shit-tier 00:08:38] drip campaigns of, “I feel like I'm starting to stalk you.” Then don't send the message, jackhole. It's about empathy and not crapping on people who are trying to find their own path in this ridiculous industry.Tim: I think you brought up recruiters, and, you know, we here at The Duckbill Group are currently recruiting for a senior cloud economist and we don't actually have a recruiter on staff. So, we're going through various ways to find this work and it has really made me appreciate the work that recruiters in the past that I've worked with have done. Some of the ones out there are doing really fantastic work, especially sourcing good candidates, vetting good candidates, making sure that the job descriptions are inclusive, making sure that the whole recruitment process is as smooth as it can be. And it can't always be. Having to deal with all the spinning plates of getting interviews with folks who have production workloads, it is pretty impressive to me to see how a lot of these folks get—pull it off and it just seems so smooth. Again, like having to actually wade through some of this stuff, it's given me a true appreciation for the work that good recruiters do.Corey: We don't have automated systems that disqualify folks based on keyword matches—I've never been a fan of that—but we do get applicants that are completely unsuitable. We've had a few come in that are actual economists who clearly did not read the job description; they're spraying their resume everywhere. And the answer is you smile, you decline it and you move on. That is the price you pay of attempting to hire people. You don't put them on blast, you don't go and yell at an entire ecosystem of people because looking for jobs sucks. It's hard work.Back when I was in my employee days, I worked harder finding new jobs than I often did in the jobs themselves. This may be related to why I get fired as much, but I had to be good at finding new work. I am, for better or worse, in a situation where I don't have to do that anymore because once again, we have people here who do the various moving parts. Plus, let's be clear here, if I'm out there interviewing at other companies for jobs, I feel like that sends a message to you and the rest of the team that isn't terrific.Tim: We might bring that up. [laugh].Corey: “Why are you interviewing for a job over there?” It's like, “Because they have free doughnuts in the office. Later, jackholes.” It—I don't think that is necessarily the culture we're building here.Tim: No, no, it's not. Specially—you know, we're more of a cinnamon roll culture anyways.Corey: No. In my case, it's one of those, “Corey, why are you interviewing for a job at AWS?” And the answer is, “Oh, it's going to be an amazing shitpost. Just wait and watch.”Tim: [laugh]. Now, speaking of AWS, I have to absolutely shout out to Emily Freeman over there who has done some fantastic work this year. It's great when you see a person get matched up with the right environment with the right team in the right role, and Emily has just been hitting out of the park ever since he got there, so I'm super, super happy to see her there.Corey: Every time I get to collaborate with her on something, I come away from the experience even more impressed. It's one of those phenomenal collaborations. I just—I love working with her. She's human, she's empathetic, she gets it. She remains, as of this recording, the only person who has ever given a talk that I have heard on ML Ops, and come away with a better impression of that space and thinking maybe it's not complete nonsense.And that is not just because it's Emily, so I—because—I'm predisposed to believe her, though I am, it's because of how she frames it, how she views these things, and let's be clear, the content that she says. And that in turn makes me question my preconceptions on this, and that is why she has that I will listen and pay attention when she speaks. So yeah, if Emily's going to try and make a point, there's always going to be something behind it. Her authenticity is unimpeachable.Tim: Absolutely. I do take my hat's off to everyone who's been doing DevRel and evangelism and those type of roles during pandemics. And we just, you know, as the past few months, I've started back to in-person events. But the folks who've been out there finding new way to do those jobs, finding a way to [crosstalk 00:12:50]—Corey: Oh, staff at re:Invent next week. Oh, my God.Tim: Yeah. Those folks, I don't know how they're being rewarded for their work, but I can assure you, they probably need to be [unintelligible 00:12:57] better than they are. So, if you are staff at re:Invent, and you see Corey and I, next week when we're there—if you're listening to this in time—we would love to shake your hand, elbow bump you, whatever it is you're comfortable with, and laud you for the work you're doing. Because it is not easy work under the best of circumstances, and we are certainly not under the best of circumstances.Corey: I also want to call out specific thanks to a group that might take some people aback. But that group is AWS marketing, which given how much grief I give them seems like an odd thing for me to say, but let's be clear, I don't have any giant companies whose ability to continue as a going concern is dependent upon my keeping systems up and running. AWS does. They have to market and tell stories to everyone because that is generally who their customers are: they round to everyone. And an awful lot of those companies have unofficial mottos of, “That's not funny.” I'm amazed that they can say anything at all, given how incredibly varied their customer base is, I could get away with saying whatever I want solely because I just don't care. They have to care.Tim: They do. And it's not only that they have to care, they're in a difficult situation. It's like, you know, they—every company that sizes is, you know, they are image conscious, and they have things that say what like, “Look, this is the deal. This is the scenario. This is how it went down, but you can still maintain your faith and confidence in us.” And people do when AWS services, they have problems, if anything comes out like that, it does make the news and the reason it doesn't make the news is because it is so rare. And when they can remind us of that in a very effective way, like, I appreciate that. You know, people say if anything happens to S3, everybody knows because everyone depends on it and that's for good reason.Corey: And let's not forget that I run The Duckbill Group. You know, the company we work for. I have the Last Week in AWS newsletter and blog. I have my aggressive shitposting Twitter feed. I host the AWS Morning Brief podcast, and I host this Screaming in the Cloud. And it's challenging for me to figure out how to message all of those things because when people ask what you do, they don't want to hear a litany that goes on for 25 seconds, they want a sentence.I feel like I've spread in too many directions and I want to narrow that down. And where do I drive people to and that was a bit of a marketing challenge that Natalie in our marketing department really cut through super well. Now, pretend I work in AWS. The way that I check this based upon a public list of parameters they stub into Systems Manager Parameter Store, there are right now 291 services that they offer. That is well beyond any one person's ability to keep in their head. I can talk incredibly convincingly now about AWS services that don't exist and people who work in AWS on messaging, marketing, engineering, et cetera, will not call me out on it because who can provably say that ‘AWS Strangle Pony' isn't a real service.Tim: I do want to call out the DevOps—shout out I should say, the DevOps term community for AWS Infinidash because that was just so well done, and AWS took that with just the right amount of tongue in cheek, and a wink and a nod and let us have our fun. And that was a good time. It was a great exercise in improv.Corey: That was Joe Nash out of Twilio who just absolutely nailed it with his tweet, “I am convinced that a small and dedicated group of Twitter devs could tweet hot takes about a completely made up AWS product—I don't know AWS Infinidash or something—and it would appear as a requirement on job specs within a week.” And he was right.Tim: [laugh]. Speaking of Twitter, I want to shout out Twitter as a company or whoever does a product management over there for Twitter Spaces. I remember when Twitter Spaces first came out, everyone was dubious of its effect, of it's impact. They were calling it, you know, a Periscope clone or whatever it was, and there was a lot of sneering and snarking at it. But Twitter Spaces has become very, very effective in having good conversations in the group and the community of folks that have just open questions, and then to speak to folks that they probably wouldn't only get to speak to about this questions and get answers, and have really helpful, uplifting and difficult conversations that you wouldn't otherwise really have a medium for. And I'm super, super happy that whoever that product manager was, hats off to you, my friend.Corey: One group you're never going to hear me say a negative word about is AWS support. Also, their training and certification group. I know that are technically different orgs, but it often doesn't feel that way. Their job is basically impossible. They have to teach people—even on the support side, you're still teaching people—how to use all of these different varied services in different ways, and you have to do it in the face of what can only really be described as abuse from a number of folks on Twitter.When someone is having trouble with an AWS service, they can turn into shitheads, I've got to be honest with you. And berating the poor schmuck who has to handle the AWS support Twitter feed, or answer your insulting ticket or whatnot, they are not empowered to actually fix the underlying problem with a service. They are effectively a traffic router to get the message to someone who can, in a format that is understood internally. And I want to be very clear that if you insult people who are in customer service roles and blame them for it, you're just being a jerk.Tim: No, it really is because I'm pretty sure a significant amount of your listeners and people initially started off working in tech support, or customer service, or help desk or something like that, and you really do become the dumping ground for the customers' frustrations because you are the only person they get to talk to. And you have to not only take that, but you have to try and do the emotional labor behind soothing them as well as fixing the actual problem. And it's really, really difficult. I feel like the people who have that in their background are some of the best consultants, some of the best DevRel folks, and the best at talking to people because they're used to being able to get some technical details out of folks who may not be very technical, who may be under emotional distress, and certainly in high stress situations. So yeah, AWS support, really anybody who has support, especially paid support—phone or chat otherwise—hats off again. That is a service that is thankless, it is a service that is almost always underpaid, and is almost always under appreciated.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: I'll take another team that's similar to that respect: Commerce Platform. That is the team that runs all of AWS billing. And you would be surprised that I'm thanking them, but no, it's not the cynical approach of, “Thanks for making it so complicated so I could have a business.” No, I would love it if it were so simple that I had to go find something else to do because the problem was that easy for customers to solve. That is the ideal and I hope, sincerely, that we can get there.But everything that happens in AWS has to be metered and understood as far as who has done what, and charge people appropriately for it. It is also generally invisible; people don't understand anything approaching the scale of that, and what makes it worst of all, is that if suddenly what they were doing broke and customers weren't built for their usage, not a single one of them would complain about it because, “All right, I'll take it.” It's a thankless job that is incredibly key and central to making the cloud work at all, but it's a hard job.Tim: It really is. And is a lot of black magic and voodoo to really try and understand how this thing works. There's no simple way to explain it. I imagine if they were going to give you the index overview of how it works with a 10,000 feet, that alone would be, like, a 300 page document. It is a gigantic moving beast.And it is one of those things where scale will show all the flaws. And no one has scale I think like AWS does. So, the folks that have to work and maintain that are just really, again, they're under appreciated for all that they do. I also think that—you know, you talk about the same thing in other orgs, as we talked about the folks that handle the billing and stuff like that, but you mentioned AWS, and I was thinking the other day how it's really awesome that I've got my AWS driver. I have the same, like, group of three or four folks that do all my deliveries for AWS.And they have been inundated over this past year-and-a-half with more and more and more stuff. And yet, I've still managed—my stuff is always put down nicely on my doorstep. It's never thrown, it's not damaged. I'm not saying it's never been damaged, but it's not damaged, like, maybe FedEx I've [laugh] had or some other delivery services where it's just, kind of, carelessly done. They still maintain efficiency, they maintain professionalism [unintelligible 00:21:45] talking to folks.What they've had to do at their scale and at that the amount of stuff they've had to do for deliveries over this past year-and-a-half has just been incredible. So, I want to extend it also to, like, the folks who are working in the distribution centers. Like, a lot of us here talk about AWS as if that's Amazon, but in essence, it is those folks that are working those more thankless and invisible jobs in the warehouses and fulfillment centers, under really bad conditions sometimes, who's still plug away at it. I'm glad that Amazon is at least saying they're making efforts to improve the conditions there and improve the pay there, things like that, but those folks have enabled a lot of us to work during this pandemic with a lot of conveniences that they themselves would never be able to enjoy.Corey: Yeah. It's bad for society, but I'm glad it exists, obviously. The thing is, I would love it if things showed up a little more slowly if it meant that people could be treated humanely along the process. That said, I don't have any conception of what it takes to run a company with 1.2 million people.I have learned that as you start managing groups and managing managers of groups, it's counterintuitive, but so much of what you do is no longer you doing the actual work. It is solely through influence and delegation. You own all of the responsibility but no direct put-finger-on-problem capability of contributing to the fix. It takes time at that scale, which is why I think one of the dumbest series of questions from, again, another group that deserves a fair bit of credit which is journalists because this stuff is hard, but a naive question I hear a lot is, “Well, okay. It's been 100 days. What has Adam Selipsky slash Andy Jassy changed completely about the company?”It's, yeah, it's a $1.6 trillion company. They are not going to suddenly grab the steering wheel and yank. It's going to take years for shifts that they do to start manifesting in serious ways that are externally visible. That is how big companies work. You don't want to see a complete change in direction from large blue chip companies that run things. Like, again, everyone's production infrastructure. You want it to be predictable, you want it to be boring, and you want shifts to be gradual course corrections, not vast swings.Tim: I mean, Amazon is a company with a population of a medium to medium-large sized city and a market cap of the GDP of several countries. So, it is not a plucky startup; it is not this small little tech company. It is a vast enterprise that's distributed all over the world with a lot of folks doing a lot of different jobs. You cannot, as you said, steer that ship quickly.Corey: I grew up in Maine and Amazon has roughly the same number employees as live in Maine. It is hard to contextualize how all of that works. There are people who work there that even now don't always know who Andy Jassy is. Okay, fine, but I'm not talking about don't know him on site or whatever. I'm saying they do not recognize the name. That's a very big company.Tim: “Andy who?”Corey: Exactly. “Oh, is that the guy that Corey makes fun of all the time?” Like, there we go. That's what I tend to live for.Tim: I thought that was Werner.Corey: It's sort of every one, though I want to be clear, I make it a very key point. I do not make fun of people personally because it—even if they're crap, which I do not believe to be the case in any of the names we've mentioned so far, they have friends and family who love and care about them. You don't want someone to go on the internet and Google their parent's name or something, and then just see people crapping all over. That's got to hurt. Let people be people. And, on some level, when you become the CEO of a company of that scale, you're stepping out of reality and into the pages of legend slash history, at some point. 200 years from now, people will read about you in history books, that's a wild concept.Tim: It is I think you mentioned something important that we would be remiss—especially Duckbill Group—to mention is that we're very thankful for our families, partners, et cetera, for putting up with us, pets, everybody. As part of our jobs, we invite strangers from the internet into our homes virtually to see behind us what is going on, and for those of us that have kids, that involves a lot of patience on their part, a lot of patients on our partners' parts, and other folks that are doing those kind of nurturing roles. You know, our pets who want to play with us are sitting there and not able to. It has not been easy for all of us, even though we're a remote company, but to work under these conditions that we have been over the past year-and-a-half. And I think that goes for a lot of the folks in industry where now all of a sudden, you've been occupying a room in the house or space in the house for some 18-plus months, where before you're always at work or something like that. And that's been a hell of an adjustment. And so we talk about that for us folks that are here pontificating on podcasts, or banging out code, but the adjustments and the things our families have had to go through and do to tolerate us being there cannot be overstated how important that is.Corey: Anyone else that's on your list of people to thank? And this is the problem because you're always going to forget people. I mean, the podcast production crew: the folks that turn our ramblings into a podcast, the editing, the transcription, all of it; the folks that HumblePod are just amazing. The fact that I don't have to worry about any of this stuff as if by magic, means that you're sort of insulated from it. But it's amazing to watch that happen.Tim: You know, honestly, I super want to thank just all the folks that take the time to interact with us. We do this job and Corey shitposts, and I shitpost and we talk, but we really do this and rely on the folks that do take the time to DM us, or tweet us, or mention us in the thread, or reach out in any way to ask us questions, or have a discussion with us on something we said, those folks encourage us, they keep us accountable, and they give us opportunities to learn to be better. And so I'm grateful for that. It would be—this role, this job, the thing we do where we're viewable and seen by the public would be a lot less pleasant if it wasn't for y'all. So, it's too many to name, but I do appreciate you.Corey: Well, thank you, I do my best. I find this stuff to be so boring if you couldn't have fun with it. And so many people can't have fun with it, so it feels like I found a cheat code for making enterprise software solutions interesting. Which even saying that out loud sounds like I'm shitposting. But here we are.Tim: Here we are. And of course, my thanks to you, Corey, for reaching out to me one day and saying, “Hey, what are you doing? Would you want to come interview with us at The Duckbill Group?”Corey: And it was great because, like, “Well, I did leave AWS within the last 18 months, so there might be a non-compete issue.” Like, “Oh, please, I hope so. Oh, please, oh, please, oh, please. I would love to pick that fight publicly.” But sadly, no one is quite foolish enough to take me up on it.Don't worry. That's enough of a sappy episode, I think. I am convinced that our next encounter on this podcast will be our usual aggressive self. But every once in a while it's nice to break the act and express honest and heartfelt appreciation. I'm really looking forward to next week with all of the various announcements that are coming out.I know people have worked extremely hard on them, and I want them to know that despite the fact that I will be making fun of everything that they have done, there's a tremendous amount of respect that goes into it. The fact that I can make fun of the stuff that you've done without any fear that I'm punching down somehow because, you know it is at least above a baseline level of good speaks volumes. There are providers I absolutely do not have that confidence towards them.Tim: [laugh]. Yeah, AWS, as the enterprise level service provider is an easy target for a lot of stuff. The people that work there are not. They do great work. They've got amazing people in all kinds of roles there. And they're often unseen for the stuff they do. So yeah, for all the folks who have contributed to what we're going to partake in at re:Invent—and it's a lot and I understand from having worked there, the pressure that's put on you for this—I'm super stoked about it and I'm grateful.Corey: Same here. If I didn't like this company, I would not have devoted years to making fun of it. Because that requires a diagnosis, not a newsletter, podcast, or shitposting Twitter feed. Tim, thank you so much for, I guess, giving me the impetus and, of course, the amazing name of the show to wind up just saying thank you, which I think is something that we could all stand to do just a little bit more of.Tim: My pleasure, Corey. I'm glad we could run with this. I'm, as always, happy to be on Screaming in the Cloud with you. I think now I get a vest and a sleeve. Is that how that works now?Corey: Exactly. Once you get on five episodes, then you end up getting the dinner jacket, just, like, hosting SNL. Same story. More on that to come in the new year. Thanks, Tim. I appreciate it.Tim: Thank you, Corey.Corey: Tim Banks, principal cloud economist here at The Duckbill Group. I am, of course, Corey Quinn, and thank you for listening.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Welcome to another episode of the Action and Ambition Podcast! Joining us today is Bryon Kroger, an Intelligence Officer who turned Product Owner. He serves as COO for Project Kessel Run at the Air Force Life Cycle Management Center. Having been on the receiving end of poor software from the Defence Acquisitions Cycle for seven years, Bryon has a deep appreciation for just how steep the cost of delay can be measured not just in dollars but sometimes in lives. His passion for reducing that cost led him to the acquisition community, where he embarked on a cloud-native journey for intelligence systems using PCF. Bryon has made a career of presenting to the highest levels of military leadership on topics ranging from national security to DevOps. Tune in to learn more!

Zac Smith, managing director Equinix Metal, is sharing how Equinix Metal runs the best hardware and networking in the industry, why pairing magical software with the right hardware is the future, and what Open19 means for sustainability in the data centre. Think modular components that slot in (including CPUs), liquid cooling that converts heat into energy, and a few other solutions that minimise the impact on the environment. But first, Zac tells us about the transition from Packet to Equinix Metal, his reasons for doing what he does, as well as the things that he is really passionate about, such as the most efficient data centres in the world and building for the love of it. This is a great follow-up to episode 18 because it goes deeper into the reasons that make Gerhard excited about the work that Equinix Metal is doing. This conversation with Zac puts it all into perspective. By the way, did you know that Equinix stands for Equality in the Internet Exchange?

Recorded live from Rubyconf 2021 in Denver, CO with an audience! Panelists from The Ruby on Rails Podcast, Code with Jason and Remote Ruby gathered to chat about why they were excited to attend Rubyconf, favorite episodes and to field listener questions. Moderated By: Jemma Issroff, The Ruby on Rails Podcast () Panelists: Andrew Mason, Remote Ruby (https://twitter.com/andrewmcodes) Jason Charnes, Remote Ruby (https://twitter.com/jmcharnes) Jason Swett, Code with Jason (https://twitter.com/JasonSwett) Emily Giurleo, The Ruby on Rails Podcast (https://twitter.com/EmilyGiurleo) Nick Schwaderer, The Ruby on Rails Podcast (https://twitter.com/Schwad4HD14) Show Notes & Links: Rubyconf 2021 (https://rubyconf.org/) WNB.rb (https://twitter.com/wnb_rb) Ruby5: A Twice-Weekly 5 Minute Ruby News Podcast (http://www.rubyinside.com/ruby5-ruby-news-podcast-2441.html) Ruby Weekly (https://rubyweekly.com/?m) Sponsored By: Honeybadger (https://www.honeybadger.io/) Honeybadger makes you a DevOps hero by combining error monitoring, uptime monitoring and check-in monitoring into a single, easy to use platform. Go to Honeybadger.io (https://www.honeybadger.io/) and discover how Starr, Josh, and Ben created a 100% bootstrapped monitoring solution. Scout APM (http://scoutapm.com/rubyonrails) Try their error monitoring and APM free for 14-days, no credit card needed! And as an added bonus for Ruby on Rails listeners: Scout will donate $5 to the open-source project of your choice when you deploy. Learn more at http://scoutapm.com/rubyonrails (http://scoutapm.com/rubyonrails).

Are Linux devs getting upset with the Python community? We weigh in on a nuanced issue. Plus the mass-mod resignation over at Rust, and Mike's thoughts on setting up a dev environment on Windows 11.

About StephanieStephanie Wong is an award-winning speaker, engineer, pageant queen, and hip hop medalist. She is a leader at Google with a mission to blend storytelling and technology to create remarkable developer content. At Google, she's created over 400 videos, blogs, courses, and podcasts that have helped developers globally. You might recognize her as the host of the GCP Podcast. Stephanie is active in her community, fiercely supporting women in tech and mentoring students.Links: Personal Website: https://stephrwong.com Twitter: https://twitter.com/stephr_wong TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the things that makes me a little weird in the universe is that I do an awful lot of… let's just call it technology explanation slash exploration in public, and turning it into a bit of a brand-style engagement play. What makes this a little on the weird side is that I don't work for a big company, which grants me a tremendous latitude. I have a whole lot of freedom that lets me be all kinds of different things, and I can't get fired, which is something I'm really good at.Inversely, my guest today is doing something remarkably similar, except she does work for a big company and could theoretically be fired if they were foolish enough to do so. But I don't believe that they are. Stephanie Wong is the head of developer engagement at Google. Stephanie, thank you for volunteering to suffer my slings and arrows about all of this.Stephanie: [laugh]. Thanks so much for having me today, Corey.Corey: So, at a very high level, you're the head of developer engagement, which is a term that I haven't seen a whole lot of. Where does that start and where does that stop?Stephanie: Yeah, so I will say that it's a self-proclaimed title a bit because of the nuance of what I do. I would say at its heart, I am still a part of developer relations. If you've heard of developer advocacy or developer evangelist, I would say this slight difference in shade of what I do is that I focus on scalable content creation and becoming a central figure for our developer audiences to engage and enlighten them with content that, frankly, is remarkable, and that they'd want to share and learn about our technology.Corey: Your bio is fascinating in that it doesn't start with the professional things that most people do with, “This is my title and this is my company,” is usually the first sentence people put in. Yours is, “Stephanie Wong is an award-winning speaker, engineer, pageant queen, and hip hop medalist.” Which is both surprising and more than a little bit refreshing because when I read a bio like that my immediate instinctive reaction is, “Oh, thank God. It's a real person for a change.” I like the idea of bringing the other aspects of what you are other than, “This is what goes on in an IDE, the end,” to your audience.Stephanie: That is exactly the goal that I had when creating that bio because I truly believe in bringing more interdisciplinary and varied backgrounds to technology. I, myself have gone through a very unconventional path to get to where I am today and I think in large part, my background has had a lot to do with my successes, my failures, and really just who I am in tech as an uninhibited and honest, credible person today.Corey: I think that there's a lack of understanding, broadly, in our industry about just how important credibility and authenticity are and even the source of where they come from. There are a lot of folks who are in the DevRel space—devrelopers, as I insist upon calling them, over their protests—where, on some level, the argument is, what is developer relations? “Oh, you work in marketing, but they're scared to tell you,” has been my gag on that one for a while. But they speak from a position of, “I know what's what because I have been in the trenches, working on these large-scale environments as an engineer for the last”—fill in the blank, however long it may have been—“And therefore because I have done things, I am going to tell you how it is.” You explicitly call out that you don't come from the traditional, purely technical background. Where did you come from? It's unlikely that you've sprung fully-formed from the forehead of some god, but again, I'm not entirely sure how Google finds and creates the folks that it winds up advancing, so maybe you did.Stephanie: Well, to tell you the truth. We've all come from divine creatures. And that's where Google sources all employees. So. You know. But—[laugh].Corey: Oh, absolutely. “We climbed to the top of Olympus and then steal fire from the gods.” “It's like, isn't that the origin story of Prometheus?” “Yeah, possibly.” But what is your background? Where did you come from?Stephanie: So, I have grown up, actually, in Silicon Valley, which is a little bit ironic because I didn't go to school for computer science or really had the interest in becoming an engineer in school. I really had no idea.Corey: Even been more ironic than that because most of Silicon Valley appears to never have grown up at all.Stephanie: [laugh]. So, true. Maybe there's a little bit of that with me, too. Everybody has a bit of Peter Pan syndrome here, right? Yeah, I had no idea what I wanted to do in school and I just knew that I had an interest in communicating with one another, and I ended up majoring in communication studies.I thought I wanted to go into the entertainment industry and go into production, which is very different and ended up doing internships at Warner Brothers Records, a YouTube channel for dance—I'm a dancer—and I ended up finding a minor in digital humanities, which is sort of this interdisciplinary minor that combines technology and the humanities space, including literature, history, et cetera. So, that's where I got my start in technology, getting an introduction to information systems and doing analytics, studying social media for certain events around the world. And it wasn't until after school that I realized that I could work in enterprise technology when I got an offer to be a sales engineer. Now, that being said, I had no idea what sales engineering was. I just knew it had something to do with enterprise technology and communications, and I thought it was a good fit for my background.Corey: The thing that I find so interesting about that is that it breaks the mold of what people expect, when, “If someone's going to talk to me about technology—especially coming from a”—it's weird; it's one of the biggest companies on the planet, and people still on some level equate Google with the startup-y mentality of being built in someone's garage. That's an awfully big garage these days, if that's even slightly close to true, which it isn't. But there's this idea of, “Oh, you have to go to Stanford. You have to get a degree in computer science. And then you have to go and do this, this, this, this, and this.”And it's easy to look dismissively at what you're doing. “Communications? Well, all that would teach you to do is communicate to people clearly and effectively. What possible good is that in tech?” As we look around the landscape and figure out exactly why that is so necessary in tech, and also so lacking?Stephanie: Exactly. I do think it's an underrated skill in tech. Maybe it's not so much anymore, but I definitely think that it has been in the past. And even for developers, engineers, data scientists, other technical practitioner, especially as a person in DevRel, I think it's such a valuable skill to be able to communicate complex topics simply and understandably to a wide variety of audiences.Corey: The big question that I have for you because I've talked to an awful lot of folks who are very concerned about the way that they approach developer relations, where—they'll have ratios, for example—where I know someone and he insists that he give one deeply technical talk for every four talks that are not deeply technical, just because he feels the need to re-establish and shore up his technical bona fides. Now, if there's one thing that people on the internet love, it is correcting people on things that are small trivia aspect, or trying to pull out the card that, “Oh, I've worked on this system for longer than you've worked on this system, therefore, you should defer to me.” Do you find that you face headwinds for not having the quote-unquote, “Traditional” engineering technical background?Stephanie: I will say that I do a bit. And I did, I would say when I first joined DevRel, and I don't know if it was much more so that it was being imposed on me or if it was being self-imposed, something that I felt like I needed to prove to gain credibility, not just in my organization, but in the industry at large. And it wasn't until two or three years into it, that I realized that I had a niche myself. It was to create stories with my content that could communicate these concepts to developers just as effectively. And yes, I can still prove that I can go into an hour-long or a 45-minute-long tech talk or a webinar about a topic, but I can also easily create a five to ten-minute video that communicates concepts and inspires audiences just the same, and more importantly, be able to point to resources, code labs, tutorials, GitHub repos, that can allow the audience to be hands-on themselves, too. So really, I think that it was over time that I gained more experience and realized that my skill sets are valuable in a different way, and it's okay to have a different background as long as you bring something to the table.Corey: And I think that it's indisputable that you do. The concept of yours that I've encountered from time to time has always been insightful, it is always been extremely illuminating, and—you wouldn't think of this as worthy of occasion and comment, but I feel it needs to be said anyway—at no point in any of your content did I feel like I was being approached in a condescending way, where at every point it was always about uplifting people to a level of understanding, rather than doing the, “Well, I'm smarter than you and you couldn't possibly understand the things that I've been to.” It is relatable, it is engaging, and you add a very human face to what is admittedly an area of industry that is lacking in a fair bit of human element.Stephanie: Yeah, and I think that's the thing that many folks DevRel continue to underline is the idea of empathy, empathizing with your audiences, empathizing with the developers, the engineers, the data engineers, whoever it is that you're creating content for, it's being in their shoes. But for me, I may not have been in those shoes for years, like many other folks historically have been in for DevRel, but I want to at least go through the journey of learning a new piece of technology. For example, if I'm learning a new platform on Google Cloud, going through the steps of creating a demo, or walking through a tutorial, and then candidly explaining that experience to my audience, or creating a video about it. I really just reject the idea of having ego in tech and I would love to broaden the opportunity for folks who came from a different background like myself. I really want to just represent the new world of technology where it wasn't full of people who may have had the privilege to start coding at a very early age, in their garages.Corey: Yeah, privilege of, in many respects, also that privilege means, “Yes, I had the privilege of not having to have friends and deal with learning to interact with other human beings, which is what empowered me to build this company and have no social skills whatsoever.” It's not the aspirational narrative that we sometimes are asked to believe. You are similar in some respects to a number of things that I do—by which I mean, you do it professionally and well and I do it as basically performance shitpost art—but you're on Twitter, you make videos, you do podcasts, you write long-form and short-form as well. You are sort of all across the content creation spectrum. Which of those things do you prefer to do? Which ones of those are things you find a little bit more… “Well, I have to do it, but it's not my favorite?” Or do you just tend to view it as content is content; you just look at different media to tell your story?Stephanie: Well, I will say any form of content is queen—I'm not going to say king, but—[laugh] content is king, content is queen, it doesn't matter.Corey: Content is a baroness as it turns out.Stephanie: [laugh]. There we go. I have to say, so given my background, I mentioned I was into production and entertainment before, so I've always had a gravitation towards video content. I love tinkering with cameras. Actually, as I got started out at Google Cloud, I was creating scrappy content using webcams and my own audio equipment, and doing my own research, and finding lounges and game rooms to do that, and we would just upload it to our own YouTube channel, which probably wasn't allowed at the time, but hey, we got by with it.And eventually, I got approached by DevRel to start doing it officially on the channel and I was given budget to do it in-studio. And so that was sort of my stepping stone to doing this full-time eventually, which I never foresaw for myself. And so yeah, I have this huge interest in—I'm really engaged with video content, but once I started expanding and realizing that I could repurpose that content for podcasting, I could repurpose it for blogs, then you start to realize that you can shard content and expand your reach exponentially with this. So, that's when I really started to become more active on social media and leverage it to build not just content for Google Cloud, but build my own brand in tech.Corey: That is the inescapable truth of DevRel done right is that as you continue doing it, in time, in your slice of the industry, it is extremely likely that your personal brand eclipses the brand of the company that you represent. And it's in many ways a test of corporate character—if it makes sense—as do how they react to that. I've worked in roles before I started this place where I was starting to dabble with speaking a lot, and there was always a lot of insecurity that I picked up of, “Well, it feels like you're building your personal brand, not advancing the company here, and we as a company do not see the value in you doing that.” Direct quote from the last boss I had. And, well, that partially explains why I'm here, I suppose.But there's insecurity there. I'd see the exact opposite coming out of Google, especially in recent times. There's something almost seems to be a renaissance in Google Cloud, and I'm not sure where it came from. But if I look at it across the board, and you had taken all the labels off of everything, and you had given me a bunch of characteristics about different companies, I would never have guessed that you were describing Google when you're talking about Google Cloud. And perhaps that's unfair, but perceptions shape reality.Stephanie: Yeah, I find that interesting because I think traditionally in DevRel, we've also hired folks for their domain expertise and their brand, depending on what you're representing, whether it's in the Kubernetes space or Python client library that you're supporting. But it seems like, yes, in my case, I've organically started to build my brand while at Google, and Google has been just so spectacular in supporting that for me. But yeah, it's a fine line that I think many people have to walk. It's like, do you want to continue to build your own brand and have that carry forth no matter what company you stay at, or if you decide to leave? Or can you do it hand-in-hand with the company that you're at? For me, I think I can do it hand-in-hand with Google Cloud.Corey: It's taken me a long time to wrap my head around what appears to be a contradiction when I look at Google Cloud, and I think I've mostly figured it out. In the industry, there is a perception that Google as an entity is condescending and sneering toward every other company out there because, “You're Google, you know how to do all these great, amazing things that are global-spanning, and over here at Twitter for Pets, we suck doing these things.” So, Google is always way smarter and way better at this than we could ever hope to be. But that is completely opposed to my personal experiences talking with Google employees. Across the board, I would say that you all are self-effacing to a fault.And I mean that in the sense of having such a limited ego, in some cases, that it's, “Well, I don't want to go out there and do a whole video on this. It's not about me, it's about the technology,” are things that I've had people who work at Google say to me. And I appreciate the sentiment; it's great, but that also feels like it's an aloofness. It also fails to humanize what it is that you're doing. And you are a, I've got to say, a breath of fresh air when it comes to a lot of that because your stories are not just, “Here's how you do a thing. It's awesome. And this is all the intricacies of the API.”And yeah, you get there, but you also contextualize that in a, “Here's why it matters. Here's the problem that solves. Here is the type of customer's problem that this is great for,” rather than starting with YAML and working your way up. It's going the other way, of, “We want to sell some underpants,” or whatever it is the customer is trying to do today. And that is the way that I think is one of the best ways to drive adoption of what's going on because if you get people interested and excited about something—at least in my experience—they're going to figure out how the API works. Badly in many cases, but works. But if you start on the API stuff, it becomes a solution looking for a problem. I like your approach to this.Stephanie: Thank you. Yeah, I appreciate that. I think also something that I've continued to focus on is to tell stories across products, and it doesn't necessarily mean within just Google Cloud's ecosystem, but across the industry as well. I think we need to, even at Google, tell a better story across our product space and tie in what developers are currently using. And I think the other thing that I'm trying to work on, too, is contextualizing our products and our launches not just across the industry, but within our product strategy. Where does this tie in? Why does it matter? What is our forward-looking strategy from here? When we're talking about our new data cloud products or analytics, [unintelligible 00:17:21], how does this tie into our API strategy?Corey: And that's the biggest challenge, I think, in the AI space. My argument has been for a while—in fact, I wrote a blog post on it earlier this year—that AI and machine learning is a marvelously executed scam because it's being pushed by cloud providers and the things that you definitely need to do a machine learning experiment are a bunch of compute and a whole bunch of data that has to be stored on something, and wouldn't you know it, y'all sell that by the pound. So, it feels, from a cynical perspective, which I excel at espousing, that approach becomes one of you're effectively selling digital pickaxes into a gold rush. Because I see a lot of stories about machine learning how to do very interesting things that are either highly, highly use-case-specific, which great, that would work well, for me too, if I ever wind up with, you know, a petabyte of people's transaction logs from purchasing coffee at my national chain across the country. Okay, that works for one company, but how many companies look like that?And on the other side of it, “It's oh, here's how we can do a whole bunch of things,” and you peel back the covers a bit, and it looks like, “Oh, but you really taught me here is bias laundering?” And, okay. I think that there's a definite lack around AI and machine learning of telling stories about how this actually matters, what sorts of things people can do with it that aren't incredibly—how do I put this?—niche or a problem in search of a solution?Stephanie: Yeah, I find that there are a couple approaches to creating content around AI and other technologies, too, but one of them being inspirational content, right? Do you want to create something that tells the story of how I created a model that can predict what kind of bakery item this is? And we're going to do it by actually showcasing us creating the outcome. So, that's one that's more like, okay. I don't know how relatable or how appropriate it is for an enterprise use case, but it's inspirational for new developers or next gen developers in the AI space, and I think that can really help a company's brand, too.The other being highly niche for the financial services industry, detecting financial fraud, for example, and that's more industry-focused. I found that they both do well, in different contexts. It really depends on the channel that you're going to display it on. Do you want it to be viral? It really depends on what you're measuring your content for. I'm curious from you, Corey, what you've seen across, as a consumer of content?Corey: What's interesting, at least in my world, is that there seems to be, given that what I'm focusing on first and foremost is the AWS ecosystem, it's not that I know it the best—I do—but at this point, it's basically Stockholm Syndrome where it's… with any technology platform when you've worked with it long enough, you effectively have the most valuable of skill sets around it, which is not knowing how it works, but knowing how it doesn't, knowing what the failure mode is going to look like and how you can work around that and detect it is incredibly helpful. Whereas when you're trying something new, you have to wait until it breaks to find the sharp edges on it. So, there's almost a lock-in through, “We failed you enough times,” story past a certain point. But paying attention to that ecosystem, I find it very disjointed. I find that there are still events that happen and I only find out when the event is starting because someone tweets about it, and for someone who follows 40 different official AWS RSS feeds, to be surprised by something like that tells me, okay, there's not a whole lot of cohesive content strategy here, that is at least making it easy for folks to consume the things that they want, especially in my case where even the very niche nature of what I do, my interest is everything.I have a whole bunch of different filters that look for various keywords and the rest, and of course, I have helpful folks who email me things constantly—please keep it up; I'm a big fan—worst case, I'd rather read something twice than nothing. So, it's helpful to see all of that and understand the different marketing channels, different personas, and the way that content approaches, but I still find things that slip through the cracks every time. The thing that I've learned—and it felt really weird when I started doing it—was, I will tell the same stories repeatedly in different forums, or even the same forum. I could basically read you a Twitter thread from a year ago, word-for-word, and it would blow up bigger than it did the first time. Just because no one reads everything.Stephanie: Exactly.Corey: And I've already told my origin story. You're always new to someone. I've given talks internally at Amazon at various times, and I'm sort of loud and obnoxious, but the first question I love to ask is, “Raise your hand if you've never heard of me until today.” And invariably, over three-quarters of the room raises their hand every single time, which okay, great. I think that's awesome, but it teaches me that I cannot ever expect someone to have, quote-unquote, “Done the reading.”Stephanie: I think the same can be said about the content that I create for the company. You can't assume that people, A) have seen my tweets already or, B) understand this product, even if I've talked about it five times in the past. But yes, I agree. I think that you definitely need to have a content strategy and how you format your content to be more problem-solution-oriented.And so the way that I create content is that I let them fall into three general buckets. One being that it could be termed definition: talking about the basics, laying the foundation of a product, defining terms around a topic. Like, what is App Engine, or Kubeflow 101, or talking about Pub/Sub 101.The second being best practices. So, outlining and explaining the best practices around a topic, how do you design your infrastructure for scale and reliability.And the third being diagnosis: investigating; exploring potential issues, as you said; using scripts; Stackdriver logging, et cetera. And so I just kind of start from there as a starting point. And then I generally follow a very, very effective model. I'm sure you're aware of it, but it's called the five point argument model, where you are essentially telling a story to create a compelling narrative for your audience, regardless of the topic or what bucket that topic falls into.So, you're introducing the problem, you're sort of rising into a point where the climax is the solution. And that's all to build trust with your audience. And as it falls back down, you're giving the results in the conclusion, and that's to inspire action from your audience. So, regardless of what you end up talking about this problem-solution model—I've found at least—has been highly effective. And then in terms of sharing it out, over and over again, over the span of two months, that's how you get the views that you want.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: See, that's a key difference right there. I don't do anything regular in terms of video as part of my content. And I do it from time to time, but you know, getting gussied up and whatnot is easier than just talking into a microphone. As I record this, it's Friday, I'm wearing a Hawaiian shirt, and I look exactly like the middle-aged dad that I am. And for me at least, a big breakthrough moment was realizing that my audience and I are not always the same.Weird confession for someone in my position: I don't generally listen to podcasts. And the reason behind that is I read very quickly, and even if I speed up a podcast, I'm not going to be able to consume the information nearly as quickly as I could by reading it. That, amongst other reasons, is one of the reasons that every episode of this show has a full transcript attached to it. But I'm not my audience. Other people prefer to learn by listening and there's certainly nothing wrong with that.My other podcast, the AWS Morning Brief, is the spoken word version of the stuff that I put out in my newsletter every week. And that is—it's just a different area for people to consume the content because that's what works for them. I'm not one to judge. The hard part for me was getting over that hump of assuming the audience was like me.Stephanie: Yeah. And I think the other key part of is just mainly consistency. It's putting out the content consistently in different formats because everybody—like you said—has a different learning style. I myself do. I enjoy visual styles.I also enjoy listening to podcasts at 2x speed. [laugh]. So, that's my style. But yeah, consistency is one of the key things in building content, and building an audience, and making sure that you are valuable to your audience. I mean, social media, at the end of the day is about the people that follow you.It's not about yourself. It should never be about yourself. It's about the value that you provide. Especially as somebody who's in DevRel in this position for a larger company, it's really about providing value.Corey: What are the breakthrough moments that I had relatively early in my speaking career—and I think it's clear just from what you've already said that you've had a similar revelation at times—I gave a talk, that was really one of my first talks that went semi-big called, “Terrible Ideas in Git.” It was basically, learn how to use Git via anti-pattern. What it secretly was, was under the hood, I felt it was time I learned Git a bit better than I did, so I pitched it and I got a talk accepted. So well, that's what we call a forcing function. By the time I give that talk, I'd better be [laugh] able to have built a talk that do this intelligently, and we're going to hope for the best.It worked, but the first version of that talk I gave was super deep into the plumbing of Git. And I'm sure that if any of the Git maintainers were in the audience, they would have found it great, but there aren't that many folks out there. I redid the talk and instead approached it from a position of, “You have no idea what Git is. Maybe you've heard of it, but that's as far as it goes.” And then it gets a little deeper there.And I found that making the subject more accessible as opposed to deeper into the weeds of it is almost always the right decision from a content perspective. Because at some level, when you are deep enough into the weeds, the only way you're going to wind up fixing something or having a problem that you run into get resolved, isn't by listening to a podcast or a conference talk; it's by talking to the people who built the thing because at that level, those are the only people who can hang at that level of depth. That stops being fodder for conference talks unless you turn it into an after-action report of here's this really weird thing I learned.Stephanie: Yeah. And you know, to be honest, the one of the most successful pieces of content I've created was about data center security. I visited a data center and I essentially unveiled what our security protocols were. And that wasn't a deeply technical video, but it was fun and engaging and easily understood by the masses. And that's what actually ended up resulting in the highest number of views.On top of that, I'm now creating a video about our subsea fiber optic cables. Finding that having to interview experts from a number of different teams across engineering and our strategic negotiators, it was like a monolith of information that I had to take in. And trying to format that into a five-minute story, I realized that bringing it up a layer of abstraction to help folks understand this at a wider level was actually beneficial. And I think it'll turn into a great piece of content. I'm still working on it now. So, [laugh] we'll see how it turns out.Corey: I'm a big fan of watching people learn and helping them get started. The thing that I think gets lost a lot is it's easy to assume that if I look back in time at myself when I was first starting my professional career two decades ago, that I was exactly like I am now, only slightly more athletic and can walk up a staircase without getting winded. That's never true. It never has been true. I've learned a lot about not just technology but people as I go, and looking at folks are entering the workforce today through the same lens of, “Well, that's not how I would handle that situation.” Yeah, no kidding. I have two decades of battering my head against the sharp edges and leaving dents in things to inform that opinion.No, when I was that age, I would have handled it way worse than whatever it is I'm critiquing at the time. But it's important to me that we wind up building those pathways and building those bridges so that people coming into the space, first, have a clear path to get here, and secondly, have a better time than I ever did. Where does the next generation of talent come from has been a recurring question and a recurring theme on the show.Stephanie: Yeah. And that's exactly why I've been such a fierce supporter of women in tech, and also, again, encouraging a broader community to become a part of technology. Because, as I said, I think we're in the midst of a new era of technology, of people from all these different backgrounds in places that historically have had more remote access to technology, now having the ability to become developers at an early age. So, with my content, that's what I'm hoping to drive to make this information more easily accessible. Even if you don't want to become a Google Cloud engineer, that's totally fine, but if I can help you understand some of the foundational concepts of cloud, then I've done my job well.And then, even with women who are already trying to break into technology or wanting to become a part of it, then I want to be a mentor for them, with my experience not having a technical background and saying yes to opportunities that challenged me and continuing to build my own luck between hard work and new opportunities.Corey: I can't wait to see how this winds up manifesting as we see understandings of what we're offering to customers in different areas in different ways—both in terms of content and terms of technology—how that starts to evolve and shift. I feel like we're at a bit of an inflection point now, where today if I graduate from school and I want to start a business, I have to either find a technical co-founder or I have to go to a boot camp and learn how to code in order to build something. I think that if we can remove that from the equation and move up the stack, sure, you're not going to be able to build the next Google or Pinterest or whatnot from effectively Visual Basic for Interfaces, but you can build an MVP and you can then continue to iterate forward and turn it into something larger down the road. The other part of it, too, is that moving up the stack into more polished solutions rather than here's a bunch of building blocks for platforms, “So, if you want a service to tell you whether there's a picture of a hot dog or not, here's a service that does exactly that.” As opposed to, “Oh, here are the 15 different services, you can bolt together and pay for each one of them and tie it together to something that might possibly work, and if it breaks, you have no idea where to start looking, but here you go.” A packaged solution that solves business problems.Things move up the stack; they do constantly. The fact is that I started my career working in data centers and now I don't go to them at all because—spoiler—Google, and Amazon, and people who are not IBM Cloud can absolutely run those things better than I can. And there's no differentiated value for me in solving those global problems locally. I'd rather let the experts handle stuff like that while I focus on interesting problems that actually affect my business outcome. There's a reason that instead of running all the nonsense for lastweekinaws.com myself because I've worked in large-scale WordPress hosting companies, instead I pay WP Engine to handle it for me, and they, in turn, hosted on top of Google Cloud, but it doesn't matter to me because it's all just a managed service that I pay for. Because me running the website itself adds no value, compared to the shitpost I put on the website, which is where the value derives from. For certain odd values of value.Stephanie: [laugh]. Well, two things there is that I think we actually had a demo created on Google Cloud that did detect hot dogs or not hot dogs using our Vision API, years in the past. So, thanks for reminding me of that one.Corey: Of course.Stephanie: But yeah, I mean, I completely agree with that. I mean, this is constantly a topic in conversation with my team members, and with clients. It's about higher level of abstractions. I just did a video series with our fellow, Eric Brewer, who helped build cloud infrastructure here at Google over the past ten decades. And I asked him what he thought the future of cloud would be in the next ten years, and he mentioned, “It's going to be these higher levels of abstraction, building platforms on top of platforms like Kubernetes, and having more services like Cloud run serverless technologies, et cetera.”But at the same time, I think the value of cloud will continue to be providing optionality for developers to have more opinionated services, services like GKE Autopilot, et cetera, that essentially take away the management of infrastructure or nodes that people don't really want to deal with at the end of the day because it's not going to be a competitive differentiator for developers. They want to focus on building software and focusing on keeping their services up and running. And so yeah, I think the future is going to be that, giving developers flexibility and freedom, and still delivering the best-of-breed technology. If it's covering something like security, that's something that should be baked in as much as possible.Corey: You're absolutely right, first off. I'm also looking beyond it where I want to be able to build a website that is effectively Twitter, only for pets—because that is just a harebrained enough idea to probably raise a $20 million seed round these days—and I just want to be able to have the barks—those are like tweets, only surprisingly less offensive and racist—and have them just be stored somewhere, ideally presumably under the hood somewhere, it's going to be on computers, but whether it's in containers, or whether it's serverless, or however is working is the sort of thing that, “Wow, that seems like an awful lot of nonsense that is not central nor core to my business succeeding or failing.” I would say failing, obviously, except you can lose money at scale with the magic of things like SoftBank. Here we are.And as that continues to grow and scale, sure, at some point I'm going to have bespoke enough needs and a large enough scale where I do have to think about those things, but building the MVP just so I can swindle some VCs is not the sort of thing where I should have to go to that depth. There really should be a golden-path guardrail-style thing that I can effectively drag and drop my way into the next big scam. And that is, I think, the missing piece. And I think that we're not quite ready technologically to get there yet, but I can't shake the feeling and the hope that's where technology is going.Stephanie: Yeah. I think it's where technology is heading, but I think part of the equation is the adoption by our industry, right? Industry adoption of cloud services and whether they're ready to adopt services that are that drag-and-drop, as you say. One thing that I've also been talking a lot about is this idea of service-oriented networking where if you have a service or API-driven environment and you simply want to bring it to cloud—almost a plug-and-play there—you don't really want to deal with a lot of the networking infrastructure, and it'd be great to do something like PrivateLink on AWS, or Private Service Connect on Google Cloud.While those conversations are happening with customers, I'm finding that it's like trying to cross the Grand Canyon. Many enterprise customers are like, “That sounds great, but we have a really complex network topology that we've been sitting on for the past 25 years. Do you really expect that we're going to transition over to something like that?” So, I think it's about providing stepping stones for our customers until they can be ready to adopt a new model.Corey: Yeah. And of course, the part that never gets said out loud but is nonetheless true and at least as big of a deal, “And we have a whole team of people who've built their entire identity around that network because that is what they work on, and they have been ignoring cloud forever, and if we just uplift everything into a cloud where you folks handle that, sure, it's better for the business outcome, but where does that leave them?” So, they've been here for 25 years, and they will spend every scrap of political capital they've managed to accumulate to torpedo a cloud migration. So, any FUD they can find, any horse-trading they can do, anything they can do to obstruct the success of a cloud initiative, they're going to do because people are people, and there is no real plan to mitigate that. There's also the fact that unless there's a clear business value story about a feature velocity increase or opening up new markets, there's also not an incentive to do things to save money. That is never going to be the number one priority in almost any case short of financial disaster at a company because everything they're doing is building out increasing revenue, rather than optimizing what they're already doing.So, there's a whole bunch of political challenges. Honestly, moving the computer stuff from on-premises data centers into a cloud provider is the easiest part of a cloud migration compared to all of the people that are involved.Stephanie: Yeah. Yeah, we talked about serverless and all the nice benefits of it, but unless you are more a digitally-born, next-gen developer, it may be a higher burden for you to undertake that migration. That's why we always [laugh] are talking about encouraging people to start with newer surfaces.Corey: Oh, yeah. And that's the trick, too, is if you're trying to learn a new cloud platform these days—first, if you're trying to pick one, I'd be hard-pressed to suggest anything other than Google Cloud, with the possible exception of DigitalOcean, just because the new user experience is so spectacularly good. That was my first real, I guess, part of paying attention to Google Cloud a few years ago, where I was, “All right, I'm going to kick the tires on this and see how terrible this interface is because it's a Google product.” And it was breathtakingly good, which I did not expect. And getting out of the way to empower someone who's new to the platform to do something relatively quickly and straightforwardly is huge. And sure, there's always room to prove, but that is the right area to focus on. It's clear that the right energy was spent in the right places.Stephanie: Yeah. I will say a story that we don't tell quite as well as we should is the One Google story. And I'm not talking about just between Workspace and Google Cloud, but our identity access management and knowing your Google account, which everybody knows. It's not like Microsoft, where you're forced to make an account, or it's not like AWS where you had a billion accounts and you hate them all.Corey: Oh, my God, I dread logging into the AWS console every time because it is such a pain in the ass. I go to cloud.google.com sometimes to check something, it's like, “Oh, right. I have to dig out my credentials.” And, “Where's my YubiKey?” And get it. Like, “Oh. I'm already log—oh. Oh, right. That's right. Google knows how identity works, and they don't actively hate their customers. Okay.” And it's always a breath of fresh air. Though I will say that by far and away, the worst login experience I've seen yet is, of course, Azure.Stephanie: [laugh]. That's exactly right. It's Google account. It's yours. It's personal. It's like an Apple iCloud account. It's one click, you're in, and you have access to all the applications. You know, so it's the same underlying identity structure with Workspace and Gmail, and it's the same org structure, too, across Workspace and Google Cloud. So, it's not just this disingenuous financial bundle between GCP and Workspace; it's really strategic. And it's kind of like the idea of low code or no code. And it looks like that's what the future of cloud will be. It's not just by VMs from us.Corey: Yeah. And there are customers who want to buy VMs and that's great. Speed up what they're doing; don't get in the way of people giving you their money, but if you're starting something net-new, there's probably better ways to do it. So, I want to thank you for taking as much time as you have to wind up going through how you think about, well, the art of storytelling in the world of engineering. If people want to learn more about who you are, what you're up to, and how you approach things, where can they find you?Stephanie: Yeah, so you can head to stephrwong.com where you can see my work and also get in touch with me if you want to collaborate on any content. I'm always, always, always open to that. And my Twitter is @stephr_wong.Corey: And we will, of course, put links to that in the [show notes 00:40:03]. Thank you so much for taking the time to speak with me.Stephanie: Thanks so much.Corey: Stephanie Wong, head of developer engagement at Google Cloud. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment telling me that the only way to get into tech these days is, in fact, to graduate with a degree from Stanford, and I can take it from you because you work in their admissions office.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Pixel 6a leaks, Moto Watch 100, iMessage reactions on Android, At a Glance What is Esper.io? What is DevOps? Android 12L deep dive: Every change, thoroughly documented Android 11 on nearly a quarter of all devices, but it's not the most used version of Google's OS Wear OS Share Surges on Samsung's Highest Quarterly Smartwatch Shipments in Q3 2021 Google Pixel 6a will run on a Tensor chip, but a lesser camera than Pixel 6 Fairphone 2 passes six years of software updates with new Android 10 beta Moto Watch 100 launches with two-week battery life, $100 price tag, and no Wear OS in sight Your Android phone now properly displays iMessage reactions — if you use Google Messages Pixel 6 preps big 'At a Glance' upgrade that shows BT device battery, doorbell & 'flashlight is on' alerts, more DuckDuckGo Wants to Stop Apps From Tracking You on Android Why does my Pixel 6 always ask for a PIN when I have a fingerprint set? Pixel 6 phantom dialing could be related to the Safety App and Car Crash Detection. Why won't my Pixel 6 play nice with my Mazda Android Auto head unit? Read our show notes here: https://bit.ly/3cJhsEO Hosts: Jason Howell and Ron Richards Guest: David Ruddock Subscribe to All About Android at https://twit.tv/shows/all-about-android. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit Sponsors: betterhelp.com/android UserWay.org/twit

Pixel 6a leaks, Moto Watch 100, iMessage reactions on Android, At a Glance What is Esper.io? What is DevOps? Android 12L deep dive: Every change, thoroughly documented Android 11 on nearly a quarter of all devices, but it's not the most used version of Google's OS Wear OS Share Surges on Samsung's Highest Quarterly Smartwatch Shipments in Q3 2021 Google Pixel 6a will run on a Tensor chip, but a lesser camera than Pixel 6 Fairphone 2 passes six years of software updates with new Android 10 beta Moto Watch 100 launches with two-week battery life, $100 price tag, and no Wear OS in sight Your Android phone now properly displays iMessage reactions — if you use Google Messages Pixel 6 preps big 'At a Glance' upgrade that shows BT device battery, doorbell & 'flashlight is on' alerts, more DuckDuckGo Wants to Stop Apps From Tracking You on Android Why does my Pixel 6 always ask for a PIN when I have a fingerprint set? Pixel 6 phantom dialing could be related to the Safety App and Car Crash Detection. Why won't my Pixel 6 play nice with my Mazda Android Auto head unit? Read our show notes here: https://bit.ly/3cJhsEO Hosts: Jason Howell and Ron Richards Guest: David Ruddock Subscribe to All About Android at https://twit.tv/shows/all-about-android. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit Sponsors: betterhelp.com/android UserWay.org/twit

Pixel 6a leaks, Moto Watch 100, iMessage reactions on Android, At a Glance What is Esper.io? What is DevOps? Android 12L deep dive: Every change, thoroughly documented Android 11 on nearly a quarter of all devices, but it's not the most used version of Google's OS Wear OS Share Surges on Samsung's Highest Quarterly Smartwatch Shipments in Q3 2021 Google Pixel 6a will run on a Tensor chip, but a lesser camera than Pixel 6 Fairphone 2 passes six years of software updates with new Android 10 beta Moto Watch 100 launches with two-week battery life, $100 price tag, and no Wear OS in sight Your Android phone now properly displays iMessage reactions — if you use Google Messages Pixel 6 preps big 'At a Glance' upgrade that shows BT device battery, doorbell & 'flashlight is on' alerts, more DuckDuckGo Wants to Stop Apps From Tracking You on Android Why does my Pixel 6 always ask for a PIN when I have a fingerprint set? Pixel 6 phantom dialing could be related to the Safety App and Car Crash Detection. Why won't my Pixel 6 play nice with my Mazda Android Auto head unit? Read our show notes here: https://bit.ly/3cJhsEO Hosts: Jason Howell and Ron Richards Guest: David Ruddock Subscribe to All About Android at https://twit.tv/shows/all-about-android. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit Sponsors: betterhelp.com/android UserWay.org/twit

Pixel 6a leaks, Moto Watch 100, iMessage reactions on Android, At a Glance What is Esper.io? What is DevOps? Android 12L deep dive: Every change, thoroughly documented Android 11 on nearly a quarter of all devices, but it's not the most used version of Google's OS Wear OS Share Surges on Samsung's Highest Quarterly Smartwatch Shipments in Q3 2021 Google Pixel 6a will run on a Tensor chip, but a lesser camera than Pixel 6 Fairphone 2 passes six years of software updates with new Android 10 beta Moto Watch 100 launches with two-week battery life, $100 price tag, and no Wear OS in sight Your Android phone now properly displays iMessage reactions — if you use Google Messages Pixel 6 preps big 'At a Glance' upgrade that shows BT device battery, doorbell & 'flashlight is on' alerts, more DuckDuckGo Wants to Stop Apps From Tracking You on Android Why does my Pixel 6 always ask for a PIN when I have a fingerprint set? Pixel 6 phantom dialing could be related to the Safety App and Car Crash Detection. Why won't my Pixel 6 play nice with my Mazda Android Auto head unit? Read our show notes here: https://bit.ly/3cJhsEO Hosts: Jason Howell and Ron Richards Guest: David Ruddock Subscribe to All About Android at https://twit.tv/shows/all-about-android. Get episodes ad-free with Club TWiT at https://twit.tv/clubtwit Sponsors: betterhelp.com/android UserWay.org/twit

This week's episode, "The Developer-Led Landscape," features Tyler Jewell, Managing Director at Dell Technologies Capital, who shares his top trends in DevOps and views on how DevOps and DevOps landscapes differ. Jewell also sheds light on causes for the growing tech talent shortage and the increasing imbalance of demand for solutions in the API economy. 

About BrianI lead the Google Cloud Product and Industry Marketing team. We're focused on accelerating the growth of Google Cloud by establishing thought leadership, increasing demand and usage, enabling our sales teams and partners to tell our product stories with excellence, and helping our customers be the best advocates for us.Before joining Google, I spent over 25 years in product marketing or engineering in different forms. I started my career at Microsoft and had a very non-traditional path for 20 years. I worked in every product division except for cloud. I did marketing, product management, and engineering roles. And, early on, I was the first speech writer for Steve Ballmer and worked on Bill Gates' speeches too. My last role was building up the Microsoft Surface business from scratch and as VP of the hardware businesses. After Microsoft, I spent a year as CEO at a hardware startup called Doppler Labs, where we made a run at transforming hearing, and then two years as VP at Amazon Web Services leading product marketing, developer advocacy, and a bunch more marketing teams. I have three kids still at home, Barty, Noli, and Alder, who are all named after trees in different ways. My wife Edie and I met right at the beginning of our first year at Yale University, where I studied math, econ, and philosophy and was the captain of the Swim and Dive team my senior year. Edie has a PhD in forestry and runs a sustainability and forestry consulting firm she started, that is aptly named “Three Trees Consulting”. We love the outdoors, tennis, running, and adventures in my 1986 Volkswagen Van, which is my first and only car, that I can't bring myself to get rid of.Links: Twitter: https://twitter.com/IsForAt LinkedIn: https://www.linkedin.com/in/brhall/ Episode 10: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/episode-10-education-is-not-ready-for-teacherless/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. Set up a meeting with a Redis expert during re:Invent, and you'll not only learn how you can become a Redis hero, but also have a chance to win some fun and exciting prizes. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: Writing ad copy to fit into a 30 second slot is hard, but if anyone can do it the folks at Quali can. Just like their Torque infrastructure automation platform can deliver complex application environments anytime, anywhere, in just seconds instead of hours, days or weeks. Visit Qtorque.io today and learn how you can spin up application environments in about the same amount of time it took you to listen to this ad.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined today by a special guest that I've been, honestly, antagonizing for years now. Once upon a time, he spent 20 years at Microsoft, then he wound up leaving—as occasionally people do, I'm told—and going to AWS, where according to an incredibly ill-considered affidavit filed in a court case, he mostly focused on working on PowerPoint slides. AWS is famously not a PowerPoint company, and apparently, you can't change culture. Now, he's the VP of Product and Industry Marketing at Google Cloud. Brian Hall, thank you for joining me.Brian: Hi, Corey. It's good to be here.Corey: I hope you're thinking that after we're done with our conversation. Now, unlike most conversations that I tend to have with folks who are, honestly, VP level at large cloud companies that I enjoy needling, we're not going to talk about that today because instead, I'd rather focus on a minor disagreement we got into on Twitter—and I mean that in the truest sense of disagreement, as opposed to the loud, angry, mutual blocking, threatening to bomb people's houses, et cetera, nonsense that appears to be what substitutes for modern discourse—about, oh, a month or so ago from the time we're recording this. Specifically, we talked about, I'm in favor of job-hopping to advance people's career, and you, as we just mentioned, spent 20 years at Microsoft and take something of the opposite position. Let's talk about that. Where do you stand on the idea?Brian: I stand in the position that people should optimize for where they are going to grow the most. And frankly, the disagreement was less about job-hopping because I'm going to explain how I job-hopped at Microsoft effectively.Corey: Excellent. That is the reason I'm asking you rather than poorly stating your position and stuffing you like some sort of Christmas turkey straw-man thing.Brian: And I would argue that for many people, changing jobs is the best thing that you can do, and I'm often an advocate for changing jobs even before sometimes people think they should do it. What I mostly disagreed with you on is simply following the money on your next job. What you said is if a—and I'm going to get it somewhat wrong—but if a company is willing to pay you $40,000 more, or some percentage more, you should take that job now.Corey: Gotcha.Brian: And I don't think that's always the case, and that's what we're talking about.Corey: This is the inherent problem with Twitter is that first, I tend to write my Twitter threads extemporaneously without a whole lot of thought being put into things—kind of like I live my entire life, but that's neither here nor there—Brian: I was going to say, that comes across quite clearly.Corey: Excellent. And 280 characters lacks nuance. And I definitely want to have this discussion; this is not just a story where you and I beat heads and not come to an agreement on this. I think it's that we fundamentally do agree on the vast majority of this, I just want to make sure that we have this conversation in a way, in a forum that doesn't lend itself to basically empowering the worst aspects of my own nature. Read as, not Twitter.Brian: Great. Let's do that.Corey: So, my position is, and I was contextualizing this from someone who had reached out who was early in their career, they had spent a couple of years at AWS and they were entertaining an offer elsewhere for significantly more money. And this person, I believe I can—I believe it's okay for me to say this: she—was very concerned that, “I don't want to look like I'm job-hopping, and I don't dislike my team. My manager is great. I feel disloyal for leaving. What should I do?”Which first, I just want to say how touched I am that someone who is early in their career and not from a wildly overrepresented demographic like you and I felt a sense of safety and security in reaching out to ask me that question. I really wish more people would take that kind of initiative. It's hard to inspire, but here we are. And my take to her was, “Oh, my God. Take the money.” That was where this thread started because when I have conversations with people about those things, it becomes top of mind, and I think, “Hmm, maybe there's a one-to-many story that becomes something that is actionable and useful.”Brian: Okay, so I'm going to give two takes on this. I'll start with my career because I was in a similar position as she was, at one point in my career. My background, I lucked into a job at Microsoft as an intern in 1995, and then did another internship in '96 and then started full time on the Internet Explorer team. And about a year-and-a-half into that job, I—we had merged with the Windows '98 team and I got the opportunity to work on Bill Gates's speech for the Windows '98 launch event. And I—after that was right when Steve Ballmer became president of Microsoft and he started doing a lot more speeches and asked to have someone to help him with speeches.And Chris Capossela, who's now the CMO at Microsoft, said, “Hey, Brian. You interested in doing this for Steve?” And my first reaction was, well, even inside Microsoft, if I move, it will be disloyal. Because my manager's manager, they've given me great opportunities, they're continuing to challenge me, I'm learning a bunch, and they advised not doing it.Corey: It seems to me like you were in a—how to put this?—not to besmirch the career you have wrought with the sweat of your brow and the toil of your back, but in many ways, you were—in a lot of ways—you were in the right place at the right time, riding a rocket ship, and built opportunities internally and talked to folks there, and built the relationships that enabled you to thrive inside of a company's ecosystem. Is that directionally correct?Brian: For sure. Yet, there's also, big companies are teams of teams, and loyalty is more often with the team and the people that you work with than the 401k plan. And in this case, you know, I was getting this pressure that says, “Hey, Brian. You're going to get all these opportunities. You're doing great doing what you're doing.”And I eventually had the luck to ask the question, “Hey, if I go there and do this role”—and by the way, nobody had done it before, and so part of their argument was, “You're young, Steve's… Steve. Like, you could be a fantastic ball of flames.” And I said, “Okay, if [laugh] let's say that happens. Can I come back? Can I come back to the job I was doing before?”And they were like, “Yeah, of course. You're good at what you do.” To me, which was, “Okay, great. Then I'm gone. I might as well go try this.” And of course, when I started at Microsoft, I was 20, 21, and I thought I'd be there for two or three years and then I'd end up going back to school or somewhere else. But inside Microsoft, what kept happening as I just kept getting new opportunities to do something else that I'd learned a bunch from, and I ultimately kind of created this mentality for how I thought about next job of, “Am I going to get more opportunities if I am able to be successful in this new job?” Really focused on optionality and the ability to do work that I want to do and have more choices to do that.Corey: You are also on a I almost want to call it a meteoric trajectory. In some ways. You effectively went from—what was your first role there? It was—Brian: The lowest level of college hire you can do at Microsoft, effectively.Corey: Yeah. All the way on up to at the end of it the Corporate VP for Microsoft Devices. It seems to me that despite the fact that you spent 20 years there, you wound up having a bunch of different jobs and an entire career trajectory internal to the organization, which is, let's be clear, markedly different from some of the folks I've interviewed at various times, in my career as an employer and as a technical interviewer at a consulting company, where they'd been somewhere for 15 years, and they had one year of experience that they repeated 15 times. And it was one of the more difficult things that I encountered is that some folks did not take ownership of their career and focus on driving it forward.Brian: Yeah, that, I had the opposite experience, and that is what kept me there that long. After I would finish a job, I would say, “Okay, what do I want to learn how to do next, and what is a challenge that would be most interesting?” And initially, I had to get really lucky, honestly, to be able to get these. And I did the work, but I had to have the opportunity, and that took luck. But after I had a track record of saying, “Hey, I can jump from being a product marketer to being a speechwriter; I can do speechwriting and then go do product management; I can move from product management into engineering management.”I can do that between different businesses and product types, you build the ability to say, “Hey, I can learn that if you give me the chance.” And it, frankly, was the unique combination of experiences I had by having tried to do these other things that gave me the opportunity to have a fast trajectory within the company.Corey: I think it's also probably fair to say that Microsoft was a company that, in its dealings with you, is operating in good faith. And that is a great thing to find when you see it, but I'm cynical; I admit that. I see a lot of stories where people give and sacrifice for the good of the company, but that sacrifice is never reciprocated. And we've all heard the story of folks who will put their nose to the grindstone to ship something on time, only to be rewarded with a layoff at the end, and stories like that resonate.And my argument has always been that you can't love a company because the company can't love you back. And when you're looking at do I make a career move or do I stay, my argument is that is the best time to be self-interested.Brian: Yeah, I don't think—companies are there for the company, and certainly having a culture that supports people that wants to create opportunity, having a manager that is there truly to make you better and to give you opportunity, that all can happen, but it's within a company and you have to do the work in order to try and get into that environment. Like, I worked hard to have managers who would support my growth, would give me the bandwidth and leash early on to not be perfect at what I'm doing, and that always helped me. But you get to go pick them in a company like that, or in the industry in general, you get—just like when a manager is hiring you, you also get to understand, hey, is this a person I want to work for?But I want to come back to the main point that I wanted to make. When I changed jobs, I did it because I wanted to learn something new and I thought that would have value for me in the medium-term and long-term, versus how do I go max cash in what I'm already good at?Corey: Yes.Brian: And that's the root of what we were disagreeing with on Twitter. I have seen many people who are good at something, and then another company says, “Hey, I want you to do that same thing in a worse environment, and we'll pay you more.”Corey: Excellence is always situational. Someone who is showered in accolades at one company gets fired at a different company. And it's not because they suddenly started sucking; it's because the tools and resources that they needed to succeed were present in one environment and not the other. And that varies from person to person; when someone doesn't work out of the company, I don't have a default assumption that there's something inherently wrong with them.Of course, I look at my own career and the sheer, staggeringly high number of times I got fired, and I'm starting to think, “Huh. The only consistent factor in all of these things is me. Nah, couldn't be my problem. I just worked for terrible places, for terrible people. That's got to be the way it works.” My own peace of mind. I get it. That is how it feels sometimes and it's easy to dismiss that in different ways. I don't want to let my own bias color this too heavily.Brian: So, here are the mistakes that I've seen made: “I'm really good at something; this other company will pay me to do just that.” You move to do it, you get paid more, but you have less impact, you don't work with as strong of people, and you don't have a next step to learn more. Was that a good decision? Maybe. If you need the money now, yes, but you're a little bit trading short-term money for medium-and long-term money where you're paid for what you know; that's the best thing in this industry. We're paid for what we know, which means as you're doing a job, you can build the ability to get paid more by knowing more, by learning more, by doing things that stretch you in ways that you don't already know.Corey: In 2006, I bluffed my way through a technical interview and got a job as a Unix systems administrator for a university that was paying $65,000 a year, and I had no idea what I was going to do with all of that money. It was more money than I could imagine at that point. My previous high watermark, working for an ethically challenged company in a sales role at a target comp of 55, and I was nowhere near it. So okay, let's go somewhere else and see what happens. And after I'd been there a month or two, my boss sits me down and said, “So”—it's our annual compensation adjustment time—“Congratulations. You now make $68,000.”And it's just, “Oh, my God. This is great. Why would I ever leave?” So, I stayed there a year and I was relatively happy, insofar as I'm ever happy in a job. And then a corporate company came calling and said, “Hey, would you consider working here?”“Well, I'm happy here and I'm reasonably well compensated. Why on earth would I do that?” And the answer was, “Well, we'll pay you $90,000 if you do.” It's like, “All right. I guess I'm going to go and see what the world holds.”And six weeks later, they let me go. And then I got another job that also paid $90,000 and I stayed there for two years. And I started the process of seeing what my engagement with the work world look like. And it was a story of getting let go periodically, of continuing to claw my way up and, credit where due, in my 20s I was in crippling credit card debt because I made a bunch of poor decisions, so I biased early on for more money at almost any cost. At some point that has to stop because there's always a bigger paycheck somewhere if you're willing to go and do something else.And I'm not begrudging anyone who pursues that, but at some point, it ceases to make a difference. Getting a raise from $68,000 to $90,000 was life-changing for me. Now, getting a $30,000 raise? Sure, it'd be nice; I'm not turning my nose up at it, don't get me wrong, but it's also not something that moves the needle on my lifestyle.Brian: Yeah. And there are a lot of those dimensions. There's the lifestyle dimension, there's the learning dimension, there's the guaranteed pay dimension, there's the potential paid dimension, there is the who I get to work with, just pure enjoyment dimension, and they all matter. And people should recognize that job moves should consider all of these.And you don't have to have the same framework over time as well. I've had times where I really just wanted to bear down and figure something out. And I did one job at Microsoft for basically six years. It changed in terms of scope of things that I was marketing, and which division I was in, and then which division I was in, and then which division I was in—because Microsoft loves a good reorg—but I basically did the same job for six years at one point, and it was very conscious. I was trying to get really good at how do I manage a team system at scale. And I didn't want to leave that until I had figured that out. I look back and I think that's one of the best career decisions I ever made, but it was for reasons that would have been really hard to explain to a lot of people.Corey: Let's also be very clear here that you and I are well-off white dudes in tech. Our failure mode is pretty much a board seat and a book deal. In fact, if—Brian: [laugh].Corey: —I'm not mistaken, you are on the board of something relatively recently. What was that?Brian: United Way of King County. It's a wonderful nonprofit in the Seattle area.Corey: Excellent. And I look forward to reading your book, whenever that winds up dropping. I'm sure it'll be only the very spiciest of takes. For folks who are earlier in their career and who also don't have the winds of privilege at their backs the way that you and I do, this also presents radically differently. And I've spoken to a number of folks who are not wildly over-represented about this topic, in the wake of that Twitter explosion.And what I heard was interesting in that having a manager who has your back counts for an awful lot and is something that is going to absolutely hold you to a particular company, even when it might make sense on paper for you to leave. And I think that there's something strong there. My counterargument is okay, so you turn down the offer, a month goes past and your manager gives notice because they're going to go somewhere else. What then? It's one of those things where you owe your employer a duty of confidentiality, you owe them a responsibility to do your best work, to conduct yourself in an ethical manner, but I don't believe you owe them loyalty in the sense of advancing their interests ahead of what's best for you and your career arc.And what's right for any given person is, of course, a nuanced and challenging thing. For some folks, yeah, going out somewhere else for more money doesn't really change anything and is not what they should optimize for. For other folks, it's everything. And I don't think either of those takes is necessarily wrong. I think it comes down to it depends on who you are, and what your situation is, and what's right for you.Brian: Yeah. I totally agree. For early in career, in particular, I have been a part of—I grew up in the early versions of the campus hiring program at Microsoft, and then hired 500-plus, probably, people into my teams who were from that.Corey: You also do the same thing at AWS if I'm not mistaken. You launched their first college hiring program that I recall seeing, or at least that's what scuttlebutt has it.Brian: Yes. You're well-connected, Corey. We started something called the Product Marketing Leadership Development Program when I was in AWS marketing. And then one year, we hired 20 people out of college into my organization. And it was not easy to do because it meant using, quote-unquote, “Tenured headcount” in order to do it. There wasn't some special dispensation because they were less paid or anything, and in a world where headcount is a unit of work, effectively.And then I'm at Google now, in the Google Cloud division, and we have a wonderful program that I think is really well done, called the Associate Product Marketing Manager Program, APMM. And what I'd say is for the people early in career, if you get the opportunity to have a manager who's super supportive, in a system that is built to try and grow you, it's a wonderful opportunity. And by ‘system built to grow you,' it really is, do you have the support to get taught what you need to get taught on the job? Are you getting new opportunities to learn new things and do new things at a rapid clip? Are you shipping things into the market such that you can see the response and learn from that response, versus just getting people's internal opinions, and then are people stretching roles in order to make them amenable for someone early in career?And if you're in a system that gives you that opportunity—like let's take your example earlier. A person who has a manager who's greatly supportive of them and they feel like they're learning a lot, that manager leaves, if that system is right, there's another manager, or there's an opportunity to put your hand up and say, “Hey, I think I need a new place,” and that will be supported.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: I have a history of mostly working in small companies, to the point where I consider a big company to be one that has more than 200 employees, so, the idea of radically transitioning and changing teams has never really been much on the table as I look at my career trajectory and my career arc. I have seen that I've gotten significant 30% raises by changing jobs. I am hard-pressed to identify almost anyone who has gotten that kind of raise in a single year by remaining at a company.Brian: One hundred percent. Like, I know of people who have, but it—Corey: It happens, but it's—Brian: —is very rare.Corey: —it's very rare.Brian: It's, it's, it's almost the, the, um, the example that proves the point. I getting that totally wrong. But yes, it's very rare, but it does happen. And I think if you get that far out of whack, yes. You should… you should go reset, especially if the other attributes are fine and you don't feel like you're just going to get mercenary pay.What I always try and advise people is, in the bigger companies, you want to be a good deal. You don't want to be a great deal or a bad deal. Where a great deal is you're getting significantly underpaid, a bad deal is, “Uh oh. We hired this person to [laugh] senior,” or, “We promoted them too early,” because then the system is not there to help you, honestly, in the grand scheme of things. A good deal means, “Hey, I feel like I'm getting better work from this person for what we are giving them than what the next clear alternative would be. Let's support them and help them grow.” Because at some level, part of your compensation is getting your company to create opportunities for you to grow. And part of the reason people go to a manager is they know they'll give them that compensation.Corey: I am learning this the interesting way, as we wind up hiring and building out our, currently, nine-person company. It's challenging for us to build those opportunities while bootstrapped, but it is incumbent upon us, you're right. That is a role of management is how do you identify growth opportunities for people, ideally, while remaining at the company, but sometimes that means that helping them land somewhere else is the right path for their next growth step.Brian: Well, that brings up a word for managers. What you pay your employees—and I'm talking big company here, not people like yourself, Corey, where you have to decide whether you reinvesting money or putting in an individual.Corey: Oh, yes—Brian: But at big companies—Corey: —a lot of things that apply when you own a company are radically departed from—Brian: Totally.Corey: —what is—Brian: Totally.Corey: —common guidance.Brian: Totally. At a big company, managers, you get zero credit for how much your employees get paid, what their raise is, whether they get promoted or not in the grand scheme of things. That is the company running their system. Yes, you helped and the like, but it's—like, when people tell me, “Hey, Brian, thank you for supporting my promotion.” My answer is always, “Thank you for having earned it. It's my job to go get credit where credit is due.” And that's not a big part of my job, and I honestly believe that.Where you do get credit with people, where you do show that you're a good manager is when you have the conversations with them that are harder for other people to have, but actually make them better; when you encourage them in the right way so that they grow faster; when you treat them fairly as a human being, and mostly when you do the thing that seems like it's against your own interest.Corey: That resonates. The moments of my career as a manager that I'm proud of stuff are the ones that I would call borderline subversive: telling a candidate to take the competing offer because they're going to have a better time somewhere else is one of those. But my philosophy ties back to the idea of job-hopping, where I'm going to know these people for longer than either of us are going to remain in our current role, on some level. I am curious what your approach is, given that you are now at the, I guess, other end for folks who are just starting out. How do you go about getting people into Cloud marketing? And, on some level, wouldn't you consider that being a form of abuse?Brian: [laugh]. It depends on whether they get to work with you or not, Corey.Corey: There is that.Brian: I won't tell you which one's abuse or not. So first, getting people into cloud marketing is getting people who do not have deeply technical backgrounds in most cases, oftentimes fantastic—people who are fantastic at understanding other people and communicating really well, and it gives them an opportunity to be in tech in one of the fastest-growing, fastest-changing spaces in the world. And so to go to a psych major, a marketing major, an American studies major, a history major, who can understand complex things and then communicate really well, and say, “Hey, I have an opportunity for you to join the fastest growing space in technology,” is often compelling.But their question kind of is, “Hey, will I be able to do it?” And the answer has to be, “Hey, we have a program that helps you learn, and we have a set of managers who know how to teach, and we create opportunities for you to learn on the job, and we're invested in you for more than a short period of time.” With that case, I've been able to hire and grow and work with, in some cases, people for over 15 years now that I worked with at Microsoft. I'm still in touch with many of the people from the Product Marketing Leadership Development Program at AWS. And we have a fantastic set of APMMs at Google, and it creates a wonderful opportunity for them.Increasingly, we're also seeing that it is one of the best ways to find people from many backgrounds. We don't just show up at the big CompSci schools. We're getting some wonderful, wonderful people from all the states in the nation, from the historically black colleges and universities, from majors that tend to represent very different groups than the traditional tech audiences. And so it's been a great source of broadening our talent pool, too.Corey: There's a lot to be said for having people who've been down this path and seeing the failure modes, reaching out to make so that the next generation—for lack of a better term—has an easier time than we did. The term I've heard for the concept is ‘send the elevator back down,' which is important. I think it's—otherwise we wind up with a whole industry that looks an awful lot like it did 20 years ago, and that's not ideal for anyone. The paths that you and I walked are closed, so sitting here telling people they should do what we did has very strong, ‘Okay, Boomer' energy to it.Brian: [laugh].Corey: There are different paths, and the world and industry are changing radically.Brian: Absolutely. And my—like, the biggest thing that I'd say here is—and again, just coming back to the one thing we disagreed on—look at the bigger picture and own your career. I would never say that isn't the case, but the bigger picture means not just what you're getting paid tomorrow, but are you learning more? What new options is it creating for you? And when I speak options, I mean, will you have more jobs that you can do that excite you after you do that job? And those things matter in addition to the pay.Corey: I would agree with that. Money is not everything, but it's also not nothing.Brian: Absolutely.Corey: I will say though you spent 20 years at Microsoft. I have no doubt that you are incredibly adept at managing your career, at managing corporate politics, at advancing your career and your objectives and your goals and your aspirations within Microsoft, but how does that translate to companies that have radically different corporate cultures? We see this all the time with founders who are ex-Google or ex-Microsoft, and suddenly it turns out that the things that empower them to thrive in the large corporate environment doesn't really work when you're a five-person startup, and you don't have an entire team devoted to that one thing that needs to get done.Brian: So, after Microsoft, I went to a company called Doppler Labs for a year. It was a pretty well-funded startup that made smart earbuds—this was before AirPods had even come out—and I was really nervous about the going from big company to startup thing, and I actually found that move pretty easy. I've always been kind of a hands-on, do-it-yourself, get down in the details manager, and that's served me well. And so getting into a startup and saying, “Hey, I get to just do stuff,” was almost more fun. And so after that—we ended up folding, but it was a wonderful ride; that's a much longer conversation—when I got to Amazon and I was in AWS—and by the way, the one division I never worked at Microsoft was Azure or its predecessor server and tools—and so part of the allure of AWS was not only was it another trillion-dollar company in my backwater hometown, but it was also cloud computing, was the space that I didn't know well.And they knew that I knew the discipline of product marketing and a bunch of other things quite well, and so I got that opportunity. But I did realize about four months in, “Oh, crap. Part of the reason that I was really successful at Microsoft is I knew how everything worked.” I knew where things have been tried and failed, I knew who to go ask about how to do things, and I knew none of that at Amazon. And it is a—a lot of what allows you to move fast, make good decisions, and frankly, be politically accepted, is understanding all that context that nobody can just tell you. So, I will say there is a cost in terms of your productivity and what you're able to get done when you move from a place that you're good at to a place that you're not good at yet.Corey: Way back in episode 10 of this podcast—as we get suspiciously close to 300 as best I can tell—I had Lynn Langit get on as a guest. And she was in the Microsoft MVP program, the AWS Hero program, and the Google Expert program. All three at once—Brian: Lynn is fantastic.Corey: It really is.Brian: Lynn is fantastic.Corey: I can only assume that you listened to that podcast and decided, huh, all three, huh? I can beat that. And decided that—Brian: [laugh].Corey: —instead of being in the volunteer to do work for enormous multinational companies group, you said, “No, no, no. I'm going to be a VP in all three of those.” And here we are. Now that you are at Google, you have checked all three boxes. What is the next mountain to climb for you?Brian: I have no clue. I have no clue. And honestly—again, I don't know how much of this is privilege versus by being forward-looking. I've honestly never known where the heck I was going to go in my career. I've just said, “Hey, let's have a journey, and let's optimize for doing something you want to do that is going to create more opportunities for you to do something you want to do.”And so even when I left Microsoft, I was in a great position. I ran the Surface business, and HoloLens, and a whole bunch of other stuff that was really fun, but I also woke up one day and realized, “Oh, my gosh. I've been at Microsoft for 20 years. If I stay here for the next job, I'm earning the right to get another job at Microsoft, more so than anything else, and there's a big world out there that I want to explore a bit.” And so I did the startup; it was fun, I then thought I'd do another startup, but I didn't want to commute to San Francisco, which I had done.And then I found most of the really, really interesting startups in Seattle were cloud-related and I had this opportunity to learn about cloud from, arguably, one of the best with AWS. And then when I left AWS, I left not knowing what I was going to do, and I kind of thought, “Okay, now I'm going to do another cloud-oriented startup.” And Google came, and I realized I had this opportunity to learn from another company. But I don't know what's next. And what I'm going to do is try and do this job as best I can, get it to the point where I feel like I've done a job, and then I'll look at what excites me looking forward.Corey: And we will, of course, hold on to this so we can use it for your performance review, whenever that day comes.Brian: [laugh].Corey: I want to thank you for taking so much time to speak with me today. If people care more about what you have to say, perhaps you're hiring, et cetera, et cetera, where can they find you?Brian: Twitter, IsForAt: I-S-F-O-R-A-T. I'm certainly on Twitter. And if you want to connect professionally, I'm happy to do that on LinkedIn.Corey: And we will, of course, put links to those things in the [show notes 00:36:03]. Thank you so much for being so generous with your time. I appreciate it. I know you have a busy week of, presumably, attempting to give terrible names to various cloud services.Brian: Thank you, Corey. Appreciate you having me.Corey: Indeed. Brian Hall, VP of Product and Industry Marketing at Google Cloud. I am Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment in the form of a PowerPoint deck.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

In this episode we look at can AI help me see better in a cost effective way! Grant Everybody, welcome to another episode of click AI radio. Okay, I have in the house today with me, someone I've been very excited to talk with. He and his organization reached out to me and I was quite surprised when I saw the cool AI solution that they have been bringing to the market. And Carlos has been giving me a little background on this. And I think you'll be excited to hear what it is he's putting together. But first and foremost, welcome, Carlos Anchia. You got Yeah. All right. There you go. Carlos, please, welcome and introduce yourself. Carlos Hey, Grant. Thanks a lot for having us on. Like you said, my name is Carlos Anchia. I'm the CEO of Plainsight AI. And we're bringing to market an end to end computer vision AI platform. I'm really, really happy to be here love talking about AI, computer vision, and how we can get more people to use it. Grant So okay, so tell me a little bit about what got you going down here. As you and I were just chatting a moment ago, there's so many components to AI, or it's such a broad range of technologies there. What got you thinking about the CV or the computer vision space? What problem? What How did you get started there? Carlos Yeah, that's a really good question. So like you said, AI, the breadth of AI is huge, you have deep learning, you have machine learning, forecasting, prediction, computer vision. And these are just a few. There's a lot of different applications for AI and places you can go down and succeed in. From our respect, we really, we really focus in on computer vision, specifically how to apply that to imagery and video. Today, there's a huge amounts of data going throughout the internet and in enterprise storage classes, where you can't really extract the value of that data unless you actually perform some sort of computer vision machine learning on that type of data. So we're really extracting the value of the picture or the video. So it can be understood by machines. So think of a dog and a cat in a in a picture, right? Those cases, the machine doesn't know it's a dog and a cat, you have to train it. And that's where computer vision comes in. And really, we got into it because we were pulled in by customers, customers of ours wanted to start doing more computer vision and leveraging our platform that we had around high throughput, ingestion, and event driven pipelines. So these customers came to us and hey, you know, this is great, we'd love to really use this for computer vision. And the more and more that kept happening, we kept retooling around the platform. And finally, the platform from end to end is purpose built to do computer vision technology. And it really allows us to focus in on on what we're good at today. Right? And that's really delivering value within the computer vision space. Grant So I remember the first time I wrote some of the OpenCV framework code, right. And that was my first introduced introduction to it. This is a number of years ago. And I started thinking, Oh, this is so cool. So I'm writing all this Python code, right, building this stuff out. And then I'm thinking, how many people you know, are actually leveraging this platform and look at even though open CV is cool, and it's got a lot of capability, it still takes a lot, you know, to get everything out of there. So can you talk about how you relate to that open CV? And what is it that you're doing relative to that? And how much easier do you guys make this? Carlos Yeah, so I mean, you hit the nail on the head there, right? So from a developer perspective, it's really around, I need to learn open CV, I need to learn Python, I need to learn containerization I need to learn deployments. There's a variety of different companies that, you know, they're all great in their own right, right. Every one of those companies that we just talked about organizations are contributing tremendously to AI. But from a developer's perspective, you really have to learn a little bit of everything to be able to orchestrate a solution. And finally, when you get to, hey, I use AI. Let's pretend we're looking at strawberries. Hey, look, I built a model that the Texas strawberry that is your over the moon excited, but the very next thing is around, okay, how do I take that and deploy it 1000 times over in a field across the world and understand how to make that in an operational fashion where you know, it can be supported, maintained update, and that's really where we have this this crux of an organization where it's really different building something on a on a desk for a one time use. And and there's a lot of wins through that process. But then taking that and operationalizing for business driving revenues driving corporate goals around, why would that feature is being implemented, that's really where we come in, we want to be able to take off that that single single path of workflow where it's a little bit of everything to orchestrate a solution, and provide a centralized place where other people, including developers can go and help build that workflow in a meaningful way where it's complete. Grant So operationalizing, those models, I find, that's one of the biggest, or the most challenging aspects to this, it's one thing as you know, to, to build out enough to sort of prove something out and get some of the initial feedback, but to actually get it into production. I think I saw MIT not long ago, maybe this a year ago, now, they had come out with this report, it was through the Boston Consulting Group as well, they'd mentioned something about, hey, you know, 10% of organizations doing AI are getting return on their investment. And, and, of course, when you look at all of the investment of the takes for the business to really stand up all the data scientists and all the ML work. And you can see why the numbers translate that way. So to me, it feels like not only doing this in the area of CV, but the problem you're really trying to solve it feels like is you're attacking that ROI problem, which is you could take this kind of capability into business say you don't have to stand up all of these deep technical capabilities. Rather, you can achieve ROI sooner than rather than laters. Is that Is that accurate? Carlos That's correct. And I think it's really through the adoption of technology and you hit a you hit a really strong point for us there around the the difference between it works and it's operational. That's that's really the path of your your, you're less there in the CV world and more there with the DevOps ml ops portion of it, getting machines running consistently, with the right versions of deployment strategy, that latter half of it, just as important as the model building pieces of it. But even after you get to that piece, you need a way to improve, and improvement in the model is very costly if it's not automated. Because I mean, you can just look at the the loss for a simple detector, like a strawberry, where, you know, if the model starts to perform poorly, you're not pulling as many strawberries out of the field. So you need a way to be able to update that model, quickly, get training data into a platform and push a new model back out. And it's really around how fast you can go end to end with that workflow. Again, and again. And again. And again, this is that continuous improvement that we have born into us from previous software development life, but really in in machine learning and computer vision, your ability to train, retrain, and redeploy. That is where you really get the benefit out of your workflow. Grant Well, that really confirms my experience with AI will. Typically I'll refer to the term that we've been using, as we call it a SmartStep. It's that notion that I need to be able to refactor my models and take in consideration that changed context around me, whether it comes in from the world or from the customer, or whatever that means, some level of adjustments taken place that begins to invalidate my previous AI model. And I need to be able to quickly make those adjustments. That's fascinating. How long typically does it take for you to do that kind of refactoring of your models? Is that Is it a day? Is it a year, a month? Or the answer is? Well, it depends. Carlos So it's twofold, right? So it's, it's hours to do that. But it really depends on the complexity of the model, and how long you have to train. But in an automated workflow, you're you're continuously adding data to your training set, that are lower quality predictions, where you can retrain automatically when you hit a certain threshold, and then validate the model and push that back out into your production alized environment. So it's it when you go to develop these sort of workflows. You really have to start with whatever I build, I know I have to improve on later. So that improvement cycle ends up costing a lot if it's not part of the initial discussion around how do we count strawberries, right? So it's evident you and I can nerd out on this. Let me shift the focus a little bit and ask about it from say your your customer Write from their perspective, what is it that they need to be able to do to be successful with your solution? What skills or capabilities do they have to bring to the table? Yeah, and I think it's, I think I have this conversation a lot with our clients. And it's really less about them having technology around data science and building model, and more around a collaborative environment, where organizations, you know, they have a culture of success. But that culture of success is really borne by holding hands through the fire, it's, it's being able to commit and lean in when the organization sees something that's really important to them, either either from a technical perspective or revenue perspective. And it's these companies and these types of people that get to rally around a centralized platform where they can build and collaborate with machine learning computer vision applications. And, you know, it's it's a, it's really interesting to see the companies that succeed here, because it's really based on a culture of winning, right, where the wind doesn't have to be the hardest, most technical, logically difficult problem, because complexity really drives timelines. And if you're looking to change from an organization's perspective, start getting the little wins, get the little wins, start having some adoption within the company around, wow, computer vision is working. We've identified these problems in a few hours, we have a solution deployed, you start building this sense of confidence in the organization where you can take on those larger tasks. But you have to start with a build up, you can't just go right to the highest ROI problem. No one starts at human genome sequencing. Grant Have you, or do we got a problem? Yeah. Back up, back up. So So all right. So it means to me it sounds like as an organization to succeed with this getting my problem definition, understood or crisply put together first, what would be an obvious thing to do? But how long does it take for me to iterate? Before I know that I've got value, that I've pursued the right level of the problem? You You made an interesting comment a minute ago, you're like, oh, within a couple hours, I could potentially retrain the model and have that back operationally. That means if I can fail fast, right, if I can pick my problem space, get something out there operation, try it fail fast, and then continue to iterate with AI as my helper that that's really, really quite powerful is that the model that the your person? Carlos It is the model? That's exactly right. And it's not just hours to retrain, it takes hours to start, right. And just to highlight you kind of started with, we have to define that problem set first. So even after we define that problem set, a lot of times we have to go back and redefine that problem set, and really the piece around failing fast. It's it's experimentation. And do we have the right cameras? Do we have the right vantage point is the model correct, you want to be able to cycle as fast as you can through that experimentation phase. And sometimes you have to go back and redefine that problem set. Because you're learning more as you go, right? And you're evolving into okay, I now understand the corner cases a bit better. And with the platform, you really can cycle that quickly. I mean, machine learning at scale is really how fast you can iterate through improvement. Grant It's quite, I think it's quite a testament to how the AI just world in general is improving. I know that you and I were talking earlier, years ago, you know, when I first started writing some TensorFlow code and Keras code, you know, the, the time it took to fail was much longer, right. And then the cycles were huge and, and getting this down to a matter of hours or even a few days, you know, for an enterprise's is massive. What's the, from what you've seen terms of different industries? Are there certain industries that tend to be leaning into this and adopting it or the is there no pattern yet? Carlos No, there's a there's a definite pattern. And in 2022, we'll all see kind of what that what that's looking like. And it's really an industry that has traditionally not been able to go through that digital transformation. So think of think of think of a piece that's a very manual piece, right, like physical inspection, where humans would look at something, they'd write down their notes on a piece of paper, then that that item would go through either pass or fail or some criteria for rework. That's all possible. Now with computer vision years ago, that was impossible the accuracy wasn't high enough, plain and simple. It just took it wasn't, it wasn't better than a human. Now we have models that are better than a human for visual inspection. And these industries are digitizing their workflow. So it's not only the feature of computer vision, but it's also now I have a digital record of all the transactions, I have extracted video information, it makes auditing easy for those sectors that have a lot of regulatory compliance, those that require proactive compliance to audit requirements, as well as visibility. Visibility has always been an it's funny when we talk about visibility, but it's computer vision, but like a lot of human processes, that there's zero visibility in it there, it's really difficult to audit, you know, why is this working better or not? So having that, that digitization of the flow with the feature of computer vision allows us to extract the value. And industries like agriculture are going I mean, agriculture has been a leader in technology for a while, but now you're really seeing adoption at livestock and row crop with drone technology. It's a very rich image environment, medical space, medical space for computer vision in 2022 to 2028, is estimated to be billions of dollars just with medical imaging. And that's not that's not the the total addressable market for the hardware, it's just the imaging piece. So we see we see a lot of growth in sectors that are going through this digital transformation that are adopting technologies that are now getting to the point where they can get pushed down into the masses instead of just the top five companies in the world. Grant Excellent, it seems to you part of your comment earlier made me think about process optimization for organizations, and the ability to extract processes, you're familiar with process mining, right? The ability to extract, you know, out of logs of these organizations and doing something like that where you can produce this visual representation of that, and then building models against that, to optimize your process might be an interesting use case. Yeah, that's fascinating. Carlos That's a really good point, right? Because that's, that's a different portion of AI that can be applied to like just log analysis, that then would allow you to go back and Okay, now that we have the process mind, where can we improve along the process? Grant Yeah, yeah. Amazing. So many uses and use cases around around this CV area, for sure. So let's say that someone listening to this wanted to learn more about it, where would they go? How would they? How would they find more about your organization? Carlos You can find us everywhere, right? We have a website, plainsight.ai. We're all over LinkedIn, we have Twitter, we're on Reddit, we have a Medium blog, there's a Slack channel where we geek out around computer vision use cases and how we can improve the world through computer vision. We're really we're really out there and feel free if you have questions come reach out to us. We have amazing staff that are looking to empower people in AI. So if it's through just just a question around how does this thing work, we'd love to talk to you if it's Hey, we're kind of stuck in our journey. We need some help reach out to us we can help you. Grant That's awesome. Carlos I can't thank you enough for reaching out to me and for a listening to click AI radio, but also for reaching out and sharing what it is you are you and your organization are bringing to the market think you're solving some awesome problems. Carlos Thanks a lot, Grant. Appreciate it. always appreciated talking about computer vision and AI and thank you to you and your listeners and really appreciate what you're doing to the AI space. Grant Alright, thanks again, Carlos. And again, everybody. Thanks for joining and until next time, go get some computer vision from Plainsight.

Tiago Mendes joins Mat, Jon, and Johnny to discuss eventual consistency and strategies for changing data at scale.

About Micheal BenedictMicheal Benedict leads Engineering Productivity at Pinterest. He and his team focus on developer experience, building tools and platforms for over a thousand engineers to effectively code, build, deploy and operate workloads on the cloud. Mr. Benedict has also built Infrastructure and Cloud Governance programs at Pinterest and previously, at Twitter -- focussed on managing cloud vendor relationships, infrastructure budget management, cloud migration, capacity forecasting and planning and cloud cost attribution (chargeback). Links: Pinterest: https://www.pinterest.com Twitter: https://twitter.com/micheal LinkedIn: https://www.linkedin.com/in/michealb/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You know how git works right?Announcer: Sorta, kinda, not really Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Sometimes when I have conversations with guests here, we run long. Really long. And then we wind up deciding it was such a good conversation, and there's still so much more to say that we schedule a follow-up, and that's what happened today. Please welcome back Micheal Benedict, who is, as of the last time we spoke and presumably still now, the head of engineering productivity at Pinterest. Micheal, how are you?Micheal: I'm doing great, and thanks for that introduction, Corey. Thankfully, yes, I am still the head of engineering productivity; I'm really glad to speak more about it today.Corey: The last time that we spoke, we went up one side and down the other of large-scale environments running on AWS and billing aspects thereof, et cetera, et cetera. I want to stay away from that this time and instead focus on the rest of engineering productivity, which is always an interesting and possibly loaded term. So, what is productivity engineering? It sounds almost like it's an internal dev tools team, or is it something more?Micheal: Well, thanks for asking because I get this question asked a lot of times. So, for one, our primary job is to enable every developer, at least at our company, to do their best work. And we want to do this by providing them a fast, safe, and a reliable path to take any idea into production without ever worrying about the infrastructure. As you clearly know, learning anything about how AWS works—or any public cloud provider works—is a ton of investment, and we do want our product engineers, our mobile engineers, and all the other folks to be focused on delivering amazing experiences to our Pinners. So, we could be doing some of the hard work in providing those abstractions for them in such way, and taking away the pain of managing infrastructure.Corey: The challenge, of course, that I've seen is that a lot of companies take the approach of, “Ah. We're going to make AWS available to all of our engineers in it's raw, unfiltered form.” And that lasts until the first bill shows up. And then it's, “Okay. We're going to start building some guardrails around that.” Which makes a lot of sense. There then tends to be a move towards internal platforms that effectively wrap cloud services.And for a while now, I've been generally down on the concept and publicly so in the general sense. That said, what I say that applies as a best practice or something that most people should consider does tend to fall apart when we talk about specific use cases. You folks are an extremely large environment; how do you view it? First off, do you do internal platforms like that? And secondly, would you recommend that other companies do the same thing?Micheal: I think that's such a great question because every company evolves with its own pace of development. And I wouldn't say Pinterest by itself had a developer productivity or an engineering productivity organization from the get-go. I think this happens when you start realizing that your core engineers who are working on product are now spending a certain fraction of time—which starts ballooning pretty fast—in managing the underlying systems and the infrastructure. And at that point in time, it's probably a good question to ask, how can I reduce the friction in those people's lives such that they could be focused more on the product. And, kind of, centralize or provide some sort of common abstractions through a central team which can take away all that pain.So, that is generally a good guiding principle to think about when your engineers are spending at least 30% of their time on operating the systems rather than building capabilities, that's probably a good time to revisit and see whether a central team would make sense to take away some of that. And just simple examples, right? This includes upgrading OS on your EC2 machines, or just trying to make sure you're patching all the right versions on your next big Kubernetes cluster you're running for serving x number of users. The moment you start seeing that, you want to start thinking about, if there is a central team who could take away that pain, what are the things they could be investing on to help up-level every other engineer within your organization. And I think that's one of the best ways to be thinking about it.And it was also a guiding principle for us within Pinterest to view what investments we could make in these central teams which can up-level each and every different type of engineer in the company as well. And just an example on that could be your mobile engineer would have very different expectations from your backend engineer who was working on certain aspects of code in your product. And it is truly important to understand where you want to centralize capabilities, which both these types of engineers could use, or you want to divest and have unique capabilities where it's going to make them productive. There's no one-size-fits-all solution for this, but I'm happy to talk about what we have at Pinterest, which has been reasonably working well. But I do think there's a lot more improvements we could be doing.Corey: Yeah, but let's also be clear that, as you've mentioned, you are heavily biased towards EC2 instances for a lot of what you do. If we look at the AWS console and we see hundreds of different services now, and it's easy to sit here and say, “Oh, internal platforms are terrible because all of those services are going to be enhanced in various ways and you're never going to be able to keep up with feature parity.” Yeah, but if you can wrap something like EC2 in an internal platform wrapper, that begins to be a different story because sure, someone's going to go and try something new with a different AWS service, they're going to need direct access. But the EC2 product across the board generally does not evolve in leaps and bounds with transformative changes overnight. Let's also not forget that at a company with the scale that Pinterest operates at, “Hey, AWS just dusted off a new feature and docs are still rolling out, and it's not in CloudFormation yet, but we're going to roll it out to production,” probably seems like the wrong direction to go in, I would assume.Micheal: And yes, I think that brings one of the key guardrails, I think, which these groups provide. So, when we start thinking about what teams, centralized teams like engineering productivity, developer tools, developer platforms actually do is they help with a couple of things. The top three are: they can help pave a path for the most common use cases. Like to your point, provisioning EC2 does take a set of steps, all the time. If you're going to have a thousand people doing that every time they're building a new service or trying to expand capacity playing with their launch templates, those are things you can start streamlining and making it simple by some wrapper because you want to address those 80% use cases which are usually common, and you can have a wrapper or could just automate that. And that's one of the key things: can you provide a paved path for those use cases?The second thing is, can you do that by having the right guardrails in place? How often have you heard the story that, “I just clicked a button and that now spun up, like, a thousand-plus instances.” And now you have to juggle between trying to stop them or do something about it.Corey: Back in 2013, you folks were still focusing on this fair bit. I remember because Jeremy Carroll, who I believe was your first SRE there once upon a time, wound up doing a whole series of talks around how Pinterest approached doing an AMI Factory. And back in those days, the challenges were, “Okay. We have the baseline AMI, and that's great, but we also want to do deployments of things and we don't really want to do a new deploy of an entire fleet of EC2 instances for a single line of config change, so how do we wind up weighing off of when you bake a new AMI versus when you just change something that has—in what is deployed to them?” And it was really a complicated problem back then.I'm not convinced it's not still a complicated problem, but the answers are a lot more cohesive. And making sure that every team—when you're talking about a company as large as Pinterest with that many teams—is doing things in the same way, seems like it's critically important otherwise you wind up with a whole bunch of unique-looking instances that each have to be managed by hand as opposed to something that can be reasoned around collectively.Micheal: Yep. And that last part you mentioned is extremely crucial as well because like I said, our audience or our customers are just not the engineers; we do work with our product managers and business partners as well because at times, we have to tie or change our architecture based on certain cost optimizations which would make sense, like you just articulated. We don't want to have all the instance types. It does not add much value to a developer unless they're explicitly seeking a high-memory instance or a [GP-based instance in a 00:10:25] certain way. So, we can then work with our business partners to make sure that we're committing to only a certain type of instances, and how we can abstract our tools to only give you that. For example, our deployment system, Teletraan which is an open-source system, actually condenses down all these instance types to a couple of categories like high-compute, high-memory—and you've probably seen that in many of the new cloud providers as well—so people don't have to learn or know the underlying instance type.When we moved from c3 to c5, it was just called as a high-compute system, so the next time someone provisioned a new service or deployed it using our system, they would just select high-compute as the de facto instance type and we would just automatically provision a C5 for them. So, that just reduces the extra complexity or the cognitive overhead individuals would have to go through in learning each instance type, what is the base AMI that comes on it, what are the different configurations that need to go in terms of setting up your AZ-scaling properties. We give them a good reasonable set of defaults to get started with, and then they can then work on optimizing or making changes to it.Corey: Ignoring entirely your mispronunciation of AMI, which is, of course, three syllables—and that is a petty hill upon which I will die—it occurs to me the more I work with AWS in various ways, the easier it gets. And I used to think in some respects, it was because the platform was so—it was improving so dramatically around me. But no, in many cases, it's because the first time you write some CloudFormation by hand, it's a nightmare and you keep smacking into weird issues. But the second or third time, it's super easy because you just copy the thing you've already built and change the relevant bits around. And that was the learning curve that I went through playing around with a lot of these things.When you start looking at this from a large-scale environment where it's not just about upskilling the people that you have to understand how these things integrate in AWS land, but also the consistent onboarding of engineers at a fairly progressive clip is, great, you effectively have to start doing trainings on all these things, and there's a lot of knobs and dials that can blow up and hurt people. At some point, building the guardrails or building the environment in which you are getting all the stuff abstracted away from where the application engineers have to think about this at all, it eventually reaches a tipping point where it starts to feel like it's no longer optional if you want to continue growing as a company because you don't have the luxury of spending six months of onboarding before you let someone touch the thing they were hired to build.Micheal: And you will see that many companies very often have very similar programming practices like you just described. Even I learned that the same way: you have a base template, you just copy-paste it and start from there on. And no one goes through the bootstrapping process manually anymore; you want to—I think we call it cargo-culting, but in general, just get something to bootstrap and start from there. But one of the things we learned in sort of the hard way is that can also lead to, kind of, you pushing, you know, not great practices because people don't know what is a blessed version of a good template or what actually would make sense. So, some of those things, we have been working on.And this is where centralized teams like engineering productivity are really helpful is we provide you with the blessed or the canonical way to do certain things. Case in point example is a CI/CD pipeline or delivery of software services. We have invested enough in experimenting on what works with some of the more nuanced use cases at Pinterest, in helping generate, sort of, a canonical version which would cover 80% of the use cases. Someone could just go and try to build a service and they could just use the same canonical pipeline without learning much or making changes to it. This also reduces that cargo-culting nature which I called, rather than copying it from unknown sources and trying to like—again, it may cause havoc to our systems, so we can avoid a lot of that because of these practices.Corey: So, let's step a little bit beyond AWS—I know I hate doing it, too—but I'm going to assume that your remit is broader than, oh, AWS whisperer-slash-Wrangler. So, tell me a little bit more about what it is that your day-to-day looks like if there is anything that could be said not to focus purely around AWS whispering.Micheal: So, one of the challenges—and I want to talk about this a bit more—is our environments have become extremely complex over time. And it's the nature of, like, rising entropy. Like, we've just noticed that there's two things: we have a diverse set of customer base, and these include everyone trying to do different workloads or work service types. What that essentially translates into is that we realized that our solution may not fit all of them. For example, what works for a machine-learning engineer in terms of iterating on building a model and delivering a model is not the same as someone working on a long-running service and trying to deploy that. The same would apply for someone trying to operate a Kafka system.And that has made, I think, definitely our job a bit challenging in trying to assess where do you actually draw the line on the abstraction? What is the right layer of abstraction across your local development experience, across when you move over to staging your code in a PR model and getting feedback and subsequently actually releasing it to production? Because this changes dramatically based on what is the workload type you're working on. And we feel like that has been one of the biggest challenges where I know I spent my day-to-day and my team does too, in trying to help provide some of the right solutions for these individuals. There's—very often we'll also get asked from individuals trying to do a very nuanced thing.Of late, we have been talking about thinking about how you operate functions, like provide Functions as a Service within the company? It just put us in a difficult spot at times because we have to ask the hard question, “Is this required?” I know the industry is doing it; it's definitely there. I personally believe, yes, it could be a future, but is that absolutely important? Is that going to benefit Pinterest in any formal way if we invest on some core abstractions?And those are difficult conversations to have because we have exciting engineers coming in trying to do amazing things; it puts us in a hard spot, as well, as to sometimes saying graciously, no. I know many companies deal with it when they have these centralized teams, but I think it's part of that job. Like when you say it's day-to-day, I would say I'm probably saying no a couple of times in that day.Corey: Let's pretend for the sake of argument that I am, tomorrow morning, starting another company—Twitter for Pets—and over the next ten years, it grows to be larger than Pinterest in terms of infrastructure, probably not revenue because it turns out pets are not the lucrative source of ad revenue that I was hoping it would be but, you know, directionally the same thing. It seems to me that building out this sort of function with this sort of approach to things is dramatically early as far as optimizations go when it's just me puttering around on something. I'm always cognizant of the wrong people taking the wrong message when we're talking about things that happen like this at scale. When does having an engineering productivity group begin to make sense?Micheal: I mentioned this earlier; like, yeah, there is definitely not a right answer, but we can start small. For example, this group actually started more as a delivery team. You know, when we started, we realized that we had different ways of deploying services or software at Pinterest, so we first gathered together to figure out, okay, what are the different ways and can we start simplifying that part? And that's where it started expanding. Okay, we are doing button-based deployments right now we have thousand-plus microservices, and we are seeing more incidents than we wanted to because anything where there's a human involved means there's a potential gap for error. I myself was involved in a SEV 0 incident, and I will be honest; we ended up deploying a Hello World application in one of our production fleet. Not the thing I wanted to be associated with my name, but, you know—Corey: And you were suddenly saying hello to the world, in fact—Micheal: [laugh].Corey: —and oops-a-doozy.Micheal: Yeah. So—and that really prompted us to rethink how we need to enable guardrails to do safe production rollouts. And that's how those conversations start ballooning out.Corey: And the healthy correct way. We've all broken production in various ways, and it's—you correctly are identifying, I believe, the direction you're heading in where this is a process problem and a tooling problem; it is not that you are secretly crap and should never have been allowed near anything in production. I mean, that's my excuse for me, but in your case, this is a common thing where it's, if someone can unintentionally cause issues like that, there needs to be better processes and procedures as the organization matures.Micheal: Yep. And that's kind of like always the route or the starting point for these discussions. And it starts growing from there on because, okay, you've helped improve the deploy process but now we're seeing insane amount of slowness, say on the build processes, or even post-deploy, there's, like, issues on how we monitor and look into data.And that I think forces these conversations, okay, where do we have these bespoke tools available? What are people doing today? And you have to ask those hard questions, like what can we actually remove from here? The goal is not to introduce yet another new system. Many a times, to be honest bash just gets the job done. [laugh].Personally, I'm okay with that as long as it's consistent and people, you know, are able to contribute to it and you have good practices in validating it, if it works, we should go for it rather than introducing yet another YAML [laugh] and some of that other aspects of doing that work. And that's what we encourage as well. That's how I think a lot of this starts connecting together in terms of, okay, now this is becoming a productivity group; they're focused on certain challenges where investing probably one person here may up-level a few other engineers who don't have to do that on a day-to-day basis. And I think that's one of the key items for, especially, folks who are running mid-sized companies to realize and start investing in these type of teams to really up-level, sort of, the rest of the engineering.Corey: You've been doing this for a fair while. If you were to go back and start over again on day one—which is always a terrifying question, on some level—what would you have done differently about building out this function as Pinterest continued to scale out?Micheal: Well, first, I must acknowledge that this was just not me, and there's, like, ton of people involved in helping make this happen.Corey: No, that's fair. We'll blame them for the missteps; that is—Micheal: [laugh].Corey: —just fine with me. I kid. I kid.Micheal: I think, definitely the nuances. If I look back, all the decisions that were made then at that point in time, there was a decision made to move to Phabricator, which was back then a great open-source code management system where with the current information at that point in time. And I'm not—I think it's very hard to always look back and say, “Oh, we could have chosen x at one point in time.” And I think in reality, that's how engineering organizations always evolve, that you have to make do with the information you have right now to make a decision that works for you over a couple of years.And I'll give you a small example of this. There was a time when Pinterest was actually on GitHub Enterprise—this was like circa 2013, I would say—and it really served as well for, like, five-plus years. Only then at certain point, we realized that it's hard to hire PHP engineers to support a tool like that, and we had to rethink what is the ROI and the investments we've made here? Can we ever map up or match back to one of the offerings in the industry today? And that's when you make decisions that, okay, at this point in time, it's clear that business continuity talks, you know, and it's hard to operate a system, which is, at this moment not supported, and then you make a call about making a shift or moving.And I think that's the key item. I don't think there's anything dramatically I would have changed since the start. Perhaps definitely investing a bit more individuals into the group and going from there. But that said, I'm really, sort of, at least proud of the fact that usually these teams are extremely lean and small, and they always have an outsized impact, especially when they're working with other engineers, other [opinionated 00:22:13] engineers for what it's worth.This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: Most folks show up intending to do good today, and you make the best decision at the time with the context and constraints that you have, but my question I think is less around, “Well, what were the biggest mistakes you made?” But more to do with the idea of, based upon what you've learned and as you have shown—as you've shined light on these dark areas, as you have been exploring it, has anything jumped out at you that is, “Oh, yeah. Now, that I know—if I had known then what I know now, I would definitely have made this other decision.” Ideally, something that applies a little more globally than specific within Pinterest, just because the whole idea, aspirationally, is that people might learn something from our conversation. At least I will, if nothing else.Micheal: No, I think that's a great question. And I think the three things that jump to me, top of mind. I think technology is means to an end unless it gives you a competitive edge. And it's really hard to figure out at what point in time what technology and why we adopted it, it's going to make the biggest difference. Humans always tend to have a bias towards aligning towards where we want to go. So, that's the first one in my mind.The second one is, and we spoke about this last time, embrace your cloud provider as much as possible. You'd want to avoid taking on operational burden which is not going to add value to the business. If there is something you see your operating which can be offloaded—because your provider can, trust me, do a way better job than you or your team of few can ever do—embrace that as soon as possible. It's better that way because then it frees up your time to focus on the most important thing, which I've realized over time is—I really think teams like ours are actually—we're probably the most value as a glue to all the different experiences a software engineer would go through as part of their SDLC lifecycle.If we can simplify someone's life by giving them a clear view as to where their commit or the work is in this grand scheme of rolling out and giving them the right amount of data to take action when something goes wrong, trust me, they will love you for what you're doing because you're saving them ton of time. Many times, we don't realize that when we publish 11 different ways for you to go and check to just get your basic validation of work done. We tend to so much focus on the technological aspect of what the tool does, rather than the experience of it, and I've realized, if you can bridge the experience, especially for teams like ours, people really don't even need to know whether you're running Kubernetes or any of those solutions behind the scenes. And I think that's one of the biggest takeaways I have.Corey: I want to double down on something you said about the fact that you are not going to be able to run these services as effectively as your provider can. And relatively recently—in fact, since the first time we spoke—AWS has released a investment report in Virginia. And from 2011 through 2020, they have invested in building AWS data centers there, $35 billion. I promise almost no company that employs people listening to this that are not themselves a cloud provider is going to make that kind of investment in running these things themselves.Now, do cloud providers have sharp edges? Yes, absolutely. That is what my entire career is about, unfortunately. But you're not going to do a better job of running things more sustainably, more reliably, et cetera, et cetera. But there are other problems with this—and that's what I want to start exploring here—where in the olden days, when I ran things in data centers and they went down a lot more as a result, sometimes when there were outages, I would have the CEO of the company just standing there nervous worrying over my shoulder as I frantically typed to fix things.Spoiler: my typing accuracy did not improve by having someone looming over me. Now, when there's an outage that your cloud provider takes, in many cases the thing that you are doing to fix it is reloading the status page and waiting for an update because it is completely out of your hands. Is that something that you've had to encounter? Because you can push buttons and turn dials when things are broken and you control it, but in an AWS—or other cloud provider—outage, all you can really do is wait unless you have a DR plan that is large-scale and effective enough that you won't feel foolish or have wasted a huge amount of time and energy migrating off and then—because then it gets repaired in ten minutes. How do you approach that, from your perspective? I guess, the expectation management piece?Micheal: It's definitely I know something which keeps a lot of folks within infrastructure up at night because, like you just said, at times we can feel extremely powerless when we obviously don't have direct control—or visibility at times, as well—on what's happening. One of the things we have realized over time as part of running on our cloud provider for over a decade now, it forces us to rethink a bit on our priority workflows, what we want our Pinners to always have access to, what they need to see, what is not important or critical. Because it puts into perspective, even for the infrastructure teams, is to what is the most important thing we should always have it available and running, what is okay to be in a degraded state, until what time, right? So, it actually forces us to define SLOs and availability criteria within the team where we can broadcast that to the larger audience including the executives. So, none of this comes as a surprise at that point.I mean, it's not the answer, probably, you're looking for because is there's nothing we can do except set expectations clearly on what we can do and how when you think about the business when these things do happen. So, I know people may have I have a different view on this; I'm definitely curious to hear as well, but I know at Pinterest at least we have converged on our priority workflows. When something goes out, how do we jump in to provide a degraded experience? We have very clear run books to do that, and especially when it's a SEV 0, we do have clear processes in place on how often we need to update our entire company on where things are. And especially this is where your partnership with the cloud provider is going to be a big, big boon because you really want to know or have visibility, at the minimum some predictability on when things can get resolved, and how you want to work with them on some creative solutions. This is outside the DR strategy, obviously; you should still be focused on a DR strategy, but these are just simple things we've learned over time on how to just make it predictable for individuals within the company, so not everyone is freaking out.Corey: Yeah, from my perspective, I think the big things that I found that have worked, in my experience—mostly by getting them wrong the first time—is explain that someone else running the infrastructure when they take an outage; there's not much we can do. And no, it's not the sort of thing where picking up the phone and screaming at someone is going to help us, is the sort of thing that is best to communicate to executive stakeholders when things are running well, not in the middle of that incident.Then when things break, it's one of those, “Great, you're an exec. You know what your job is? Literally anything other than standing in the middle of the engineering floor, making everyone freak out even more. We'll have a discussion later about what the contributing factors were when you demand that we fire someone because of an outage. Then we're going to have a long and hard talk about what kind of culture you're trying to build here again?” But there are no perfect answers here.It's easy to sit here in the silver light of day with things working correctly and say, “Oh, yeah. This is how outages should be handled.” But then when it goes down, we're all basically an inch away at best from running around with our hair on fire, screaming, “Fix it, fix it, fix it, fix it, now.” And I am empathetic to that. There's a reason but I fix AWS bills for a living, and one of those big reasons is that it's a strictly business-hours problem and I don't have to run production infrastructure that faces anything that people care about, which is kind of amazing and freeing for someone who spent too many years on call.Micheal: Absolutely. And one of the things is that this is not only with the cloud provider, I think in today's nature of how our businesses are set up, there's probably tons of other APIs you are using or you're working with you may not be aware of. And we ended up finding that the hard way as well. There were a certain set of APIs or services we were using in the critical path which we were not aware of. When these outages happen, that's when you find that out.So, you're not only beholden to your provider at that point in time; you have to have those SLO expectations set with your other SaaS providers as well, other folks you're working with. Because I don't think that's going to change; it's probably only going to get complicated with all the different types of tools you're using. And then that's a trade-off you need to really think about. An example here is just like—you know, like I said, we moved in the past from GitHub to Phabricator—I didn't close the loop on that because we're moving back to GitHub right now [laugh] and that's one of the key projects I'm working with. Yeah, it's circle of life.But the thing is, we did a very strong evaluation here because we felt like, “Okay, there's a probability that GitHub can go down and that means people will be not productive for that couple of hours. What do we do then?” And we had to put a plan together to how we can mitigate that part and really build that confidence with the engineering teams, internally. And it's not the best solution out there; the other solution was just run our own, but how is that going to make any other difference because we do have libraries being pulled out of GitHub and so many other aspects of our systems which are unknowingly dependent on it anyways. So, you have to still mitigate those issues at some point in your entire SDLC process.So, that was just one example I shared, but it's not always on the cloud provider; I think there are just many aspects of—at least today how businesses are run, you're dependent; you have critical dependencies, probably, on some SaaS provider you haven't really vetted or evaluated. You will find out when they go down.Corey: So, I don't think I've told this story before, but before I started this place, I was doing a fair bit of consulting work for other companies. And I was doing a project at Pinterest years ago. And this was one of the best things I've ever experienced at a company site, let alone a client site, where I was there early in the morning, eight o'clock or so, so you know, engineers love to show up at the crack of 11:30. But so I was working a little early; it was great. And suddenly my SSH session that I was using to remote into something or other hung.And it's tap up, tap enter a couple of times, tap it a couple more. It was hung hard. “What's the—” and then someone gently taps me on the shoulder. So, I take the headphones off. It was someone from corporate IT was coming around saying, “Hey, there's a slight problem with our corporate firewall that we're fixing. Here's a MiFi device just for you that you can tether to get back online and get worked on until the firewall gets back.”And it was incredible, just the level of just being on top of things, and the focus on keeping the people who were building things and doing expensive engineering work that was awesome—and also me—productive during that time frame was just something I hadn't really seen before. It really made me think about the value of where do you remove bottlenecks from people getting their jobs done? It was—it remains one of the most impressive things I've seen.Micheal: That is great. And as you were telling me that I did look up our [laugh] internal system to see whether a user called Corey Quinn existed, and I should confirm this with you. I do see entries over here, a couple of commits, but this was 2015. Was that the time you were around, or is this before that even?Corey: That would have been around then, yes. I didn't start this place until late 2016.Micheal: I do see your commits, like, from 2015, and I—Corey: And they're probably terrible, I have no doubt. There's a reason I don't read code for a living anymore.Micheal: Okay, I do see a lot of GIFs—and I hope it's pronounced as GIF—okay, this is cool. We should definitely have a chat about this separately, Corey?Corey: Oh, yeah. “Would you explain this code?” “Absolutely not. I wrote it. Of course, I have no idea what it does. That's the rule. That's the way code always works.”Micheal: Oh, you are an honorary Pinterest engineer at this point, and you have—yes—contributed to our API service and a couple of Puppet profiles I see over here.Corey: Oh, yes—Micheal: [Amazing 00:36:11]. [laugh].Corey: You don't wind up thinking that's a risk factor that should be disclosed. I kid. I kid. It's, I made a joke about this when VMware acquired SaltStack and I did some analytics and found that 60 some odd lines of code I had written, way back when that were still in the current version of what was being shipped. And they thought, “Wait, is this actually a risk?”And no, I am making a joke. The joke is, is my code is bad. Fortunately, there are smart people around me who review these things. This is why code review is so important. But there was a lot to admire when I was there doing various things at Pinterest. It was a fun environment to work in, the level of professionalism was phenomenal, and I was just a big fan of a lot of the automation stuff.Phabricator was great. I love working with it, and, “Great, I'm going to use this to the next place I go.” And I did and then it was—I looked at what it took to get it up and running, and oh, yeah, I can see why GitHub is so popular these days. But it was neat. It was interesting seeing that type of environment up close.Micheal: That is great to hear. You know, this is what I enjoy, like, hearing some of these war stories. I am surprised; you seem to have committed way more than I've ever done in my [laugh] duration here at Pinterest. I do managing for a living, but then again—Corey, the good news is your code is still running on production. And we—Corey: Oh dear.Micheal: —haven't—[laugh]. We haven't removed or made any changes to it, so that's pretty amazing. And thank you for all your contributions.Corey: Oh, please, you don't have to thank me. I was paid, it was fine. That's the value of—Micheal: [laugh].Corey: —[work 00:37:38] for hire. It's kind of amazing. And the best part about consultants is, is when we're done with a project, we get the hell out everyone's happy about it.More happy when it's me that's leaving because of obvious personality-related reasons. But it was just an interesting company from start to finish. I remember one other time, I wound up opening a ticket about having a slight challenge with a flickering on my then Apple-branded display that everyone was using before they discontinued those. And I expected there to be, “Oh, okay. You're a consultant. Great. How did we not put you in the closet with a printer next to that thing, breathing the toner?” Like most consulting clients tend to do, and sure enough, three minutes later, I'm getting that tap on the shoulder again; they have a whole replacement monitor. “Can you go grab a cup of coffee? We'll run the cable for it. It'll just be about five minutes.” I started to feel actively bad about requesting things because I did a lot of consulting work for a lot of different companies, and not to be unkind, but treating consultants and contractors super well is not something that a lot of companies optimize for. I can't necessarily blame them for that. It just really stood out.Micheal: Yep, I do hope we are keeping up with that right now because I know our team definitely has a lot of consultants working with us as well. And it's always amazing to see; we do want to treat them as FTs. It doesn't even matter at that point because we're all individuals and we're trying to work towards common goals. Like you just said, I think I personally have learned a few items as well from some of these folks. Which is again, I think speaks to how we want to work and create a culture of, like, we're all engineers; we want to be solving problems together, and as you were doing it, we want to do it in such a way that it's still fun, and we're not having the restrictions of titles or roles and other pieces. But I think I digressed. It was really fun to see your commits though, I do want to track this at some point before we move completely over to GitHub, at least keep this as a record, for what it's worth.Corey: Yeah basically look at this graffiti in the codebase of, “A shit-poster was here,” and here I am. And that tends to be, on some level, the mark we live on the universe. What's always terrifying is looking at things I did 15 years ago in my first Linux admin job. Can I still ping the thing that I built there? Yes, I can. And how is that even possible? That should not have outlived me; honestly, it should never have seen the light of day in production, but here we are. And you never know how long that temporary kluge you put together is going to last.Micheal: You know, one of the things I was recalling, I was talking to someone in my team about this topic as well. We always talk about 10x engineers. I don't know what your thoughts are on that, but the fact that you just mentioned you built something; it still pings. And there's a bunch of things, in my mind, when you are writing code or you're working on some projects, the fact that it can outlast you and live on, I think that's a big, big contribution. And secondly, if your code can actually help up-level, like, ten other people, I think you've really made the mark of 10x engineer at that point.Corey: Yeah, the idea of the superhuman engineer is always been a strange and dangerous one. If for nothing else, from where I sit, excellence is inherently situational. Like we just talked about someone at Pinterest: is potentially going to be able to have that kind of impact specifically because—to my worldview—that there's enough process and things around there that empower them to succeed. Then if you were to take that engineer and drop them into a five-person startup where none of those things exist, they might very well flounder. It's why I'm always a little suspicious of this is a startup founded by engineers from Google or Facebook, or wherever it is.It's, yeah, and what aspects of that culture do you think are one-to-one matches with the small scrappy startup in the garage? Right, I predicting some challenges here. Excellence is always situational. An amazing employee at one company can get fired at a second one for lack of performance, and that does not mean that there's anything wrong with them and it does not mean that they are a fraud. It means that what they needed to be successful was present in one of those shops, but not the other.Micheal: This is so true. And I really appreciate you bringing this up because whenever we discuss any form of performance management, that is a—in my view personally—I think that's an incorrect term to be using. It is really at that point in time, either you have outlived the environment you are in, or the environment is going in a different direction where I think your current skill set probably could be best used in the environment where it's going to work. And I know it's very fuzzy at that point, but like you said, yes, excellence really means you don't want to tie it to the number of commits you have pushed out, or any specific aspect of your deliverables or how you work.Corey: There are no easy answers to any of these things, and it's always situational. It's why I think people are sometimes surprised when I will make comments about the general case of how things should be, then I talk to a specific environment where they do the exact opposite, and I don't yell at them for it. It's there—in a general sense, I have some guidance, but they are usually reasons things are the way they are, and I'm interested in hearing them out. Everything's situational, the worst consultant in the world is the one that shows up, has no idea what's going on, and then asked, “What moron set this up?” Invariably, two said, quote-unquote, “Moron.” And the engagement doesn't go super well from there. It's, “Okay, why is this the way that it is? What constraints shaped it? What was the context behind the problem you were trying to solve?” And, “Well, why didn't you use this AWS service?” “Because it didn't exist for another three years when we were building that thing,” is a—Micheal: Yes.Corey: —common answer.Micheal: Yes, you should definitely appreciate that of all the decisions that have been made in past. People tend to always forget why they were made. You're absolutely right; what worked back then will probably not work now, or vice versa, and it's always situational. So, I think I can go on about this for hours, but I think you hit that to the point, Corey.Corey: Yeah, I do my best. I want to thank you for taking another block of time out of your day to wind up talking with me about various aspects of what it takes to effectively achieve better levels of engineering productivity at large companies, with many teams, working on shared codebases. If people want to learn more about what you're up to, where can they find you?Micheal: I'm definitely on Twitter. So, please note that I'm spelled M-I-C-H-E-A-L on Twitter. So, you can definitely read on to my tweets there. But otherwise, you can always reach out to me on LinkedIn, too.Corey: Fantastic and we will, of course, include a link to that in the [show notes 00:44:02]. Thanks once again for your time. I appreciate it.Micheal: Thanks a lot, Corey.Corey: Micheal Benedict, head of engineering productivity at Pinterest. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment telling me that you work at Pinterest, have looked at the codebase, and would very much like a refund and an apology.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

This week Gerhard is chatting with Romano Roth, Head of DevOps at Zühlke, a company founded by Gerhard Zühlke in 1968. Nowadays they help companies all over the world build, ship and run anything from factory robots, to AI assistants in complex regulatory environments, and even medical devices that perform autonomous robotic surgery. When Romano is not leading a team of 30 software engineers that specialise in operations, infrastructure and cloud, he is one of the organisers of DevOps Days Zürich, and also the DevOps Meetup group which is how Gerhard and Romano met in 2019. Having started his career as a .Net developer back in 2002, Romano had his fair share of dev and ops challenges, and he always enjoys seeing real business value delivered continuously in an automated way. In recent years, Romano's perspective broadened, and now he sees DevOps challenges across many companies. If you are curious about what good DevOps looks like, and what are the real challenges, then Romano has some good insights for you.

Rob Hirschfeld is CEO and co-founder of RackN, leaders in physical and hybrid DevOps software. He has been in the cloud and infrastructure space for nearly 15 years This is a special episode with Rob returning as the guest for his 4th podcast and for the commemorative 200th episode!  We discuss how to unlock the power of multi-cloud automation, the challenge of human ops, and how we are finally reaching an "overnight success" of true bare-metal provisioning and multi-cloud automation and operations.   Follow Rob on Twitter here: https://twitter.com/zehicle  Check out the awesome work by RackN here: https://rackn.com  Subscribe and listen to the 2030.cloud podcast here: https://soundcloud.com/user-410091210  This episode is brought to you by Veeam Software and the 4-Step Guide to Delivering Extraordinary Software Demos that Win Deals and Diabolical Coffee plus the great folks over at Fiverr!  Want to ensure real privacy online? Check out ExpressVPN and keep your online life protected. 

No holiday break on the recruiting front! Brittany and Brian catch up on the Rails recruiting trends. They discuss how companies need to up the ante now that so many are remote first, how interviews should be structured and whether it is worth getting in the door, if it is not for the role you want. Show Notes & Links: Mirror Placement (https://www.mirrorplacement.com/) Rubyconf 2021 Job Fair (https://rubyconf.org) Sponsored By: Honeybadger (https://www.honeybadger.io/) Honeybadger makes you a DevOps hero by combining error monitoring, uptime monitoring and check-in monitoring into a single, easy to use platform. Go to Honeybadger.io (https://www.honeybadger.io/) and discover how Starr, Josh, and Ben created a 100% bootstrapped monitoring solution. Atlantis Technology (https://www.atlantistech.com/careers) Atlantis is looking for great engineers! Why work at Atlantis? You'll work with great people. You'll work on projects that change the world. No matter where you are in your career, they're prepared to help you advance it. Find out more here (https://www.atlantistech.com/careers).

We get some spicy emails, dig into why Mike just picked up another Linux laptop, and then share our real thoughts on Web3.

About Guang Ming Guang Ming Whitley was elected to Mount Pleasant Town Council in 2017 and resides in Old Mount Pleasant with her husband, four children, and a dog.She earned a B.S. in Chemical Engineering from the University of Southern California and a J.D. from the University of Chicago Law School, where she was a member of Law Review and a moot court semi-finalist. After completing her law degree, Guang Ming taught at the University of Chicago and practiced intellectual property law in Los Angeles. She then retired from active practice to serve as Chief Operating Officer of the Whitley Household. In 2020, she cofounded Lattice Climbers, a company dedicated to teaching soft and life skills to young adults.Guang Ming is also President of the Girls State Alumnae Foundation and attended the American Legion Auxiliary Girls State in 1996, where she was elected governor. She has volunteered with the ALA Girls State program in a variety of capacities since 2000.Links:Lattice Climbers: https://www.latticeclimbers.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: You know how git works right?Announcer: Sorta, kinda, not really Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Sometimes people like to ask me what this show is really about and my answer has always been, “The business of cloud,” which is intentionally overbroad; really gives me an excuse to talk about anything that strikes my fancy at a given time. A recurring theme has always been, “Where does the next generation of folks working on cloud come from?”That's not strictly bounded to engineers; that goes throughout the entire ecosystem. There are a lot of jobs that are important to the functioning of businesses that don't require a whole bunch of typing into a text editor and being mad about YAML all day long. Today, my guest is Guang Ming Whitley. Guang Ming, thank you for joining me, I'll let you tell the story. Who are you exactly?Guang Ming: Oh, my goodness. That's a tough question. Well, I am someone who has lived my life in a series of segments. I started off as an engineer—a chemical engineer—then went off to law school, taught for a year—Corey: Well, let's interject as well. That is how I got looped into this whole nonsense; you were law school classmates with my spouse. And whatever you're in town, she gets very excited at the chance to see you, and we finally got to meet not that long ago, had a great conversation. It was, “Oh, my God, you need to come on the podcast.” Which is neither here nor there. Please, continue.Guang Ming: So, then I had a segment as a stay-at-home mother. I started having babies and I had a lot of them. I had one daughter, then a son, and then I had identical twin boys. And once I started having them in litters, we decided that it was time to stop. So, four kids in and about a decade as a stay-at-home mom, during which time I wrote some books.And then ran for office back in 2017. And then in 2020, was working with someone just, kind of, over coffee, just having, you know, conversation, and we came up with the idea to start a business, and Lattice Climbers was born out of that.Corey: And Lattice Climbers is what I think we're going to be talking about the most today because there's an entire episode baked into every one of those steps. Maybe not every one of them would fit on a cloud-oriented podcast, but there's a lot of interesting backstory there and it resonates with me because my entire life has been lived in phases as well. And the more I talk to people, the more I start to realize that maybe I'm not that bizarre. People go through stages and they'd love to retcon what the story was at the time and make it all look like there's a common thread and narrative running through, but when we're going through it, it feels—to me at least—like I've been careening from thing to thing to thing without ever really having an end goal in mind. But in hindsight, looking back, it just seems like it was inevitable that I would go from where I was to here. It never feels that way at the time for me.Guang Ming: Well, I think for me, where I've ended up with Lattice Climbers has felt sort of inevitable because one of the through-lines of all my segments that I've gone through is a program called Girls State, and it is one that I have volunteered with. It's sponsored by the American Legion Auxiliary and it's a government simulation program. Over the course of one week, you simulate city, county, and state government. And it's all about civic engagement and education of young women, and empowerment. So, it's such a fantastic program.And I love it, but one of the things that I've seen with this program is, as the young women come through the program, some of them have skills, and some of them don't have skills. And there's elements that are missing, and that's something that I want to try to help with, with Lattice Climbers.Corey: So, what is Lattice Climbers in a nutshell? It's still very early days, which is fine, terrific; the fact that you care enough about a problem that is clearly plaguing not just our industry, but arguably our entire society is worth exploring in-depth. And with the understanding that the narrative may very well shift as times go on what is Lattice Climbers today?Guang Ming: So, Lattice Climbers steps into the gap between formal education and the skills necessary to actually adult at life, to survive in the real world.Corey: That is an area that is of intense interest to me. For listeners who may not have listened to every single episode here, my academic background is checkered, to put it politely. On paper, I have an eighth-grade education and no one can take that away from me. I was expelled from two boarding schools in high school, I wound up getting a diploma from a homeschooling organization that years later I discovered was not accredited, then I failed out of college. But again, no one can take that eighth-grade education away from me.But also look at me. I am a white dude in tech where my failure mode is a board seat and a book deal somewhere, and there are winds of privilege at my back when I do that. What also has been a strong contributing factor is that when I was 12 years old, my dad sat me down and had a long conversation with me about how to handle a job interview, what a job interview was—because when I was 12, I had no idea—and what they're looking to gain from asking you these questions, and why they're asking you the things that they do, what answers they're looking for, and the purpose behind the meeting that you're in. And that more than almost anything else as a single moment in my childhood shaped the reason that I became moderately successful [laugh] in my career, depending on what phase of my career we're talking about. That's stuff is super important and they don't teach it formally in any program I've ever seen. How do you approach it?Guang Ming: So, what we do is we have an intake quiz that assesses your skill gaps, sort of like a self-assessment, and then it gives you a customized curriculum just meant to fill your specific skill gaps. So professionalism, where we cover things like interview skills, behavior at events, table manners, those kinds of things. Financial literacy, we have little mantras like, “Credit cards are not free money,” [laugh] which some people never learned that. And then there's different tracks, so depending on whether or not you're college-bound, or vocational school, or military-bound, you can pick a different track for that and receive two-minute lessons, sort of the gems, distilled down. And there's little animations; we try to keep it as brief and information-packed as possible.Corey: Would it be fair to categorize this as more or less micro-lessons in how to adult?Guang Ming: Exactly. That is exactly what we're trying to do.Corey: I somewhat recently read one of the best stories I've ever heard about teaching students in middle school about financial literacy. And invariably, the financial literacy courses are all sponsored by financial institutions, and that's great. So, what happened was, someone from the bank came in and spoke to the students and then took them all to the bank and had them all open a bank account and deposit $5 into it. Great. A couple of years go by and it earns interest—not much because $5—the bank was then acquired and acquired again and eventually became rolled into Wells Fargo, and had a small balance fee, which then of course wiped out all of these accounts.And I don't think that there is any better lesson in the way the financial system works—in some ways—than that. And yes, that's cynical, but that idea of, if you are sort of toward the bottom, this system is basically stacked against you in a bunch of different ways. Look, I'm not here to rail against capitalism or society as it stands, but understanding that basic concept is foundational to realizing that maybe the credit card company isn't always your friend with your very best interests in mind.Guang Ming: Mm-hm. And we tried to explain that, too, you that when you get a credit limit, that is based on what your ability to pay the minimum balance every month. They don't care if you can pay it off. They care about making that interest off of you, and I think that's something that children and young adults need to understand.Corey: It feels like it ties into the idea of thinking critically. The problem with that is the root of that entire financial literacy anecdote that I came out with just now, is that the financial literacy program was developed and promoted by financial institutions. What I like is that I checked your website very briefly, and given the significant absence of a pile of disclosures at the bottom, I don't believe you're a bank.Guang Ming: We are not a bank, and we are not sponsored by a bank. We want to provide practical real-life advice that is useful, and in digestible chunks.Corey: A while back, before I wound up starting down the path that I'm on now, I basically yelled at people for fun on the internet. I know, imagine that. I was the moderator of two particular subreddits: personal finance—which, great, I spent my 20s in crippling debt; there's no one as passionate about that stuff as someone who has been converted. Great. And the other was the legal advice subreddit, which is probably horrifying to people like you who are actual attorneys.But it turns out that an awful lot of what I was doing in both of those subreddits was giving life advice to people on how to function in society. On the legal side of it, “You can't sue a dog.” “Okay, you are not going to be able to go down to the police station and explain your way out of troubles. Get an attorney.” It's baseline-level stuff.“Oh, you've been given a contract that seems unreasonable, but they'd say that you need you to sign it.” “Yeah. How about don't do that without having someone review it?” It's not actually legal advice. It is how to function in society as an adult, but that's a less catchy subreddit title as it turns out.Guang Ming: Well, it's all about raising your awareness level. So, I have a friend and she tells this story, MBA grad and spent her first six months on the job wearing sneakers every day to work—they were cute, fashionable sneakers, but they were sneakers—and they were not part of appropriate business attire for the work environment she was in because she just was oblivious to that as being an issue. And it took someone who was more senior to finally sit her down and say, “You shouldn't do this. You need to wear appropriate shoes to work.” And she was mortified but learned from that experience.So, what if you never had to have that? What if you never had to have that sit-down conversation with someone correcting you? What if you had a little, sort of, pocket guide that gave you that level of awareness? It's like, “Take a look at your office. See what people are wearing. You can't wear what the CEO is wearing because you're not the CEO”—I mean, unless you are the CEO, then you can wear whatever you want, but if you're just an underling at the company, if you're just starting out, you need to understand what the company culture is and you need to conform to that culture. Unfortunately, that's just, like… the truth of the matter.Corey: The common wisdom is, “Oh, if you don't know how to dress or how to behave in a certain scenario, reach out to one of your mentors and ask them for advice.” Not everyone has one of those things. I get some crap sometimes through it, but one of the big reasons I have open DMs on Twitter is specifically so people can message me and ask me questions about the industry generally, life in general; I'm always willing to talk to folks who are trying to figure things out. That's important. Since a disproportionate number of the listeners to this show do work in tech and the idea of having a dress code is ridiculous, yeah, in a lot of tech culture at t-shirt and jeans is just fine, but in other cases, it's not.And, for example, I'll get on stage wearing a full bespoke three-piece suit and give a talk. And it's fun. It's hilarious. It plays with people's expectations, but it's important to understand I view that more as costuming than I do how I believe someone should necessarily dress in that environment. I am, for better or worse, a very distinctive personality in this space, and using me as a blueprint for someone who is starting out their career is going to lead to disaster.Yes, I'm mouthy and I make fun of big companies because that's my thing. I also got fired an awful lot in—Guang Ming: [laugh].Corey: —my career, and those two things are not entirely unrelated, let's be very clear here. There's a lot that we can learn through observation, but dialing it in and figuring out what the expectations, are important.Guang Ming: Well, I think a lot of young adults—one of the things we focus on, as well, is the importance of mentoring and finding good mentors. And then you being the kind of person that a mentor would want to mentor. Because I think there's a lot of formal mentoring in work environments, and those don't always work as well as the organic relationships. So, we want to be that mentor that you never knew that you needed, the mentor that you wish you always had, to give you all that baseline information so that when you do meet with your substantive mentor, they can truly help you in ways that we cannot with our scalable mentoring micro-lessons.Corey: I have to ask, what is your revenue model? Because if this turns into charging kids money to learning these things, that has a giant exploitative flashing warning sign around it.Guang Ming: So, what we're planning to do is work with school districts and with nonprofits, and do sort of like a B2B model where we pilot with the school district, we pilot with the technical college, and give them an opportunity to add 30 to 50 students, work with the program. And if they find it something valuable, they find that it's a value-add and it's helping their students land jobs and have a better career, I think that then they'll use our program for their full technical school.Corey: I'm done a fair number of mentorships in the course of my career. I helped administer and run the LOPSA 00:13:43—or League Of Professional System Administrators—mentorship program for a couple of years. The reason that I have a career at all is that people did favors for me, and you can never repay that; you can only pay it forward. So, I had a number of people assigned to me through that program and through other areas as well, and what I've learned is that the success of a mentorship is almost entirely on the person seeking guidance: how diligent are they about following up, about going and asking great questions? Because otherwise, if someone comes and says, “Hey, can you mentor me?”—they never frame it quite like that, but that's fine; the terminology is always squishy here.Like, “Hey, can you give me advice on things?” “Sure.” And then they don't ask any questions. Well, if I just butt in with unsolicited advice, that's not helping them in a mentoring capacity; that's being a dude on Twitter. So, I'm trying to figure out the way of solving for that, and I don't know if there is an answer. What's your take?Guang Ming: I think that for many young people, there is a baseline level of information that they need, that almost any mentor can give, but it takes up a lot of time to get to that point. So, for example, I had a young woman reach out to me, and she wanted to get a foot in the door in the legal world and wanted some advice. And I couldn't. It was like, pulling teeth. I couldn't get her to say a word about herself. And our conversation lasted less than five minutes because I couldn't get her to speak about herself.And I almost let it end at that. But then I circled back with her a week later, and called her and said, “You know, I'm going to connect you to someone because I want to help you in your journey. But I need you to think before you get to that conversation about who you are, what you want, where you're going, what's your story. You know, I know just from the person who connected us that you're the first in your family to go to college. Speak to that.” And just really tried to help her understand that she needed to craft a narrative around herself. And I think a lot of young adults don't know how to craft that narrative.Corey: The problem that I see when I look at this systemically is that all of this stuff seems like it's very bespoke. It's [spreading an 00:15:45] opportunity, but it is incumbent upon folks to learn about it for themselves. One of the most foundational memories of my ill-fated academic career was in public school for my first sophomore year of high school, where the US history teacher said, all right. Today, we're not doing our traditional stuff, what I'm about to do is not in the curriculum. Please feel free to complain to your parents and then have them take it to the school board.And what he did was he passed at a flyer where each one of us had different numbers on it, and it was a, “You are a family of x number of people; you made this much money last year.” And then he passed out 1040-EZ forms. And he taught us how to file a tax return in the course of that 45-minute session. And it was, instead of learning a series of whitewashed facts about American history, I was learning how to function as an adult in society. And the fact that he had to do this almost as a subversive thing as opposed to being an accepted part of the curriculum is just mind-boggling to me. I see what you're doing is important and valuable, but it also in some level kind of feels like a band-aid over a massive societal failing. Is that accurate or am I missing something?Guang Ming: No, I think that certain school districts are trying to do this, they're trying to integrate financial literacy into calculus. Some schools will even offer a course, but the course isn't an AP course; it doesn't give you special credit, and so students don't take it, or it's viewed as a less valuable course even though it's probably the most valuable course. And there's also a level of embarrassment. Like, for certain things, we cover personal hygiene. The importance of brushing your teeth every day, and taking a shower, and wearing deodorant.Which is something you wouldn't necessarily think you would need to teach someone, but wait till you're in certain work environments, and that is actually something that people need to know that they're bothering their coworkers by this lack. That can be really embarrassing.With Lattice Climbers, you can do this in the privacy of your own home, you can do it in your bedroom, you can do it wherever you are, and you can get these little lessons and not feel embarrassed. Or sometimes you are afraid to ask a question because you feel dumb asking it. When we did a pilot with 17-to 19-year-olds, the favorite video was actually making an appointment. Just giving tips on how to gather the appropriate documentation you would need to, say for example, make a doctor's appointment, and sample scripts—we have downloadables that go along with—sample scripts of how a conversation would potentially run if you were to call.Corey: The way you describe this and the problem you're solving, I have a hard time seeing this as the business opportunity that becomes a $60 billion company because to do that, you would have to do something that is abjectly terrifying. So apparently, becoming rich beyond the wildest dreams of avarice is not the reason that you're doing this. What made you decide that this was a problem you wanted to address?Guang Ming: So, I am the daughter of an immigrant and a first-generation college student. And there were so many things that my parents just didn't know to teach me. They were very focused on academics and there was no focus on anything outside of book smarts. So, when I had my first college interview, my mom took me to the Fashion for Price Boutique next to the Drug Emporium in the strip mall near our home and bought me an interview suit—we didn't have a ton of money—and the interview suit involved zebra print zippers and a very short skirt. And that is what I wore to my Harvard interview. The one [laugh] school I didn't get into.And not only that, not only was I dressed wholly inappropriately, I also was a deer in the headlights. I had never done a mock interview, I had never done anything that would help prepare me for this situation. And I look back at that 17-year-old and I think, “How can I help her? How can I help people like her who don't have the social or cultural capital to know these things, to know how to move in the world that they want to be in desperately? How do I help them overcome that obstacle?” And that is how Lattice Climbers was born.Corey: The idea of having an experience like that as being necessary to forge this is—it's moving. It's the sort of thing that you hear about other people—you [unintelligible 00:20:04] secondhand cringing from hearing that sort of story, at least I do. And I can definitely understand not wanting other folks to have to go through this. We talk about hilarious interview mistakes that we've made, that we've had candidates make, and in some cases, most of the ones that I like to talk about are the folks who are—let's [unintelligible 00:20:23] here—25 years into their career or so, where they really should know better. Because making fun of some naive kid who'd never been in an interview scenario before is just being shitty, let's be clear.At some point, though, you should learn how to comport yourself in a working environment that makes sense. But without having mentorships or guidance like that, it feels like a lot of people have stories like this. I think what makes your story different than most of them is that you're willing to talk about it in public. Most of us bury those things down the memory hole, I would think.Guang Ming: Yes. I very much own the zebra print story, and it is something that I share when I speak at Girls State. I speak at Girls State just about every year to the young women, and I talk a lot about some of these things that we go over in Lattice Climbers to just try to impart, even in a six-minute speech, some of the key nuggets that I want them to take away with them, as they move through life.Corey: Tell me a little more about Girls State. I've heard the term a couple of times, but know remarkably little about it because, for better or worse, my daughters are still at a point where—I regret this constantly—I have to know entirely too much about the Paw Patrol.Guang Ming: So, Girls Day is a program sponsored by the American Legion Auxiliary. It is a week-long civic engagement program that simulates government over the course of one week. And for California Girls State, it is one girl from each sponsored high school, with about 540 young women—and of course, we've had to be virtual for the past couple of years, but they've done it in a webinar virtual sessions—and the program is all about women empowerment and encouraging civic engagement. And one of the things that has really impacted Lattice Climbers has been my observations in Girls State as a counselor for over 20 years. Because we work with young women from all different backgrounds, whether their parents are migrant workers in Modesto or doctors in Big Sur, there are gaps that these young women have that differ based on their backgrounds.And what I'm hoping to do with Lattice Climbers is fill those gaps and help them avoid these missteps and increase their trajectory as they climb the lattice. And that is one thing that we do is we don't talk about climbing the ladder because a ladder implies that there is one pathway to the top, there's room for only one there. We approach it as you're climbing a lattice: we're all in it together and there are infinite paths to success.Corey: All of these things that you talk about are challenging at the best of times, and these are very clearly not the best of times. One of the reasons that, to date, we at The Duckbill Group have not hired junior folks is because in a full-remote environment—and to be clear, even without the pandemic The Duckbill Group has been full-remote since its inception—I don't know that's necessarily the best way to expose someone new to the workforce. It feels to me like there's not a lot of examples around there. There's a requirement to be a lot more self-directed, and it's likely, for example, that someone will get stuck and spin on something for a while rather than asking for help because they don't want to appear like they don't know what they're doing and inadvertently make things worse. Do you think that remote as we move forward is going to be an increasing burden on folks like this, or—which I'm perfectly willing to accept—am I completely wrong, and that in fact having a full-remote environment like this is in fact a terrific opportunity for folks new to the workforce?Guang Ming: No, I think full-remote is an issue. I think that it takes so much more emotional energy to connect through a video than it does to connect in person. And there's also the lack of organic interactions. There are so many mentorships that develop just from walking down the hall and running into someone over coffee, or at the wat—I mean literally at the watercooler and having the opportunity to chat with someone about something non-work-related that can then evolve into a mentoring relationship. And there is just a lack of that.All these young people entering the work environment, they can wear pajamas all day and lay in bed with their laptop on their laps and work, and they may love that, but I think that if you want to work in a professional office environment, you need to understand appropriate attire, you need to understand appropriate behavior at events. I think that, especially if you're from certain backgrounds and you've never been around an open buffet before, it can be very tempting to just pile that plate as high as you can with crab legs or, you know, shrimp cocktail. And it's not appropriate in that setting. And so we cover those—Corey: Wait. It's not?Guang Ming: [laugh]. Well, it depe—if you're the CEO of the company, Corey, you can do whatever you want.Corey: No, no. That's my business partner. I am just the chief cloud economist because it's not professional to put the word shit-poster on a business card. Or so they tell me.Guang Ming: [laugh].Corey: In my experience, the worst of all worlds, though, is not the full-remote; it's not the in-office; it's the hybrid scenario where you have some people that are in an office together working and then you have folks who are remote, and regardless of what your intentions are, it is almost impossible to avoid having a striated structure where the in-person folks collaborate in different ways and make decisions informally to which remote folks are not privy. And it's not to do with cliques or anything like that, but the watercooler discussions, or, “I'm going to go grab lunch. Do you want to come with me?” Type of engagement stories. And I can't shake the feeling that remote really needs to be all or nothing, at least within the bounds of a team, if not company-wide.Guang Ming: I think that a hybrid version could work if there was a concerted effort to include the remote individuals if there was a scheduled Zoom happy hour. So, one of the things that happened during the COVID times is there's a group that I'm a part of, and we just had a happy hour on Saturday nights. At 8 p.m. everyone just kind of logged on and hung out for a period of time. And it was really good to connect with people in that casual environment; there wasn't always pressure to speak, there wasn't always pressure to perform, it was just being together and having that togetherness. So, I think that in a work environment, you could create opportunities for that. And then also, I think, bringing people into the office for specific meetings and things that are important like that. And then potentially—I don't like assigning mentors, but I think you almost have to assign mentors when you have remote workforce.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: The challenge also becomes one of, great for junior folks, that makes an awful lot of sense. Hire someone with 15 years of experience, and, “Oh, we're going to assign you a mentor here.” And they're like, “Oh, really. So, that's what condescending means. I was always looking for a perfect example.”Guang Ming: [laugh].Corey: That's a delicate balance to strike in my experience.Guang Ming: Oh, very true. Very true. I was thinking more of, like, the young adults starting out in their career because our focus is really early career, as well as young adults in high school.Corey: And that's, I guess, my question for you next is why is that the target age range that you think is best served by this? Now, having, again, spent too much time gazing into the mess that is the Paw Patrol, I understand why preschoolers are not the target market for this, but my approach has generally been targeting folks who are entering the workforce. Although let's be very clear, a large part of that is because I generally don't appreciate the optics of going and hanging out at the local high school trying to talk to kids.Guang Ming: Well, I think that high school is really where it starts. This is the age at which brains are starting to develop a little bit more; they're starting to have more social awareness. This is where beginnings of your network are important. And I think that the sooner that we can convey to young people that they're only as strong as their networks, the better. If they can understand that it's their teachers, their coaches, the parents of their friends are all the beginnings of their network.That's how you get internships, that's how you get a leg up is through these connections because if you're just a resume floating out there, your chances of getting looked at—and we all know how the world works—well, we should all know how the world works, which is it's all about your connections that helps you launch to the next thing.Corey: That's the thing that I think is understated in this is that we wind up telling students a whole bunch of things that are well-intentioned lies. The, “Oh, put your nose to the grindstone and work hard, and one day you will surely be promoted.” Now, I get flack when I say this sometimes from folks who've been at the same company for 15 years and demonstrated growth trajectory internally, but that's the exception, not the rule. Big moves generally look a lot like transitioning between companies.“Oh, you don't want to be a job hopper. It looks bad on the resume.” Yeah, you know who says that? People who don't want you to quit your job because you're unhappy because then they have to backfill you, or people who are trying to recruit you in and want to make sure that you when you show up at this new job, you stay there for a while. It's self-serving.Yeah, there's going to be some questions about it in the interview process, but you should have an answer ready to go for it. It's the interview skills piece of it and make sure that you don't inadvertently torpedo your own candidacy with conversations like that. And this is stuff that I find that is—it's not just the newer generation that we're talking about here; people well into their careers still haven't cracked a lot of these codes, mostly because, for better or worse, it turns out that people aren't nearly as cynical [laugh] about things as I am.Guang Ming: Well, and we also cover things like how to leave a job professionally. Because as we live in a world where you're not going to go work for one company for the next 30 years, or where you shouldn't go work for the same company for 30 years necessarily, but there are stories out there of people just ghosting on the job, ghosting on job interviews, and that burns bridges. And everyone you meet is a potential connection in your network as you climb the lattice, and so you need to preserve those relationships moving forward because you never know who you help out along the way, or who helps you out along the way, you never know how that connection is going to play out later on in life.Corey: That's the trick is that it's talking to people and being friendly with them. And there are ways to do networking properly in my world, and there are ways not to. And, “Oh, I should talk to you because down the road you might be useful to me,” is just cynical and terrible. I hate the pattern.Whereas, I like keeping in touch with people because I find them interesting. My default assumption has always been that I'm going to be talking to someone for longer than either one of us is going to be doing whatever it is we're currently doing, and trying to treat relationships as transactional is a mistake. But that's what networking is often interpreted as.Guang Ming: It's so true. And people can tell. They can tell when you're being fake. They can tell when you're being transactional. They can tell when you just are waiting for the ask.I think it actually is really hard to be genuine and natural for some people that comes across as transactional, and one of the ways that we talked about avoiding that is through just an ongoing relationship. So, you don't only reach out to the person when you have an ask, you reach out to the person quarterly. And you can have a spreadsheet—almost—about it, and of the people that you want to contact and maintain cont—and even if it's just a text message that says, “Hey, this is what I'm up to. Hope all is well with you.” And even if they don't respond, or just it's a one-word answer, you've at least had that touchpoint with them over the course of time.Corey: There's often a criticism levied at folks who are advocating for networking, that it is a lot harder when you're an introvert or when you are neurodivergent, in certain ways. To be clear, I've neurodivergent in ways that do not directly negatively impact my ability to socialize with folks; it just means they think I'm a jerk. But there are folks who definitely have different expressions of different divergences. And that's fine. How do you view the networking aspect for folks who do not work nearly as well interpersonally?Guang Ming: That's so hard because interpersonal skills are something that is so necessary, and I think that unfortunately, there are people who get by one hundred percent on their social skills. Like, their people skills are all they need to move forward in the world. And I think that you have to work at it, and you have to study how to behave in those situations. It's almost like—so for example, my husband is an introvert, but he was also an actor in college. And when he goes into these situations, it's almost like putting on a show.Like you talked about putting on your three-piece suit. There is the extrovert persona that he wears in these environments, and then he takes it off when he gets home. And I think that you almost have to create that persona for yourself. And you can acknowledge that you're neurodivergent, and you can acknowledge that you're an introvert, and I think that's way more acceptable these days than it used to be. And there are lots of people that are in the world that are neurodivergent and are introverts, and so I think it's completely fine to be that way.Corey: I've never had a good answer for folks who ask those questions, just because it is so different from my lived experience that I don't have an answer that's worth listening to, and I try very hard to stay in my lane. I don't ever want that to be interpreted as it's not important because it very much is.So, one last question I have for you is I love, love, love your zebra print suit story, but it's also back when you were applying to school, back in early career, which you are very clearly not now; it's decades old. Do you have any other similar stories from folks that you've been working through, either at Lattice Climbers or through Girls State, that illustrate this in a somewhat more modern era?Guang Ming: Oh, absolutely. So, there was a young woman at Girls State; we were all in a room and they were talking about colleges, the girls were talking about colleges. And this one young woman remains silent during this conversation. And so I approached her. I said, “Well, what about you? What are your plans for college?” And she shared that she wasn't going to go to college because her parents didn't go to college, they didn't have a lot of money, and she just didn't think that college was in the cards for her.And I disabused her of that notion. I told her absolutely not. You're here at Girls State, which means you're the top girl from your high school. You absolutely should go to college, and I told her that there were so many paths to college. You could go to community college and then transfer; you could go to technical college; there's so many different options.And there's so many scholarships out there, especially for low-income individuals. Well, we became friends on social media, and about three years after Girls State—because they attend the summer after their junior years—I received a message from her, sort of, out of the blue, and she let me know that that conversation that we had changed her life because she had gone to community college—she had taken my advice; she had gone to community college and had just been accepted as a transfer student to UC Berkeley. And that story just makes me tear up every time I think about it. And that one conversation had that huge impact on her life, and I'm hoping that through Lattice Climbers and our little lessons, that we can have that kind of impact on young lives, that we can help them avoid these missteps that could have huge impacts on their trajectory, and we could help them increase their trajectory on the lattice.Corey: It's similar in some respects to the folks I talk to who are building products for the cloud industry. It's, “Yes, yes, of course. You're always going to have a story about how it works for you. That's fine. Let's talk about your customers.” Like, “Find me a customer, someone else in the world who has a story like this that really demonstrates the value you provide.”And I love the fact that it is so easy for you to come up with these things off the top of your head, even when you weren't necessarily expecting the question. So, you're onto something. This is a clear problem, and it's not going away anytime soon, and it's largely underserved because there's no opportunity to invest venture capital into it and make a ridiculous return on that investment because there's not money in solving it that I can see—and apparently, most the industry can see—compared to another Twitter for Pets app.Guang Ming: [laugh]. Well, there is not that much money in them there hills because no one owns the problem, and because no one owns the problem, it's very hard to find people willing to pay to solve the problem. But that doesn't mean that the problem isn't there and that doesn't mean that it doesn't need to be solved. And I actually think that companies should have an incentive to do it because it will help with employee retention, it will help with employee performance if they do invest in their workers, and in high school students who, the sooner that they know these things, the better it will be for their long-term careers.Corey: And if nothing else, I think that's the lesson to take away from this for the young folk—the youth, as it were—that this is the single greatest thing I look at and credit my professional trajectory has been in learning to handle expectations in corporate environments. And sure, I have fun with them and I play games with them, but you have to know the rules before you can break them in this context. And there are business meetings in which I assure you, you would question whether it was the same person. And that's what it comes down to, I think, on some level is, if you know how to handle a job interview, you will always be able to find something to put food on the table. Conversely, if you're terrific at any number of different things, but absolutely cannot handle the dynamics of a job interview, you are going to struggle to find work anywhere until you find someone willing to alter their corporate process just in order to bring you aboard.It's a skill that you need to be at least conversant with. And what makes it even worse, as it's a skill that you only really get to practice when you're looking for jobs. I want to thank you for taking the time to speak with me so much about all this stuff. If people want to learn more about what you're up to and how you're approaching it, where can they find you?Guang Ming: So, we are at latticeclimbers.com. And we are currently in waitlist mode, so you can sign up on our waitlist and get more information about when we're ready to launch. We are working with some nonprofits and some school districts on some pilot programs, and we're hoping to have that going, hopefully by the end of the year.Corey: And we will, of course, put a link to that in the [show notes 00:38:07]. Thank you so much for taking the time to speak with me today. I really appreciate it.Guang Ming: Well, thank you so much for having me. I really appreciate the opportunity to share what we're doing with Lattice Climbers, and I just hope, like I said, if I can get one person to not wear zebra print to that Harvard interview, [laugh] then I will view Lattice Climbers as a success.Corey: [laugh]. Excellent. Thank you so much. Once again, Guang Ming Whitley, co-founder of Lattice Climbers. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, along with a rambling comment explaining why we're wrong and that a zebra-print suit for a college interview is in fact a best practice.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

This week Brian speaks with Chethan Athreyas, CEO of SynctacticAI SynctacticAI is a platform to power the future of Artificial Intelligence and Machine Learning. With the proliferation in Data Science and its fraternity, the platform aims at providing a complete automated Data engineering and DevOps suit. SynctacticAI provides an end to end platform where anyone can come connect their data sources, define workflows, visualize insights and build ML/AI models. Teams can collaborate with each other while working on a single source of truth for all their data needs. The flexibility of Hybrid Multicloud provides data teams the power to optimize their workflows and bring in their data or store their data anywhere they please, right from On-Premise deployments to any of the cloud providers. For more information, visit https://synctactic.ai/ If you have the next big idea, apply to the Expert Dojo Accelerator: www.expertdojo.com

This week, the Cloudbusting team is joined by Mike Pfeiffer, Founder and CEO of CloudSkills. Mike educates and advises individuals and teams on their Cloud and DevOps engineering journey using blended learning experiences. Listen in for career inspiration about cloud-native technologies and “Get in the Game!”

About MatthewMatthew Prince is co-founder and CEO of Cloudflare. Cloudflare's mission is to help build a better Internet. Today the company runs one of the world's largest networks, which spans more than 200 cities in over 100 countries. Matthew is a World Economic Forum Technology Pioneer, a member of the Council on Foreign Relations, winner of the 2011 Tech Fellow Award, and serves on the Board of Advisors for the Center for Information Technology and Privacy Law. Matthew holds an MBA from Harvard Business School where he was a George F. Baker Scholar and awarded the Dubilier Prize for Entrepreneurship. He is a member of the Illinois Bar, and earned his J.D. from the University of Chicago and B.A. in English Literature and Computer Science from Trinity College. He's also the co-creator of Project Honey Pot, the largest community of webmasters tracking online fraud and abuse.Links: Cloudflare: https://www.cloudflare.com Blog post: https://blog.cloudflare.com/aws-egregious-egress/ Bandwidth Alliance: https://www.cloudflare.com/bandwidth-alliance/ Announcement of R2: https://blog.cloudflare.com/introducing-r2-object-storage/ Blog.cloudflare.com: https://blog.cloudflare.com Duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Writing ad copy to fit into a 30 second slot is hard, but if anyone can do it the folks at Quali can. Just like their Torque infrastructure automation platform can deliver complex application environments anytime, anywhere, in just seconds instead of hours, days or weeks. Visit Qtorque.io today and learn how you can spin up application environments in about the same amount of time it took you to listen to this ad.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. Today, my guest is someone I feel a certain kinship with, if for no other reason than I spend the bulk of my time antagonizing AWS incredibly publicly. And my guest periodically descends into the gutter with me to do the same sort of things. The difference is that I'm a loudmouth with a Twitter account and Matthew Prince is the co-founder and CEO of Cloudflare, which is, of course, publicly traded. Matthew, thank you for deigning to speak with me today. I really appreciate it.Matthew: Corey, it's my pleasure, and appreciate you having me on.Corey: So, I'm mostly being facetious here, but not entirely, in that you have very publicly and repeatedly called out some of the same things I love calling out, which is AWS's frankly egregious egress pricing. In fact, that was a title of a blog post that you folks put out, and it was so well done I'm ashamed I didn't come up with it myself years ago. But it's something that is resonating with a large number of people in very specific circumstances as far as what their company does. Talk to me a little bit about that. Cloudflare is a CDN company and increasingly looking like something beyond that. Where do you stand on this? What got you on this path?Matthew: I was actually searching through really old emails to find something the other day, and I found a message from all the way back in 2009, so actually even before Michelle and I had come up with a name for Cloudflare. We were really just trying to understand the pricing on public clouds and breaking it all down. How much does the compute cost? How much does storage cost? How much does bandwidth cost?And we kept running the numbers over and over and over again, and the storage and compute costs actually seemed relatively reasonable and you could understand it, but the economics behind the bandwidth just made no sense. It was clear that as bandwidth usage grew and you got scale that your costs eventually effectively went to zero. And I think it was that insight that led to us starting Cloudflare. And the self-service plans at Cloudflare have always been unlimited bandwidth, and from the beginning, we didn't charge for bandwidth. People told us at the time we were crazy to not do that, but I think that that realization, that over time and at scale, bandwidth costs do go to zero is really core to who Cloudflare is.Cloudflare launched a little over 11 years ago now, and as we've watched the various public clouds and AWS in particular just really over that same 11 years not only not follow the natural price of bandwidth down, but really hold their costs steady. At some point, we've got a lot of mutual customers and it's a complaint that we hear from our mutual customers all the time, and we decided that we should do something about it. And so that started four years ago, when we launched the Bandwidth Alliance, and worked with almost all the major public clouds with the exception of Amazon, to say that if someone is sending traffic from a public cloud network to Cloudflare's network, we're not going to charge them for the bandwidth. It's going across a piece of fiber optic cable that yeah, there's some cost to put it in place and maybe there's some maintenance costs associated with it, but there's not—Corey: And the equipment at the end costs money, but it's not cloud cost; it just cost on a per second, every hour of your lifetime basis. It's a capital expense that is amortized across a number of years et cetera, et cetera.Matthew: And it's a fixed cost. It's not a variable cost. You put that fiber optic cable and you use a port on a router on each side. There's cost associated with that, but it's relatively de minimis. And so we said, “If it's not costing us anything and it's not costing a cloud provider anything, why are we charging customers for that?”And I think it's an argument that resonated with almost every other provider that was out there. And so Google discounts traffic when it's sent to us, Microsoft discounts traffic when it's sent to us, and we just announced that Oracle has joined this discounting their traffic, which was already some of the most cost-effective bandwidth from any cloud provider.Corey: Oh, yeah. Oracle's fantastic. As you were announced, I believe today, the fact that they're joining the Bandwidth Alliance is both fascinating and also, on some level, “Okay. It doesn't matter as much because their retail starting cost is 10% of Amazon's.” You have to start pushing an awful lot of traffic relative to what you would do AWS before it starts to show up. It's great to see.Matthew: And the fact that they're taking that down to effectively zero if you're using us is even better, right? And I think it again just illustrates how Amazon's really alone in this at being so egregious in how they do that. And it's, when we've done the math to calculate what their markups are, it's almost 80 times what reasonable assumptions on what their wholesale costs are. And so we really do believe in fighting for our customers and being customer-centric, and this seems like a place where—again, Amazon provides an incredible service and so many things, but the data transfer costs are just completely outrageous. And I'm glad that you're calling them out on it, and I'm glad we're calling them out on it and I think increasingly they look isolated and very anti-customer.Corey: What's interesting to me is that ingress to AWS at all the large public tier-one cloud providers is free. Which has led, I think, to the assumption—real or not—that bandwidth doesn't actually cost anything, whereas going outbound, all I can assume is that one day, some Amazon VP was watching a rerun of Meet the Parents and they got to the line where Ben Stiller says, “Oh, you can milk anything with nipples,” and said, “Holy crap. Our customers all have nipples; we can milk them with egress charges.” And here we are. As much as I think the cloud empowers some amazing stuff, the egress charges are very much an Achilles heel to a point where it starts to look like people won't even consider public cloud for certain workloads based upon that.People talk about how Netflix is a great representation of the ideal AWS customers. Yeah, but they don't stream a single byte to customers from AWS. They have their own CDN called Open Connect that they put all around the internet, specifically for that use case because it would bankrupt them otherwise.Matthew: If you're a small customer, bandwidth does cost something because you have to pay someone to do the work of interconnecting with all of the various networks that are out there. If you start to be, though, a large customer—like a Cloudflare, like an AWS, like an Azure—that is sending serious traffic to the internet, then it starts to actually be in the interest of ISPs to directly interconnect with you, and the costs of your bandwidth over time will approach zero. And that's the just economic reality of how bandwidth pricing works. I think that the confusion, to some extent, comes from all of us having bought our own home internet connection. And I think that the fact that you get more bandwidth up in most internet connections, and you get down, people think that there's some physics, which is associated with that.And there are; that turns out just to be the legacy of the cable system that was really designed to send pictures down to your—Corey: It wasn't really a listening post. Yeah.Matthew: Right. And so they have dedicated less capacity for up and again, in-home network connections, that makes a ton of sense, but that's not how internet connections work globally. In fact, you pay—you get a symmetric connection. And so if they can demonstrate that it's free to take the traffic in, we can't figure out any reason that's not simply about customer lock-in; why you would charge to take data out, but you wouldn't charge to put it in. Because actually cost more from writing data to a disk, it costs more than reading it from a disk.And so by all reasonable accounts, if they were actually charging based on what their costs were, they would charge for ingress but they want to charge for egress. But the approach that we've taken is to say, “For standard bandwidth, we just aren't going to charge for it.” And we do charge for if you use our premium routing services, which is something called Argo, but even then it's relatively cheap compared with what is just standard kind of internet connectivity that's out there. And as we see more of the clouds like Microsoft and Google and Oracle show that this is a place where they can be much more customer-centric and customer-friendly, over time I'm hopeful that will put pressure on Amazon and they will eliminate their egress fees.Corey: People also tend to assume that when I talk about this, that I'm somehow complaining about the level of discounting or whatnot, and they yell at me and say, “Oh, well, you should know by now, Corey, that no one at significant scale pays retail pricing.” “Thanks, professor. I appreciate that, but four years ago, or so I sat down with a startup founder who was sketching out the idea for a live video streaming service and said, ‘There's something wrong with my math because if I built this on AWS—which he knew very well, incidentally—it looks like it would cost me at our scale of where we're hoping to hit $65,000 a minute.'” And I checked and yep, sure enough, his math was not wrong, so he obviously did not build his proof of concept on top of AWS. And the last time I checked, they had raised several 100 million dollars in a bunch of different funding rounds.That is a company now that will not be on AWS because it was never an option. I want to talk as well about your announcement of R2, which is just spectacular. It is—please correct me if I get any of this wrong—it's an object store that lives in your existing distributed-points-of-presence-slash-data-centers-slash-colo-slash-a-bunch-of-computers-in-fancy-warehouse-rooms-with-the-lights-are-always-on-And-it's-always-cold-and-noisy. And people can store data there—Matthew: [crosstalk 00:10:23] aisles it's cold; in the other aisles, it's hot. But yes.Corey: Exactly. But it turns out when you lurk around to the hot aisle, that's not where all the buttons are and the things you're able to plug into, so it's freeze or sweat, and there's never a good answer. But it's an object store that costs a fair bit less than retail pricing for Amazon S3, or most other object stores out there. Which, okay, great. That's always good to see competition in the storage space, but specifically, you're not charging any data transfer costs whatsoever for doing this. First, where did this come from?Matthew: So, we needed it ourselves. I think all of the great products at Cloudflare start with an internal need. If you look at why do we build our zero-trust solutions? It's because we said we needed a security solution that was fast and reliable and secure to protect our employees as they were going out and using the internet.Why did we build Cloudflare Workers? Because we needed a very flexible compute platform where we could build systems ourselves. And that's not unique to us. I mean, why did Amazon build AWS? They built it because they needed those tools in order to continue to grow and expand as quickly as possible.And in fact, I think if you look at the products that Google makes that are really great, it ends up being the ones that Google's employees use themselves. Gmail started as Caribou once upon a time, which was their internal email system. And so we needed an object store and the sometimes belligerent CEO of Cloudflare insisted that our team couldn't use any of the public cloud object stores. And so we had to build it.That was the start of it and we've been using it internally for products over time. It powers, for example, Cloudflare Images, it powers a lot of our streaming video services, and it works great. And at some point, we said, “Can we take this and make it available to everyone?” The question that you've asked on Twitter, and I think a lot of people reasonably ask us, “What's the catch?”Corey: Well, in my defense, I think it's fair. There was an example that I gave of, “Okay, I'm going to go ahead and keep—because it's new, I don't trust new object stores. Great. I'm going to do the same experiment twice, keep one the pure AWS story and the other, I'm just going to add Cloudflare R2 to the mix so that I have to transfer out of AWS once.” For a one gigabyte file that gets shared out for a petabyte's worth of bandwidth, on AWS it costs roughly $52,000 to do that. If I go with the R2 solution, it cost me 13 cents, all of which except for a penny-and-a-half are AWS charges. And that just feels—when you're looking at that big of a gap, it's easy to look at that and think, “Okay, someone is trying to swindle me somewhere. And when you can't spot the sucker, it's probably me. What's the catch?”Matthew: I guess it's not really a catch; it's an explanation. We have been able to drive our bandwidth costs down low enough that in that particular use case, we have to store the file, and that, again, that—there's a hard disk in there and we replicate it to make sure that it's available so it's not just one hard disk, but it's multiple hard disks in various places, but that amortized over time, isn't that big a cost. And then bandwidth is effectively zero. And so if we can do that, then that's great.Maybe a different way of framing the question is like, “Why would we do that?” And I think what we see is that there is an opportunity for customers to be able to use the best of various cloud providers and hook the different parts together. So, people talk about multi-cloud all the time, and for a while, the way that I think people thought about that was you take the exact same workload and you run it in Azure and AWS. That turns out not to be—I mean, maybe some people do that, but it's super rare and it's incredibly hard.Corey: It has been a recurring theme of most things I say where, by default, that is one of the dumbest things I can imagine.Matthew: Yeah, that isn't good. But what people do want to do is they want to say, “Listen, there's some really great services that Amazon provides; we want to use those. And there's some really great services that Azure provides, and we want to use those. And Google's got some great machine learning, and so does IBM. And I want to sort of mix and match the various pieces together.”And the challenge in doing that is the egress fees. If everyone just had a detente and said there's going to be no egress fees for us to be able to hook these various [pits 00:14:48] together, then you would be able to take advantage of a lot of the different technologies and we would actually get stronger applications. And so the vision of what we're trying to build is how can we be the fabric that can stitch the various cloud providers together so that you can do that. And when we looked at that, and we said, “Okay, what's the path to getting there?” The big place where there's the just meatiest cost on egress fees is object stores.And so if you could have a centralized object store, and you can say then from that object go use whatever the best service is at Amazon, go use whatever the best service is at Google, go use whatever the best service is at Azure, that then allows, I think, actually people to take advantage of the cloud in a way which is what people really should mean when they talk about multi-cloud. Which is, there should be competition on the various features themselves, and you should be able to pick and choose the best of all of the different bits. And I think we as consumers then benefit from that. And so when we're looking at how we can strategically enable that future, building an object store was a real key part of that, and that's part of what we're doing. Now, how do we make money off of that? Well, there's a little bit off the storage, and again, even [laugh]—Corey: Well, that is the Amazonian answer there. It's like, “Your margin is my opportunity,” is a famous Bezos quote, and I figure you're sitting there saying, “Ah, it would cost $52,000 to do that in Amazon. Ah, we can make a penny-and-a-half.” That's very Amazonian, you could probably get hired over there with that philosophy.Matthew: Yeah. And this is a commodity service, just [laugh] storing data. If you look across the history of what Cloudflare has done, in 2014, we made encryption free because it's absurd to pay for math, right? I mean, it's just crazy right?Corey: Or to pay for security as a value-add. No, that should be baked into whatever you're doing, in an ideal world.Matthew: Domain registration. Like, it's writing something down in a ledger. It's a commodity; of course it should go to whatever the absolute cost is. On the other hand, there are things that we do that aren't commodities where we are able to better protect people because we see so much traffic, and we've built the machine learning models, and we've done those things, and so we charge for those things. So commodities, we think over time, go to effectively, whatever their cost is, and then the value is in the actual intelligent services that are on top of it.But an object store is a commodity and so we should be trying to drive that pricing down. And in the case of bandwidth, it's effectively free for us. And so if we can be that fabric that connects the different class together, I think that makes sense is a strategy for us and that's why R2 made a ton of sense for us to build and to launch.Corey: There seems to be a lack of ability for lots of folks, at least on the internet to imagine a use case other than theirs. I cheated by being a consultant, I get to borrow other people's use cases at a high degree of turnover. But the question I saw raised was, “Well, how many workloads really do that much egress from static objects that don't change? Doesn't sound like there'd be a whole lot of them.” And it's, “Oh, my sweet summer child. Sure, your app doesn't do a lot of that, but let me introduce it to my friends who are hosting videos on their website, for example, or large images that get accessed a whole bunch of times; things that are written once and then read forever by the internet.”Matthew: And we sit in a position where because of the role that Cloudflare plays where we sit in front of a number of these different cloud providers, we could actually look at the use cases and the data, and then build products in order to solve that. And that's why we started with Workers; that's why we then built the KV store that was on top of that; we built object-store next. And so you can see as we're sort of marching through these things, it is very much being informed by the data that we actually see from real customers. And one of the things that I really like about R2 is in exactly the example that you gave where you can keep everything in S3; you can set R2 in front of it and put it in slurp mode, and effectively it just—as those objects get pulled out, it starts storing them there. And so the migration path is super easy; you don't have to actually change anything about your application and will cut your bills substantially.And so I think that's the right thing to enable a multi-cloud world where, again, it's not you're running the exact same workload in different places, but you get to take advantage of the really great tack that all of these companies are building and use that. And then the companies will compete on building that tech well. So, it's not just about how do I get the data in and then kind of underinvest in all of the different services that I provide. It's how can we make sure that on a service-by-service basis, you actually are having real competition over time. And again, I think that's the right thing for customers, and absolutely R2 might not be the right thing for every use case that's out there, but I think that it wi—enabling more competition is going to make the cloud better for everyone.Corey: Oh, yeah. It's always fun hearing it from Amazonians. It's, “You have a service that talks to satellites in orbit. You really think that's a general-purpose thing that every company out there has to deal with?” No. Well, not yet, anyway.It also just feels to me like their transfer approach is antithetical to almost every other aspect of how they have built their cloud. Amazonians have told me repeatedly—I believe them—that their network is effectively magic. The fact that you can get near line rate between any two points without melting various [unintelligible 00:20:14], which shows that there was significant thought, work, effort, planning, technology, et cetera, put into the network. And I don't dispute that. But if I'm trying to build a workload and put it inside of AWS, I can control how it performs tied to budget; I can have a lot of RAM for things that are memory intensive, or I can have a little RAM; I can have great CPU performance or terrible CPU performance.The challenge with data transfer is it is uniformly great. “I want to get that data over there super quickly.” Yeah, awesome. I'm fine paying a premium for that. But I have this pile of data right here. I want to get it over there, ideally by Tuesday. There's no good way to do that, even with their Snowball—or Snow Family devices—when you fill them with data and send them into AWS, yeah, that's great. Then you just pay for the use of the device.Use them to send data out of AWS, they tack on an additional per-gigabyte fee for getting the data out. You're training as a lawyer, you went to the same law school that my wife did, the University of Chicago, which, oh, interesting stories down that path. But if we look at this, my argument is that the way to do an end-run around this is to sue Amazon for something, and then demand access to the data you have living in their environment during discovery. Make them give it to you for free, though, they'd probably find a way to charge it there, too. It's just a complete lack of vision and lack of awareness because it feels like they're milking a cash cow until it dies.Matthew: Yeah, they probably would charge for it and you'd also have to pay a lot of lawyers. So, I'm not sure that's the cost [crosstalk 00:21:44]—Corey: Its only works above certain volumes, I figure.Matthew: I do think that if your pricing strategy is designed to lock people in to prevent competition, then that does create other challenges. And there are certainly some University of Chicago law professors out there that have spent their careers arguing why antitrust laws don't make any sense, but I think that this is definitely one of those areas where you can see very clearly that customers are actually being harmed by the pricing strategy that's there. And the pricing strategy is not tied in any way to the underlying costs which are associated with that. And so I do think that, especially as you see other providers in the space—like Oracle—taking their bandwidth costs to effectively zero, that's the sort of thing that I think will have regulators start to scratch their heads. If tomorrow, AWS took egress costs to zero, and as a result, R2 was not as advantaged as it is today against them, you know, I think there are a lot of people who would say, “Oh, they showed Cloudflare.” I would do a happy dance because that's the best thing [thing they can do 00:22:52] for our customers.Corey: Our long-term goals, it sounds like, are relatively aligned. People think that I want to see AWS reign ascendant; people also say I want to see them burning and crashing into the sea, and neither one of those are true. What I want is, I want someone in a few years from now to be doing a startup and trying to figure out which cloud provider they should pick, and I want that to be a hard decision. Ideally, if you wind up reducing data transfer fees enough, it doesn't even have to be only one. There are stories that starts to turn into an actual realistic multi-cloud story that isn't, at its face, ridiculous. But right now, you have to pick a horse and ride it, for a variety of reasons. And I don't like that.Matthew: It's entirely egress-based. And again, I think that customers are better off if they are able to pick who is the best service at any time. And that is what encourages innovation. And over time, that's even what's good for the various cloud providers because it's what keeps them being valuable and keeps their customers thinking that they're building something which is magical and that they aren't trapped in the decision that they made, which is when we talk to a lot of the customers today, they feel that way. And it's I think part of why something like R2 and something like the Bandwidth Alliance has gotten so much attention because it really touches a nerve on what's frustrating customers today. And if tomorrow Amazon announced that they were eliminating egress fees and going head-to-head with R2, again, I think that's a wonderful outcome. And one that I think is unlikely, but I would celebrate it if it happened.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: My favorite is people who don't do research on this stuff. They wind up saying, “Oh, yeah. Cloudflare is saying that bandwidth is a fixed cost. Of course not. They must be losing their shirt on this.”You are a publicly-traded company. Your gross margins are 76% or 77%, depending upon whether we're talking about GAAP or non-GAAP. Point being, you are clearly not selling this at a loss and hoping to make it up in volume. That's what a VC-backed company does. Is something that is real and as accurate.I want to, on some level, I guess, low-key apologize because I keep viewing Cloudflare through a lens that is increasingly inaccurate, which is as a CDN. But you've had Cloudflare Workers for a while, effectively Functions as a Service that run at the edge, which has this magic aura around it, that do various things, which is fascinating to me. You're launching R2; it feels like you are in some ways aiming at becoming a cloud provider, but instead of taking the traditional approach of building it from the region's outward, you're building it from the outward in. Is that a fair characterization?Matthew: I think that's right. I think fundamentally what Cloudflare is, is a network. And I remember early on in the pandemic, we did a series of fireside chats with people we thought we could learn from. And so was everyone from Andre Iguodala, the basketball player, to Mark Cuban, the entrepreneur, to we had a [unintelligible 00:25:56] governor and all kinds of things. And we these were just internal on off the record.And I got to do one with Eric Schmidt, the former CEO of Google. And I said, “You know, Eric, one of the things that we struggle with is describing what is Cloudflare.” And without hesitation, he said, “Oh, that's easy. You're the network I plug into and don't have to worry about anything else.” And I think that's better than I could say it, myself, and I think that's what it is that we fundamentally are: we're the network that fits together.Now, it turns out that in the process of being that network and enabling that network, we are going to build things like R2, which start to be an object store and starts to sort of step into some of the cloud provider space. And Workers is really just a way of programming that network in order to do that, but it turns out that there are a bunch of workloads that if you move them into the network itself, make sense—not going to be every workload, but a lot of workloads that makes sense there. And again, I think that you can actually be very bullish on all of the big public cloud providers and bullish on Cloudflare at the same time because what we want to do is enable the ability for people to mix and match, and change, and be the fabric that connects all of those things together. And so over time, if Amazon says, “We're going to drop egress fees,” it may be that R2 isn't a product that exists—I don't think they're going to do that, so I think it's something that is going to be successful for us and get a lot of new users to us—but fundamentally, I think that where the traditional public clouds think of themselves as the place you put data and you process data, I think we think of ourselves as the place you move data. And that's somewhat different.That then translates into it as we're building out the different pieces, where it does feel like we're building from the outside in. And it may be that over time, that put versus move distinction becomes narrower and narrower as we build more and more services like R2, and durable objects, and KV, and we're working on a database, and all those things. And it could be that we converge in a similar place.Corey: One thing I really appreciate about your vision because it is so atypical these days, is that you aren't trying to build the multifunction printer of companies. You are not trying to be all things to all people in every scenario. Which is impossible to do, but companies are still trying their level best to do it. You are staking out the bounds of where you were willing to start and where you're willing to stop, in a variety of different ways. I would be—how do I put it?—surprised if you at some point in the next five years come out with, “And this is our own database that we have built out that directly competes with the following open-source project that we basically have implemented their API and gone down that particular path.” It does not sound like it is in your core wheelhouse at that point. You don't need—to my understanding—to write your own database engine in order to do what you do.Matthew: Maybe. I mean, we actually are kind of working on a database because—Corey: Oh, no, here we go again.Matthew: [laugh]—and yeah—in a couple of different ways. So, the first way is, we want to make sure that if you're using Workers, you can connect to whatever database you want to use anywhere in the world. And that's something that's coming and we'll be there. At the same time, the challenge of distributed computing turns out not to be the computing, it turns out to be the data and figuring out how to—CAP theorem is real, right? Consistency, Availability, and Partition tolerance; you can pick any two out of the three, but you can't get all three.And so you there's always going to be some trade-off that's there. And so we don't see a lot of good examples. There's some really cool companies that are working on things in the space, but we don't see a lot of really good examples of who has built a database that can be run on a distributed workload system, like Cloudflare to it do well. And so our team internally needs that, and so we're trying to figure out how to build it for ourselves, and I would imagine that after we build it for ourselves—if it works the way we expect it will—that that will then be something that we open up.Our motivation and the way we think about products is we need to build the tools for our own team. Our team itself is customer zero, and then some of those things are very specific to us, but every once in a while, when there are functions that makes sense for others, then we'll build them as well. And that does maybe risk being the multifunction printer, but again, I think that because the customer for that starts with ourselves, that's how we think about it. And if there's someone else's making a great tool, we'll use that. But in this case, we don't see anyone that's built a multi-tenant, globally-distributed, ACID-compliant relational database.Corey: I can't let it pass on challenge. Sure they have, and you're running it yourself. DNS: the finest database in the world. You stuff whatever you want to text records, and now you have taken a finely crafted wrench and turned it into a barely acceptable hammer, which is what I love about doing that terrible approach. Yeah, relational is not going to quite work that way. But—Matthew: Yes. That's a fancy key-value store, right? So—and we've had that for a long time. As we're trying to build those things up, the good news is that, again, we've run data at scale for quite some time and proven that we can do it efficiently and reliably.Corey: There's a lot that can be said about building the things you need to deliver your product to customers. And maybe a database is a poor example here, but I don't see that your motivation in this space is to step into something completely outside your areas of expertise solely because there's money to be made over there. Well, yeah, fortune passes everywhere. The question is, which are you best positioned to wind up delivering an actual transformative solution to that space, and what parts of it are just rent-seeking where it's okay, we're going to go and wherever the money is, we're chasing that down.Matthew: Yeah, we're still a for-profit business, and we've been able to grow revenue well, but I think it is that what motivates us and what drives us comes back to our mission, which is how do you help build a better internet? And you can look at every single thing that we've done, and we try to be very long-term-oriented. So, for instance, when we in 2014 made encryption free, the number one reason at the time, when people upgraded for the free version of our service, the paid version of our service is they got encryption for that. And so it was super scary to say, “Hey, we're going to take the biggest feature and give it away for free,” but it was clearly the direction of history and we wanted to be on the right side of history. And we considered it a bug that the internet wasn't built in an encrypted way from the beginning.So, of course, that was going to head that direction. And so I think that we and then subsequently Let's Encrypt, and a bunch of others have said, it's absurd that you're charging for math. And again, I think that's a good example of how we think about products. And we want to continue to disrupt ourselves and take the things that once upon a time were reserved for our customers that spend $10 million-plus with us, and we want to keep pushing those things down because, over time, the real opportunity is if you do right by customers, there will be plenty of ways that you can earn some of their budget. And again, we think that is the long-term winning strategy.Corey: I would agree with this. You're not out there making sneakers and selling them because you see people spend a lot of money on that; you're delivering value for customers. I say this as one of your paying customers. I have zero problem paying you every month like clockwork, and it is the least cloud-like experience because I know exactly what the bill is going to be in advance, which is apparently not how things should be done in this industry, yadda, yadda, yadda. It is a refreshingly delightful experience every time.The few times I've had challenges with the service, it has almost always been a—I'll call it a documentation gap, where the way it was explained in the formal documentation was not how I conceptualize things, which, again, explaining what these complex things are to folks who are not steeped in certain areas of them is always going to be a challenge. But I cannot think back to a single customer service failure I've had with you folks. I can't look back at any point where you have failed me as a customer, which is a strange thing to say, given how incredibly efficient I am at stumbling over weird bugs.Matthew: Terrific to have you as a customer. We are hardly perfect and we make mistakes, but one of the things I think that we try to do and one of the core values of Cloudflare is transparency. If I think about, like, the original sins of tech, a lot of it is this bizarre secrecy which pervades the entire industry. When we make mistakes, we talk about them, and we explain them. When there's an error, we don't throw up a white page; we put up a page that has our logo on it because we want to own it.And that sometimes gets blowback because you're in front of it, but again, I think it's the right thing to do for customers. And it's and I think it's incredibly important. One of the things that's interesting is you mentioned that you know what your bill is going to be. If you go back and look at the history of hosting on the internet, in the early days of internet hosting, it looks a lot like AWS.Corey: Oh, 95th percentile transit billing; go for one five minutes segment over and boom, your bill explodes. Oh, I remember those days. Unkindly.Matthew: And it was super complicated. And then what happened is the hosting world switched from this incredibly complicated billing to much more simplified, predictable, unlimited bandwidth with maybe some asterisks, but largely that was in place. And then it's strange that Amazon came along and then has brought us back to the more complicated world that's out there. I would have predicted that that's a sine wave—Corey: It has to be. I mean—Matthew: —and it's going to go back and forth over time. But I would have predicted that we would be more in the direction of coming back toward simplify, everything included. And again, I think that's how we've priced our things from the beginning. I'm surprised that it has held on as long as it has, but I do think that there's going to be an opportunity for—and I don't think Amazon will be the leader here, but I think there will be an opportunity for one of the big clouds.And again, I think Oracle is probably doing this the best of any of them right now—to say, “How can we go away from that complexity? How can we make bills predictable? How can we not nickel and dime everything, but allow you to actually forecast and budget?” And it just seems like that's the natural arc of history, and we will head back toward that. And, again, I think we've done our part to push that along. And I'm excited that other cloud providers seem to be thinking about that now as well.Corey: Oh, yeah. What I do with fixing AWS bills is the same thing folks were doing in the 70s and 80s with long-distance bills for companies. We're definitely hitting that sine wave. I know that if I were at AWS in a leadership role, I would be actively embarrassed that the company that is delivering a better customer experience around financial things is Oracle of all companies, given their history of audits and surprising people and the rest. It is ridiculous to me.One last topic that I want to cover with you before we call it an episode is, back in college, you had a thesis that you have done an excellent job of effectively eliminating from the internet. And the theme of this, to my understanding, was that the internet is a fad. And I am so aligned with that because I'm someone who has said for years that emerging technologies are fads. I've said it about cloud, about virtualization, about containers. And I just skipped Kubernetes. And now I'm all-in on serverless, which means, of course it's going to fail because I'm always wrong on these things. But tell me about that.Matthew: When I was seven years old in 1980, my grandmother gave me an Apple ][+ computer for Christmas. And I took to it like a just absolute duck to water and did things that made me very popular in junior high school, like going to computer camp. And my mom used to sign up for continuing education classes at the local university in computer science, and basically sneak me in, and I'd do all the homework and all that. And I remember when I got to college, there was a small group of students that would come around and help other students set their computer up, and I had it all set up and was involved. And so, got pretty deeply involved in the computer science program at college.And then I remember there was a group of three other students—so they were four of us—and they wanted to start an online digital magazine. And at the time, this was pre-web, or right in the early days of the web; it was sort of nineteen… ninety-three. And we built it originally on old Apple technology called HyperCard. And we used to email out the old HyperCard stacks. And the HyperCard stacks kept getting bigger and bigger and bigger, and we'd send them out to the school so [laugh] that we—so we kept crashing the mail servers.But the college loved this, so they kept buying bigger and bigger mail servers. But they were—at some point, they said, “This won't scale. You got to switch technologies.” And they introduced us to two different groups. One was a printer company based out in San Francisco that had this technology called PDF. And I was a really big fan of PDF. I thought PDF was the future, it was definitely going to be how everything got published.And then the other was this group of dorky graduate students at the University of Illinois that had this thing called a browser, which was super flaky, and crashed all the time, and didn't work. And so of the four of us, I was the one who voted for PDF and the other three were like, “Actually, I think this HTML thing is going to be a hit.” And we built this. We won an award from Wired—which was only a print magazine at the time—that called us the first online-only weekly publication. And it was such a struggle to get anyone to write for it because browsers sucked and, you know, trying to get students on campus, but no one on campus cared.We would get these emails from the other side of the world, where I remember really clearly is this—in broken English—email from Japan saying, “I love the magazine. Please keep writing more for the magazine.” And I remember thinking at the time, “Why do I care if someone in Japan is reading this if the girl down the hall who I have a crush on isn't?” Which is obviously what motivates dorky college students like myself. And at that same time, you saw all of this internet explosion.I remember the moment when Netscape went public and just blew through all the expectations. And it was right around the time I was getting ready to graduate for college, and I was kind of just burned out on the entire thing. And I thought, “If I can't even get anyone to write for this dopey magazine and yet we're winning awards, like, this stuff has to all just be complete garbage.” And so wrote a thesis on—ehh, it was not a very good [laugh] thesis. It's—but one of the things I said was that largely the internet was a fad, and that if it wasn't, that it had some real risks because if you enabled everyone to connect with whatever their weird interests and hobbies were, that you would very quickly fall to the lowest common denominator. And predicted some things that haven't come true. I thought for sure that you would have both a liberal and conservative search engine. And it's a miracle to this day, I think that doesn't exist.Corey: Now, that you said it, of course, it's going to.Matthew: Well, I don't know I've… [sigh] we'll see. But it is pretty amazing that Google has been able to, again, thread that line and stay largely apolitical. I'm surprised there aren't more national search engines; the fact that it only Russia and China have national search engines and France and Germany don't is just strange to me. It seems like if you're controlling the source of truth and how people find it, that seems like something that governments would try and take over. There are some things that in retrospect, look pretty wise, but there were a lot more things that looked really, really stupid. And so I think at some level, I had to build Cloudflare to atone for that stupidity all those years ago.Corey: There's something to be said for looking back and saying, “Yeah, I had an opinion, and with the light of new information, I am changing my opinion.” For some reason, in some circles, it feels like that gets interpreted as a sign of weakness, but I couldn't disagree more, it's, “Well, I had an opinion based upon what I saw at the time. Turns out, I was wrong, and here we are.” I really wish more people were capable of doing that.Matthew: It's one of the things we test for in hiring. And I think the characteristic that describes people who can do that well is really empathy. The understanding that the experiences that you have lead you to have a unique set of insights, but they also create a unique set of blind spots. And it's rare that you find people that are able to do that. And whenever you do—whenever we do we hire them.Corey: To that end, as far as hiring and similar topics go, if people want to learn more about how you view things, and how you see the world, and what you're releasing—maybe even potentially work with you—where can they find you?Matthew: [laugh]. So, the joke, sometimes, internal at Cloudflare is that Cloudflare is a blogging company that runs this global network just to have something to write about. So, I think we're unlike most corporate blogs, which are—if our corporate blog were typical, we'd have articles on, like, “Here are the top six reasons you need a fast website,” which would just be, you know, shoot me. But instead, I think we write about the things that are going on online and our unique view into them. And we have a core value of transparency, so we talk about that. So, if you're interested in Cloudflare, I'd encourage you to—especially if you're of the sort of geekier variety—to check out blog.cloudflare.com, and I think that's a good place to learn about us. And I still write for that occasionally.Corey: You're one of the only non-AWS corporate blogs that I pay attention to, for that exact reason. It is not, “Oh, yay. More content marketing by folks who just feel the need to hit a quota as opposed to talking about something valuable and interesting.” So, it's appreciated.Matthew: The secret to it was we realized at some point that the purpose of the blog wasn't to attract customers, it was to attract potential employees. And it turns out, if you sort of change that focus, then you talk to people like their peers, and it turns out then that the content that you create is much more authentic. And that turns out to be a great way to attract customers as well.Corey: I want to thank you for taking so much time out of your day to speak with me. I really appreciate it.Matthew: Thanks for all you're doing. And we're very aligned, and keep fighting the good fight. And someday, again, we'll eliminate cloud egress fees, and we can share a beer when we do.Corey: I will absolutely be there for it. Matthew, Prince, CEO, and co-founder of Cloudflare. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a rambling comment explaining that while data packets into a cloud provider are cheap and crappy, the ones being sent to the internet are beautiful, bespoke, unicorn snowflakes, so of course they cost money.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

In this episode, we cover: 00:00:00 - Introduction 00:02:45 - Adopting the Cloud 00:08:15 - POC Process  00:12:40 - Infrastructure Team Building 00:17:45 - “Disaster Roleplay”/Communicating to the Non-Technical Side  00:20:20 - Leadership 00:22:45 - Tomas' Horror Story/Dashboard Organziation 00:29:20 - Outro Links: Productboard: https://www.productboard.com Scaling Teams: https://www.amazon.com/Scaling-Teams-Strategies-Successful-Organizations/dp/149195227X Seeking SRE: https://www.amazon.com/Seeking-SRE-Conversations-Running-Production/dp/1491978864/ TranscriptJason: Welcome to Break Things on Purpose, a podcast about failure and reliability. In this episode, we chat with Tomas Fedor, Head of Infrastructure at Productboard. He shares his approach to testing and implementing new technologies, and his experiences in leading and growing technical teams.Today, we've got with us Tomas Fedor, who's joining us all the way from the Czech Republic. Tomas, why don't you say hello and introduce yourself?Tomas: Hello, everyone. Nice to meet you all, and my name is Tomas, or call me Tom. And I've been working for a Productboard for past two-and-a-half year as infrastructure leader. And all the time, my experience was in the areas of DevOps, and recently, three and four years is about management within infrastructure teams. What I'm passionate about, my main technologies-wise in cloud, mostly Amazon Web Services, Kubernetes, Infrastructure as Code such as Terraform, and recently, I also jumped towards security compliances, such as SOC 2 Type 2.Jason: Interesting. So, a lot of passions there, things that we actually love chatting about on the podcast. We've had other guests from HashiCorp, so we've talked plenty about Terraform. And we've talked about Kubernetes with some folks who are involved with the CNCF. I'm curious, with your experience, how did you first dive into these cloud-native technologies and adopting the cloud? Is that something you went straight for, or is that something you transitioned into?Tomas: I actually slow transition to cloud technologies because my first career started at university when I was like, say, half developer and half Unix administrator. And I had experience with building very small data center. So, those times were amazing to understand all the hardware aspects of how it's going to be built. And then later on, I got opportunity to join a very famous startup at Czech Republic [unintelligible 00:02:34] called Kiwi.com [unintelligible 00:02:35]. And that time, I first experienced cloud technologies such as Amazon Web Services.Jason: So, as you adopted Amazon, coming from that background of a university and having physical servers that you had to deal with, what was your biggest surprise in adopting the cloud? Maybe something that you didn't expect?Tomas: So, that's great question, and what comes to my mind first, is switching to completely different [unintelligible 00:03:05] because during my university studies and career there, I mostly focused on networking [unintelligible 00:03:13], but later on, you start actually thinking about not how to build a service, but what service you need to use for your use case. And you don't have, like, one service or one use case, but you have plenty of services that can suit your needs and you need to choose wisely. So, that was very interesting, and it needed—and it take me some time to actually adopt towards new thinking, new mindset, et cetera.Jason: That's an excellent point. And I feel like it's only gotten worse with the, “How do you choose?” If I were to ask you to set up a web service and it needs some sort of data store, at this point you've got, what, a half dozen or more options on Amazon? [laugh].Tomas: Exactly.Jason: So, with so many services on providers like Amazon, how do you go about choosing?Tomas: After a while, we came up with a thing like RFCs. That's like ‘Request For Comments,' where we tried to sum up all the goals, and all the principles, and all the problems and challenges we try to tackle. And with that, we also tried to validate all the alternatives. And once you went through all these information, you tried to sum up all the possible solutions. You typically had either one or two options, and those options were validated with all your team members or the whole engineering organization, and you made the decision then you try to run POC, and you either are confirmed, yeah this is the technology, or this is service you need and we are going to implement it, or you revised your proposal.Jason: I really like that process of starting with the RFC and defining your requirements and really getting those set so that as you're evaluating, you have these really stable ideas of what you need and so you don't get swayed by all of the hype around a certain technology. I'm curious, who is usually involved in the RFC process? Is it a select group in the engineering org? Is it broader? How do you get the perspectives that you need?Tomas: I feel we have very great established process at Productboard about RFCs. It's transparent to the whole organization, that's what I love the most. The first week, there is one or two reporters that are mainly focused on writing and summing up the whole proposal to write down goals, and also non-goals because that is going to define your focus and also define focus of reader. And then you're going just to describe alternatives, possible options, or maybe to sum up, “Hey, okay, I'm still unsure about this specific decision, but I feel this is the right direction.” Maybe I have someone else in the organization who is already familiar with the technology or with my use case, and that person can help me.So, once—or we call it a draft state, and once you feel confident, you are going to change the status of RFC to open. The time is open to feedback to everyone, and they typically geared, like, two weeks or three weeks, so everyone can give a feedback. And you have also option to present it on engineering all-hands. So, many engineers, or everyone else joining the engineering all-hands is aware of this RFC so you can receive a lot of feedback. What else is important to mention there that you can iterate over RFCs.So, you mark it as resolved after through two or three weeks, but then you come up with a new proposal, or you would like to update it slightly with important change. So, you can reopen it and update version there. So, that also gives you a space to update your RFC, improve the proposal, or completely to change the context so it's still up-to-date with what you want to resolve.Jason: I like that idea of presenting at engineering all-hands because, at least in my experience, being at a startup, you're often super busy so you may know that the RFC is available, but you may not have time to actually read through it, spend the time to comment, so having that presentation where it's nicely summarized for you is always nice. Moving from that to the POC, when you've selected a few and you want to try them out, tell me more about that POC process. What does that look like?Tomas: So typically, in my infrastructure team, it's slightly different, I believe, as you have either product teams focus on POCs, or you have more platform teams focusing on those. So, in case of the infrastructure team, we would like to understand what code is actually going to be about because typically the infrastructure team has plenty of services to be responsible for, to be maintained, and we try to first choose, like, one specific use case and small use case that's going to suit the need.For instance, I can share about implementation of HashiCorp Vault, like our adoption. We leveraged firstly only key-value engine for storing secrets. And what was important to understand here, whether we want to spend hours of building the whole cluster, or we can leverage their cloud service and try to integrate it with one of our services. And we need to understand what service we are going to adopt with Vault.So, we picked cloud solution. It was very simple, the experience that were seamless for us, we understood what we needed to validate. So, is developer able to connect to Vault? Is application able to connect to Vault? What roles does it offer? Was the difference for cloud versus on-premise solution?And at the end, it's often the cost. So, in that case, POC, we spin up just cloud service integrated with our system, choose the easiest possible adaptable service, run POC, validate it with developers, and provide all the feedback, all the data, to the rest of the engineering. So, that was for us, some small POC with large service at the end.Jason: Along with validating that it does what you want it to do, do you ever include reliability testing in that POC?Tomas: It is, but it is in, like, let's say, it's in a later stage. For example, I can again mention HashiCorp Vault. Once we made a decision to try to spin up first on-premise cluster, we started just thinking, like, how many master nodes do we need to have? How many availability zones do we need to have? So, you are going to follow quorum?And we are thinking, “Okay, so what's actually the reliability of Amazon Web Services regions and their availability zones? What's the reliability of multi-cross-region? And what actually the expectations that is going to happen? And how often they happen? Or when in the past, it happened?”So, all those aspects were considered, and we ran out that decision. Okay, we are still happy with one region because AWS is pretty stable, and I believe it's going to be. And we are now successfully running with three availability zones, but before we jumped to the conclusion of having three availability zones, we run several tests. So, we make sure that in case one availability zone being down, we are still fully able to run HashiCorp Vault cluster without any issues.Jason: That's such an important test, especially with something like HashiCorp Vault because not being able to log into things because you don't have credentials or keys is definitely problematic.Tomas: Fully agree.Jason: You've adopted that during the POC process, or the extended POC process; do you continue that on with your regular infrastructure work continuing to test for reliability, or maybe any chaos engineering?Tomas: I actually measure something about what we are working on, like, what we have so far improved in terms of post-mortem process that's interesting. So, we started two-and-a-half year ago, and just two of us as infrastructure engineers. At the time, there was only one incident response on-call team, our first iteration within the infrastructure team was with migration from Heroku, where we ran all our services, to Amazon Web Services. And that time, we needed to also start thinking about, okay, the infrastructure team needs to be on call as well. So, that required to update in the process because until then, it works great; you have one team, people know each other, people know the whole stack. Suddenly, you are going to add new people, you're going to add new people a separate team, and that's going to change the way how on-call should be treated, and how the process should look like.You may ask why. You have understanding within the one team, you understand the expectations, but then you have suddenly different skill set of people, and they are going to be responsible for different part of the technical organization, so you need to align the expectation between two teams. And that was great because guys at Productboard are amazing, and they are always helpful. So, we sat down, we made first proposal of how new team is going to work like, what are going to be responsibilities. We took inspirations from the already existing on-call process, and we just updated it slightly.And we started to run with first test scenarios of being on call so we understand the process fully. Later on, it evolved to more complex process, but it's still very simple. What is more complex: we have more teams that's first thing being on call; we have better separation of all the alerts, so you're not going to route every alert to one team, but you are able to route it to every team that's responsible for its service; the team have also prepared a set of runbooks so anyone else can easily follow runbook and fix the incident pretty easily, and then we also added section about post-mortems, so what are our expectations of writing down post-mortem once incident is resolved.Jason: That's a great process of documenting, really—right—documenting the process so that everybody, whether they're on a different team and they're coming over or new hires, particularly, people that know nothing about your established practices can take that runbook and follow along, and achieve the same results that any other engineer would.Tomas: Yeah, I agree. And what was great to see that once my team grew—we are currently five and we started two—we saw excitement of the team members to update the process so everybody else we're going to join the on-call is going to be excited, is going to take it as an opportunity to learn more. So, we added disaster roleplay, and that section talks about you are new person joining on-call team, and we would like to make sure you are going to understand all the processes, all the necessary steps, and you are going to be aligned with all the expectations. But before you will actually going to have your first alerts of on-call, we would like to try to run roleplay. Imagine what a HashiCorp Vault cluster is going down; you should be the one resolving it. So, what are the first steps, et cetera?And that time you're going to realize whatever is being needs to be done, it's not only from a technical perspective, such as check our go to monitoring, check runbook, et cetera, but also communication-wise because you need to communicate not only with your shadowing buddy, but you also need to communicate internally, or to the customers. And that's going to change the perspective of how an incident should be handled.Jason: That disaster roleplay sounds really amazing. Can you chat a little bit more about the details of how that works? Particularly you mentioned engaging the non-technical side—right—of communication with various people. Does the disaster roleplay require coordinating with all those people, or is it just a mock, you would pretend to do, but you don't actually reach out to those people during this roleplay?Tomas: So, we would like to also combine the both aspects. We would like to make sure that person understands all the communication channels that are set within our organization, and what they are used for, and then we would like to make sure that that person understand how to involve other engineers within the organization. For instance, what was there the biggest difference is that you have plenty of options how to configure assigning or creating an alert. And so for those, you may have a different notification settings. And what happened is that some of the people have settings only for newly created alert, but when you made a change of assigned person of already existing alert, someone else, it might happen that that person didn't notice it because the notification setting was wrong. So, we encountered even these kind of issues and we were able to fix it, thanks to disaster roleplay. So, that was amazing to be found out.Jason: That's one of the favorite things that I like to do when we're using chaos engineering to do a similar thing to the disaster roleplay, is to really check those incident response processes, and validating those alerts is huge. There's so many times that I've found that we thought that someone would be alerted for some random thing, and turns out that nobody knew anything was going on. I love that you included that into your disaster roleplay process.Tomas: Yeah, it was also great experience for all the engineers involved. Unfortunately, we run it only within our team, but I hope we are going to have a chance to involve all other engineering on-call teams, so the onboarding experience to the engineering on-call teams is going to rise and is going to be amazing.Jason: So, one of the things that I'm really interested in is, you've gone from being a DevOps engineer, an SRE individual contributor role, and now you're leaving a small team. I think a lot of folks, as they look at their career, and I think more people are starting to become interested in this is, what does that progression look like? This is sort of a change of subject, but I'm interested in hearing your thoughts on what are the skills that you picked up and have used to become an effective technical leader within Productboard? What's some of that advice that our listeners, as individual contributors, can start to gain in order to advance where they're going with their own careers?Tomas: Firstly, it's important to understand what makes you passionate in your career, whether it's working with people, understanding their needs and their future, or you would like to be more on track as individual contributor and you would like to enlarge your scope of responsibilities towards leading more technical complex initiatives, that are going to take a long time to be implemented. In case all the infrastructure, or in case of the platform leaders, I would say the position of manager or technical leader also requires certain technical knowledge so you can be still in close touch with your team or with your most senior engineers, so you can set the goals and set the strategic clearly. But still, it's important to be, let's say, people person and be able to listen because in that case, people are going to be more open to you, and you can start helping them, and you can start making their dreams true and achievable.Jason: Making their dreams true. That's a great take on this idea because I feel like so many times, having done infrastructure work, that you start to get a mindset of maybe that people just are making demands of you, all the time. And it's sometimes hard to keep that perspective of working together as a team and really trying to excel to give them a platform that they can leverage to really get things done. We were talking about disaster roleplaying, and that naturally leads to a question that we like to ask of all of our guests and that's, do you have any horror stories from your career about an incident, some horror story or outage that you experienced and what you've learned from it?Tomas: I have one, and it actually happened at the beginning of my career of DevOps engineer. What is interesting here that it was one of the toughest incidents I experienced. It happened after midnight. So, the time I was still new to a company, and we have received an alert informing about too many 502, 504 errors written from API. At the time API process thousands of requests per second, and the incident had a huge impact on the services we were offering.And as I was shadowing my on-call buddy, I tried to check our main alerting channel, see what's happening, what's going on there, how can I help, and I started with checking monitoring system, reviewing all the reports from the engineers of being on-call, and I initiated the investigation on my own. I realized that something is wrong or something is not right, and I realized I was just confused and I want sleep, so it took me a while to get back on track. So, I made the side note, like, how can I start my brain to be working as during the day? And then I got back to the incident resolution process.So, it was really hard for me to start because I didn't know what [unintelligible 00:24:27] you knew about the channel, you knew about your engineers working on the resolution, but there were plenty of different communication funnels. Like, some of the engineers were deep-focused on their own investigation, and some of them were on call. And we needed to provide regular updates to the customers and internally as well. I had that inner feeling of let's share something, but I realized I just can't drop a random message because the message with all the information should have certain format and should have certain information. But I didn't know what kind of information should be there.So, I tried to ping someone, so, “Hey, can you share something?” And in the meantime, actually, more other people send me direct message. And I saw there are a lot of different tracks of people who tried to solve the incident, who tries to provide the status, but we were not aligned. So, this all showed me how important is to have proper communication funnel set. And we got the lucky to actually end up in one channel, we got lucky to resolve incident pretty quickly.And what else I learned that I would recommend to make sure you know where to work. I know it's pretty obvious sentence, but once your company has plenty of dashboards and you need to find one specific metric, sometime it looks like mission impossible.Jason: That's definitely a good lesson learned and feeds back to that disaster roleplays, practicing how you do those communications, understanding where things need to be communicated. You mentioned that it can be difficult to find a metric within a particular dashboard when you have so many. Do you have any advice for people on how to structure their dashboards, or name their dashboards, or organize them in a certain way to make that easier to find the metric or the information that you're looking for?Tomas: I will have a different approach, and that do have basic dashboard that provides you SLOs of all the services you have in the company. So, we understand firstly what service actually impacts the overall stability or reliability. So, that's my first advice. And then you should be able to either click on the specific service, and that should redirect you to it's dashboard, or you're going to have starred one of your favorite dashboards you have. So, I believe the most important is really have one main dashboard where you have all the services and their stability resourced, then you have option to look.Jason: Yeah, when you have one main dashboard, you're using that as basically the starting point, and from there, you can branch out and dive deeper, I guess, into each of the services.Tomas: Exactly, exactly true.Jason: I like that approach. And I think that a lot of modern dashboarding or monitoring systems now, the nice thing is that they have that ability, right, to go from one particular dashboard or graphic and have links out to the other information, or just click on the graph and it will show you the underlying host dashboard or node dashboard for that metric, which is really, really handy.Tomas: And I love the connection with other monitoring services, such as application monitoring. That gives you so much insight and when it's even connected with your work management tool is amazing so you can have all the important information in one place.Jason: Absolutely. So, oftentimes we talk about—what is it—the three pillars of observability, which I know some of our listeners may hate that, but the idea of having metrics and performance monitoring/APM and logs, and just how they all connect to each other can really help you solve a lot, or uncover a lot of information when you're in the middle of an incident. So Tomas, thanks for being on the show. I wanted to wrap up with one more question, and that's do you have any shoutouts, any plugs, anything that you want to share that our listeners should go take a look at?Tomas: Yeah, sure. So, as we are talking about management, I would like to promote one book that helped make my career, and that's Scaling Teams. It's written by Alexander Grosse and David Loftesness.And another one book is from Google, they have, like, three series, one of those is Seeking SRE, and I believe other parts are also useful to be read in case you would like to understand whether your organization needs SRE team and how to implement it within organization, and also, technically.Jason: Those are two great resources, and we'll have those linked in the show notes on the website. So, for anybody listening, you can find more information about those two books there. Tomas, thanks for joining us today. It's been a pleasure to have you.Tomas: Thanks. Bye.Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called “Battle of Pogs” by Komiku, and it's available on loyaltyfreakmusic.com.

In this episode, Jaspreet Sachdev discusses the Telecom Infra Project, his passion for open-source technology, and how TIP aims to bring connectivity to underserved communities around the world.

It's time we recognize that the way we build digital products is broken. That's because the products we use today represent the people who build them more than the people who use them. There is a digital divide between the experiences of people with disabilities and people who are able-bodied. Bridging this divide is about more than compliance or checking a box. Fable is helping companies practice accessibility at scale with the goal of operationalizing accessibility in the same way we already do for things like DesignOps and DevOps. Listen as Fable's CEO and Co-founder, Alwar Pillai, and Fable's Engineering Manager, Perry Trinier, talk about the importance of inclusive design, the need for digital accessibility and how to integrate accessibility into the development process. Join our Discord Community ►► discord.gg/devinterruptedOur Website ►► devinterrupted.com/Want to try LinearB?  Book a LinearB Demo and use the "Dev Interrupted Podcast" discount code.Have 60 seconds? Review the show on Apple Podcasts

In this episode, Gerhard is joined by Cyrille Le Clerc, Product Manager Lead on Observability at Elastic, and Oleg Nenashev, Principal Engineer at CloudBees. It all started with Oleg's tweet back in July, in which he was promoting Akihiro Kiuchi's work on Jenkins monitoring with OpenTelemetry. This was done in the context of Google's Summer of Code - a link to Akihiro's demo is in the show notes. As you may remember from episode 20, instrumenting our changelog.com pipeline is on Gerhard's mind, and this conversation helped him clarify a few things. If you are thinking of instrumenting your CI/CD pipeline with OpenTelemetry, this episode is for you.

Natalie and Mat explore hacking in Go from the eyes of 2 security researchers. Joakim Kennedy and JAGS have both used Go for hacking: writing malware, hardware hacking, reverse engineering Go code, and more.

Eric and Brandon look at Edge architecture, where it fits into a traditional infrastructure, what industries could take advantage of IT at the Edge, and what pitfalls this could bring. Destination Linux Network (https://destinationlinux.network) Sudo Show Website (https://sudo.show) Sponsor: Bitwarden (https://bitwarden.com/dln) Sponsor: Digital Ocean (https://do.co/dln) Sudo Show Swag (https://sudo.show/swag) Contact Us: DLN Discourse (https://sudo.show/discuss) Email Us! (mailto:contact@sudo.show) Sudo Matrix Room (https://sudo.show/matrix) Follow our Hosts: Brandon's Website (https://open-tech.net) Eric's Website (https://itguyeric.com) Red Hat Streaming (https://www.redhat.com/en/livestreaming) What Does the Future Hold for Edge Computing (https://builtin.com/cloud-computing/future-edge-computing) Challenge of Scaling the Intelligent Edge (https://thenewstack.io/the-challenge-of-scaling-the-intelligent-edge) Compiler 02: What Can Video Games Teach Us About Edge Computing (https://www.redhat.com/en/compiler-podcast/video-games-edge-computing) K3S (https://k3s.io) Tensor Flow (https://www.tensorflow.org/) Chapters 00:00 Intro 00:42 Welcome 01:49 Sponsor: Digital Ocean 03:12 Sponsor: Bitwarden 04:39 What is the Edge 10:44 Returning to On Prem? 19:27 Gaming and Other Use Cases 25:44 How to use the Edge 36:58 Wrap Up

About RichardHe's also an instructor at Pluralsight, a frequent public speaker, and the author of multiple books on software design and development. Richard maintains a regularly updated blog (seroter.com) on topics of architecture and solution design and can be found on Twitter as @rseroter. Links: Twitter: https://twitter.com/rseroter LinkedIn: https://www.linkedin.com/in/seroter Seroter.com: https://seroter.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: You know how git works right?Announcer: Sorta, kinda, not really Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Once upon a time back in the days of VH1, which was like MTV except it played music videos, would have a show that was, “Where are they now?” Looking at former celebrities. I will not use the term washed up because that's going to be insulting to my guest.Richard Seroter is a returning guest here on Screaming in the Cloud. We spoke to him a year ago when he was brand new in his role at Google as director of outbound product management. At that point, he basically had stars in his eyes and was aspirational around everything he wanted to achieve. And now it's a year later and he has clearly failed because it's Google. So, outbound products are clearly the things that they are going to be deprecating, and in the past year, I am unaware of a single Google Cloud product that has been outright deprecated. Richard, thank you for joining me, and what do you have to say for yourself?Richard: Yeah, “Where are they now?” I feel like I'm the Leif Garrett of cloud here, joining you. So yes, I'm still here, I'm still alive. A little grayer after twelve months in, but happy to be here chatting cloud, chatting whatever else with you.Corey: I joke a little bit about, “Oh, Google winds up killing things.” And let's be clear, your consumer division which, you know, Google is prone to that. And understanding a company's org chart is a challenge. A year or two ago, I was of the opinion that I didn't need to know anything about Google Cloud because it would probably be deprecated before I really had to know about it. My opinion has evolved considerably based upon a number of things I'm seeing from Google.Let's be clear here, I'm not saying this to shine you on or anything like that; it's instead that I've seen some interesting things coming out of Google that I consider to be the right moves. One example of that is publicly signing multiple ten-year deals with very large, serious institutions like Deutsche Bank, and others. Okay, you don't generally sign contracts with companies of that scale and intend not to live up to them. You're hiring Forrest Brazeal as your head of content for Google Cloud, which is not something you should do lightly, and not something that is a short-term play in any respect. And the customer experience has continued to improve; Google Cloud products have not gotten worse, and I'm seeing in my own customer conversations that discussions about Google Cloud have become significantly less dismissive than they were over the past year. Please go ahead and claim credit for all of that.Richard: Yeah. I mean, the changes a year ago when I joined. So, Thomas Kurian has made a huge impact on some of that. You saw us launch the enterprise APIs thing a while back, which was, “Hey, here's, for the most part, every one of our products that has a fixed API. We're not going to deprecate it without a year's notice, whatever it is. We're not going to make certain types of changes.” Maybe that feels like, “Well, you should have had that before.” All right, all we can do is improve things moving forward. So, I think that was a good change.Corey: Oh, I agree. I think that was a great thing to do. You had something like 80-some-odd percent coverage of Google Cloud services, and great, that's going to only increase with time, I can imagine. But I got a little pushback from a few Googlers for not being more congratulatory towards them for doing this, and look, it's a great thing. Don't get me wrong, but you don't exactly get a whole lot of bonus points and kudos and positive press coverage—not that I'm press—for doing the thing you should have been doing [laugh] all along.It's, “This is great. This is necessary.” And it demonstrates a clear awareness that there was—rightly or wrongly—a perception issue around the platform's longevity and that you've gone significantly out of your way to wind up addressing that in ways that go far beyond just yelling at people on Twitter they don't understand the true philosophy of Google Cloud, which is the right thing to do.Richard: Yeah, I mean, as you mentioned, look, the consumer side is very experimental in a lot of cases. I still mourn Google Reader. Like, those things don't matter—Corey: As do we all.Richard: Of course. So, I get that. Google Cloud—and of course we have the same cultural thing, but at the same time, there's a lifecycle management that's different in Google Cloud. We do not deprecate products that much. You know, enterprises make decade-long bets. I can't be swap—changing databases or just turning off messaging things. Instead, we're building a core set of things and making them better.So, I like the fact that we have a pretty stable portfolio that keeps getting a little bit bigger. Not crazy bigger; I like that we're not just throwing everything out there saying, “Rock on.” We have some opinions. But I think that's been a positive trend, customers seem to like that we're making these long-term bets. We're not going anywhere for a long time and our earnings quarter after quarter shows it—boy, this will actually be a profitable business pretty soon.Corey: Oh, yeah. People love to make hay, and by people, I stretch the term slightly and talk about, “Investment analysts say that Google Cloud is terrible because at your last annual report you're losing something like $5 billion a year on Google Cloud.” And everyone looked at me strangely, when I said, “No, this is terrific. What that means is that they're investing in the platform.” Because let's be clear, folks at Google tend to be intelligent, by and large, or at least intelligent enough that they're not going to start selling cloud services for less than it costs to run them.So yeah, it is clearly an investment in the platform and growth of it. The only way it should be turning a profit at this point is if there's no more room to invest that money back into growing the platform, given your market position. I think that's a terrific thing, and I'm not worried at all about it losing money. I don't think anyone should be.Richard: Yeah, I mean, strategically, look, this doesn't have to be the same type of moneymaker that even some other clouds have to be to their portfolio. Look, this is an important part, but you look at those ten-year deals that we've been signing: when you look at Univision, that's a YouTube partnership; you look at Ford that had to do with Android Auto; you look at these others, this is where us being also a consumer and enterprise SaaS company is interesting because this isn't just who's cranking out the best IaaS. I mean, that can be boring stuff over time. It's like, who's actually doing the stuff that maybe makes a traditional company more interesting because they partner on some of those SaaS services. So, those are the sorts of deals and those sorts of arrangements where cloud needs to be awesome, and successful, and make money, doesn't need to be the biggest revenue generator for Google.Corey: So, when we first started talking, you were newly minted as a director of outbound product management. And now, you are not the only one, there are apparently 60 of you there, and I'm no closer to understanding what the role encompasses. What is your remit? Where do you start? Where do you stop?Richard: Yeah, that's a good question. So, there's outbound product management teams, mostly associated with the portfolio area. So network, storage, AI, analytics, database, compute, application modernization-y sort of stuff—which is what I cover—containers, dev tools, serverless. Basically, I am helping make sure the market understands the product and the product understands the market. And not to be totally glib, but a lot of that is, we are amplification.I'm amplifying product out to market, analysts, field people, partners: “Do you understand this thing? Can I help you put this in context?” But then really importantly, I'm trying to help make sure we're also amplifying the market back to our product teams. You're getting real customer feedback: “Do you know what that analyst thinks? Have you heard what happened in the competitive space?”And so sometimes companies seem to miss that, and PMs poke their head up when I'm about to plan a product or I'm about to launch a product because I need some feedback. But keeping that constant pulse on the market, on customers, on what's going on, I think that can be a secret weapon. I'm not sure everybody does that.Corey: Spending as much time as I do on bills, admittedly AWS bills, but this is a pattern that tends to unfold across every provider I've seen. The keynotes are chock-full of awesome managed service announcements, things that are effectively turnkey at further up the stack levels, but the bills invariably look a lot more like, yeah, we spend a bit of money on that and then we run 10,000 virtual instances in a particular environment and we just treat it like it's an extension of our data center. And that's not exciting; that's not fun, quote-unquote, but it's absolutely what customers are doing and I'm not going to sit here and tell them that they're wrong for doing it. That is the hallmark of a terrible consultant of, “I don't understand why you're doing what you're doing, so it must be foolish.” How about you stop and gain some context into why customers do the things that they do?Richard: No, I send around a goofy newsletter every week to a thousand or two people, just on things I'm learning from the field, from customers, trying to make sure we're just thinking bigger. A couple of weeks ago, I wrote an idea about modernization is awesome, and I love when people upgrade their software. By the way, most people migration is a heck of a lot easier than if I can just get this into your cloud, yeah love that; that's not the most interesting thing, to move VMs around, but most people in their budget, don't have time to rewrite every Java app to go. Everybody's not changing .NET framework to .NET core.Like, who do I think everybody is? No, I just need to try to get some incremental value first. Yes, then hopefully I'll swap out my self-managed SQL database for a Spanner or a managed service. Of course, I want all of that, but this idea that I can turn my line of business loan processing app into a thousand functions overnight is goofy. So, how are we instead thinking more pragmatically about migration, and then modernizing some of it? But even that sort of mindset, look, Google thinks about innovation modernization first. So, also just trying to help us take a step back and go, “Gosh, what is the normal path? Well, it's a lot of migration first, some modernization, and then there's some steady-state work there.”Corey: One of the things that surprised me the most about Google Cloud in the market, across the board, has been the enthusiastic uptake for enterprise workloads. And by enterprise workloads, I'm talking about things like SAP HANA is doing a whole bunch of deployments there; we're talking Big Iron-style enterprise-y things that, let's be honest, countervene most of the philosophy that Google has always held and espoused publicly, at least on conference stages, about how software should be built. And I thought that would cut against them and make it very difficult for you folks to gain headway in that market and I could not have been more wrong. I'm talking to large enterprises who are enthusiastically talking about Google Cloud. I've got a level with you, compared to a year or two ago, I don't recognize the place.Richard: Mmm. I mean, some of that, honestly, in the conversations I have, and whatever I do a handful of customer calls every week, I think folks still want something familiar, but you're looking for maybe a further step on some of it. And that means, like, yes, is everybody going to offer VMs? Yeah, of course. Is everyone going to have MySQL? Obviously.But if I'm an enterprise and I'm doing these generational bets, can I cheat a little bit, and maybe if I partner with a more of an innovation partner versus maybe just the easy next step, am I buying some more relevance for the long-term? So, am I getting into environment that has some really cool native zero-trust stuff? Am I getting into environment with global backend services and I'm not just stitching together a bunch of regional stuff? How can I cheat by using a more innovation vendor versus just lifting and shifting to what feels like hosted software in another cloud? I'm seeing more of that because these migrations are tough; nobody should be just randomly switching clouds. That's insane.So, can I make, maybe, one of these big bets with somebody who feels like they might actually even improve my business as a whole because I can work with Google Pay and improve how I do mobile payments, or I could do something here with Android? Or, heck, all my developers are using Angular and Flutter; aren't I going to get some benefit from working with Google? So, we're seeing that, kind of, add-on effect of, “Maybe this is a place not just to host my VMs, but to take a generational leap.”Corey: And I think that you're positioning yourselves in a way to do it. Again, talk about things that you wouldn't have expected to come out of Google of all places, but your console experience has been first-rate and has been for a while. The developer experience is awesome; I don't need to learn the intricacies of 12 different services for what I'm trying to do just in order to get something basic up and running. I can stop all the random little billing things in my experimental project with a single click, which that admittedly has a confirm, which you kind of want. But it lets you reason about these things.It lets you get started building something, and there's a consistency and cohesiveness to the console that, again, I am not a graphic designer, by any stretch of the imagination. My most commonly used user interface is a green-screen shell prompt, and then I'm using Vim to wind up writing something horrifying, ideally in Python, but more often in YAML. And that has been my experience, but just clicking around the console, it's clear that there was significant thought put into the design, the user experience, and the way of approaching folks who are starting to look very different, from a user persona perspective.Richard: I can—I mean, I love our user research team; they're actually fun to hang out with and watch what they do, but you have to remember, Google as a company, I don't know, cloud is the first thing we had to sell. Did have to sell Gmail. I remember 15 years ago, people were waiting for invites. And who buys Maps or who buys YouTube? For the most part, we've had to build things that were naturally interesting and easy-to-use because otherwise, you would just switch to anything else because everything was free.So, some of that does infuse Google Cloud, “Let's just make this really easy to use. And let's just make sure that, maybe, you don't hate yourself when you're done jumping into a shell from the middle of the console.” It's like, that should be really easy to do—or upgrade a database, or make changes to things. So, I think some of the things we've learned from the consumer good side, have made their way to how we think of UX and design because maybe this stuff shouldn't be terrible.Corey: There's a trope going around, where I wound up talking about the next million cloud customers. And I'm going to have to write a sequel to it because it turns out that I've made a fundamental error, in that I've accepted the narrative that all of the large cloud vendors are pushing, to the point where I heard from so many folks I just accepted it unthinkingly and uncritically, and that's not what I should be doing. And we'll get to what I was wrong about in a minute, but the thinking goes that the next big growth area is large enterprises, specifically around corporate IT. And those are folks who are used to managing things in a GUI environment—which is fine—and clicking around in web apps. Now, it's easy to sit here on our high horse and say, “Oh, you should learn to write code,” or YAML, which is basically code. Cool.As an individual, I agree, someone should because as soon as they do that, they are now able to go out and take that skill to a more lucrative role. The company then has to backfill someone into the role that they just got promoted out of, and the company still has that dependency. And you cannot succeed in that market with a philosophy of, “Oh, you built something in the console. Now, throw it away and do it right.” Because that is maddening to that user persona. Rightfully so.I'm not that user persona and I find it maddening when I have to keep tripping over that particular thing. How did that come to be, from your perspective? First, do you think that is where the next million cloud customers come from? And have I adequately captured that user persona, or am I completely often the weeds somewhere?Richard: I mean, I shared your post internally when that one came out because that resonated with me of how we were thinking about it. Again, it's easy to think about the cloud-native operators, it's Spotify doing something amazing, or this team at Twitter doing something, or whatever. And it's not even to be disparaging. Like, look, I spent five years in enterprise IT and I was surrounded by operators who had to run dozen different systems; they weren't dedicated to just this thing or that. So, what are the tools that make my life easy?A lot of software just comes with UIs for quick install and upgrades, and how does that logic translate to this cloud world? I think that stuff does matter. How are you meeting these people a little better where they are? I think the hard part that we will always have in every cloud provider is—I think you've said this in different forums, but how do I not sometimes rub the data center on my cloud or vice versa? I also don't want to change the experience so much where I degrade it over the long term, I've actually somehow done something worse.So, can I meet those people where they are? Can we pull some of those experiences in, but not accidentally do something that kind of messes up the cloud experience? I mean, that's a fine line to walk. Does that make sense to you? Do you see where there's a… I don't know, you could accidentally cater to a certain audience too much, and change the experience for the worse?Corey: Yes, and no. My philosophy on it is that you have to meet customers where they are, but only to a point. At some point, what they're asking for becomes actively harmful or disadvantageous to wind up providing for them. “I want you to run my data center for me,” is on some level what some cloud environments look like, and I'm not going to sit here and tell people they're inherently wrong for that. Their big reason for moving to the cloud was because they keep screwing up replacing failed hard drives in their data center, so we're going to put it in the cloud.Is it more expensive that way? Well, sure in terms of actual cash outlay, it almost certainly is, but they're also not going down every month when a drive fails, so once the value of that? It's a capability story. That becomes interesting to me, and I think that trying to sit here in isolation, and say that, “Oh, this application is not how we would build it at Google.” And it's, “Yeah, you're Google. They are insert an entire universe of different industries that look nothing whatsoever like Google.” The constraints are different, the resources are different, and—Richard: Sure.Corey: —their approach to problem-solving are different. When you built out Google, and even when you're building out Google Cloud, look at some of the oldest craftiest stuff you have in your entire all of Google environment, and then remember that there are companies out there that are hundreds of years old. It's a different order of magnitude as far as era, as far as understanding of what's in the environment, and that's okay. It's a very broad and very diverse world.Richard: Yeah. I mean, that's, again, why I've been thinking more about migration than even some of the modernization piece. Should you bring your network architecture from on-prem to the cloud? I mean, I think most cases, no. But I understand sometimes that edge firewall, internal trust model you had on-prem, okay, trying to replicate that.So, yeah, like you say, I want to meet people where they are. Can we at least find some strategic leverage points to upgrade aspects of things as you get to a cloud, to save you from yourself in some places because all of a sudden, you have ten regions and you only had one data center before. So, many more rooms for mistakes. Where are the right guardrails? We're probably more opinionated than others at Google Cloud.I don't really apologize for that completely, but I understand. I mean, I think we've loosened up a lot more than maybe people [laugh] would have thought a few years ago, from being hyper-opinionated on how you run software.Corey: I will actually push back a bit on the idea that you should not replicate your on-premises data center in your cloud environment. Sure, are there more optimal ways to do it that are arguably more secure? Absolutely. But a common failure mode in moving from data center to cloud is, “All right, we're going to start embracing this entirely new cloud networking paradigm.” And it is confusing, and your team that knows how the data center network works really well are suddenly in way over their heads, and they're inadvertently exposing things they don't intend to or causing issues.The hard part is always people, not technology. So, when I glance at an environment and see things like that, perfect example, are there more optimal ways to do it? Oh, from a technology perspective, absolutely. How many engineers are working on that? What's their skill set? What's their position on all this? What else are they working on? Because you're never going to find a team of folks who are world-class experts in every cloud? It doesn't work that way.Richard: No doubt. No doubt, you're right. There's areas where we have to at least have something that's going to look similar, let you replicate aspects of it. I think it's—it'll just be interesting to watch, and I have enough conversations with customers who do ask, “Hey, where are the places we should make certain changes as we evolve?” And maybe they are tactical, and they're not going to be the big strategic redesign their entire thing. But it is good to see people not just trying to shovel everything from one place to the next.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: Now, to follow up on what I was saying earlier, what I think I've gotten wrong by accepting the industry talking points on is that the next million cloud customers are big enterprises moving from data centers into the cloud. There's money there, don't get me wrong, but there is a larger opportunity in empowering the creation of companies in your environment. And this is what certain large competitors of yours get very wrong, where it's we're going to launch a whole bunch of different services that you get to build yourself from popsicle sticks. Great. That is not useful.But companies that are trying to do interesting things, or people who want to found companies to do interesting things, want something that looks a lot more turnkey. If you are going to be building cloud offerings, that for example, are terrific building blocks for SaaS companies, then it behooves you to do actual investments, rather than just a generic credit offer, into spurring the creation of those types of companies. If you want to build a company that does payroll systems, in a SaaS, cloud way, “Partner with us. Do it here. We will give you a bunch of credits. We will introduce you to your first ten prospective customers.”And effectively actually invest in a company success, as opposed to pitch-deck invest, which is, “Yeah, we'll give you some discounting and some credits, and that's our quote-unquote, ‘investment.'” actually be there with them as a partner. And that's going to take years for folks to wrap their heads around, but I feel like that is the opportunity that is significantly larger, even than the embedded existing IT space because rather than fighting each other for slices of the pie, I'm much more interested in expanding that pie overall. One of my favorite questions to get asked because I think it is so profoundly missing the point is, “Do you think it's possible for Google to go from number three to number two,” or whatever the number happens to be at some point, and my honest, considered answer is, “Who gives a shit?” Because number three, or number five, or number twelve—it doesn't matter to me—is still how many hundreds of billions of dollars in the fullness of time. Let's be real for a minute here; the total addressable market is expanding faster than any cloud or clouds are going to be able to capture all of.Richard: Yeah. Hey, look, whoever who'll be more profitable solving user problems, I really don't care about the final revenue number. I can be the number one cloud tomorrow by making Google Cloud free. What's the point? That's not a sustainable business. So, if you're just going for who can deploy the most VCPUs or who can deploy the most whatever, there's ways to game that. I want to make sure we are just uniquely solving problems better than anybody else.Corey: Sorry, forgive me. I just sort of zoned out for a second there because I'm just so taken aback and shocked by the idea of someone working at a large cloud provider who expresses a philosophy that isn't lying awake at night fretting over the possibility of someone who isn't them as making money somewhere.Richard: [laugh]. I mean, your idea there, it'll be interesting to watch, kind of, the maker's approach of are you enabling that next round of startups, the next round of people who want to take—I mean, honestly, I like the things we're doing building block-wise, even with our AI: we're not just handing you a vision API, we're giving you a loan processing AI that can process certain types of docs, that more packaged version of AI. Same with healthcare, same with whatever. I can imagine certain startups or a company idea going, “Hey, maybe I could disrupt or serve a new market.”I always love what Square did. They've disrupted emerging markets, small merchants here in North America, wherever, where I didn't need a big expensive point of sale system. You just gave me the nice, right building blocks to disrupt and run my business. Maybe Google Cloud can continue to provide better building blocks, but I do like your idea of actually investment zones, getting part of this. Maybe the next million users are founders and it's not just getting into some of these companies with, frankly, 10, 20, 30,000 people in IT.I think there's still plenty of room in these big enterprises to unlock many more of those companies, much more of their business. But to your point, there's a giant market here that we're not all grabbing yet. For crying out loud, there's tons of opportunity out here. This is not zero-sum.Corey: Take it a step further beyond that, and today, if you have someone who's enterprising, early on in their career, maybe they just got out of school, maybe they have just left their job and are ready to snap, or they have some severance money that they want to throw into something. Great. What do they want to do if they have an idea for a company? Well today, that answer looks a lot like, well, time to go to a boot camp and learn to code for six months so you can build a badly done MVP well enough to get off the ground and get some outside investment, and then go from there. Well, what if we cut that part out entirely?What if there were building blocks of I don't need to know or care that there's a database behind it, or what a database looks like. Picture Visual Basic in a web browser for building apps, and just take this bit of information I give you and store it and give it back to me later. Sure, you're going to have some significant challenges in the architecture or something like that as it goes from this thing that I'm talking about as an MVP to something planet-scale—like a Spotify for example—but that's not most businesses, and that's okay. Get out of the way and let people innovate and iterate on what it is they're doing more rapidly, and make it more accessible to teach people. That becomes huge; that gets the infrastructure bits that cloud providers excel at out of the way, and all it really takes is packaging those things into a golden path of what a given company of a particular profile should be doing, if—unless they have reason to deviate from it—and instead of having this giant paradox of choice issue, it's, “Oh, okay, I'll drag-drop, build things accordingly.”And under the hood, it's doing all the configuration of services and that's great. But suddenly, you've made being a founder of a software company—fundamentally—accessible to people who are not themselves software engineers. And I know that's anathema to some people, and I don't even slightly care because I am done with gatekeeping.Richard: Yeah. No, it's exciting if that can pull off. I mean, it's not the years ago where, how much capital was required to find the rack and do all sorts of things with tech, and hire some developers. And it's an amazing time to be software creators, now. The more we can enable that—yeah, I'm along for that journey, sign me up.Corey: I'm looking forward to seeing how it winds up shaking out. So, I want to talk a little bit about the paradox of choice problem that I just mentioned. If you take a look at the various compute services that every cloud provider offers, there are an awful lot of different choices as far as what you can run. There's the VM model, there's containers—if you're in AWS, you have 17 ways to run those—and you wind up—any of the serverless function story, and other things here and there, and managed services, I mean and honestly, Google has a lot of them, nowhere near as many as you do failed messaging products, but still, an awful lot of compute options. How do customers decide?What is the decision criteria that you see? Because the worst answer you can give someone who doesn't really know what they're doing is, “It depends,” because people don't know how to make that decision. It's, “What factors should I consider then, while making that decision?” And the answer has to be something somewhat authoritative because otherwise, they're going to go on the internet and get yelled at by everyone because no one is ever going to agree on this, except that everyone else is wrong.Richard: Mm-hm. Yeah, I mean, on one hand, look, I like that we intentionally have fewer choices than others because I don't think you need 17 ways to run a container. I think that's excessive. I think more than five is probably excessive because as a customer, what is the trade-off? Now, I would argue first off, I don't care if you have a lot of options as a vendor, but boy, the backends of those better be consistent.Meaning if I have a CI/CD tool in my portfolio and it only writes to two of them, shame on me. Then I should make sure that at least CI/CD, identity management, log management, monitoring, arguably your compute runtime should be a late-binding choice. And maybe that's blasphemous because somebody says, “I want to start up front knowing it's a function,” or, “I want to start it's a VM.” How about, as a developer, I couldn't care less. How about I just build cool software and maybe even at deploy time, I say, “This better fits in running in Kubernetes.” “This is better in a virtual machine.”And my cost of changing that later is meaningless because, hey, if it is in the container, I can switch it between three or four different runtimes, the identity management the same, it logs the exact same way, I can deploy CI/CD the same way. So, first off, if those things aren't the same, then the vendor is messing up. So, the customer shouldn't have to pay the cost of that. And then there gets to be other actual criteria. Look, I think you are looking at the workload itself, the team who makes it, and the strategy to figure out the runtime.It's easy for us. Google Compute Engine for VMs, containers go in GKE, managed services that need some containers, there are some apps around them, are Cloud Functions and Cloud Run. Like, it's fairly straightforward and it's going to be an OR situation—or an AND situation not an OR, which is great. But we're at least saying the premium way to run containers in Google Cloud for systems is GKE. There you go. If you do have a bunch of managed services in your architecture and you're stitching them together, then you want more serverless things like Cloud Run and Cloud Functions. And if you want to just really move some existing workload, GCE is your best choice. I like that that's fairly straightforward. There's still going to be some it depends, but it feels better than nine ways to run Kubernetes engines.Corey: I'm sure we'll see them in the fullness of time.Richard: [laugh].Corey: So, talk about Anthos a bit. That was a thing that was announced a while back and it was extraordinarily unclear what it was. And then I looked at the pricing and it was $10,000 a month with a one-year minimum commitment, and is like, “Oh, it's not for me. That's why I don't get it.” And I haven't really looked back at it since. But it is something else now. It almost feels like a wrapper brand, in some respects. How's it going? [unintelligible 00:29:26]?Richard: Yeah. Consumption, we'll talk more upcoming months on some of the adoption, but we're finally getting the hockey stick, which always comes delayed with platforms because nobody adopts platforms quickly. They buy the platform and a year later they start to actually build new development, migrate the things they have. So, we're starting to see the sort of growth. But back to your first point. And I even think I poorly tried to explain it a year ago with you. Basically, look, Anthos is the ability to manage fleets of GKE clusters, wherever they are. I don't care if they're on-prem, I don't care if they're in Google Cloud, I don't care if they're Amazon. We have one customer who only uses Anthos on AWS. Awesome, rock on.So, how do I put GKE clusters everywhere, but then do fleet management because look, some people are doing an app per cluster. They don't want to jam 50 apps in the cluster from different teams because they don't like the idea that this app requires root access; now you can screw around with mine. Or, you didn't update; that broke the cluster. I don't want any of that. So, you're going to see companies more, doing even app per cluster, app per developer per cluster.So, now I have a fleet problem. How do I keep it in sync? How do I make sure policy is consistent? Those sorts of things. So, Anthos is kind of solving the fleet management challenge and replacing people's first-gen app platform.Seeing a lot of those use cases, “Hey, we're retiring our first version of Docker Enterprise, Mesos, Cloud Foundry, even OpenShift,” saying, “All right, now's the time for our next version of our app platform. How about GKE, plus Cloud Run on top of it, plus other stuff?” Sounds good. So, going well is a, sort of—as you mentioned, there's a brand story here, mainly because we've also done two things that probably matter to you. A, we changed the price a lot.No minimum commit, remarkably at 20% of the cost it was when we launched, on purpose because we've gotten better at this. So, much cheaper, no minimum commit, pay as you go. Be on-premises, on bare metal with GKE. Pay by the hour, I don't care; sounds great. So, you can do that sort of stuff.But then more importantly, if you're a GKE customer and you just want config management, service mesh, things like that, now you can buy all of those independently as well. And Anthos is really the brand for fleet management of GKE. And if you're on Google Cloud only, it adds value. If you're off Google Cloud, if you're multi-cloud, I don't care. But I want to manage fleets of compute clusters and create them. We're going to keep doubling down on that.Corey: The big problem historically for understanding a lot of the adoption paradigm of Kubernetes has been that it was, to some extent, a reimagining of how Google ran and built software internally. And I thought at the time, the idea was—from a cynical perspective—that, “All right, well, your crappy apps don't run well on Google-style infrastructure so we're going to teach the entire world how to write software the way that we do.” And then you end up with people running their blog on top of Kubernetes, where it's one of those, like, the first blog post is, like, “How I spent the last 18 months building Kubernetes.” And, okay, that is certainly a philosophy and an approach, but it's almost approaching Windows 95 launch level of hype, where people who didn't own computers were buying copies of it, on some level. And I see the term come up in conversations in places where it absolutely has no place being brought up. “How do I run a Kubernetes cluster inside of my laptop?” And, “It's what you got going on in there, buddy?”Richard: [laugh].Corey: “What do you think you're trying to do here because you just said something that means something that I think is radically different to me than it is to you.” And again, I'm not here to judge other people's workflows; they're all terrible, except for mine, which is an opinion held by everyone about their own workflow. But understanding where people are, figuring out how to get there, how to meet customers where they are and empower them. And despite how heavily Google has been into the Kubernetes universe since its inception, you're very welcoming to companies—and loud-mouth individuals on Twitter—who have no use for Kubernetes. And working through various products you offer, I don't ever feel like a second-class citizen. There's really something impressive about that, of not letting the hype dictate the product and marketing decisions of it.Richard: Yeah, look, I think I tweeted it recently, I think the future of software is managed services with containers in the gap, for the most part. Whereas—if you can use managed services, please do. Use them wherever you can. And if you have to sling some code, maybe put it in a really portable thing that's really easy to run in lots of places. So, I think that's smart.But for us, look, I think we have the best container workflow from dev tools, and build tools, and artifact registries, and runtimes, but plenty of people are running containers, and you shouldn't be running Kubernetes all over the place. That makes sense for the workload, I think it's better than a VM at the retail edge. Can I run a small cluster, instead of a weird point-of-sale Windows app? Maybe. Maybe it makes sense to have a lightweight Kubernetes cluster there for consistency purposes.So, for me, I think it's a great medium for a subset of software. Google Cloud is going to take whatever you got, which is great. I think containers are great, but at the same time, I'm happily going to let you deploy a function that responds to you adding a storage item to a bucket, where at the same time give you a SaaS service that replaces the need for any code. All of those are terrific. So yeah, we love Kubernetes. We think it's great. We're going to be the best version to run it. But that's not going to be your whole universe.Corey: No, and I would argue it absolutely shouldn't be.Richard: [laugh]. Right. Agreed. Now again, for some companies, it's a great replacement for this giant fleet of VMs that all runs at eight percent utilization. Can I stick this into a bunch of high-density clusters? Absolutely you should. You're going to save an absolute fortune doing that and probably pick up some resilience and functionality benefits.But to your point, “Do I want to run a WordPress site in there?” I don't know, probably not. “Do I need to run my own MySQL?” I'd prefer you not do that. So, in a lot of cases, don't use it unless you have to. That should go for all compute nowadays. Use managed services.Corey: I'm a big believer in going down that approach just because it is so much easier than trying to build it yourself from popsicle sticks because you theoretically might have to move it someday in the future, even though you're not.Richard: [laugh]. Right.Corey: And it lets me feel better about a thing that isn't going to be used by anything that I'm doing in the near future. I just don't pretend to get it.Richard: No, I don't install a general purpose electric charger in my garage for any electric car I may get in the future; I charge for the one I have now. I just want it to work for my car; I don't want to plan for some mythical future. So yeah, premature optimization over architecture, or death in IT, especially nowadays where speed matters, don't waste your time building something that can run in nine clouds.Corey: Richard, I want to thank you for coming on again a year later to suffer my slings, arrows, and other various implements of misfortune. If people want to learn more about what you're doing, how you're doing it, possibly to pull a Forrest Brazeal and go work with you, where can they find you?Richard: Yeah, we're a fun place to work. So, you can find me on Twitter at @rseroter—R-S-E-R-O-T-E-R—hang out on LinkedIn, annoy me on my blog seroter.com as I try to at least explore our tech from time to time and mess around with it. But this is a fun place to work. There's a lot of good stuff going on here, and if you work somewhere else, too, we can still be friends.Corey: Thank you so much for your time today. Richard Seroter, director of outbound product management at Google. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment into which you have somehow managed to shove a running container.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Justin Searls helped start Test Double—a software agency of experienced developers who work with clients to build great software together, as a team. He gave Brittany the scoop on Mocktail, an opinionated alternative to minitest's mocks, rr, mocha, and rspec-mocks. As a bonus: he offers his insights and advice on being a confident speaker. Show Notes & Links: testdouble/mocktail (https://github.com/testdouble/mocktail) Test Double | An Agency Improving the World's Software (https://testdouble.com/) Mockito framework site (https://site.mockito.org/) Jim Weirich's Tweet (https://twitter.com/jimweirich/status/77124604602228737) searls/gimme (https://github.com/searls/gimme) jimweirich/flexmock (https://github.com/jimweirich/flexmock) Please don't mock me - Justin Searls - JSConf US 2018 (https://www.youtube.com/watch?v=x8sKpJwq6lY) VCR - GitHub (https://github.com/vcr/vcr) How to Make a Gem of a Gem | Rubyconf 2021 (https://rubyconf.org/program/sessions#session-1203) Justin Searls (@searls) · Twitter (https://twitter.com/searls) Sponsored By: Honeybadger (https://www.honeybadger.io/) Honeybadger makes you a DevOps hero by combining error monitoring, uptime monitoring and check-in monitoring into a single, easy to use platform. Go to Honeybadger.io (https://www.honeybadger.io/) and discover how Starr, Josh, and Ben created a 100% bootstrapped monitoring solution. Scout APM (http://scoutapm.com/rubyonrails) Try their error monitoring and APM free for 14-days, no credit card needed! And as an added bonus for Ruby on Rails listeners: Scout will donate $5 to the open-source project of your choice when you deploy. Learn more at http://scoutapm.com/rubyonrails (http://scoutapm.com/rubyonrails).

About Micheal Micheal Benedict leads Engineering Productivity at Pinterest. He and his team focus on developer experience, building tools and platforms for over a thousand engineers to effectively code, build, deploy and operate workloads on the cloud. Mr. Benedict has also built Infrastructure and Cloud Governance programs at Pinterest and previously, at Twitter -- focussed on managing cloud vendor relationships, infrastructure budget management, cloud migration, capacity forecasting and planning and cloud cost attribution (chargeback). Links: Pinterest: https://www.pinterest.com Teletraan: https://github.com/pinterest/teletraan Twitter: https://twitter.com/micheal Pinterestcareers.com: https://pinterestcareers.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You know how git works right?Announcer: Sorta, kinda, not really. Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while, I like to talk to people who work at very large companies that are not in fact themselves a cloud provider. I know it sounds ridiculous. How can you possibly be a big company and not make money by selling managed NAT gateways to an unsuspecting public? But I'm told it can be done here to answer that question. And hopefully at least one other is Pinterest. It's head of engineering productivity, Micheal Benedict. Micheal, thank you for taking the time to join me today.Micheal: Hi, Corey, thank you for inviting me today. I'm really excited to talk to you.Corey: So, exciting times at Pinterest in a bunch of different ways. It was recently reported—which of course, went right to the top of my inbox as 500,000 people on Twitter all said, “Hey, this sounds like a ‘Corey would be interested in it' thing.” It was announced that you folks had signed a $3.2 billion commitment with AWS stretching until 2028. Now, if this is like any other large-scale AWS contract commitment deal that has been made public, you were probably immediately inundated with a whole bunch of people who are very good at arithmetic and not very good at business context saying, “$3.2 billion? You could build massive data centers for that. Why would anyone do this?” And it's tiresome, and that's the world in which we live. But I'm guessing you heard at least a little bit of that from the peanut gallery.Micheal: I did, and I always find it interesting when direct comparisons are made with the total amount that's been committed. And like you said, there's so many nuances that go into how to perceive that amount, and put it in context of, obviously, what Pinterest does. So, I at least want to take this opportunity to share with everyone that Pinterest has been on the cloud since day one. When Ben initially started the company, that product was launched—it was a simple Django app—it was launched on AWS from day one, and since then, it has grown to support 450-plus million MAUs over the course of the decade.And our infrastructure has grown pretty complex. We started with a bunch of EC2 machines and persisting data in S3, and since then we have explored an array of different products, in fact, sometimes working very closely with AWS, as well and helping them put together a product roadmap for some of the items they're working on as well. So, we have an amazing partnership with them, and part of the commitment and how we want to see these numbers is how does it unlock value for Pinterest as a business over time in terms of making us much more agile, without thinking about the nuances of the infrastructure itself. And that's, I think, one of the best ways to really put this into context, that it's not a single number we pay at the end [laugh] of the month, but rather, we are on track to spending a certain amount over a period of time, so this just keeps accruing or adding to that number. And we basically come out with an amazing partnership in AWS, where we have that commitment and we're able to leverage their products and full suite of items without any hiccups.Corey: The most interesting part of what you said is the word partner. And I think that's the piece that gets lost an awful lot when we talk about large-scale cloud negotiations. It's not like buying a car, where you can basically beat the crap out of the salesperson, you can act as if $400 price difference on a car is the difference between storm out of the dealership and sign the contract. Great, you don't really have to deal with that person ever again.In the context of a cloud provider, they run your production infrastructure, and if they have a bad day, I promise you're going to have a bad day, too. You want to handle those negotiations in a way that is respectful of that because they are your partner, whether you want them to be or not. Now, I'm not suggesting that any cloud provider is going to hold an awkward negotiation against the customer, but at the same time, there are going to be scenarios in which you're going to want to have strong relationships, where you're going to need to cash in political capital to some extent, and personally, I've never seen stupendous value in trying to beat the crap out of a company in order to get another tenth of a percent discount on a service you barely use, just because someone decided that well, we didn't do well in the last negotiation so we're going to get them back this time.That's great. What are you actually planning to do as a company? Where are you going? And the fact that you just alluded to, that you're not just a pile of S3 and EC2 instances speaks, in many ways, to that. By moving into the differentiated service world, suddenly you're able to do things that don't look quite as much like building a better database and start looking a lot more like servicing your users more effectively and well.Micheal: And I think, like you said, I feel like there's like a general skepticism in viewing that the cloud providers are usually out there to rip you apart. But in reality, that's not true. To your point, as part of the partnership, especially with AWS and Pinterest, we've got an amazing relationship going on, and behind the scenes, there's a dedicated team at Pinterest, called the Infrastructure Governance Team, a cross-functional team with folks from finance, legal, engineering, product, all sitting together and working with our AWS partners—even the AWS account managers at the times are part of that—to help us make both Pinterest successful, and in turn, AWS gets that amazing customer to work with in helping build some of their newer products as well. And that's one of the most important things we have learned over time is that there's two parts to it; when you want to help improve your business agility, you want to focus not just on the bottom line numbers as they are. It's okay to pay a premium because it offsets the people capital you would have to invest in getting there.And that's a very tricky way to look at math, but that's what these teams do; they sit down and work through those specifics. And for what it's worth, in our conversations, the AWS teams always come back with giving us very insightful data on how we're using their systems to help us better think about how we should be pricing or looking things ahead. And I'm not the expert on this; like I said, there's a dedicated team sitting behind this and looking through and working through these deals, but that's one of the important takeaways I hope the users—or the listeners of this podcast then take away that you want to treat your cloud provider as your partner as much as possible. They're not always there to screw you. That's not their goal. And I apologize for using that term. It is important that you set that expectations that it's in their best interest to actually make you successful because that's how they make money as well.Corey: It's a long-term play. I mean, they could gouge you this quarter, and then you're trying to evacuate as fast as possible. Well, they had a great quarter, but what's their long-term prospect? There are two competing philosophies in the world of business; you can either make a lot of money quickly, or you can make a little bit of money and build it over time in a sustained way. And it's clear the cloud providers are playing the long game on this because they basically have to.Micheal: I mean, it's inevitable at this point. I mean, look at Pinterest. It is one of those success stories. Starting as a Django app on a bunch of EC2 machines to wherever we are right now with having a three-plus billion dollar commitment over a span of couple of years, and we do spend a pretty significant chunk of that on a yearly basis. So, in this case, I'm sure it was a great successful partnership.And I'm hoping some of the newer companies who are building the cloud from the get-go are thinking about it from that perspective. And one of the things I do want to call out, Corey, is that we did initially start with using the primitive services in AWS, but it became clear over time—and I'm sure you heard of the term multi-cloud and many of that—you know, when companies start evaluating how to make the most out of the deals they're negotiating or signing, it is important to acknowledge that the cost of any of those evaluations or even thinking about migrations never tends to get factored in. And we always tend to treat that as being extremely simple or not, but those are engineering resources you want to be spending more building on the product rather than these crazy costly migrations. So, it's in your best interest probably to start using the most from your cloud provider, and also look for opportunities to use other cloud providers—if they provide more value in certain product offerings—rather than thinking about a complete lift-and-shift, and I'm going to make DR as being the primary case on why I want to be moving to multi-cloud.Corey: Yeah. There's a question, too, of the numbers on paper look radically different than the reality of this. You mentioned, Pinterest has been on AWS since the beginning, which means that even if an edict had been passed at the beginning, that, “Thou shalt never build on anything except EC2 and S3. The end. Full stop.”And let's say you went down that rabbit hole of, “Oh, we don't trust their load balancers. We're going to build our own at home. We have load balancers at home. We'll use those.” It's terrible, but even had you done that and restricted yourselves just to those baseline building blocks, and then decide to do a cloud migration, you're still looking back at over a decade of experience where the app has been built unconsciously reflecting the various failure modes that AWS has, the way that it responds to API calls, the latency in how long it takes to request something versus it being available, et cetera, et cetera.So, even moving that baseline thing to another cloud provider is not a trivial undertaking by any stretch of the imagination. But that said—because the topic does always come up, and I don't shy away from it; I think it's something people should go into with an open mind—how has the multi-cloud conversation progressed at Pinterest? Because there's always a multi-cloud conversation.Micheal: We have always approached it with some form of… openness. It's not like we don't want to be open to the ideas, but you really want to be thinking hard on the business case and the business value something provides on why you want to be doing x. In this case, when we think about multi-cloud—and again, Pinterest did start with EC2 and S3, and we did keep it that way for a long time. We built a lot of primitives around it, used it—for example, my team actually runs our bread and butter deployment system on EC2. We help facilitate deployments across a 100,000-plus machines today.And like you said, we have built that system keeping in mind how AWS works, and understanding the nuances of region and AZ failovers and all of that, and help facilitate deployments across 1000-plus microservices in the company. So, thinking about leveraging, say, a Google Cloud instance and how that works, in theory, we can always make a case for engineering to build our deployment system and expand there, but there's really no value. And one of the biggest cases, usually, when multi-cloud comes in is usually either negotiation for price or actually a DR strategy. Like, what if AWS goes down in and us-east-1? Well, let's be honest, they're powering half the internet [laugh] from that one single—Corey: Right.Micheal: Yeah. So, if you think your business is okay running when AWS goes down and half the internet is not going to be working, how do you want to be thinking about that? So, DR is probably not the best reason for you to be even exploring multi-cloud. Rather, you should be thinking about what the cloud providers are offering as a very nuanced offering which your current cloud provider is not offering, and really think about just using those specific items.Corey: So, I agree that multi-cloud for DR purposes is generally not necessarily the best approach with the idea of being able to failover seamlessly, but I like the idea for backups. I mean, Pinterest is a publicly-traded company, which means that among other things, you have to file risk disclosures and be responsive to auditors in a variety of different ways. There are some regulations to start applying to you. And the idea of, well, AWS builds things out in a super effective way, region separation, et cetera, whenever I talk to Amazonians, they are always surprised that anyone wouldn't accept that, “Oh, if you want backups use a different region. Problem solved.”Right, but it is often easier for me to have a rehydrate the business level of backup that would take weeks to redeploy living on another cloud provider than it is for me to explain to all of those auditors and regulators and financial analysts, et cetera why I didn't go ahead and do that path. So, there's always some story for okay, what if AWS decides that they hate us and want to kick us off the platform? Well, that's why legal is involved in those high-level discussions around things like risk, and indemnity, and termination for convenience and for cause clauses, et cetera, et cetera. The idea of making an all-in commitment to a cloud provider goes well beyond things that engineering thinks about. And it's easy for those of us with engineering backgrounds to be incredibly dismissive of that of, “Oh, indemnity? Like, when does AWS ever lose data?” “Yeah, but let's say one day they do. What is your story going to be when asked some very uncomfortable questions by people who wanted you to pay attention to this during the negotiation process?” It's about dotting the i's and crossing the t's, especially with that many commas in the contractual commitments.Micheal: No, it is true. And we did evaluate that as an option, but one of the interesting things about compliance, and especially auditing as well, we generally work with the best in class consultants to help us work through the controls and how we audit, how we look at these controls, how to make sure there's enough accountability going through. The interesting part was in this case, as well, we were able to work with AWS in crafting a lot of those controls and setting up the right expectations as and when we were putting proposals together as well. Now, again, I'm not an expert on this and I know we have a dedicated team from our technical program management organization focused on this, but early on we realized that, to your point, the cost of any form of backups and then being able to audit what's going in, look at all those pipelines, how quickly we can get the data in and out it was proving pretty costly for us. So, we were able to work out some of that within the constructs of what we have with our cloud provider today, and still meet our compliance goals.Corey: That's, on some level, the higher point, too, where everything is everything comes down to context; everything comes down to what the business demands, what the business requires, what the business will accept. And I'm not suggesting that in any case, they're wrong. I'm known for beating the ‘Multi-cloud is a bad default decision' drum, and then people get surprised when they'll have one-on-one conversations, and they say, “Well, we're multi-cloud. Do you think we're foolish?” “No. You're probably doing the right thing, just because you have context that is specific to your business that I, speaking in a general sense, certainly don't have.”People don't generally wake up in the morning and decide they're going to do a terrible job or no job at all at work today, unless they're Facebook's VP of Integrity. So, it's not the sort of thing that lends itself to casual tweet size, pithy analysis very often. There's a strong dive into what is the level of risk a business can accept? And my general belief is that most companies are doing this stuff right. The universal constant in all of my consulting clients that I have spoken to about the in-depth management piece of things is, they've always asked the same question of, “So, this is what we've done, but can you introduce us to the people who are doing it really right, who have absolutely nailed this and gotten it all down?” “It's, yeah, absolutely no one believes that that is them, even the folks who are, from my perspective, pretty close to having achieved it.”But I want to talk a bit more about what you do beyond just the headline-grabbing large dollar figure commitment to a cloud provider story. What does engineering productivity mean at Pinterest? Where do you start? Where do you stop?Micheal: I want to just quickly touch upon that last point about multi-cloud, and like you said, every company works within the context of what they are given and the constraints of their business. It's probably a good time to give a plug to my previous employer, Twitter, who are doing multi-cloud in a reasonably effective way. They are on the data centers, they do have presence on Google Cloud, and AWS, and I know probably things have changed since a couple of years now, but they have embraced that environment pretty effectively to cater to their acquisitions who were on the public cloud, help obviously, with their initial set of investments in the data center, and still continue to scale that out, and explore, in this case, Google Cloud for a variety of other use cases, which sounds like it's been extremely beneficial as well.So, to your point, there is probably no right way to do this. There's always that context, and what you're working with comes into play as part of making these decisions. And it's important to take a lot of these with a grain of salt because you can never understand the decisions, why they were made the way they were made. And for what it's worth, it sort of works out in the end. [laugh]. I've rarely heard a story where it's never worked out, and people are just upset with the deals they've signed. So, hopefully, that helps close that whole conversation about multi-cloud.Corey: I hope so. It's one of those areas where everyone has an opinion and a lot of them do not necessarily apply universally, but it's always fun to take—in that case, great, I'll take the lesser trod path of everyone's saying multi-cloud is great, invariably because they're trying to sell you something. Yeah, I have nothing particularly to sell, folks. My argument has always been, in the absence of a compelling reason not to, pick a provider and go all in. I don't care which provider you pick—which people are sometimes surprised to hear.It's like, “Well, what if they pick a cloud provider that you don't do consulting work for?” Yeah, it turns out, I don't actually need to win every AWS customer over to have a successful working business. Do what makes sense for you, folks. From my perspective, I want this industry to be better. I don't want to sit here and just drum up business for myself and make self-serving comments to empower that. Which apparently is a rare tactic.Micheal: No, that's totally true, Corey. One of the things you do is help people with their bills, so this has come up so many times, and I realize we're sort of going off track a bit from that engineering productivity discussion—Corey: Oh, which is fine. That's this entire show's theme, if it has one.Micheal: [laugh]. So, I want to briefly just talk about the whole billing and how cost management works because I know you spend a lot of time on that and you help a lot of these companies be effective in how they manage their bills. These questions have come up multiple times, even at Pinterest. We actually in the past, when I was leading the infrastructure governance organization, we were working with other companies of our similar size to better understand how they are looking into getting visibility into their cost, setting sort of the right controls and expectations within the engineering organization to plan, and capacity plan, and effectively meet those plans in a certain criteria, and then obviously, if there is any risk to that, actively manage risk. That was like the biggest thing those teams used to do.And we used to talk a lot trade notes, and get a better sense of how a lot of these companies are trying to do—for example, Netflix, or Lyft, or Stripe. I recall Netflix, content was their biggest spender, so cloud spending was like way down in the list of things for them. [laugh]. But regardless, they had an active team looking at this on a day-to-day basis. So, one of the things we learned early on at Pinterest is that start investing in those visibility tools early on.No one can parse the cloud bills. Let's be honest. You're probably the only person who can reverse… [laugh] engineer an architecture diagram from a cloud bill, and I think that's like—definitely you should take a patent for that or something. But in reality, no one has the time to do that. You want to make sure your business leaders, from your finance teams to engineering teams to head of the executives all have a better understanding of how to parse it.So, investing engineering resources, take that data, how do you munch it down to the cost, the utilization across the different vectors of offerings, and have a very insightful discussion. Like, what are certain action items we want to be taking? It's very easy to see, “Oh, we overspent EC2,” and we want to go from there. But in reality, that's not just that thing; you will start finding out that EC2 is being used by your Hadoop infrastructure, which runs hundreds of thousands of jobs. Okay, now who's actually responsible for that cost? You might find that one job which is accruing, sort of, a lot of instance hours over a period of time and a shared multi-tenant environment, how do you attribute that cost to that particular cost center?Corey: And then someone left the company a while back, and that job just kept running in perpetuity. No one's checked the output for four years, I guess it can't be that necessarily important. And digging into it requires context. It turns out, there's no SaaS tool to do this, which is unfortunate for those of us who set out originally to build such a thing. But we discovered pretty early on the context on this stuff is incredibly important.I love the thing you're talking about here, where you're discussing with your peer companies about these things because the advice that I would give to companies with the level of spend that you folks do is worlds apart from what I would advise someone who's building something new and spending maybe 500 bucks a month on their cloud bill. Those folks do not need to hire a dedicated team of people to solve for these problems. At your scale, yeah, you probably should have had some people in [laugh] here looking at this for a while now. And at some point, the guidance changes based upon scale. And if there's one thing that we discover from the horrible pages of Hacker News, it's that people love applying bits of wisdom that they hear in wildly inappropriate situations.How do you think about these things at that scale? Because, a simple example: right now I spend about 1000 bucks a month at The Duckbill Group, on our AWS bill. I know. We have one, too. Imagine that. And if I wind up just committing admin credentials to GitHub, for example, and someone compromises that and start spinning things up to mine all the Bitcoin, yeah, I'm going to notice that by the impact it has on the bill, which will be noticeable from orbit.At the level of spend that you folks are at, at company would be hard-pressed to spin up enough Bitcoin miners to materially move the billing needle on a month-to-month basis, just because of the sheer scope and scale. At small bill volumes, yeah, it's pretty easy to discover the thing that spiking your bill to three times normal. It's usually a managed NAT gateway. At your scale, tripling the bill begins to look suspiciously like the GDP of a small country, so what actually happened here? Invariably, at that scale, with that level of massive multiplier, it's usually the simplest solution, an error somewhere in the AWS billing system. Yes, they exist. Imagine that.Micheal: They do exist, and we've encountered that.Corey: Kind of heartstopping, isn't it?Micheal: [laugh]. I don't know if you remember when we had the big Spectre and the Meltdown, right, and those were interesting scenarios for us because we had identified a lot of those issues early on, given the scale we operate, and we were able to, sort of, obviously it did have an impact on the builds and everything, but that's it; that's why you have these dedicated teams to fix that. But I think one of the points you made, these are large bills and you're never going to have a 3x jump the next day. We're not going to be seeing that. And if that happens, you know, God save us. [laugh].But to your point, one of the things we do still want to be doing is look at trends, literally on a week-over-week basis because even a one percentage move is a pretty significant amount, if you think about it, which could be funding some other aspects of the business, which we would prefer to be investing on. So, we do want to have enough rigor and controls in place in our technical stack to identify and alert when something is off track. And it becomes challenging when you start using those higher-order services from your public cloud provider because there's no clear insights on how do you, kind of, parse that information. One of the biggest challenges we had at Pinterest was tying ownership to all these things.No, using tags is not going to cut it. It was so difficult for us to get to a point where we could put some sense of ownership in all the things and the resources people are using, and then subsequently have the right conversation with our ads infrastructure teams, or our product teams to help drive the cost improvements we want to be seeing. And I wouldn't be surprised if that's not a challenge already, even for the smaller companies who have bills in the tunes of tens and thousands, right?Corey: It is. It's predicting the spend and trying to categorize it appropriately; that's the root of all AWS bill panic on the corporate level. It's not that the bill is 20% higher, so we're going to go broke. Most companies spend far more on payroll than they do on infrastructure—as you mentioned with Netflix, content is a significantly larger [laugh] expense than any of those things; real estate, it's usually right up there too—but instead it's, when you're trying to do business forecasting of, okay, if we're going to have an additional 1000 monthly active users, what will the cost for us be to service those users and, okay, if we're seeing a sudden 20% variance, if that's the new normal, then well, that does change our cost projections for a number of years, what happens? When you're public, there starts to become the question of okay, do we have to restate earnings or what's the deal here?And of course, all this sidesteps past the unfortunate reality that, for many companies, the AWS bill is not a function of how many customers you have; it's how many engineers you hired. And that is always the way it winds up playing out for some reason. “It's why did we see a 10% increase in the bill? Yeah, we hired another data science team. Oops.” It's always seems to be the data science folks; I know I'd beat up on those folks a fair bit, and my apologies. And one day, if they analyze enough of the data, they might figure out why.Micheal: So, this is where I want to give a shout out to our data science team, especially some of the engineers working in the Infrastructure Governance Team putting these charts together, helping us derive insights. So, definitely props to them.I think there's a great segue into the point you made. As you add more engineers, what is the impact on the bottom line? And this is one of the things actually as part of engineering productivity, we think about as well on a long-term basis. Pinterest does have over 1000-plus engineers today, and to large degree, many of them actually have their own EC2 instances today. And I wouldn't say it's a significant amount of cost, but it is a large enough number, were shutting down a c5.9xl can actually fund a bunch of conference tickets or something else.And then you can imagine that sort of the scale you start working with at one point. The nuance here is though, you want to make sure there's enough flexibility for these engineers to do their local development in a sustainable way, but when moving to, say production, we really want to tighten the flexibility a bit so they don't end up doing what you just said, spin up a bunch of machines talking to the API directly which no one will be aware of.I want to share a small anecdote because when back in the day, this was probably four years ago, when we were doing some analysis on our bills, we realized that there was a huge jump every—I believe Wednesday—in our EC2 instances by almost a factor of, like, 500 to 600 instances. And we're like, “Why is this happening? What is going on?” And we found out there was an obscure job written by someone who had left the company, calling an EC2 API to spin up a search cluster of 500 machines on-demand, as part of pulling that ETL data together, and then shutting that cluster down. Which at times didn't work as expected because, you know, obviously, your Hadoop jobs are very predictable, right?So, those are the things we were dealing with back in the day, and you want to make sure—since then—this is where engineering productivity as team starts coming in that our job is to enable every engineer to be doing their best work across code building and deploying the services. And we have done this.Corey: Right. You and I can sit here and have an in-depth conversation about the intricacies of AWS billing in a bunch of different ways because in different ways we both specialize in it, in many respects. But let's say that Pinterest theoretically was foolish enough to hire me before I got into this space as an engineer, for terrifying reasons. And great. I start day one as a typical software developer if such a thing could be said to exist. How do you effectively build guardrails in so that I don't inadvertently wind up spinning up all the EC2 instances available to me within an account, which it turns out are more than one might expect sometimes, but still leave me free to do my job without effectively spending a nine-month safari figuring out how AWS bills work?Micheal: And this is why teams like ours exist, to help provide those tools to help you get started. So today, we actually don't let anyone directly use AWS APIs, or even use the UI for that matter. And I think you'll soon realize, the moment you hit, like, probably 30 or 40 people in your organization, you definitely want to lock it down. You don't want that access to be given to anyone or everyone. And then subsequently start building some higher-order tools or abstraction so people can start using that to control effectively.In this case, if you're a new engineer, Corey, which it seems like you were, at some point—Corey: I still write code like I am, don't worry.Micheal: [laugh]. So yes, you would get access to our internal tool to actually help spin up what we call is a dev app, where you get a chance to, obviously, choose the instance size, not the instance type itself, and we have actually constrained the instance types we have approved within Pinterest as well. We don't give you the entire list you get a chance to choose and deploy to. We actually have constraint to based on the workload types, what are the instance types we want to support because in the future, if we ever want to move from c3 to c5—and I've been there, trust me—it is not an easy thing to do, so you want to make sure that you're not letting people just use random instances, and constrain that by building some of these tools. As a new engineer, you would go in, you'd use the tool, and actually have a dev app provisioned for you with our Pinterest image to get you started.And then subsequently, we'll obviously shut it down if we see you not being using it over a certain amount of time, but those are sort of the guardrails we've put in over there so you never get a chance to directly ever use the EC2 APIs, or any of those AWS APIs to do certain things. The similar thing applies for S3 or any of the higher-order tools which AWS will provide, too.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: How does that interplay with AWS launches yet another way to run containers, for example, and that becomes a valuable potential avenue to get some business value for a developer, but the platform you built doesn't necessarily embrace that capability? Or they release a feature to an existing tool that you use that could potentially be a just feature capability story, much more so than a cost savings one. How do you keep track of all of that and empower people to use those things so they're not effectively trying to reimplement DynamoDB on top of EC2?Micheal: That's been a challenge, actually, in the past for us because we've always been very flexible where engineers have had an opportunity to write their own solutions many a times rather than leveraging the AWS services, and of late, that's one of the reasons why we have an infrastructure organization—an extremely lean organization for what it's worth—but then still able to achieve outsized outputs. Where we evaluate a lot of these use cases, as they come in and open up different aspects of what we want to provide say directly from AWS, or build certain abstractions on top of it. Every time we talk about containers, obviously, we always associate that with something like Kubernetes and offerings from there on; we realized that our engineers directly never ask for those capabilities. They don't come in and say, “I need a new container orchestration system. Give that to me, and I'm going to be extremely productive.”What people actually realize is that if you can provide them effective tools and that can help them get their job done, they would be happy with it. For example, like I said, our deployment system, which is actually an open-source system called Teletraan. That is the bread and butter at Pinterest at which my team runs. We operate 100,000-plus machines. We have actually looked into container orchestration where we do have a dedicated Kubernetes team looking at it and helping certain use cases moved there, but we realized that the cost of entire migrations need to be evaluated against certain use cases which can benefit from being on Kubernetes from day one. You don't want to force anyone to move there, but give them the right incentives to move there. Case in point, let's upgrade your OS. Because if you're managing machines, obviously everyone loves to upgrade their OSes.Corey: Well, it's one of the things I love savings plans versus RIs; you talk about the c3 to c5 migration and everyone has a story about one of those, but the most foolish or frustrating reason that I ever saw not to do the upgrade was what we bought a bunch of Reserved Instances on the C3s and those have a year-and-a-half left to run. And it's foolish not on the part of customers—it's economically sound—but on the part of AWS where great, you're now forcing me to take a contractual commitment to something that serves me less effectively, rather than getting out of the way and letting me do my job. That's why it's so important to me at least, that savings plans cover Fargate and Lambda, I wish they covered SageMaker instead of SageMaker having its own thing because once again, you're now architecturally constrained based upon some ridiculous economic model that they have imposed on us. But that's a separate rant for another time.Micheal: No, we actually went through that process because we do have a healthy balance of how we do Reserved Instances and how we look at on-demand. We've never been big users have spot in the past because just the spot market itself, we realized that putting that pressure on our customers to figure out how to manage that is way more. When I say customers, in this case, engineers within the organization.Corey: Oh, yes. “I want to post some pictures on Pinterest, so now I have to understand the spot market. What?” Yeah.Micheal: [laugh]. So, in this case, when we even we're moving from C3 to C5—and this is where the partnership really plays out effectively, right, because it's also in the best interest of AWS to deprecate their aging hardware to support some of these new ones where they could also be making good enough premium margins for what it's worth and give the benefit back to the user. So, in this case, we were able to work out an extremely flexible way of moving to a C5 as soon as possible, get help from them, actually, in helping us do that, too, allocating capacity and working with them on capacity management. I believe at one point, we were actually one of the largest companies with a C3 footprint and it took quite a while for us to move to C5. But rest assured, once we moved, the savings was just immense. We were able to offset any of those RI and we were able to work behind the scenes to get that out. But obviously, not a lot of that is considered in a small-scale company just because of, like you said, those constraints which have been placed in a contractual obligation.Corey: Well, this is an area in which I will give the same guidance to companies of your scale as well as small-scale companies. And by small-scale, I mean, people on the free tier account, give or take, so I do mean the smallest of the small. Whenever you wind up in a scenario where you find yourself architecturally constrained by an economic barrier like this, reach out to your account manager. I promise you have one. Every account, even the tiny free tier accounts, have an account manager.I have an account manager, who I have to say has probably one of the most surreal jobs that AWS, just based upon the conversations I throw past him. But it's reaching out to your provider rather than trying to solve a lot of this stuff yourself by constraining how you're building things internally is always the right first move because the worst case is you don't get anywhere in those conversations. Okay, but at least you explored that, as opposed to what often happens is, “Oh, yeah. I have a switch over here I can flip and solve your entire problem. Does that help anything?”Micheal: Yeah.Corey: You feel foolish finding that out only after nine months of dedicated work, it turns out.Micheal: Which makes me wonder, Corey. I mean, do you see a lot of that happening where folks don't tend to reach out to their account managers, or rather treat them as partners in this case, right? Because it sounds like there is this unhealthy tension, I would say, as to what is the best help you could be getting from your account managers in this case.Corey: Constantly. And the challenge comes from a few things, in my experience. The first is that the quality of account managers and the technical account managers—the folks who are embedded many cases with your engineering teams in different ways—does vary. AWS is scaling wildly and bursting at the seams, and people are hard to scale.So, some are fantastic, some are decidedly less so, and most folks fall somewhere in the middle of that bell curve. And it doesn't take too many poor experiences for the default to be, “Oh, those people are useless. They never do anything we want, so why bother asking them?” And that leads to an unhealthy dynamic where a lot of companies will wind up treating their AWS account manager types as a ticket triage system, or the last resort of places that they'll turn when they should be involved in earlier conversations.I mean, take Pinterest as an example of this. I'm not sure how many technical account managers you have assigned to your account, but I'm going to go out on a limb and guess that the ratio of technical account managers to engineers working on the environment is incredibly lopsided. It's got to be a high ratio just because of the nature of how these things work. So, there are a lot of people who are actively working on things that would almost certainly benefit from a more holistic conversation with your AWS account team, but it doesn't occur to them to do it just because of either perceived biases around levels of competence, or poor experiences in the past, or simply not knowing the capabilities that are there. If I could tell one story around the AWS account management story, it would be talk to folks sooner about these things.And to be clear, Pinterest has this less than other folks, but AWS does themselves no favors by having a product strategy of, “Yes,” because very often in service of those conversations with a number of companies, there is the very real concern of are they doing research so that they can launch a service that competes with us? Amazon as a whole launching a social network is admittedly one of the most hilarious ideas I [laugh] can come up with and I hope they take a whack at it just to watch them learn all these lessons themselves, but that is again, neither here nor there.Micheal: That story is very interesting, and I think you mentioned one thing; it's just that lack of trust, or even knowing what the account managers can actually do for you. There seems to be just a lack of education on that. And we also found it the hard way, right? I wouldn't say that Pinterest figured this out on day one. We evolved sort of a relationship over time. Yes, our time… engagements are, sort of, lopsided, but we were able to negotiate that as part of deals as we learned a bit more on what we can and we cannot do, and how these individuals are beneficial for Pinterest as well. And—Corey: Well, here's a question for you, without naming names—and this might illustrate part of the challenge customers have—how long has your account manager—not the technical account managers, but your account manager—been assigned to your account?Micheal: I've been at Pinterest for five years and I've been working with the same person. And he's amazing.Corey: Which is incredibly atypical. At a lot of smaller companies, it feels like, “Oh, I'm your account manager being introduced to you.” And, “Are you the third one this year? Great.” What happens is that if the account manager excels, very often they get promoted and work with a smaller number of accounts at larger spend, and whereas if they don't find that AWS is a great place for them for a variety of reasons, they go somewhere else and need to be backfilled.So, at the smaller account, it's, “Great. I've had more account managers in a year than you've had in five.” And that is often the experience when you start seeing significant levels of rotation, especially on the customer engineering side where you wind up with you have this big kickoff, and everyone's aware of all the capabilities and you look at it three years later, and not a single person who was in that kickoff is still involved with the account on either side, and it's just sort of been evolving evolutionarily from there. One thing that we've done in some of our larger accounts as part of our negotiation process is when we see that the bridges have been so thoroughly burned, we will effectively request a full account team cycle, just because it's time to get new faces in where the customer, in many cases unreasonably, is not going to say, “Yeah but a year-and-a-half ago you did this terrible thing and we're still salty about it.” Fine, whatever. I get it. People relationships are hard. Let's go ahead and swap some folks out so that there are new faces with new perspectives because that helps.Micheal: Well, first off, if you had so many switches in account manager, I think that's something speaks about [laugh] how you've been working, too. I'm just kidding. There are a bu—Corey: Entirely possible. In seriousness, yes. But if you talk to—like, this is not just me because in my case, yeah, I feel like my account manager is whoever drew the short straw that week because frankly, yeah, that does seem like a great punishment to wind up passing out to someone who is underperforming. But for a lot of folks who are in the mid-tier, like, spending $50 to $100,000 a month, this is a very common story.Micheal: Yeah. Actually, we've heard a bit about this, too. And like you said, I think maintaining context is the most thing. You really want your account manager to vouch for you, really be your champion in those meetings because AWS, like you said is so large, getting those exec time, and reviews, and there's so many things that happen, your account manager is the champion for you, or right there. And it's important and in fact in your best interest to have a great relationship with them as well, not treat them as, oh yet another vendor.And I think that's where things start to get a bit messy because when you start treating them as yet another vendor, there is no incentive for them to do the best for you, too. You know, people relationships are hard. But that said though, I think given the amount of customers like these cloud companies are accruing, I wouldn't be surprised; every account manager seems to be extremely burdened. Even in our case, although I've been having a chance to work with this one person for a long time, we've actually expanded. We have now multiple account managers helping us out as we've started scaling to use certain aspects of AWS which we've never explored before.We were a bit constrained and reserved about what service we want to use because there have been instances where we have tried using something and we have hit the wall pretty immediately. API rate limits, or it's not ready for primetime, and we're like, “Oh, my God. Now, what do we do?” So, we have a bit more cautious. But that said, over time, having an account manager who understands how you work, what scale you have, they're able to advocate with the internal engineering teams within the cloud provider to make the best of supporting you as a customer and tell that success story all the way out.So yeah, I can totally understand how this may be hard, especially for those small companies. For what it's worth, I think the best way to really think about it is not treat them as your vendor, but really go out on a limb there. Even though you signed a deal with them, you want to make sure that you have the continuing relationship with them to have—represent your voice better within the company. Which is probably hard. [laugh].Corey: That's always the hard part. Honestly, if this were the sort of thing that were easy to automate, or you could wind up building out something that winds up helping companies figure out how to solve these things programmatically, talk about interesting business problems that are only going to get larger in the fullness of time. This is not going away, even if AWS stopped signing up new customers entirely right now, they would still have years of growth ahead of them just from organic growth. And take a company with the scale of Pinterest and just think of how many years it would take to do a full-on exodus, even if it became priority number one. It's not realistic in many cases, which is why I've never been a big fan of multi-cloud as an approach for negotiation. Yeah, AWS has more data on those points than any of us do; they're not worried about it. It just makes you sound like an unsophisticated negotiator. Pick your poison and lean in.Micheal: That is the truth you just mentioned, and I probably want to give a call out to our head of infrastructure, [Coburn 00:42:13]. He's also my boss, and he had brought this perspective as well. As part of any negotiation discussions, like you just said, AWS has way more data points on this than what we think we can do in terms of talking about, “Oh, we are exploring this other cloud provider.” And it's—they would be like, “Yeah. Do tell me more [laugh] how that's going.”And it's probably in the best interest to never use that as a negotiation tactic because they clearly know the investments that's going to build on what you've done, so you might as well be talking more—again, this is where that relationship really plays together because you want both of them to be successful. And it's in their best interest to still keep you happy because the good thing about at least companies of our size is that we're probably, like, one phone call away from some of their executive team, where we could always talk about what didn't work for us. And I know not everyone has that opportunity, but I'm really hoping and I know at least with some of the interactions we've had with the AWS teams, they're actively working and building that relationship more and more, giving access to those customer advisory boards, and all of them to have those direct calls with the executives. I don't know whether you've seen that in your experience in helping some of these companies?Corey: Have a different approach to it. It turns out when you're super loud and public and noisy about AWS and spend too much time in Seattle, you start to spend time with those people on a social basis. Because, again, I'm obnoxious and annoying to a lot of AWS folks, but I'm also having an obnoxious habit of being right in most of the things I'm pointing out. And that becomes harder and harder to ignore. I mean, part of the value that I found in being able to do this as a consultant is that I begin to compare and contrast different customer environments on a consistent ongoing basis.I mean, the reason that negotiation works well from my perspective is that AWS does a bunch of these every week, and customers do these every few years with AWS. And well, we do an awful lot of them, too, and it's okay, we've seen different ways things can get structured and it doesn't take too long and too many engagements before you start to see the points of commonality in how these things flow together. So, when we wind up seeing things that a customer is planning on architecturally and looking to do in the future, and, “Well, wait a minute. Have you talked to the folks negotiating the contract about this? Because that does potentially have bearing and it provides better data than what AWS is gathering just through looking at overall spend trends. So yeah, bring that up. That is absolutely going to impact the type of offer you get.”It just comes down to understanding the motivators that drive folks and it comes down to, I think understanding the incentives. I will say that across the board, I have never yet seen a deal from AWS come through where it was, “Okay, at this point you're just trying to hoodwink the customer and get them to sign on something that doesn't help them.” I've seen mistakes that can definitely lead to that impression, and I've seen areas where they're doing data is incomplete and they're making assumptions that are not borne out in reality. But it's not one of those bad faith type—Micheal: Yeah.Corey: —of negotiations. If it were, I would be framing a lot of this very differently. It sounds weird to say, “Yeah, your vendor is not trying to screw you over in this sense,” because look at the entire IT industry. How often has that been true about almost any other vendor in the fullness of time? This is something a bit different, and I still think we're trying to grapple with the repercussions of that, from a negotiation standpoint and from a long-term business continuity standpoint, when your faith is linked—in a shared fate context—with your vendor.Micheal: It's in their best interest as well because they're trying to build a diversified portfolio. Like, if they help 100 companies, even if one of them becomes the next Pinterest, that's great, right? And that continued relationship is what they're aiming for. So, assuming any bad faith over there probably is not going to be the best outcome, like you said. And two, it's not a zero-sum game.I always get a sense that when you're doing these negotiations, it's an all-or-nothing deal. It's not. You have to think they're also running a business and it's important that you as your business, how okay are you with some of those premiums? You cannot get a discount on everything, you cannot get the deal or the numbers you probably want almost everything. And to your point, architecturally, if you're moving in a certain direction where you think in the next three years, this is what your usage is going to be or it will come down to that, obviously, you should be investing more and negotiating that out front rather than managed NAT [laugh] gateways, I guess. So, I think that's also an important mindset to take in as part of any of these negotiations. Which I'm assuming—I don't know how you folks have been working in the past, but at least that's one of the key items we have taken in as part of any of these discussions.Corey: I would agree wholeheartedly. I think that it just comes down to understanding where you're going, what's important, and again in some cases knowing around what things AWS will never bend contractually. I've seen companies spend six weeks or more trying to get to negotiate custom SLAs around services. Let me save everyone a bunch of time and money; they will not grant them to you.Micheal: Yeah.Corey: I promise. So, stop asking for them; you're not going to get them. There are other things they will negotiate on that they're going to be highly case-dependent. I'm hesitant to mention any of them just because, “Well, wait a minute, we did that once. Why are you talking about that in public?” I don't want to hear it and confidentiality matters. But yeah, not everything is negotiable, but most things are, so figuring out what levers and knobs and dials you have is important.Micheal: We also found it that way. AWS does cater to their—they are a platform and they are pretty clear in how much engagement—even if we are one of their top customers, there's been many times where I know their product managers have heavily pushed back on some of the requests we have put in. And that makes me wonder, they probably have the same engagement even with the smallest of customers, there's always an implicit assumption that the big fish is trying to get the most out of your public cloud providers. To your point, I don't think that's true. We're rarely able to negotiate anything exclusive in terms of their product offerings just for us, if that makes sense.Case in point, tell us your capacity [laugh] for x instances or type of instances, so we as a company would know how to plan out our scale-ups or scale-downs. That's not going to happen exclusively for you. But those kind of things are just, like, examples we have had a chance to work with their product managers and see if, can we get some flexibility on that? For what it's worth, though, they are willing to find a middle ground with you to make sure that you get your answers and, obviously, you're being successful in your plans to use certain technologies they offer or [unintelligible 00:48:31] how you use their services.Corey: So, I know we've gone significantly over time and we are definitely going to do another episode talking about a lot of the other things that you're involved in because I'm going to assume that your full-time job is not worrying about the AWS bill. In fact, you do a fair number of things beyond that; I just get stuck on that one, given that it is but I eat, sleep, breathe, and dream about.Micheal: Absolutely. I would love to talk more, especially about how we're enabling our engineers to be extremely productive in this new world, and how we want to cater to this whole cloud-native environment which is being created, and make sure people are doing their best work. But regardless, Corey, I mean, this has been an amazing, insightful chat, even for me. And I really appreciate you having me on the show.Corey: No, thank you for joining me. If people want to learn more about what you're up to, and how you think about things, where can they find you? Because I'm also going to go out on a limb and assume you're also probably hiring, given that everyone seems to be these days.Micheal: Well, that is true. And I wasn't planning to make a hiring pitch but I'm glad that you leaned into that one. Yes, we are hiring and you can find me on Twitter at twitter dot com slash M-I-C-H-E-A-L. I am spelled a bit differently, so make sure you can hit me up, and my DMs are open. And obviously, we have all our open roles listed on pinterestcareers.com as well.Corey: And we will, of course, put links to that in the [show notes 00:49:45]. Thank you so much for taking the time to speak with me today. I really appreciate it.Micheal: Thank you, Corey. It was really been great on your show.Corey: And I'm sure we'll do it again in the near future. Micheal Benedict, Head of Engineering Productivity at Pinterest. I am Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a long rambling comment about exactly how many data centers Pinterest could build instead.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Microsoft has a bunch of new goodies for developers, but Mike is becoming more and more concerned about an insidious new feature.

On this week's episode, Jay Gordon is here to walk us through how to use the features in Microsoft Teams to streamline communication when changes to GitHub repos occur. He'll demo how to improve collaboration and efficiency using the tools we have at our fingertips!Jump to:[02:52] Connecting GitHub to Teams[11:37] Committing a change to GitHub[13:42] Viewing notification in TeamsLearn more:Collaborate in teams and channels with Microsoft Teamshttps://aka.ms/GitHubTeamsTDLGitHub Enterprise | Microsoft AppSource https://aka.ms/GHETeamsTDLGitHub for Teams | Microsoft AppSource https://aka.ms/GHTeamsTDLNext-level Collaboration with GitHub and Microsoft Teamshttps://aka.ms/GHTeamsAzure DevOps Docs https://aka.ms/AzureDevOps/Docs​Azure DevOps YouTube https://aka.ms/AzDevOpsYouTubeCreate your free Azure DevOps account https://aka.ms/AzureDevOpsStartFreeFollow April https://twitter.com/TheAprilEdwardsFollow Jay https://twitter.com/jaydestroCheck out new episodes every week at https://aka.ms/TheDevOpsLab and https://aka.ms/AzDevOpsYouTube

Should you inject a little chaos into your systems? Julie Gunderson says yes! Richard chats with Julie about how deliberate failing portions of your system teach you more about your dependency and resiliency. But it's not just about breaking things - you need to build a hypothesis around how you see the reliability of the system and then create tests to challenge that reliability - and how to recover quickly if the test does harm to productivity! Julie talks about starting small, maybe just a development pipeline before moving on to more significant systems like production. The goal is to be resilient, but you're only resilient if you test!Links:Chaos MonkeyGremlin BootcampsPage it to the Limit PodcastRecorded October 12, 2021

This week, Stephanie Wong and Carter Morgan are talking about the recently released State of DevOps Report. Guests Dustin Smith and Nathen Harvey tell us all about DORA, the research group working to study DevOps, and the findings of their years-long study aimed at improving workplace environments, fostering sustainable increased productivity, and ensuring quality output across industries. During their years of research, the DORA team has developed ways to measure team results and workplace culture. Our guests tell us about the five measures they use, including deployment frequency and reliability. The shared responsibility and collaboration of teams at a company to optimize these five metrics is what makes good DevOps performance. Through a real-life example, we hear how the coordination of goals and incentives across departments can improve results of the DevOps metrics, thus improving the speed and stability of finished products. Once businesses identify problems, they need realistic expectations of the time and energy required to solve these issues. Learning from each change made and growing during the process is an important part of optimization, and our guests talk about the best practices their research has identified for facilitating smoother transitions. High quality documentation is a vital part of optimizing DevOps, and this year’s report examined internal documentation for the first time. Nathan describes what makes good documentation, like clear ownership of the documents and docs that are regularly updated for easy sharing and scaling of up-to-date material across the company. Dustin elaborates, explaining other factors that make quality, reliable documents. Later, we talk SRE and how companies can measure and optimize Site Reliability Engineering. A supportive team culture and ensuring a secure product and supply chain are some important factors in optimal SRE, the DORA study found. Our guests offer advice for companies looking to get started with DevOps practices. Nathen Harvey Nathen Harvey is a developer relations engineer at Google who has built a career on helping teams realize their potential while aligning technology to business outcomes. Nathen has had the privilege of working with some of the best teams and open source communities, helping them apply the principles and practices of DevOps and SRE. Dustin Smith Dustin Smith is a UX Research Manager and the DORA research lead. He studies the factors that influence a team’s ability to deliver software quickly and reliably. Cool things of the week Email is 50 years old, and still where it's @ blog Make the most of hybrid work with Google Workspace blog We analyzed 80 million ransomware samples – here's what we learned blog Interview DevOps site DORA site SRE site 2021 Accelerate State of DevOps report addresses burnout, team performance report

About AmyAmy (she/her) has spent the better part of the last 15 years in the tech start-up world, starting off as a front-end software engineer before transitioning into leadership. She has built and led teams across the software and product development spectrum, including web and mobile development, QA, operations and infrastructure, customer support, and IT.These days, Amy is building the software engineering team at EdTech startup, Unicycle, and challenging the archetype of what a tech leader should be. She strives to be a real-life success story for other leaders who believe that safe, welcoming, and equitable environments can exist in tech. Links: Unicycle: https://www.unicycle.co AmyChanta: https://twitter.com/AmyChanta TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: Writing ad copy to fit into a 30 second slot is hard, but if anyone can do it the folks at Quali can. Just like their Torque infrastructure automation platform can deliver complex application environments anytime, anywhere, in just seconds instead of hours, days or weeks. Visit Qtorque.io today and learn how you can spin up application environments in about the same amount of time it took you to listen to this ad.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. A famous quote was once uttered by Irena Dunn who said, “A woman without a man is like a fish without a bicycle.” Now, apparently at some point, people just, you know, looked at the fish without a bicycle thing, thought, “That was overwrought. We can do a startup and MVP it. Why do two wheels? We're going to go with one.”And I assume that's the origin story of Unicycle. My guest today is Amy Chantasirivisal who is the Director of Engineering at Unicycle. Amy, thank you for putting up with that incredibly tortured opening. But that's okay; we torture metaphors to death here.Amy: [laugh]. Thank you for having me. That was a great intro.Corey: So, you are, at the time of this recording at least, a relatively new hire to Unicycle, which to my understanding is a relatively new company. What do you folks do over there?Amy: Yes, so Unicycle is not even a year old, so a company born out of the pandemic. But we are building a product to reimagine what the digital classroom looks like. The product itself was thought up right during a time during the pandemic when it became very clear how much students and teachers are struggling with converting their experience into online platforms. And so we are trying to just bring better workflows, more efficiency into that. And right now we're starting with email, but we'll be expanding to other things in the future.Corey: I am absolutely the wrong person to ask about a lot of this stuff, just because my academic background, tortured doesn't really begin to cover it. I handle academia about as well as I handled working for other people. My academic and professional careers before I started this place were basically a patchwork of nonsense and trying to pretend I was something other than I was. You, on the other hand, have very much been someone who's legitimate as far as what you do and how you do it. Before Unicycle, you were the Director of Engineering at Wildbit, which is a name I keep hearing about and a bunch of odd places. What did you do there?Amy: [laugh]. I will have to follow up and ask what the odd places are but—so I was leading a team there of engineers that were fully distributed across the US and also in Europe. And we were building an email product called Postmark, which some of your listeners might use, and then also a couple of other smaller things like People-First Jobs and Beanstalk—not AWS's Beanstalk, but a developer repository and workflow tool.Corey: Forget my listeners for a minute; I use Postmark. That's where I keep seeing you on the invoices because it's different branding. As someone who has The Duckbill Group, but also the Last Week in AWS things, it's the brand confusion problem is very real. That does it. Sorry. Thank you for collapsing the waveform on that one. And of course, before that you were at PagerDuty, which is a company that most folks in the ops space are aware of, founded to combat the engineer's true enemy: sleep.Amy: Absolutely. It's the product that engineers love to hate, but also can't live without, to some degree. Or maybe they want to live without it, but uh… [laugh] are not able to.Corey: So, I have a standing policy on this show of not talking to folks who are not wildly over-represented—as I am—and effectively disregarding the awesome stuff that they've done professionally in favor of instead talking about, “Wow, what's it like not to be a white guy in the room? I can't even imagine such a thing. It sounds hard.” However, in your case, an awful lot of the work you have done and are most proud of centers around DEI, diversity, equity, and inclusion. Tell me about that.Amy: Absolutely. I would say that it's the work that I've spent my time focusing on in recent years, but also that I'm still learning, right, and as someone who is Asian American, and also from a middle-class socioeconomic background, I have a bunch of privileges that I still have to unpack and that show up in the way that I work every day, as well. And so just acknowledging that, you know, while I spend a lot of time on DEI, still have just barely scratched the surface on it, really, in the grand scheme of things. But what I will say is that, you know, I've been really fortunate in my career in that I started in tech 15 or so years ago, and I started at a time when it wasn't super hard for someone who has no CS degree to actually get into some sort of coding job. And so I fell into my first role; I was building HTML and CSS landing pages for a marketing team, for an ISP that was based in San Francisco.So, I was cobbling together a bunch of technical skills, and I got better and better. And then I reached this point in my career where I didn't really have a lot of mentors, and so I was like, “I don't know what's next for me.” But then I am also frustrated that it is so hard for our team to get things done. And so I took it upon myself to figure out Scrum and project management type of stuff for my team, and then made the jump into people management from there. So, people management and leadership through project management.But when I look back on my career, I think about, “Oh, if I had a mentor, would that still have been my fate? Would I have continued down this track of becoming a very senior technical person and just doing that for my whole career?” Because letting go of the code was definitely a hard, hard thing. And I was lucky enough that I really did enjoy the people and the process side of all of this. And so [laugh] this relates to DEI in the fact that there's research and everything that backs this up, but that women and women of color generally tend to get less mentorship overall and get less actionable feedback about their job performance.And you think about how that potentially compounds over time, over the course of someone's career and that may be one of the reasons why women and people of color get pushed out of tech because they're not getting the support that they need, potentially. They're not getting feedback, they're not being advocated for in meetings, and then there's also all the stuff that you can add on around microaggressions, or just aggressions period, potentially, depending on the culture of the team that you're working on. And so all of those things compounded are the types of things that I think about now when I reflect on my own career and the types of teams that I want to be building in the future.Corey: Back when I was stumbling my way through piecing my career together. I mean, as mentioned, I don't have a degree; I don't have a high school diploma, as it turns out, and—that was a surprise when I discovered midway through my 20s that the school I had graduated from wasn't accredited—but I would tell stories, and I found ways to weasel my way through and I gave a talk right around 2015 or 2016, about, “Weasel Your Way to the Top: How to Handle a Job Interview,” and looking back, I would never give that talk again. I canceled it as soon as someone pointed out something that was only obvious in hindsight, that the talk was built out of things that had worked for me. And it's easy to sit here and say that, well, I had to work for what I have; none of this was handed to me. And there's an element of truth to that, except for the part where there was nothing fighting against me as I went.There was not this headwind of a presumed need for me to have to prove myself; I am presumed competent. I sometimes say that as a white guy in tech, my failure mode is a board seat and a book deal, and it's not that far from wrong. It takes, I guess, a lot of listening and a lot of interaction with folks from wildly different backgrounds before you start to see some of these things. It takes time. So, if you're listening to this, and you aren't necessarily convinced that this might be real or whatnot, talk less, listen more. There are a lot of stories out there in the world that I think that it's not my place to tell but listen. That's how I approach it.What's interesting about your pathway into management is it's almost the exact opposite of mine, where I was craving novelty, and okay, I wanted to try and managing a team of people. Years later, in hindsight—I'm not a good manager and I know that about myself, and I explicitly go out of my way these days to avoid managing people wherever possible, for a variety of reasons, but at the time, I didn't know. I didn't know that. I wanted to see how it went.First, I had to disabuse myself of this notion that, oh, management is a promotion. It's not. It's an orthogonal skill.Amy: Yes.Corey: The thing I really learning—management or not—now, is that the higher in the hierarchy you rise, if you want to view it that way, the less hands-on work you do, which means everything that you are responsible for that—and oh, you are responsible—isn't something you can jump in and do yourself. You can only impact the outcome via influence. And that was a hard lesson to learn.Amy: Right. And there are some schools of thought, though, where you can affect the outcome by control. And that's not what I'm about. I think I'm more aligned with what you're saying in terms of, it's really the influence and the ability to clear the way for people who are smarter than you to do the things that they need to do. Just get out of their way, and remove the roadblocks, and just help give them what they need. That's really, sort of like, my overall approach. But I know that there are some folks out there who lead the opposite way of, “It's my way, and I'm going to dictate how things should be done, and really you're here to take and follow orders.”Corey: It's always fun interviewing people to manage teams. “So, why do you want to be a manager?” It's, “Oh, I want to tell people what to do.” And I have to say that as an interviewer, there is nothing that takes the pressure off nearly as well as a perfectly wrong answer. And, yes, that at least to my world, is a perfectly wrong answer to this. There aren't that many pass-fail questions, but you can fail any question if you try hard enough.Amy: [laugh]. Oh, gosh, yeah, it's true. But also, at the same time, I would say that there are organizations that are built that way. Because—all it takes is the one person who wants to tell people what to do, and then they start a company, and then they hire other people who want to tell people what to do. And so there are ways where organizations like that exist and come into being even today, I would say.Corey: The question that I have for you about engineering leadership is, back when I was an engineer, and thinking, all right, it's time for me to go ahead and try being a manager—let's be clear, I joke about it, but the actual reason I wanted to try my hand at management was that I found people problems more interesting than computer problems at that point. I still do, but these days, especially when it comes to, you know, cloud services marketing and such, yeah, generally, the technical problems are, in fact, people problems at their core. But talking to my manager friends of how do I go and transition from being an engineer into being a manager, the universal response I got at the time was, “Ehh, I don't know.” Every person I knew who'd had made that transition was in the right place at the right time, and quote-unquote, “Got lucky.”Amy: Absolutely.Corey: And then once they had management on their resume, then they could go and transition back to being an IC and then to management again. But it's that initial breakthrough that becomes a challenge.Amy: Absolutely. And I fell into it as well. I mean, I got into it, partially for selfish reasons because I was, an IC, I was doing development work, and I was frustrated, and I had teammates who were coming to me and they were frustrated about how hard it was for us to get our work done, or the friction involved in shipping code. And so I took it upon myself to say, “I think I see a pattern about why this is happening, and so I will try to solve this problem for the team.” And so that's where the Agile and Scrum thing come in, and the project management side.And then, when I was at this company—this was One Kings Lane; this was, like, the heyday of flash sales websites and stuff like that, so it was kind of a rocket ship at that time—and because we were also growing so fast and I was interviewing folks as well, I just fell into this management role of, “Well, if I'm interviewing these people, then I guess I should be [laugh] managing them, too.” And that happens for so many people, similar stories of getting into management. And I think that's where it starts to go wrong for a lot of organizations because, like you said, it's not an up-leveling; it's a changing of your role, and it requires training and learning and figuring out how to be effective as a manager. And a lot of people just stumble their way through it and make a lot of mistakes—myself included—through that process.And that becomes really troubling knowing that you can make these really big mistakes, but these mistakes that you make don't affect just yourself. It's the careers of the people that you manage as well and sort of where they're headed in their lives. And so it's troubling to think that most leaders that are out there today have not received any sort of training on how to be a good manager and how to be effective as a manager.Corey: I would agree with that wholeheartedly. It seems that in many cases, companies take the best engineer that they have on their team and promote them to manager. It's brilliant in some respects in just how short-sighted it is. You are taking a great engineer and trading them for a junior and unproven manager, and hoping for the best. And there is no training on any of these things, at least—Amy: Right.Corey: —not the companies that I ever worked at. Of course, there are ways you can learn to be a better manager; there are people who specialize in exactly this. There are companies that do exactly this. But tech has this weird thing where it just tries to solve itself from first principles rather than believing for a minute that someone might possibly have prior experience that could be useful for these things. And—Amy: Absolutely.Corey: —that was a challenge. I had a lot of terrible managers before I entered management myself, and I figured, ah, I'll do the naive thing and I'm just going to manage based upon doing the exact opposite of what those terrible managers all did. And I got surprisingly far with it, on some level. But you don't see the whole picture when you're an individual contributor who's writing code—crappy in my case—most of the time, and then only seeing the aspects of your manager that they allow you to see. They don't share—if they're any good—the constraints that they have to deal with, that they're managing expectations around the team, conflicting priorities, strategic objectives, et cetera because it's not something that gets shown to folks. So—Amy: Absolutely.Corey: —if you bias for that, in my experience you become an empathetic manager to the people on your team, but completely ineffective at managing laterally or upwards.Amy: Mm-hm, absolutely. And you know, I'm exploring this idea of further. Being at a very small company, I think allows me to do that. And exploring this idea of, does it have to be that way? Can you be transparent about what the constraints are as a leader while still caring for your team and supporting them in the ways that they need and helping them grow their careers and just being open about one of the challenges that you have in building the company?And I don't know, I feel like I have some things to prove there, but I think it's possible to achieve some sort of balance there, something better or more beyond just what exists now of having that entire leadership layer typically be very opaque and just very unclear why certain decisions are made.Corey: The hard part that extends that these to me beyond that is it's difficult to get meaningful feedback, on some level, when you're suddenly thrust into that position. I also, in hindsight, realize that an awful lot of those terrible managers that I had weren't nearly as terrible as I thought they were. I will say that being on the other side of that divide definitely breeds empathy. Now that I'm the co-owner of The Duckbill Group, and we're building out a leadership team and the rest, hiring managers of managers is starting to be the sort of thing that I have to think about.It's effectively, how do I avoid inadvertently doing end-runs around people? And oh, I'm just going to completely undermine a manager by reaching out to one of their team and retasking them on something because obviously whatever I have in mind is much more important. What could they possibly be working on that's better than the Twitter shitpost I'm borrowing them to help out with? Yeah, you learn a lot by getting it wrong, and there becomes a power imbalance that even if you try your best to ignore it—which you should not—I assure you, the person who has less power in that relationship cannot set that aside. Even when I have worked with people I consider close friends, that friendship gained some distance during the duration of their employment because there has to be that professional level of separation. It's a hard thing to learn.Amy: It's a very hard line to walk in terms of recognizing the power that you have over someone's career and the power over, you know, making decisions for them and for the team and for the company, and still being empathetic towards their personal needs. And if they're going through a tough time, but then you also know from a business perspective that X, Y, or Z needs to happen, and how do you push but not push too hard, and try to balance needs of people who are humans and have things that happen and go on sometimes, and the fact that we work in a capitalist society and we still need to make money to make the business run. And that's definitely one of the hardest things to learn, and I am still learning. I definitely don't have that figured out, but I err on the side of, let's listen to what people are saying because ultimately, I'm not going to be the one to write the code. I haven't done that in years, and also I would probably suck at it now. And so it behooves leaders to listen to the people who were doing the work and to try, to the best of their abilities in whatever role whether that's exec-level leadership or mid-level… sort of like, middle management type of stuff to do what is in your power to help set them up to succeed.Corey: I want to get back a little bit to the idea of building diverse teams. It's something that you spend an inordinate amount of time and effort on. I do too. It's one of those areas where it's almost fraught to talk about it because I don't want to sound like I'm breaking my arm by patting myself on the back here. I certainly have a hell of a lot to learn, and mostly—and I'm ashamed to admit this—I very often learn only by really putting my foot in it sometimes. And it's painful, but that is, I think, a necessary prerequisite for growth. From your perspective, what is the most challenging part of building diverse teams?Amy: I think it's that piece that you said of making the mistakes or just putting yourself in a position where you are going to be uncomfortable. And I think that a lot of organizations that I've been in talk about DEI on a very surface level in terms of, “Oh, well, you know, we want to have more candidates from diverse backgrounds in our pipelines for hiring,” and things like that. But then not really just thinking about, but how do we work as a team in a way that potentially makes retention of those folks a lot harder? And for myself, I would say that when I was earlier on in all of this in my learning, I would say that I was able to kickstart my learning by thinking about my own identity, the fact that I was often the only Asian person on my team, the only woman on my team, and then more recently, the only mom on my team. And that has happened to me so many times in my career. More often than not.And so being able to draw on those experiences and those feelings of oh, okay, no one wants to hear about my kid because everyone else is, you know, busy going out to drink or something on the weekends. And like that feeling of, you know, that not belonging, and feeling of feeling excluded from things, and then thinking about how then this might manifest for folks with different identities for myself. And then going there and learning about it, listening, doing more listening than talking, and yeah, and that's, that's really just been the hardest part of just removing myself from that equation and just listening to the experiences of other people. And it's uncomfortable. And I think a lot of people are—you have to be in the right mindset, I guess, to be uncomfortable; you have to be willing to accept that you will be uncomfortable. And I think a lot of folks maybe are not ready to do that on a personal level.Corey: The thing that galls me the most is I do try on these things, and I get it wrong a fair bit. And my mistakes I find personally embarrassing, and I strive not to repeat them. But then I look around the industry—and let's be clear, a lot of this is filtered through the unhealthy amount of time I spend on Twitter—but it seems that I'm trying and I'm failing and attempting to do better as I go, and then I see people who are just, “Nope. Not at all. In fact, we're not just going to lean into bias, we're going to build a startup around it.”And I look at this and it's at some level hard to reconcile the fact that… at first, that I'm doing badly at all, which is the easy cop-out of, “Oh, well, if that is considered acceptable on some level, then I certainly don't even have to try,” which I think is a fallacy. But further it's—I have to step beyond myself on that and just, I cannot fathom how discouraging that must be, particularly to people who are early in their careers because it looks like it's just a normal thing that everyone thinks and does that just someone got a little too loud with it. And it's abhorrent. And if people are listening to this and thinking that is somehow just entrenched, and normalized, and everyone secretly thinks that… no. I assure you it is not something that is acceptable, even in the quote-unquote, “Private white dude who started companies” gathering holes. Yeah, people articulating sentiments like that suddenly find themselves not welcome there anymore, at least in every one of those types of environments I've ever found myself in.Amy: Yeah, the landscape is shifting. It's slow, but it is shifting. And, myself on Twitter, like, I do a lot of rant-y stuff too sometimes, but despite all of that, I feel like I am ultimately an optimist because I have to be. Otherwise, I would have left tech already because every time I am faced with a job search for myself, I'm like, “Should I—is this it? Am I done in tech? Do I want to go do something else? Am I going to finally go open that bakery that I've always wanted to open?” [laugh].And so… I have to be an optimist. And I see that—even in the most recent job search I've done—have seen so many new founders and new CEOs, really, with this mindset of, “We want to build a diverse team, but we're also doing it—and we're using diversity as a foundation for what we want to build; it's part of our decision-making process and this is how we're going to hold ourselves accountable to it.” And so it is shifting, and while there are those bad actors out there still, I'm seeing a lot of good in the industry now. And so that's why I stick around; that's why I'm still here.Corey: I want to actually call something out as concrete here because it's easy for me to fall into the trope of just saying vague things. I'll be specific about something, give us a good example. We've done a decent job, I think, of hiring a diverse team, but—and this is a problem that I see spread across an awful lot of companies—as you look at the leadership team, it gets a lot wider and a lot more male. And that is an inherent challenge. In our particular case, my business partner is someone who I've been close friends with for a decade.I would not be able to start a business with someone I didn't have that kind of relationship with just because your values have to be aligned or there's trouble down the road. And beyond that, it winds up rapidly, on some level, turning into what appears to be a selection bias. When you're trying to hire senior leaders, for example, there's a prerequisite to being a senior leader, which is embodied in the word senior, which implies tenure of having spent a fair bit of time in an industry that is remarkably unfriendly in a lot of different ways to a lot of different people. So, there's a prerequisite of being willing to tolerate the shit for as long as it takes to get to that level of seniority, rather than realizing at any point as any of us can, there are easier jobs that don't have this toxicity inherent to them and I'll go do that instead. So, there's a tenure question; there's a survivorship bias question.And I don't have the answers to any of this, but it's something that I'm seeing, and it's one of those once you see it, you can't unsee it any more moments. At least for me.Amy: Yeah, absolutely.Corey: Please tell me I'm not the only person who see [laugh]—who is encountering these problems. Like, “Wow, you just sound terrible.” Which might very well be a fair rejoinder here. I'm just trying to wrap my head around how to think about this properly.Amy: Yeah. I mean, this is why I was saying that I am very optimistic about [laugh] new companies that are coming—like, up-and-coming these days, new startups, primarily, because you're right that a lot of people just end up quitting tech before they get to that point of experience and seniority, to get into leadership. I mean, obviously, there's a lot of bias and discrimination that happens at those leadership levels, too, but I will say that, you know, it's both of those things. There are also more things on top of that. But this is why I'm like, so excited to see people from diverse backgrounds as founders of new companies and why I think that being able to be in a position to potentially either help fund, or advocate, or sponsor, or amplify those types of orgs, I think is where the future is that because ultimately, I think a lot of the established companies that are out there these days, it's going to be really hard for them to walk back on what their leadership team looks like now, especially if it is a sizable leadership team and they're all white men.Corey: Yeah. I'm going to choose to believe we say sizable leadership team that it's also not—we're talking about the horizontal scaling that happens to some of us, especially during a pandemic as we continue to grow into our seats. You're right, it's a problem as well, where you can cut a bit of slack in some cases to small teams. It's, “Okay, we don't have any Black employees, but we're three people,” is a lot more understandable-slash-relatable than, “We haven't hired any Black people yet and we're 3000 people.” One of those is acceptable—or at least understandable, if not acceptable—the other is just completely egregious.Amy: Yes. And I think then the question that you have to ask if you're looking at, you know, a three-person company, or [laugh] I guess, like in my case, I was looking at the seven-person company, is that, “Okay. There are currently no Black people on your team. And why is that?” And then, “What are you doing to change that? And how are you going to make sure that you're holding ourselves accountable to it?”Because I think it's easy to say, “Oh, you know, the first couple of hires were people we just worked with in the past, and they just happened to, you know, look like us and whatnot.” And then you blink becau—and you do that a handful of times, and you blink, and then suddenly you have a team of 25 and there are no people of color on your team. And maybe you have, like, one woman on the team or something. And you're like, “Huh. That's strange. I guess we should think about this and figure out what we can do.”And then I think what ends up happening at that point is that there are so many already established behaviors, and cultural norms, and things like that, that have organically grown within a team that are potentially not welcoming towards people from different backgrounds who have different backgrounds. So, you go and attempt to hire someone who is different, and they come in, and they're just sort of like, “This is how you work? I don't feel like I belong here.” And then they don't stay, and then they leave. And then people sit there and scratch their heads like, “Oh, what did we do wrong?” And, “I don't get it.”And so there's this conversation, I think, in the industry of like, “Oh, it's a pipeline problem, and if we were just able to hire a lot of people from diverse backgrounds, the problem is solved.” Which really isn't the case because once people are there and at your company, are they getting promoted at the same rate as white men? Are they staying with the company for as long? And who's in leadership? And how are you working to break down the biases that you may have?All those sorts of things, I think, generally are not considered as part of all of this DEI work. Especially when, in my experience in startups, the operational side of all that is so immature a lot of the times, just not well developed that deeper thought process and reflection doesn't really happen.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: I do my best to have these conversations in public as frequently as is practical for me to do, just because I admit, I get things wrong. I say things that are wrong and I'm doing a fair bit of learning in public around an awful lot of that. Because frankly, I can withstand the heat, if it comes down to someone on Twitter gets incredibly incensed by something I've said on this podcast, for example. Because it isn't coming from a place of ill intent when someone accuses me of being ableist or expressing bias. My response is generally to suppress the initial instinctive flash of defensiveness and listen and ask.And that is, even if I don't necessarily agree with what they're saying after reflection, I have to appreciate on some level the risk-taking inherent in calling someone out who is in my position where, if I were a trash fire, I could use the platform to turn it into, “All right. Now, let's go hound the person that called me out.” No. I don't do that, full stop. If I'm going to harass people, it's going to be—not people, despite what the Supreme Court might tell us—but it's going to be a $2 trillion company—one in particular—because that's who I am and that's how I roll.Whenever I get a DM—which I leave open because I have the privilege to do that—from folks who are early career who are not wildly over-represented, I just have to stop and marvel for a minute at the level of risk-taking inherent to that because there is risk to that. For me, when I DM people, the only risk I feel like I'm running at any given point is, “Are they going to think that I'm bothering them? Oh, the hell with it. I'm adorable. They'll love me.” And the fact that I'm usually right is completely irrelevant to that. There's just that sense of I don't really risk a damn thing in the grand scheme of things compared to the risk that many people are taking just living who they are.Amy: Yeah. And someone DMs you and you suppress that initial sort of defensiveness: I would say that that is an underrated skill. [laugh].Corey: Well, a DM is a privilege, too. A call in—Amy: Yes.Corey: —is deeply appreciated; no one owes it to me. I often will get people calling me out on Twitter and I generally stop and think about that; I have a very close circle of friends who I trust to be objective on these things, and I'll ask them, “Did I get this wrong?” And very often the answer is yes. And, “Well, I thought the joke was funny and I spent time building it.” “Yeah, but if people hear a joke I'm making and feel bad about it, then is it really that good of a joke or should I try harder?” It's a process, and I look back at who I was ten years ago and I feel a sense of shame. And I believe that if anyone these days doesn't, either they were effectively a saint, or they haven't grown.Amy: Yes.Corey: And that's my personal philosophy on this stuff, anyway.Amy: Yeah, absolutely. And that growth is so important. And part of that growth really is being able to suppress your desire to make it about you, [laugh] right? That initial, “Oh, I did something bad,” or, “I'm a horrible person because I said this thing,” right? It's not about you, there's, like, the impact that you had on someone else.And I've been giving this some thought recently, and I—you know, I also similarly have a group of trusted friends who I often talk about these things with, and you know, we always kind of check ourselves in terms of, did we mess something up? Did we, you know, put our foot in our mouths? Stuff like that. And think what it really comes down to is being able to say, “Maybe I did something wrong and I need to suppress that desire to become defensive and put up walls and guard and protect myself from feeling vulnerable, in order to actually learn and grow from this experience.”Corey: It's hard to do, but it's required because I—Amy: Extremely, yes.Corey: —used to worry about, “Ohh, what if I get quote-unquote, ‘canceled?'” well, I've done a little digging into this and every notable instance of this I can find is when someone is called out for something crappy, they get defensive, and they double-down and triple-down and quadruple-down, and they keep digging a hole nice and deep to the point where no one with a soul can really be on their side of this issue, and now they have a problem. I have never gotten to that point because let's be honest with you, there are remarkably few things I care that passionately about that I'm going to pick those fights publicly. The ones that I do, I am very much on the other side [laugh] of those issues. That has not been a realistic concern.I used to warn every person here before I hired them—to get this back to engineering management—that there was a risk that I could have a bad tweet and we don't have a company anymore. I don't give that warning anymore because I no longer believe that it's true.Amy: Mm-hm. Mm-hm. I also wonder about, in general, because of the world that we live in, and our history with white supremacy and oppression and all those things, I also wonder if this skill of being able to self-reflect and be uncomfortable and manage your own reaction and your emotions, I wonder if that's just a thing that white people generally haven't had a lot of practice for because of the inherent privileges that are afforded to white people. I wonder if a lot of this just stems from the fact that white people get to navigate this world and not get called out, and thus don't have this opportunity to exercise this skill of holding on to that and listening more than talking.Corey: Absolutely agree. And it gets piled on by a lot of folks, for example—I'll continue to use myself as an example in this case—I live in San Francisco. I would argue that I'm probably not, “In tech,” quote-unquote, the way that I once was, but I'm close enough that there's no discernible difference. And my social circle is as well. Back before I entered tech, I did a bunch of interesting jobs, telemarketing to pay the bills, I was a recruiter for a while, I worked construction a couple of summers.These days, everyone that I engage with for meaningful periods of time is more or less fairly tech adjacent. It really turns into a one-sided perspective. And I can sit here and talk about what folks who are not living in the tech bubble should be doing or how they should think about this, but it's incredibly condescending, it's incredibly short-sighted, and fails to appreciate a very different lived experience. And I can remind myself of this now, but that lack of diversity and experience is absolutely something where it feels like the tech bubble, especially for those folks in this bubble who look a lot like me, it is easy to fall into a pattern of viewing ourselves as the modern aristocracy where we deserve the nice things that we have, and the rest. And that's a toxic pattern. It takes vigilance to avoid it. I'm not saying I get it right all the time, by a landslide, but ugh, the perils of not doing that are awful.Amy: Agreed. And it shows up, you know, getting back to the engineering manager and leadership and org building piece of things, that shows up even in the way that we talk about career development and career ladders, for those of us in tech, and software engineering specifically for me, where we've kind of like come up with all these matrices of job levels, and competencies, all that, and humans just are so vastly different. Every person is an individual, and yet we talked about career ladders and how to advance your career in this two-dimensional matrix. And, like, how does that actually work, right?And I've seen some good career ladders that account for a larger variety of competencies than just, “Can you code?” And, “What are your system design skills?” And, “Do you understand distributed systems?” And so on and so forth, but I think a lot gets left behind and gets left on the table when it comes to thinking about the fact that when you get a group of people together working on some sort of common cause or a product, that there's so much more to the dynamic than just the writing of the code. It's how do you work with each other? How do you support each other? How do you communicate with each other? And then all my glue work—that is what I call it—like, the glue work that goes into a successful team and building products, a lot of that is just not captured in the way that we talk about career development for folks. And it's just incredibly two-dimensional, I think.Corey: One last question that I have for you before we wrap the episode here is, you spend a lot of time focusing on this, and I have some answers, but I'm very interested to hear yours instead because I assure you, the world hears enough from me and people who look like me, what is the biggest mistake that you see companies making in their attempts to build diverse teams?Amy: I would say that there's two major things. One is that there have been a lot of orgs in my own past that think about diversity, equity, inclusion as a program and not a mindset that everyone should be embracing. And that manifests itself into, sort of like, this secondary problem of stopping at the D part of D, E, and I. That's the whole, “We're going to hire a bunch of people from different backgrounds and then just we're going to stop with that because we've solved the problem.” But by not adopting that mindset of the equity, the inclusion, and also the welcoming and the belonging piece of things internally, then anyone that you hire who comes in from those marginalized or minority backgrounds is not going to want to stay long-term because they don't feel like they fit in, they don't feel like they belong.And so, it becomes this revolving door of you hire in people and then those people leave after some amount of time because they're not getting what they need out of either the role or for themselves personally in terms of just emotional support, even. And so I would say that's the problem that I see is not a numbers game—although the metrics and the numbers help hold you accountable—but the metrics and the numbers are not the end goal. The end goal is really around the mindset that you have in building the org and the way that people behave. And the way that you work together is really core to that.Corey: What I tend to see on the other side is the early intake funnels. People will reach out to me sometimes, “Hey, do you know any diverse speakers we can hire to do a speaking engagement here?” It doesn't… work that way. There's a lot more to it than that. It is not about finding people who check boxes, it is not about quote-unquote, “Diversity hires.”It's about—at least in my experience—structuring job ads, for example, in ways that are not coded—unconsciously in most cases, but ehh—that are going to resonate towards folks who are in certain cultures and not in others. It's about being more equitable. It's about understanding that not everyone is going to come across in a job interview as the most confident person in the room. Part of the talk that I gave on how to handle job interviews, there was a strong section in it on salary negotiation. Well, turns out when I do it, I'm an aggressive hard-charger and they like that, whereas if someone who is not male does that, well, in that case, they look like they're being difficult and argumentative and pushy and rising above their station. It was awful.One of the topics I'm most proud of was the redone version of that talk that I gave with a friend, Sonia Gupta, who has since left tech because of how shitty it is, and that was a much better talk. She was a former attorney who had spent time negotiating in much higher-stakes situations.Amy: Yeah.Corey: And it was terrific to see during the deconstruction and rebuilding of that talk, just how much of my own unconscious bias had crept in. It's, again, I look back at the early version of those talks and I'm honestly ashamed. It wasn't from ill will, but it's always impact over intent as far as how this has potentially made things worse. It's, if nothing else, if I don't say the right things when I should speak up, that's not great, but I always prefer that to saying things that are actively harmful. So—Amy: Absolutely.Corey: —it's hard. I deserve no sympathy for this, to be clear. It is incumbent upon all of us because again, as mentioned, my failure mode is a non-issue in the world compared to the failure mode for folks for against whom the deck has been stacked unfairly for a very long time. At least, that's how I see it.Amy: Right. And that's why I think that it's important for folks who are in positions of power to really reflect on—even operationally, right, you were mentioning your job ads, and how to structure that to include more inclusive language, and just doing that for everything, really, in the way that you work. How do decisions get made? And by whom? And why? How do you structure things like compensation? Even, like, how do you do project planning, right?Even in my own reflections, now when I think back towards Scrum and Agile and all of that, I think that the base foundation of all of that was like was good, but then ultimately the implementation of how that works at most companies is problematic in a lot of ways as well. And then to just be able to reflect and really think about all of your processes or policies—all of that—and bring that lens of equity, really, equity and inclusion to those things, and to really dig deep and think about how those things might manifest and affect people from different backgrounds in different ways.Corey: So, before we wrap, something that I think you… are something of an empathetic party on is when I see companies in the space who are doing significant DE&I initiatives, it seems like it's all flash; it feels like it's all sizzle, no steak to appropriate a phrase from the country of Texas. Is that something that you see, too?Amy: I do think that it is pretty common, and I think it's because that's… that's the easy route. That's the easy way to do it because the vanity metrics, and the photo of the team that is so diverse, and all these things that show up on a marketing website. I mean, there—it's, like, a signal for someone, potentially, who might be considering a job at your company, but ultimately the hard work that I feel like is not happening is really in that whole reflecting on the way you do business, reflecting on the way that you work. That is the hard work and it requires a leadership team to prioritize it, and to make time for it, and to make it really a core principle of the way that you build an org., and it doesn't happen enough, by far, in my opinion.Corey: It feels like it's an old trope of the company that makes a $100,000 donation and then spends $10 million dollars telling the world about it, on some level. It's about, “Oh, look at us, we're doing good things,” as opposed to buckling down and doing the work. Then the actual work falls to folks who are themselves not overrepresented as unpaid emotional labor, and then when the company still struggles with diversity issues, those people catch the blame. It's frustrating.Amy: Yeah. And as an organization, if you have the money to donate somewhere, that's great, but it can't just stop at that. And a lot of companies will just stop at that because it's the optics of, “Oh, well, we spent x millions of dollars and we've helped out this nonprofit or this charity or whatnot.” Which is great that you're able to do that, but that can't be it because then ultimately, what you have internally and within your own company doesn't improve for people from those backgrounds.Corey: I want to thank you for taking so much time to chat with me about these things. Some of these topics are challenging to talk about and finding the right forum can be difficult, and I'm just deeply appreciative that you were able to clear enough time to have that chat with me today.Amy: Yeah, thank you for having me. I mean, I think it's important for us to recognize, even between the two of us that, I mean, obviously, you as a white man have benefited a lot in this space, and then even myself as, you know, that model minority whole thing, but growing up very adjacent to white people and just being ingrained in that culture and raised in that culture, you know, that we have those privileges and there's still parts of the conversation, I think, that are not captured by [laugh] by the two of us are the nuances as well, and so just recognizing that. And it's just a learning process. And I think that everyone could benefit from just realizing that you'll never know everything. And there's always going to be something to learn in all of this. And yes, it is hard, but it's something that is worthwhile to strive for.Corey: Most things worthwhile are. If people want to learn more about who you are, how you think about these things, potentially consider working with you, et cetera. Where can they find you?Amy: So, I am on Twitter. I am the queen of very, very long threads, I should just start a blog or something, but I have not. But in any case, I'm on Twitter. I am AmyChanta, so @A-M-Y-C-H-A-N-T-A.Our website is unicycle.co, if you're thinking about applying for a role, and working with me, that would be awesome. Or just, you know, reach out. I'd also just love to network with anyone, even if there's not an open position now. I just, you know, build that relationship and maybe there will be in the future. Or if not at Unicycle, then somewhere else.Corey: And we will, of course, put links to that in the [show notes 00:48:13]. Thank you so much, once again. I appreciate your time.Amy: Thanks for having me.Corey: Amy Chantasirivisal, Director of Engineering at Unicycle. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment pointing out that it's not about making an MVP of a bicycle that turns into a unicycle so much as it is work-life balance.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Wayne Haber, is the Director of Engineering at GitLab. Gitlab is a complete DevOps platform, to help teams improve development cycle time from weeks to minutes. Wayne is also a veteran of three successful startups (including GitLab) and has experience in multiple areas including healthcare, finance, and security. Wayne talks about how bringing transparency and constantly iterating leads to growth and fulfillment. He is excited about applying ML to improve user experience and loves the environment of continuous collaboration at Gitlab.

This week, Emily Freeman of Amazon and Phoebe Goh of NetApp join us to discuss Emily's new DevOps revolution keynote, DevOps for dummies and the people aspect of DevOps.

[00:00:32] Andrew tells us they shipped a new project at work this week they've been working on for a few months, and although it went pretty smoothly, he explains some bumps they had along the way and dealing with crunch time. Chris shares an issue and why he's been postponing the launch of the new Hatchbox. [00:04:13] We hear more about propagating the DNS and how long it took.[00:08:28] Andrew mentions using the Proxyman app and what it does. [00:09:15] Chris tells us about his new Mac, and he can't believe how fast it is![00:13:56] Andrew talks about some issues with installing Ruby 2.6.3 and building things in Docker on a new M1 Mac that a developer on his team just got.[00:17:24] Chris explains his upgrading issues on an older app he was working on this week and realized it was a Sass change he made. Ironically, Andrew ran into something very similar with Sass as well. [00:20:57] We hear about the Ember CLI Rails gem and Chris brings up that there is no solution on how to take an abandoned project like this and just keep maintaining it and he wishes there was a better solution.  [00:25:43] Andrew mentions every time you add a gem, you need to be aware of the amount of code debt you will have, and he shares what happened to him when he was a beginning developer. Chris explains why he would rather build it from scratch in the app to tailor it to exactly what they need. [00:29:48] Chris announces a new GoRails Screencast coming up with Kasper and what they'll be talking about.[00:35:25] Find out more about the awesome and very thorough tutorial on “Deploying a Rails application to Kubernetes” that you should check out! [00:39:25] Chris and Andrew chat about the importance of being Rails Developers and not working on DevOps stuff. Panelists:Chris OliverAndrew MasonSponsor:HoneybadgerLinks:Ruby Radar NewsletterRuby Radar TwitterProxymanGlassWireGoRailsGoRails-YouTube SassDeploying a Rails application to Kubernetes-By Marco ColliEmber CLI Rails-GitHubRubyConf 2021

About DannDann Berg is a Senior CloudOps Analyst at Datadog, and has nearly a decade of experience working in the cloud and optimizing multi-million dollar budgets. He is also an active member of the larger technical community, hosting the monthly New York City FinOps Meetup, and has been published multiple times in places such as MSNBC, Fox News, NPR, and others. When he's not saving companies millions of dollars, he's writing plays, and has had two full-lengh plays produced in New York City and China.Links: Datadog: https://www.datadoghq.com Personal Website: https://dannb.org LinkedIn: https://www.linkedin.com/in/dannberg/ Twitter: https://twitter.com/dannberg Monthly newsletter: https://dannb.org/newsletter/ Previous SITC episode with Dann Berg, Episode 51: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/episode-51-size-of-cloud-bill-not-about-number-of-customers-but-number-of-engineers-you-ve-hired/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. If there's one thing that I love, it is certainly not AWS billing, but for better or worse, that's where my career has led me. Way back in Episode 51, I had Dann Berg, the CloudOps analyst at Datadog. And now he's back for more. Things have changed. He's now a senior CloudOps analyst, and I'm hoping my jokes have gotten better. Dann, thanks for being bold enough to come out and find out.Dann: Yeah. I'm excited to see if these jokes have gotten better. That's the main reason for coming back.Corey: Exactly. Because it turns out that death, taxes, and AWS bills are the things that are inevitable and never seem to change.Dann: Yeah. They just keep coming. They never stop, and they're always slightly different than you expect. I guess, just like death and taxes.Corey: So, when we spoke back in, I want to say 2019 is when it aired, so probably that—ish—is when we had the conversation, if not a little bit before that, you were effectively a team of one, and as mentioned, had the CloudOps analyst title. Now, you're a senior CloudOps analyst, which I assume just means you're older. Is the team larger as well? What does that process look like? How has it evolved in the last couple years?Dann: Yeah, it's been interesting, especially being a single organization and that organization being Datadog, that to be able to grow the team a little bit. So, as you said, it was just me. Now, it's a total of four people, including myself, so three others. And, yeah, it's been interesting just in terms of my own professional development, being able to identify what needs to be done, how much capacity I have, and being able to grow it over time, especially in this fairly new space of being specifically focused on cloud cost billing. So, kind of that bridge between engineering and finance, which itself is kind of a fairly new space, still.Corey: It is. And my favorite part of having these conversations with folks who have no idea what this space is, is learning—when I was starting—out how to talk about this in a way that didn't lead down weird paths. It's, “Oh, you save money on Amazon bills? Can you help me save money on socks?” It's like, “No. Well, yes. Get the Prime card, it gives you 5% off. But no.” And yeah, I talk about camelcamelcamel and other ways of working around the retail side, but that's not really what I do.It's similar to back when I was doing SRE-style work. I made it a point never to talk about being someone involved in working in tech, or suddenly you're the neighborhood printer repair person. Similarly, you have, I guess, gone in a strange direction because you weren't, to my recollection, someone who had a strong SRE background. That's not where you came from in the traditional sense, is it?Dann: No, not an SRE background at all. Yeah, I mean, it's really interesting. So, talking about this space, I mean, people are calling it a lot of different things, cloud economics, the term FinOps—financial operations—is being used a lot, now—Corey: Cloud financial management is another popular one. Oh, swing a dead cat, you'll hit 15 different words, and I give—my advice on that, even though I hate some of the terms is, cool. If people are going to pay you to have a title, even if you think it's ridiculous, you can take the money or you can die on a petty naming hill and here we are.Dann: Yeah. And it's interesting because the role that I was hired for at Datadog was very much this niche, very specific role that I didn't realize was a niche, very specific role at the time. So previously, I was at a company and I was building out their data centers, so I was working with vendors, buying servers, sometimes going on-site, installing, racking those, dealing with RMAs. And I was getting more involved as their cloud usage was growing and bringing some of those hardware capitalization cost procedures to the cloud. And so I found myself in this kind of niche role in my previous company.And a Datadog, they basically had the exact same role that was dealing with all of the billing stuff around the cloud—kind of from an engineering perspective because it was on the engineering team—but working closely with finance, and I was like, “Oh, these are the skills that I have.” And it kind of fit perfectly. And it wasn't until after I got to Datadog and was doing more research about this specific space that I discovered just how wide open it was. And I mean, meeting you was one of the earliest things that I did in the industry. Discovering the FinOps Foundation and a few other things has kind of like opened my eyes to this as an actual career path.Corey: It's an expensive problem that isn't going away anytime soon, and it is foundational and core to the entire rest of how companies are building things these days. My argument has been for a while that when it comes to cloud, cost and architecture are the exact same thing. You don't have the deep SRE architect background, but you're also now a member of a four-person team. Does everyone in the team have the same skill set as you, or do you wind up effectively tagging in subject matter experts from different areas? How is the team composed? People love to ask me this question, and I strongly believe there's no one way to do it. But what's your answer?Dann: Yeah, I mean, the team works very much in terms of everybody kind of taking on tasks that they need to do, but we did hire for specific skill sets when we tried to find people. So, the first person that we hired, we wanted them to have more of a developer engineer type background, writing code, stuff like that. The third hire, we were looking for somebody that was more of a generalist. I've seen myself more as a generalist in the space; anything that's going on, I can pick it up and make some progress on it and build something out. And then the fourth person, we were lacking some of the deeper FP&A or FinOps experience, and so we found somebody with more of that kind of background and less of the engineering experience, but they were eager to, kind of, move from finance into more of an engineering role. And I feel like this is the perfect role for that because I feel like there are a lot of non-engineers that want to break into engineering and don't really know how to do it. And if you are in finance, in FP&A, finding one of these more cloud-cost-optimization-specific roles is the great way to bridge that gap, I feel.Corey: The last time we spoke, I was independent, doing this all myself, and it turns out that taking all of the things that make me and trying to find those in other people is a relatively heavy lift, even if you discount the things like ‘obnoxious on Twitter.' So, how do you start decomposing that? Well, now we're a dozen people and we've found ways to do it. But by and large in our experience, for the way that we interact—and I want to get to that in a second—is that it's easier for us to teach engineers how finance works than it is the opposite direction. And there are exceptions to that, and as we scale, I can easily see a day in the near future where that is no longer the case.However, we also have two very specific styles of engagement. We do our cost optimization projects, where we go into an environment and, “Oh, fix this. Turn that thing off. Do you really need eight copies of those four petabytes of data? Oh, you didn't realize they were there. Great, maybe delete it.” And we look like wizards from the future and things are great.The other project that we do is contract negotiation with AWS, especially at large scale. It's never as simple as people would have you believe because, “Oh, you're doing co-marketing efforts, and you have a very specific use case, and there are business partnerships on 15 different levels, and that all factors into how this works.” It's nuanced and challenging, and of course, because it's a series of anecdata, I can't really tell too many stories in public about that. But those are the two things that we wind up focusing on. You are focusing on a very different problem.You're not moving from company to company, basically reimplementing the same global problem, solving it locally for them. You are embedded in an account for the duration, almost four years now by my count. And, “Okay, I guess I could just do a whole bunch of cost optimization projects on a quarterly basis in an environment like that,” doesn't seem like it solves the problem in any meaningful way. What does your team do?Dann: Yeah. Well, I mean, that's such an interesting question. Just in terms of—yeah, if you're doing consulting, you're starting from square one every time you get a new contract, a new engagement, and being at the same company for, like you said, about four years, going on four years now, you really have a chance to dive in and think about, “Okay, what does it mean to work cloud cost optimization into just the regular business cycle of how it works?” Because I mean, you have the triangle that everybody's familiar with: things can either be cheaper, faster, efficient and at different stages in the product lifecycle, you want to be focusing on these areas, more or less. And so, on our team, the different things that I'm thinking about is, first is visibility, is you want to provide engineers visibility into their cost. And not just numbers, right? Actionable visibility where if something needs to change, they need to do something, they know what that is.And a lot of the times, that means not just costs, but also efficiency. So, these are the metrics that this particular application should be scaling against. As this application grows, as usage grows, are we remaining as cost-efficient? Then there's also the piece—as you're saying—like discovering things within the infrastructure that, “Hey, if we make this change, or if you turn this off, if we do things this way, we'll save a bunch of money. Let's do those.”There's things like reservations, committed use discounts for GCP, all of those kinds of things we manage. And then dealing closely with verifying our bill, working with finance—FP&A—on cost modeling forecasting, both short-term—like, within a month; like, what are we going to be at the end of this month and it's the 10th right now?—and also, what does our next quarter look like? What are our next two years look like? And that bleeds into the contract negotiations, those kind of things as well.So, I mean, it's setting up the cycles of how do you prioritize this work? What is the company focusing on at the time? And what can you do when the company is not focusing explicitly on deciding to save money?Corey: One of the more interesting aspects of my work that I didn't expect is, whenever I wind up starting an engagement, or even in the prospect stage, I love asking the dumbest possible questions I can think of because it turns out they're not. And the most common one that I always love to start with is, “Oh, okay. Your AWS bill is too high. Why do you care?” And that often takes people aback, but once you dig down underneath the surface just a little bit, it becomes pretty clear that the actual goal is not that it's too much money—because spoiler, payroll always cost more than infrastructure—instead, it's, “How do I think about this? How do I rationalize what the additional costs are going to be per thousand monthly active users or whatever metric it is you're choosing to use?”And how do you wind up forecasting that because the old days of data centers where you—“Well, we're going to spend a boatload of money, and then we'll have capacity for the next, ehh, two years, maybe down to eighteen months, depending on growth,” that's easier for companies to rationalize around, rather than this idea of incremental cost on a per-unit basis, but not exactly because it also turns out that architecture changes, problems of scale, AWS pricing changes from time to time, all tend to impact that. What I think is not well understood in this space is that yeah, if you have a 20% overage this month, people are going to have some serious questions, but they're also going to have those same questions if you're 20% low.Dann: Yeah. I mean, understanding why people care about the cost is definitely the first step because with a single company, so it's just constantly looking at the numbers rather than understanding exactly what motivations a company has to contact somebody like you, like a consultant, right? Because usually, I imagine that it's going to be a bill, maybe two bills, three bills come in, and they keep going up and up and up, and they need to go down. And they're going to have an explicit reason why it needs to go down; finance is going to say, “Margins are x, y, and z,” or, “Revenue has done this; our costs can't do this.” There's going to be explicit reasons because if there aren't reasons, then they shouldn't necessarily be focusing on costs at that moment in time.What you want to do is have—I mean, this is way more complicated than just saying it out loud, but have a culture of cloud cost mindfulness, where people aren't just spinning up resources willy nilly. But also, my goal is for people not to have to really think about cost that much other than just in a way that helps them do their work. Because I mean, I want engineers to be able to build stuff and build stuff fast—that's what the cloud is all about—but I also want to be able to do it in a way that isn't inappropriately high in cost.Corey: I have my thoughts on this, and I've shared them before and I'll dive into them again, but how do you approach that? If Datadog makes a grievous error and hires me to write code somewhere as an engineer, what is the, I guess, cost approach training for me as I wind up going through my onboarding as part of an SRE team or an application team?Dann: I mean, this feels so basic as to not even be the right answer, but honestly, visibility is the easiest and best thing that you can give people, and so we've built out some visibility reports that engineers get on a regular basis. We also meet with our top—what is it—ten or fifteen spending internal engineering teams on a monthly basis to go over those costs so that they understand what they're looking at so that we understand the context behind it, so that we can understand what's on the roadmap going forward so that when things in the cost happen, we're aware. And then we're just staying on top of things. And if we have questions, we have an open dialogue with engineers and things like that.In an ideal space, it would be great to have cost, I guess, more fit into the product development lifecycle in a more deeply ingrained way, but at the same time, I really don't want to serve as a gatekeeper. Our goal is not to stop any sort of engineering process. And we haven't needed to do anything like that although I guess every company is going to be different in terms of what their needs are. But yeah, I'm totally happy to being a little bit more reactionary in terms of looking at the numbers and responding, and then proactive just in terms of the regular communication with people.Corey: I tend to take the perspective that engineers need to know enough about cost to maybe fill an index card at most because you don't want them, I guess, over-fixating on it. Left to my own devices in my personal account, I'll see a $7 a month bill and, “Oh, I'm going to spend two weeks knocking that down to $4.” And of course, I can do it, but is that the best use of my time? Absolutely not.Very often what is a lot of money to an engineer is absolutely not to the business. And vice versa when you bring in a data science team; it's, “Oh, yeah, we need at least four more exabytes of data because we never learned to do a join properly.” Yeah, maybe don't do that. Understanding the difference between those two approaches is key. But I've always been of the mindset that I would rather bias for letting developers build and experiment and have things that catch outsized things quickly, then trying to wind up putting a culture of fear around cost because I'd much rather see whether the thing they're trying to build is possible to build, then go back and optimize it later, once that's proven out. But again, this is a nuanced thing.Everyone seems to think I have this back pocket answer that will apply to all companies. And you've been doing this at Datadog for almost four years with a team of people. I am an outsider; I see the global trend, I see what works in different ways in different companies, but the idea that I can sit down and say, “Oh. Well, clearly the thing you're doing is completely wrong because that's not how I think about it,” is the hallmark of a terrible consultant. There are reasons that things are the way that they are and it's generally not that people are expecting to do a terrible job today. You know, unless they work in the Facebook ethics department, which is neither here nor there.Dann: Yeah, I mean, like I said, the product lifecycle, when you're building something new, you want to go as fast as possible. When you're launching it, you want it to be as reliable as possible. Once you're launched, once you're reliable, then you can start focusing on costs is, kind of like, not the universal rule, but kind of the flow that I tend to see. So, as you're at a company that is regularly innovating, creating new products, going through that cycle, you're going to have these kind of periods.As well as you have the products that have been around. There's a lot of legacy code, there's a lot of stuff going on, that maybe isn't the best, or some efficiency work that has been deprioritized for whatever reason, that maybe it's time to start considering doing this. So, keeping track of all of that. And like I said, if for whatever reason the business wants to focus on cloud cost efficiency, or a team has decided that in a particular quarter or for a particular reason they want to focus on that, being able to assist as much as you can, being able to save all that work so that there's kind of like a queue that you can go to when it is time to focus on cost efficiency stuff.Corey: So, here's a fun one for you. As of the time of this recording, it's a couple weeks old, but if you're anything like what we do here for some of our more sophisticated clients, we do occasionally build out prediction models, models of economics that wind up defining how some architectural patterns should be addressed, et cetera, et cetera. What's always fun is the large clients who have this significant level of spend on an outlier service. Every once in a while—it was great that we got to do a deep dive into the Washington Post's use of Lambda because normally, Lambda is a rounding error on the bill; they had a specific challenge and they did a whole blog post on this for the AWS blog. I believe the Monitoring Tools blog, but don't take that at face value; I never remember which AWS blog is which because AWS doesn't speak with a single voice on anything.But yeah, most of the time is block, tackle, baseline stuff that is the big driver of spend, but a few weeks ago, they change the pricing dimensions for S3 intelligent tiering, where there's no longer a monitoring charge for objects that are smaller than 128 kilobytes, and there's no 30-day minimum. So, the fact that those two things went away removed almost every caveat that I can picture for using S3 intelligent tiering, which means that for most use cases, that should now be the default. I imagine you caught that change as well, since that's one of those wake up and take notice, no matter what time of the world [laugh] it is where you are when that gets dropped. How did that change your modeling? Or did that not significantly shift how you view any of this?Dann: No, I mean, I think part of our role within the organization is to pay attention to stuff like that, and then you just have those conversations with the teams that I know were either exploring intelligent tiering. We do some pricing modeling for different products, S3 storage for different types, so updating those and being like, “Hey, this might be something we want to actually use and explore now.” Similar and I guess, more of something that I actively worked on that I consider in the same category is when Amazon announced savings plans as replacing convertible reservations. Because at first they announced, and being like, “Okay, well, it's going to automatically rebalance between… different instance families across regions, too”—which convertible RIs could never do it—“And it's going to be the exact same price for a compute savings plan as a convertible RI.” And we were kind of like, what's the catch? And we spent a few weeks doing a deep dive working with our data science team, kind of like being, “Where is the catch here?”Corey: Yeah, the real catch is that you can't sell it on the secondary market if it—Dann: Yeah.Corey: —turns out you bought the wrong thing, which if that's your Plan A, then good luck.Dann: Yeah. We definitely don't use that secondary market. I don't have as much experience there, although I'm sure some people can use it to their advantage.Corey: Almost no one does. In fact, the reason that it exists—my pet theory—is that once upon a time, companies would try and classify some of the reserved instance purchases as capital expenditures, which there has since been guidance from regulatory authorities not to do that. But at the time, the fact that you could sell it to a third-party on the secondary market would help shore up that argument. If you're listening to this, and you're classifying some of your RIs as CapEx, please don't do that. Feel free to reach out to me, I can dig out the actual regulation and send it to you. There are two of them. It's a nuanced topic. If you're listening to this and have no idea what I'm talking about, God, do I envy you.Dann: [laugh]. Yeah, definitely don't do that. [laugh].Corey: There was a lot that was interesting about savings plans. When I was read in the month or so in advance of them being announced, it was, “Great. I want to see this and this and these other things, too.” And some of those things came to pass. It was extended to work with Lambda.Now, I don't believe that is financially useful in almost every case, but it doesn't need to be because so much of cloud economics from where I sit is psychological in nature, where, “Oh, we have this workload that lives on EC2 instances and we want to move it to Lambda, but we already bought the reserved instances so we're not going to do it because of sunk cost fallacy.” Which is not much of a fallacy when it's that kind of money, in some cases. Okay, great. Now, if it can migrate to Lambda and still wind up getting the discounts you've paid for it, you have removed an architectural barrier. And that's significant.Now, I want to see that same thing apply to oh if you move from EC2 to RDS, or DynamoDB or anything else, that should be helpful, too. But whatever you do, don't do what SageMaker did and launch their own separate savings plan that is not compatible with the compute savings plans, so effectively, it's great; you're locked-in architecturally to one or the other because machine learning is, once again, a marvelously executed scam to sell pickaxes into a digital gold rush.Dann: I mean, I like savings plans a lot and we've been slowly, as convertible RIs have expired, replacing them with savings plans. And I think that it is pushing the other cloud providers forward—because we're definitely multi-cloud—and so that's really useful and I hope more people will take on the compute savings plan type model, just because it makes our lives so much easier. Or it makes my life so much easier in terms of planning it, selling the commitment internally, just everything about it has made my life easier. So, I mean, how many years later are we? I definitely haven't found any big gotchas, I guess, from the secondary market. But that doesn't really impact me.Corey: Yeah, I spent a lot of time looking forward, too, doing deep analyses of okay, for which instance classes in which regions is there a price discrepancy? And I finally got someone to go semi on record and say, “Yeah. There should not be any please ping us if you find one.” “Oh, okay, great. That is enough for me to work with.”Dann: Exactly, we got that, too. I didn't believe it so we were downloading price sheets and doing comparisons, doing all that stuff.Corey: Oh, trust but verify. And when we're talking this kind of money, I don't trust very far. They make mistakes on billing issues from time to time. And I get it; it's hard, but there are challenges here and there. I am glad you mentioned a minute ago that you are multi-cloud because my position on that has often been misconstrued.I think that designing something from day one to work on multiple cloud providers is generally foolish. I think that unless you have a compelling reason not to go all-in on one cloud provider, that's what you should do. Pick a cloud—I don't care which—and go all-in. Conversely, you have a product like Datadog where your customers are in multiple clouds, and first, no one wants to pay egress to send all the telemetry from where they are into AWS, and secondly, they're not going to put up, in many cases, with their data going to a cloud provider they have explicitly chosen not to work with, so you have to meet your customers where they are. In your case, it is absolutely the right thing to do. And Twitter often gets upset and calls me hypocrite on stuff like this because Twitter believes that two things that take opposite visions cannot possibly both be true, but the world is messy.Dann: Yeah. And I mean, the nice thing about us being in multiple clouds is we are our own biggest user. And that's actually one of the reasons why I love working at Datadog is because I get to use Datadog all the time. And not only that, Datadog is on everything and we have all of our products. I'm very spoiled [laugh] with all of this. But I mean, we are running in these different cloud providers; we are using Datadog in those different cloud providers, and that is just helping everything overall, too. In addition to supporting customers that are in each cloud because that is a huge reason as well.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: One of the problems that I keep running into across the board is that with things like Datadog—and again, not to single you out; every monitoring vendor to some extent has aspects of this problem—it's that when I'm a customer and I'm hooking my accounts up to Datadog, I want you to tell me about things that are going on, but the CloudWatch charges can be so egregious on the customer side, where it is bizarre and, frankly, abhorrent to me when I wind up paying more for the CloudWatch charges than I am for Datadog. And let's be clear here; I am, in fact, a Datadog customer. I pay you folks money. Not a lot of money, but I pay you money because I have certain things that I need to know are working for a variety of excellent reasons.And the problem that I keep smacking into on this is—it's not your fault; there's not anything you can do. In fact, you are one of the better providers as far as not only not being egregious with the way that you slam the CloudWatch endpoints, but also in giving guidance to customers on how to tune it further. And I really wish that more folks in your space would do things like that. It always bugs me when I wind up using a tool that tries to save money that in turn winds up costing me more than it saves.Dann: Yeah. Yeah, it's tricky there. I have less experienced myself setting up Datadog and running it in my own infrastructure as I'm more digging deep into the cost stuff and us using the cloud, so I can't speak to that specifically. But yeah, you're not the first person that I've heard have that experience. [laugh].Corey: And again, it's not your fault at all. I've been beating up the CloudWatch team for years on this, and I will continue to do so until I'm safely dead, which—depending on Amazon's level of patience—might be in mere minutes.Dann: In the larger-picture-wise, we have to remember that we're super early in the cloud adoption, even looking at the cloud economics FinOps cloud cost optimization world. I feel like most businesses at this stage in their journey are still in data centers and they're dealing with the problem of how do we move to the cloud and do it cost-efficiently? How do we set everything up? And that's where the world is right now.And I think that dealing with, “Okay, we are one hundred percent running in the cloud. What are the processes that we have in place? How do we think of finance and the finance organization not through the lens of ‘we once had data centers and now we don't,' but how do we look through that in the lens of ‘okay, we are cloud-native from day one? What does the finance department look like?'” And dealing with those problems is really interesting because Datadog has never been in a data center. We are cloud-native from the very beginning, and so it was interesting for me to join the company and build up a lot of these processes because it is different than what a lot of other people were dealing with and doing. And it presents some really interesting problems and questions that I think are going to be the foundation for the next decade of building companies and operating in the cloud.Corey: I always love having conversations with folks who are building out teams to handle these things because usually the folks I keep talking to, or who want to have conversations like this are building tools themselves to solve this problem through the miracle of SaaS, where they will bend over backwards to avoid ever talking to a customer. And we're all dealing with the same AWS APIs; there's not that much of a new spin you can put on most of these things. But understanding what customers are actually trying to do instead of falling down the rabbit hole trap of, “Hey, turn off those idle instances that are all labeled ‘drsite' because you probably don't need them,” is foolish. And after a few foolish recommendations, tooling doesn't get there. I am a big believer that tools can assist the process and narrow down what to look at.I believe they shouldn't have to exist; I think that the billing dashboard should be a hell of a lot better natively than having to pay a third party to make sense of it for me. But by and large, I do believe this is a problem that is best solved from a consultative approach. When I started this place, I was planning to build out some software, tried doing it—called DuckTools—and wound up mothballing the whole thing because what we were building was not what the industry claimed to want and, frankly, educating people into a position where then they see the value and only then will they buy is never been a game that I wanted to play.Dann: Yeah, I really liked that article that you guys published about exploring that product and the reason why you decided not to pursue it. But it's super interesting in terms of where the industry is going and building out those tools because I found that there isn't really any new thing that you can do with the tools. All the tools that exist for looking at your costs are largely the same. The main differences that I've seen is that the UI is slightly different and they have different sales teams. And if the sales teams are better, they're going to get more of the market share. And if the sales teams are not as good, it's going to be a smaller market share. And it's weird, too, be in this industry for as long as we have been, and seeing okay, well, Andreessen Horowitz just funded this new company, and this other company got invited into Y Combinator, or all of these things that are happening, and I'm kind of like, okay, but what is this tool really doing differently? And there are a few of them that are; that are doing something innovative and different, but there's also a few that are just like, this is a space where people are in, there's money here, we're doing the same thing, but we got our sales team, and we'll carve out our little corner, and then we'll get acquired, and that'll be that. Although I guess we're just at that stage of innovation in this space, I guess.Corey: Yeah, I have no earthly idea what the story is around how these companies plan to differentiate because it seems to me that they're directly attempting to compete with Cost Explorer, which—Dann: Yeah.Corey: —it's taken some time for that thing to improve to the point where it is now and it'll take further time for it to improve beyond it, but long-term, I don't think you're going to outrun AWS on a straight line like that.Dann: Yeah, I mean, when you work for one of these third-party cost tooling things, and you're working with one of your customers, and they're like, “How do I view this?” And it's kind of like, that is the easiest thing to find in Cost Explorer as well, it's—I can't imagine being like, “Well, you should pay me thousands, tens of thousands, hundreds of thousands of dollars a month to view it here,” when Cost Explorer is free. And I think Cost Explorer, it doesn't do everything, but it's gotten a lot better at what it does, and it could probably solve 90% of people's problems without using a third-party tool.Corey: You are at significant scale in multiple clouds, so the answer that these companies always give is, “Ah, but we provide a single dashboard so that you can look at costs across multiple providers in one place.” Is that even slightly useful to you?Dann: Man, if you need dashboards, get a dashboard tool. Don't get this crazy cost analysis tool. I mean, there are some great dashboard solutions that you can get where you can connect your detailed billing, cost and usage report—whatever cloud provider is calling it, but, like, that really detailed gigabytes per hour report—and then visualize it, build reports, do all that kind of stuff because that's not something that the tooling does well right now, in terms of building out cost dashboards and stuff. But that's also right now. It could in the future.Corey: Yeah. If you're a BI tool, wind up passing out templates that normalize these things? I am so tired of building it all from scratch in Tableau myself. If you're Tableau, sell me a whole bunch of things that I can use to view this stuff through, so I don't have to wind up continually reinventing that particular wheel.Dann: Yeah.Corey: Oh, I like your approach. I didn't know the answer when I was asking the question. I was about to learn something if you'd gone the other direction, but nope, but it's good to know that my impressions remain intact.Dann: Yeah, I mean, I've used different tools in the past. Again, I hesitate to name any of them, but there's a few in this space that I feel like everybody—if they're in this space, they know which tools I'm talking about—Corey: Yes, we do.Dann: —and… yeah, I've used them. They're okay—a few of them are okay, a few of them are better than others, but I mean, I was trying to evaluate the value-add over me manually setting some things up and having some sort of visualization, and just the value-add in terms of what they were charging, even if it was like a significantly smaller percent of the bill because that alone, like, percent of bill is such a difficult cost model—Corey: Oh—Dann: —to do.Corey: I hate that. Pricing is hard. Let's start there.Dann: Yeah. Yeah. Yeah, yeah.Corey: I hate the percent of bill because then it's, “Let me get this straight. I'm paying you a percentage of things like data transfer charges that I know are fixed, that I can't optimize? I'm paying you a percentage of my AWS enterprise support subscription? I'm paying you a percentage of the marketplace?” And so on and so forth. And it doesn't work. At some point of scale as well it's, I could hire a team of 20 people and save money versus what you're charging me. The other side of it though, “Ah, we'll charge you percentage of savings.” Well, then you wind up with people doing a whole bunch of things like before they bring you in, they'll make a bunch of ill-advised reserved instance purchases or savings plan purchases you have to then unwind after the fact. When I was setting this place up, I looked long and hard at different billing models and the only thing I found that worked is fixed fee. The end. Because at that point, suddenly everyone's on board with, “Hey, let's solve the problem and then get out as soon as possible.” We're not trying to build ourselves a forever job nestled in the heart of your company. And it's the only model I found that removes a whole swath of conflicts of interest. And that's the hard part. We have no partners with anyone in this space—including AWS themselves—just because as soon as we do, it becomes extremely disingenuous when we suggest doing something for your sake that happens to benefit them, such as, “Maybe back that S3 bucket up somewhere.” Well, okay, if we're partnered with them, does that mean we're trying to influence spend in the other direction? And it just becomes a morass that I never found it worth the time to deal with.Dann: Yeah, I—Corey: But that doesn't work for SaaS.Dann: Yeah, that makes a lot of sense. And I haven't actually thought about pricing model for consulting in this space that closely, but I mean, when you're charging a percent of bill or percent of savings, you have the opportunity to screw the customer, right, through all the things that you were saying. If you charge a fixed fee, you have the possibility of undervaluing yourself, which the only one that's screwed in that case is you, potentially, and if you're okay with that risk and you're okay with those dollars, that's great. Because yeah, if you're able to be like, “Okay, here's the services that I do, here's the fixed costs.” “Done.” “Done.” That just sets everybody's expectations for the relationship in a much better way that you're not constantly worried about, like, upsells and other things that might happen along the way that screws the customer.Corey: And that's the hardest part, I think, is that people lose sight of the entire customer obsession piece of it. That's one of the things Amazon gets super right. I wish more companies embrace that. Dann, I want to thank you for taking so much time out of your day to suffer my slings, and arrows, and half-formed opinions. If people want to learn more about who you are and what you're up to, where can they find you?Dann: Yeah, I have a website you guys can go to that links everywhere else. It is dannb.org. And I spell my name with two ns, so D-A-N-N-B dot org. And I have LinkedIn, I have Twitter, I have a monthly newsletter that is not really about FinOps or anything, but I really enjoy it; I've been doing it for a year, now, that you should sign up for.Corey: And links to that will, of course, be in the [show notes 00:36:26]. Dann, thanks again for your time. I really appreciate it.Dann: Yeah. Thanks so much for having me again. It's been a blast.Corey: It really has. Dann Berg, senior CloudOps analyst at Datadog. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment featuring a picture of several corkboards full of post-it notes and string, and a deranged comment telling me that you have in fact finally found the catch in savings plans.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

In this TCP Talks episode, Justin Brodley and Jonathan Baker talk with Jonathan Heiliger, co-founder and partner at Vertex Ventures: an early-stage venture capital firm backing innovative technology entrepreneurs.  Earlier in his career, at just 19, Jonathan co-founded web hosting provider GlobalCenter and served as CTO. He went on to hold engineering roles at Walmart and Danger, Inc., the latter of which was acquired by Microsoft. He was also Vice President of Infrastructure and Operations at Facebook (now Meta), and a general partner at North Bridge Ventures. The latter firm's portfolio included Quora, Periscope, and Lytro (which has been acquired by Google.) At Vertex Ventures, Jonathan has helped cutting-edge companies like LaunchDarkly and OpsLevel revolutionize the tech space with continuous delivery and IT service management solutions. Jonathan shares his insights into the shifting market of IT services and explains why decentralizing infrastructure management can help digitally native companies operate at a faster pace. According to Jonathan, the question of IT service infrastructure isn't being adequately addressed. Without properly defining service ownership, businesses looking to scale run the risk of siloing critical knowledge, and losing track of services networks.  Jonathan also discusses his own experiences running infrastructure at Facebook (oops, Meta), the merits of both centralized and decentralized IT services management, and how he and his partners at Vertex Ventures approach new investments.   Featured Guest

VTOC. Know it. Use it... Love it? If you're putting data on volumes, you're using a Volume Table of Contents to keep track of what's where, and like most things in Z, knowing just how that information gets put to use can help aid in performance and reliability. Let the experts, Kevin Miner and Mike Koester take us through the importance of this oft-overlooked aspect of storage management. 

After a little async Ruby chat and developer morality struggle, Chris explains how macOS Monterey has lapped Linux with a critical workstation feature.

About Betty Betty Junod is the Senior Director of Multi-Cloud Solutions at VMware helping organizations along their journey to cloud. This is her second time at VMware, having previously led product marketing for end user computing products.  Prior to VMware she held marketing leadership roles at Docker and solo.io in following the evolution of technology abstractions from virtualization, containers, to service mesh. She likes to hang out at the intersection of open source, distributed systems, and enterprise infrastructure software. @bettyjunod  Links: Twitter: https://twitter.com/BettyJunod Vmware.com/cloud: https://vmware.com/cloud TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You know how git works right?Announcer: Sorta, kinda, not really Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Periodically, I like to poke fun at a variety of different things, and that can range from technologies or approaches like multi-cloud, and that includes business functions like marketing, and sometimes it extends even to companies like VMware. My guest today is the Senior Director of Multi-Cloud Solutions at VMware, so I'm basically spoilt for choice. Betty Junod, thank you so much for taking the time to speak with me today and tolerate what is no doubt going to be an interesting episode, one way or the other.Betty: Hey, Corey, thanks for having me. I've been a longtime follower, and I'm so happy to be here. And good to know that I'm kind of like the ultimate cross-section of all the things [laugh] that you can get snarky about.Corey: The only thing that's going to make that even better is if you tell me, “Oh, yeah, and I moonlight on a contract gig by naming AWS services.” And then I just won't even know where to go. But I'll assume they have to generate those custom names in-house.Betty: Yes. Yes, I think they do those there. I may comment on it after the fact.Corey: So, periodically I am, let's call it miscategorized, in my position on multi-cloud, which is that it's a worst practice that when you're designing something from scratch, you should almost certainly not be embracing unless you're targeting a very specific corner case. And I stand by that, but what that has been interpreted as by the industry, in many cases because people lack nuance when you express your opinions in tweet-sized format—who knew—as me saying, “Multi-cloud bad.” Maybe, maybe not. I'm not interested in assigning value judgment to it, but the reality is that there are an awful lot of multi-cloud deployments out there. And yes, some of them started off as, “We're going to migrate from one to the other,” and then people gave up and called it multi-cloud, but it is nuanced. VMware is a company that's been around for a long time. It has reinvented itself in a few different ways at different periods of its evolution, and it's still highly relevant. What is the Multi-Cloud Solutions group over at VMware? What do you folks do exactly?Betty: Yeah. And so I will start by multi-cloud; we're really taking it from a position of meeting the customer where they are. So, we know that if anything, the only thing that's a given in our industry is that there will be something new in the next six months, next year, and the whole idea of multi-cloud, from our perspective, is giving customers the optionality, so don't make it so that it's a closed thing for them. But if they decide—it's not that they're going to start, “Hey, I'm going to go to cloud, so day one, I'm going to go all-in on every cloud out there.” That doesn't make sense, right, as—Corey: But they all gave me such generous free credit offers when I founded my startup; I feel obligated to at this point.Betty: I mean, you can definitely create your account, log in, play around, get familiar with the console, but going from zero to being fully operationalized team to run production workloads with the same kind of SLAs you had before, across all three clouds—what—within a week is not feasible for people getting trained up and actually doing that. Our position is that meeting customers where they are and knowing that they may change their mind, or something new will come up—a new service—and they really want to use a new service from let's say GCP or AWS, they want to bring that with an application they already have or build a new app somewhere, we want to help enable that choice. And whether that choice applies to taking an existing app that's been running in their data center—probably on vSphere—to a new place, or building new stuff with containers, Kubernetes, serverless, whatever. So, it's all just about helping them actually take advantage of those technologies.Corey: So, it's interesting to me about your multi-cloud group, for lack of a better term, is there a bunch of things fall under its umbrella? I believe Bitnami does—or as I insist on calling it, ‘bitten-A-M-I'—I believe that SaltStack—which I wrote a little bit of once upon a time, which tells me you folks did no due diligence whatsoever because everything I've ever written is molten garbage—Betty: Not [unintelligible 00:04:33].Corey: And—so to be clear, SaltStack is good; just the parts that I wrote are almost certainly terrible because have you met me?Betty: I'll make a note. [laugh].Corey: You have Wavefront, you have CloudHealth, you have a bunch of other things in the portfolio, and yeah, all those things do work across multiple clouds, but there's nothing that makes using any of those things a particularly bad idea even if you're all-in on one cloud provider, too. So, it's a portfolio that applies to a whole bunch have different places from your perspective, but it can be used regardless of where folks stand ideologically.Betty: Yes. So, this goes back to the whole idea that we meet the customers where they are and help them do what they want to do. So, with that, making sure these technologies that we have work on all the clouds, whether that be in the data center or the different vendors, so that if a customer wants to just use one, or two, or three, it's fine. That part's up to them.Corey: The challenge I've run into is that—and maybe this is a ‘Twitter Bubble' problem, but unfortunately, having talked to a whole bunch of folks in different contexts, I know it isn't—there's almost this idea that you have to be incredibly dogmatic about a particular technology that you're into. I joke periodically about the Rust Evangelism Strikeforce where their entire job is talking about using Rust; their primary IDE is PowerPoint because they're giving talks all the time about it rather than writing code. And great, that's a bit of an exaggeration, but there are the idea of a technology purist who is taking, “Things must be this way,” well past a point of being reasonable, and disregarding the reality that, yeah, the world is messy in a way that architectural diagrams never are.Betty: Yeah. The architectural diagrams are always 2D, right? Back to that PowerPoint slide: how can I make pretty boxes? And then I just redraw a line because something new came out. But you and I have been in this industry for a long time, there's always something new.And I think that's where the dogmatism gets problematic because if you say we're only going to do containers this way—you know, I could see Swarm and Kubernetes, or all-in on AWS and we're going to use all the things from AWS and there's only this way. Things are generational and so the idea that you want to face the reality and say that there is a little bit of everything. And then it's kind of like, how do you help them with a part of that? As a vendor, it could be like, “I'm going to help us with a part of it, or I'm going to help address certain eras of it.” That's where I think it gets really bad to be super dogmatic because it closes you off to possibly something new and amazing, new thinking, different ways to solve the same problem.Corey: That's the problem is left to our own devices, most of us who are building things, especially for random ideas, yeah, there's a whole modern paradigm of how I can build these things, but I'm going to shortcut to the thing I know best, which may very well the architectures that I was using 15 years ago, maybe tools that I was using 15 years ago. There's a reason that Vim is still as popular as it is. Would I recommend it to someone who's a new user? Absolutely not; it's user-hostile, but back in my days of being a grumpy sysadmin, you learned vi because it was on everything you could get into, and you never knew in what environment you were going to be encountering stuff. These days, you aren't logging in to remote systems to manage them, in most cases, and when it happens, it's a rarity and a bug.The world changes; different approaches change, but you have to almost reinvent your entire philosophy on how things work and what your career trajectory looks like. And you have to give up aspects of what you've considered to be part of your identity and embrace something new. It was hard for me to accept that, for example, Docker and the wave of containerization that was rolling out was effectively displacing the world that I was deep in of configuration management with Puppet and with Salt. And the world changes; I said, “Okay, now I'll work on cloud.” And if something else happens, and mainframes are coming back again, instead, well, I'm probably not going to sit here railing against the tide. It would be ridiculous to do that from my perspective. But I definitely understand the temptation to fight against it.Betty: Mm-hm. You know, we spend so much time learning parts of our craft, so it's hard to say, “I'm now not going to be an expert in my thing,” and I have to admit that something else might be better and I have to be a newbie again. That can be scary for someone who's spent a lot of time to be really well-versed in a specific technology. It's funny that you bring up the whole Docker and Puppet config management; I just had a healthy discussion over Slack with some friends. Some people that we know and comment about some of the newer areas of config management, and the whole idea is like, is it a new category or an evolution of? And I went back to the point that I made earlier is like, it's generations. We continually find new ways to solve a problem, and one thing now is it [sigh] it just all goes so much faster, now. There's a new thing every week. [laugh] it seems sometimes.Corey: It is, and this is the joy of having been in this industry for a while—toxic and broken in many ways though it is—is that you go through enough cycles of seeing today's shiny, new, amazing thing become tomorrow's legacy garbage that we're stuck supporting, which means that—at least from my perspective—I tend to be fairly conservative with adopting new technologies with respect to things that matter. That means that I'm unlikely to wind up looking at the front page of Hacker News to pick a framework to build a banking system in, and I'm unlikely to be the first kid on my block to update to a new file system or database, just because, yeah, if I break a web server, we all laugh, we make fun of the fact that it throws an error for ten minutes, and then things are back up and running. If I break the database, there's a terrific chance that we don't have a company anymore. So, it's the ‘mistakes will show' area and understanding when to be aggressive and when to hold back as far as jumping into new technologies is always a nuanced decision. And let's be clear as well, an awful lot of VMware's customers are large companies that were founded, somehow—this is possible—before 2010. Imagine that. Did people—Betty: [laugh]. I know, right?Corey: —even have businesses or lives back then? I thought we all used horse-driven carriages and whatnot. And they did not build on cloud—not because of any perception of distrust; because it functionally did not exist at the time that they were building these things. And, “Oh, come out into the cloud. It's fine now.” It… yeah, that application is generating hundreds of millions in revenue every quarter. Maybe we treat that with a little bit of respect, rather than YOLO-ing it into some Lambda-driven monster that's constructed—Betty: One hundred—Corey: —out of popsicle sticks and glue.Betty: —percent. Yes. I think people forget that. And it's not that these companies don't want to go to cloud. It's like, “I can't break this thing. That could be, like, millions of dollars lost, a second.”Corey: I write my weekly newsletters in a custom monstrosity of a system that has something like 30-some-odd Lambda functions, a bunch of API gateways that are tied together with things, and periodically there are challenges with it that break as the system continues to evolve. And that's fine. And I'm okay with using something like that as a part of my workflow because absolute worst case, I can go back to the way that my newsletter was originally written: in Google Docs, and it doesn't look anywhere near the same way, and it goes back to just a text email that starts off with, “I have messed up.” And that would be a better story than most of the stuff I put out as a common basis. Similarly, yeah, durability is important.If this were a serious life-critical app, it would not just be hanging out in a single region of a single provider; it would probably be on one provider, as I've talked about, but going multi-region and having backups to a different cloud provider. But if AWS takes a significant enough outage to us-west-2 in Oregon, to the point where my ridiculous system cannot function to write the newsletter, that too, is a different handwritten email that goes out that week because there's no announcement they've made that anyone's going to give the slightest toss about, given the fact that it's basically Cloud Armageddon. So, we'll see. It's about understanding the blast radius and understanding your use case.Betty: Yep. A hundred percent.Corey: So, you've spent a fair bit of time doing interesting things in your career. This is your second outing at VMware, and in the interim, you were at solo.io for a bit, and before that you were in a marketing leadership role at Docker. Let's dive in, if you will. Given that you are no longer working at Docker, they recently made an announcement about a pricing model change, whereas it is free to use Docker Desktop for anyone's personal projects, and for small companies.But if you're a large company, which they define is ten million in revenue a year or 250 employees—those two things don't go alike, but okay—then you have to wind up having a paid plan. And I will say it's a novel approach, but I'm curious to hear what you have to say about it.Betty: Well, I'd say that I saw that there was a lot of flutter about that news, and it's kind of a, it doesn't matter where you draw the line in the sand for the tier, there's always going to be some pushback on it. So, you have to draw a line somewhere. I haven't kept up with the details around the pricing models that they've implemented since I left Docker a few years ago, but monetization is a really important part for a startup. You do have to make money because there are people that you have to pay, and eventually, you want to get off of raising money from VCs all the time. Docker Desktop has been something that has been a real gem from a local developer experience, right, giving the—so that has been well-received by the community.I think there was an enterprise application for it, but when I saw that, I was like, yeah, okay, cool. They need to do something with that. And then it's always hard to see the blowback. I think sometimes with the years that we've had with Docker, it's kind of like no matter what they do, the Twitterverse and Hacker News is going to just give them a hard time. I mean, that is my honest opinion on that. If they didn't do it, and then, say, they didn't make the kind of revenue they needed, people would—that would become another Twitter thread and Hacker News blow up, and if they do it, you'll still have that same reaction.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: It seems to be that Docker has been trying to figure out how to monetize for a very long time because let's be clear here; I think it is difficult to overstate just how impactful and transformative Docker was to the industry. I gave a talk “Heresy in the Church of Docker” that listed a bunch of things that didn't get solved with Docker, and I expected to be torn to pieces for it, and instead I was invited to give it at ContainerCon one year. And in time, a lot of those things stopped being issues because the industry found answers to it. Now, unfortunately, some of those answers look like Kubernetes, but that's neither here nor there. But now it's, okay, so giving everything that you do that is core and central away for free is absolutely part of what drove the adoption that it saw, but goodwill from developers is not the sort of thing that generally tends to lead to interesting revenue streams.So, they had to do something. And they've tried a few different things that haven't seemed to really pan out. Then they spun off that pesky part of their business that made money selling support contracts, over to Mirantis, which was apparently looking for something now that OpenStack was no longer going to be a thing, and Kubernetes is okay, “Well, we'll take Docker enterprise stuff.” Great. What do they do, as far as turning this into a revenue model?There's a lot of the, I guess, noise that I tend to ignore when it comes to things like this because angry people on Twitter, or on Hacker News, or other terrible cesspools on the internet, are not where this is going to be decided. What I'm interested in is what the actual large companies are going to say about it. My problem with looking at it from the outside is that it feels as if there's significant ambiguity across the board. And if there's one thing that I know about large company procurement departments, it's that they do not like ambiguity. This change takes effect in three or four months, which is underwear-outside-the-pants-superhero-style speed for a lot of those companies, and suddenly, for a lot of developers, they're so far removed from the procurement side of the house that they are never going to have a hope of getting that approved on a career-wide timespan.And suddenly, for a lot of those companies, installing and running Docker Desktop just became a fireable offense because from the company's perspective, the sheer liability side of it, if they were getting subject to audit, is going to be a problem. I don't believe that Docker is going to start pulling Oracle-like audit tactics, but no procurement or risk management group in the world is going to take that on faith. So, the problem is not that it's expensive because that can be worked around; it's not that there's anything inherently wrong with their costing model. The problem is the ambiguity of people who just don't know, “Does this apply to me or doesn't this apply to me?” And that is the thing that is the difficult, painful part.And now, as a result, the [unintelligible 00:17:28] groups and their champions of Docker Desktop are having to spend a lot more time, energy, and thought on this than it would simply be for cutting a check because now it's a risk org-wide, and how do we audit to figure out who's installed this previously free open-source thing? Now what?Betty: Yeah, I'll agree with you on that because once you start making it into corporate-issued software that you have to install on the desktop, that gets a lot harder. And how do you know who's downloaded it? Like my own experience, right? I have a locked-down laptop; I can't just install whatever I want. We have a software portal, which lets me download the approved things.So, it's that same kind of model. I'd be curious because once you start looking at from a large enterprise perspective, your developers are working on IP, so you don't want that on something that they've downloaded using their personal account because now it sits—that code is sitting with their personal account that's using this tool that's super productive for them, and that transition to then go to an enterprise, large enterprise and going through a procurement cycle, getting a master services agreement, that's no small feat. That's a whole motion that is different than someone swiping a credit card or just downloading something and logging in. It's similar to what you see sometimes with the—how many people have signed up for and paid 99 bucks for Dropbox, and then now all of a sudden, it's like, “Wow, we have all of megacorp [laugh] signed up, and then now someone has to sell them a plan to actually manage it and make sure it's not just sitting on all these personal drives.”Corey: Well, that's what AWS's original sales motion looked a lot like they would come in and talk to the CTO or whatnot at giant companies. And the CTO would say, “Great, why should we pick AWS for our cloud needs?” And the answer is, “Oh, I'm sorry. You have 87 distinct accounts within your organization that we've [unintelligible 00:19:12] up for you. We're just trying to offer you some management answers and unify the billing and this, and probably give you a discount as well because there is price breaks available at certain sizing.” It was a different conversation. It's like, “I'm not here to sell you anything. We're already there. We're just trying to formalize the relationship.” And that is a challenge.Again, I'm not trying to cast aspersions on procurement groups. I mean, I do sell enterprise consulting here at The Duckbill Group; we deal with an awful lot of procurement groups who have processes and procedures that don't often align to the way that we do things as a ten-person, fully remote company. We do not have commercial vehicle insurance, for example, because we do not have a commercial vehicle and that is a prerequisite to getting the insurance, for one. We're unlikely to buy one to wind up satisfying some contractual requirements, so we have to go back and forth and get things like that removed. And that is the nature of the beast.And we can say yes, we can say no on a lot of those questionnaires, but, “It depends,” or, “I don't know,” is the sort of thing that's going to cause giant red flags and derail everything. But that is exactly what Docker is doing. Now, it's the well, we have a sort of sloppy, weird set of habits with some of our engineers around the bring your own device to work thing. So, that's the enterprise thing. Let me be very clear, here at The Duckbill Group, we have a policy of issuing people company machines, we manage them very lightly just to make sure the drives are encrypted, so they—and that the screensaver comes out with a password, so if someone loses a laptop, it's just, “Replace the hardware,” not, “We have a data breach.”Let's be clear here; we are responsible about these things. But beyond that, it's oh, you want to have some personal thing installed on your machine or do some work on that stuff? Fine. By all means. It's a situation of we have no policy against it; we understand this is how work happens, and we trust people to effectively be grownups.There are some things I would strongly suggest that any employee—ours or anyone else—not cross the streams on for obvious IP ownership rights and the rest, we have those conversations with our team for a reason. It's, understand the nuances of what you're doing, and we're always willing to throw hardware at people to solve these problems. Not every company is like that. And ten million in revenue is not necessarily a very large company. I was doing the math out for ten million in revenue or 250 employees; assuming that there's no outside investment—which with VC is always a weird thing—it's possible—barely—to have a $10 million in revenue company that has 250 employees, but if they're full time they are damn close to a $15 an hour minimum wage. So, who does it apply to? More people than you might believe.Betty: Yeah, I'm really curious to how they're going to like—like you say, if it takes place in three or four months, roll that out, and how would you actually track it and true that up for people? So.Corey: Yeah. And there are tools and processes to do this, but it's also not in anyone's roadmap because people are not sitting here on their annual planning periods—which is always aspirational—but no one's planning for, “Oh, yeah, Q3, one of our software suppliers is going to throw a real procurement wrench at us that we have to devote time, energy, resources, and budget to figure out.” And then you have a problem. And by resources, I do mean resources of basically assigning work and tooling and whatnot and energy, not people. People are humans, they are not resources; I will die on that hill.Betty: Well, you know, actually resource-wise, the thing that's interesting is when you say supplier, if it's something that people have been able to download for free so far, it's not considered a supplier. So, it's—now they're going to go from just a thing I can use and maybe you've let your developers use to now it has to be something that goes through the official internal vetting as being a supplier. So, that's just—it's a whole different ball game entirely.Corey: My last job before I started this place, was a highly regulated financial institution, and even grabbing things were available for free, “Well, hang on a minute because what license is it using and how is it going to potentially be incorporated?” And this stuff makes sense, and it's important. Now, admittedly, I have the advantage of a number of my engineering peers in that I've been married to a corporate attorney for 11 years and have insight into that side of the world, which to be clear, is all about risk mitigation which is helpful. It is a nuanced and difficult field to—as are most things once you get into them—and it's just the uncertainty that befuddles me a bit. I wish them well with it, truly I do. I think the world is better with an independent Docker in it, but I question whether this is going to find success. That said, it doesn't matter what I think; what matters is what customers say and do, and I'm really looking forward to seeing how it plays out.Betty: A hundred percent; same here. As someone who spent a good chunk of my life there, their mark on the industry is not to be ignored, like you said, with what happened with containers. But I do wish them well. There's lot of good people over there, it's some really cool tech, and I want to see a future for them.Corey: One last topic I want to get into before we wind up wrapping this episode is that you are someone who was nominated to come on the show by a couple of folks, which is always great. I'm always looking for recommendations on this. But what's odd is that you are—if we look at it and dig a little bit beneath the titles and whatnot, you even self-describe as your history is marketing leadership positions. It is uncommon for engineering-types to recommend that I talk to marketing folks.s personally I think that is a mistake; I consider myself more of a marketer than not in some respects, but it is uncommon, which means I have to ask you, what is your philosophy of marketing because it very clearly is differentiated in the public eye.Betty: I'm flattered. I will say that—and this goes to how I hire people and how I coach teams—it's you have to be super curious because there's a ton of bad marketing out there, where it's just kind of like, “Hey, we do these five things and we always do these five things: blah, blah, blah, blah, blah.” But I think it's really being curious about what is the thing that you're marketing? There are people who are just focused on the function of marketing and not the thing. Because you're doing your marketing job in the service of a thing, this new widget, this new whatever, and you got to be super curious about it.And I'll tell you that, for me, it's really hard for me to market something if I'm not excited about it. I have to personally be super excited about the tech or something happening in the industry, and it's, kind of like, an all-in thing for me. And so in that sense, I do spend a ton of time with engineers and end-users, and I really try to understand what's going on. I want to understand how the thing works, and I always ask them, “Well”—so I'll ask the engineers, like, “So… okay, this sounds really cool. You just described this new feature and you're super excited about it because you wrote it, but how is your end-user, the person you're building this for, how did they do this before? Help me understand. How did they do this before and why is this better?”Just really dig into it because for me, I want to understand it deeply before I talk about it. I think the thing is, it shows a tremendous amount of respect for the builder, and then to try to really be empathetic, to understand what they're doing and then partner with them—I mean, this sounds so business-y the way I'm talking about this—but really be a partner with them and just help them make their thing really successful. I'm like the other end; you're going to build this great thing and now I'm going to make it sound like it's the best thing that's ever happened. But to do that, I really need to deeply understand what it is, and I have to care about it, too. I have to care about it in the way that you care about it.Corey: I cannot effectively market or sell something that I don't believe in, personally. I also, to be clear because you are a marketing professional—or at least far more of one than I ever was—I do not view what I do is marketing; I view it as spectacle. And it's about telling stories to people, it's about learning what the market thinks about it, and that informs product design in many respects. It's about understanding the product itself. It's about being able to use the product.And if people are listening to this and think, “Wait a minute, that sounds more like DevRel.” I have news for you. DevRel is marketing, they're just scared to tell you that. And I know people are going to disagree with me on that. You're wrong. But that's okay; reasonable people can disagree.And that's how I see it is that, okay, I'll talk to people building the service, I'll talk to people using the service, but then I'm going to build something with the service myself because until then, it's all a game of who sounds the most convincing in the stories that they tell. But okay, you can tell an amazing story about something, but if it falls over when I tried to use it, well, I'm sorry, you're not being accurate in your descriptions of it.Betty: A hundred percent. I hate to say, like, you're storytellers, but that's a big part of it, but it's kind of like you want to tell the story, so you do something to that people believe a certain thing. But that's part of a curated experience because you want them to try this thing in a certain way. Because you've designed it for something. “I built a spoon. I want you to use that to eat your soup because you can't eat soup with a fork.”So, then you'll have this amazing soup-eating experience, but if I build you a spoon and then not give you any directions and you start throwing it at cars, you're going to be like, “This thing sucks.” So, I kind of think of it in that way. To your point of it has to actually work, it's like, but they also need to know, “What am I supposed to use it for?”Corey: The problem I've always had on some visceral level with formal marketing departments for companies is that they can say that a product that they sell is good, they can say that the product is great, or they can choose to say nothing at all about that product, but when there's a product in the market that is clearly a turd, a marketing department is never going to be able to say that, which I think erodes its authenticity in many respects. I understand the constraints behind, that truly I do, but it's the one superpower I think that I bring to the table where even when I do sponsorship stuff it's, you can buy my attention but not my opinion. Because the authenticity of me being trusted to call them like I see them, for lack of a better term, to my mind at least outweighs any short-term benefit from saying good things about a product that doesn't deserve them. Now, I've been wrong about things, sure. I have also been misinformed in both directions, thinking something is great when it's not, or terrible when it isn't or not understanding the use case, and I am thrilled to engage in those debates. “But this is really expensive when you run for this use case,” and the answer can be, “Well, it's not designed for that use case.” But the answer should not be, “No it's not.” I promise you, expensive is in the eye of the customer not the person building the thing.Betty: Yes. This goes back to I have to believe in the thing. And I do agree it's, like not [sigh]—it's not a panacea. You're not going to make Product A and it's going to solve everything. But being super clear and focused on what it is good for, and then please just try it in this way because that's what we built it for.Corey: I want to thank you for taking the time to have a what for some people is no doubt going to be perceived as a surprisingly civil conversation about things that I have loud, heated opinions about. If people want to learn more, where can they find you?Betty: Well, they can follow me on Twitter. But um, I'd say go to vmware.com/cloud for our work thing.Corey: Exactly. VM where? That's right. VM there. And we will, of course, put links to that in the [show notes 00:30:07].Betty: [laugh].Corey: Thank you so much for taking the time to speak with me. I appreciate it.Betty: Thanks, Corey.Corey: Betty Junod, Senior Director of Multi-Cloud Solutions at VMware. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a loud, ranting comment at the end. Then, if you work for a company that is larger than 250 people or $10 million in revenue, please also Venmo me $5.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

This week, we welcome Peter Klimek, Director of Technology, Office of the CTO at Imperva! Peter will talk to the challenges he's hearing from customers and partners about managing the security of APIs and what considerations organizations need to make in 2022 to better protect these growing ecosystems. In the AppSec News, Mike & John talk: Discourse SNS webhook RCE, a checklist for a Minimum Viable Secure Product, WhatsApp security assessment, privacy engineering specialties, & DevOps presentations!   Show Notes: https://securityweekly.com/asw172 Visit https://securityweekly.com/imperva to learn more about them!   Visit https://www.securityweekly.com/asw for all the latest episodes! Follow us on Twitter: https://www.twitter.com/securityweekly Like us on Facebook: https://www.facebook.com/secweekly

About EdEd Boyajian, President and CEO of EDB, drives the development and execution of EDB's strategic vision and growth strategy in the database industry, steering the company through 47 consecutive quarters of recurring revenue growth. He also led EDB's acquisition of 2ndQuadrant, a deal that brought together the world's top PostgreSQL experts and positioned EDB as the largest dedicated provider of PostgreSQL products and solutions worldwide. A 15+ year veteran of the open source software movement, Ed is a seasoned enterprise software executive who emphasizes that EDB must be a technology-first business in order to lead the open source data management ecosystem. Ed joined EDB in 2008 after serving at Red Hat, where he rose to Vice President and General Manager of North America. While there, he played a central leadership role in the development of the modern business model for bringing open source to enterprises.Links:EDB: https://enterprisedb.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring. Corey: This episode is sponsored in part by our friends at Jellyfish. So, you're sitting in front of your office chair, bleary eyed, parked in front of a powerpoint and—oh my sweet feathery Jesus its the night before the board meeting, because of course it is! As you slot that crappy screenshot of traffic light colored excel tables into your deck, or sift through endless spreadsheets looking for just the right data set, have you ever wondered, why is it that sales and marketing get all this shiny, awesome analytics and inside tools? Whereas, engineering basically gets left with the dregs. Well, the founders of Jellyfish certainly did. That's why they created the Jellyfish Engineering Management Platform, but don't you dare call it JEMP! Designed to make it simple to analyze your engineering organization, Jellyfish ingests signals from your tech stack. Including JIRA, Git, and collaborative tools. Yes, depressing to think of those things as your tech stack but this is 2021. They use that to create a model that accurately reflects just how the breakdown of engineering work aligns with your wider business objectives. In other words, it translates from code into spreadsheet. When you have to explain what you're doing from an engineering perspective to people whose primary IDE is Microsoft Powerpoint, consider Jellyfish. Thats Jellyfish.co and tell them Corey sent you! Watch for the wince, thats my favorite part. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today's promoted episode is a treasure and a delight. Longtime listeners of this show know that it's not really a database—unless of course, it's Route 53—and of course, I don't solve pronunciation problems with answers that make absolutely everyone hate me. Longtime listeners of the show know that if there's one thing I adore when it comes to databases—you know, other than Route 53—it is solving pronunciation holy wars in such a way that absolutely everyone is furious with me as a result, and today is no exception. My guest is Ed Boyajian, the CEO of EDB, a company that effectively is the driving force behind the Postgres-squeal database. Ed, thank you for joining me.Ed: Hey, Corey.Corey: So, I know that other people pronounce it ‘post-gree,' ‘Postgresql,' ‘Postgres-Q-L,' all kinds of other things. We know it's decidedly not ‘Postgres-squeal,' which is how I go for it. How do you pronounce it?Ed: We say ‘Postgres,' and this is one of the great branding challenges this fantastic open-source project has endured over many years.Corey: So, I want to start at the very beginning because when I say that you folks are the driving force behind Postgres—or Postgres-squeal—I mean it. I've encountered folks from EDB—formerly EnterpriseDB—in the wild in consulting engagements before, and it's great because whenever we found an intractable database problem, back at my hands-on keyboard engineering implementation days, very quickly after you folks got involved, it stopped being a problem, which is kind of the entire point. A lot of companies will get up there and say, “Oh, it's an open-source project,” with an asterisk next to it and 15 other things that follow from it, or, “Now, we're changing our license so the big companies can't compete with us.” Your company's not named after Postgres-squeal and you're also—when you say you have people working on it, we're not talking just one or two folks; your fingerprints are all over the codebase. How do you engage with an open-source project in that sense?Ed: First and foremost, Postgres itself is, as you know, an independent open-source project, a lot like Linux. And that means it's not controlled by a company. I think that's inherently one of Postgres's greatest strengths and assets. With that in mind, it means that a company like EDB—and this started when I came to the company; I came from Red Hat, so I've been in open-source for 20 years—when I came to the company back in 2008, it starts with a commitment and investment in bringing technology leaders in and around Postgres into a business like EDB, to help enterprises and customers. And that dynamic intersection between building the core database in the community and addressing customer needs in a business, at that intersection is where the magic happens. And we've been doing that since I joined EDB in 2008; it was really an explicit focus for the company.Corey: I'd like to explore a little bit, well first and foremost, this story of is there a future for running databases in cloud environments yourself? And I have my own angry, loud opinion on this that I'm sure we'll get to momentarily, but I want to start with yours. Who is writing their own databases in the Year of our Lord 2021, rather than just using whatever managed thing is their cloud provider of choice today is offering for them?Ed: Well, let me give you context, Corey, because I think it matters. We've been bringing enterprise Postgres solutions to companies now, since the inception of the company, which dates back to 2004, and over that trajectory, we've been helping companies as they've done really two things: migrate away, in particular from Oracle, and land on Postgres, and then write new apps. Probably the first ten of the last 13 years since I've been in the company, the focus was in traditional on-prem database transformations that companies were going through. In the last three years, we've really seen an acceleration of that intersection of their traditional deployments and their cloud deployments. Our customers now, who are represented mostly in the Fortune 500 and Global 2000, 40% of our customers report they're deploying EDB's Postgres in the cloud, not in a managed context, but in a traditional EC2 or GCP self-managed cloud deployment.Corey: And that aligns with what I've seen, a fair bit. Years ago, I wound up getting the AWS Cloud Practitioner Certification—did a whole blog post on it—not because it was opening any doors for me, but because it let me get into the certified lounge at re:Invent, and ideally charge a battery and have some mostly crappy coffee. The one question I got wrong was I was honest when I answered, “How long does it take to restore an RDS database from snapshot backup?” Rather than giving the by-the-book answer, which is way shorter than I found in practice a fair bit of the time. And that's the problem I always ran into is that when you're starting out and building something that needs a database, and it needs a relational database that runs in that model so all the no SQL options are not viable for whatever reason, great, RDS is great for getting you started, but there's only so much that you can tune and tweak before you start to run into issues were, for particular workloads as they scale-out, it's no longer a fit for a variety of reasons.And most of the large companies that I work with that are heavily relational-database-driven have either started off or migrated to the idea of, “Oh, we're going to run our own databases on top of EC2 instances,” for a variety of reasons that, again, the cloud providers will say, “Oh, that's not accurate, and they're doing the wrong thing.” But, you know, it takes a certain courage to tell a large-scale customer, “You're doing it wrong.” “Well, why is that?” “Because I have things to sell you,” is kind of a terrible answer. How do you see it? Let's not pick on RDS, necessarily, because all of the cloud providers offered managed database offerings. Where do those make sense and where do they fall down?Ed: Yeah, I think many of our customers who made their first step into cloud picked a single vendor to do it, and we often hear AWS is been that early, early—Corey: Yeah, a five-year head start makes a pretty compelling story.Ed: That's right. And let's remember what these vendors are mostly. They are mostly infrastructure companies, they build massive data centers and set those up, and they do that beautifully well. And they lean on software, but they're not software companies themselves. And I think the early implementation of many of our customers in cloud relied on what I'll call relatively lightweight software offerings from their cloud vendor, including database.They traded convenience, ease of use, an easy on-ramp, and they traded some capability in some depth for that. And it was a good trade, in fact. And for a large number of workloads it may still be a good trade. But our more sophisticated customers, enterprise customers who are running Postgres or databases at scale in their traditional environments have long depended on a very intimate relationship with their database technology vendor. And that relationship is the intersection of their evolving and emerging needs and the actual development of the database capabilities in support of that.And that's the heart of who we are at EDB and what we do with Postgres and the many people we have committed to doing that. And we don't see our customers changing that appetite. So, I think for those customers, they've emerged more aware of the need to have a primary relationship with a database vendor and still be in cloud. And so I think that's how this evolves to see two different kinds of services side-by-side, what they really want is a Database as a Service from the database vendor, which is what we just announced here at Microsoft Ignite event.Corey: So, talk to me a little bit more about that, where it's interesting in 2021 to see a company launching a managed service offering, especially in the database space, when there's been so much pushback in different ways against the large cloud providers—[cough] Amazon—who tend to effectively lose sleep at night over the haunting fear that someone who isn't them is making money, somehow. And they will take whatever is available to them and turn it into a managed service offering. That's always been the fear, so people play games with licenses and the rest. Well, they've been running Postgres offerings for a long time. It is an independent open-source project.I don't think you can wind up forcing a license change through that says everyone except big companies can run this themselves and don't do a managed service with it because that cat is very much out of the bag. How is it that you're taking something to market now and expecting that to fare competitively?Ed: So, I think there's a few things that our customers are clearly telling us they want, and I think this is the most important thing: they want control of their data. And if you step back, Corey, look at it historically, they made a huge trade to big proprietary database companies, companies like Oracle, and they made that trade actually for convenience. They traded data to that database vendor. And we all know the successes Oracle's had, and the sheer extraordinary expense of those technologies. So, it felt like a walled garden.And that's where EDB and Postgres entered to really change that equation. What's interesting is the re-platforming that happened and the transformation to cloud actually had the same, kind of, binding effect; we now moved all that data over to the public cloud vendors, arguably in an even stickier context, and now I think customers are realizing that's created a dimension of inflexibility. It's also created some—as you rightly pointed out—some deficiencies in technical depth, in database, and in software. So, our customers have sorted that out and are kind of coming back to middle. And what they're saying is, “Well, we want all the advantages of an open-source database like a Postgres, but we want control of the data.”And so what control looks like is more the ability to take one version of that software—in our case, we're worrying about Postgres—and deploy the same thing everywhere they go. And that opens the door up for EDB to be their partner as a traditional on-prem partner, in the cloud where they run our Postgres and they manage it themselves, and as their managed service, Postgres Database as a Service Provider, which is what we're doing.Corey: I've been something of a bear on the idea of, “I'm going to build a workload to run everywhere in every cloud provider,” which I get. I think that's generally foolish, and people chasing that, with remarkably few exceptions, are often going after the wrong thing. That said, I'm also a fan of having a path to strategic Exodus, where Google's Cloud Spanner is fascinating, DynamoDB is revelatory, Cosmos DB is a security nightmare, which is neither here nor there, but the idea that I can take a provider's offering that even if it solves a bunch of problems for me, well, if I ever need to move this somewhere else for any reason, I'm re-architecting, my data model and re-architecting the built-in assumptions around how the database acts and behaves, and that is a very heavy lift. We have proof of that from Amazon, who got up on stage and told a story about how much they hate Oracle, and they're migrating everything off of Oracle to Aurora, which they had to build in order to get off of Oracle, and it took them three years to migrate things. And Oracle loves telling that story, too.And it's, you realize you both sound terrible when you tell that story? It's, “This is a massive undertaking that even we struggle with, so you should probably not attempt it.” Well, what I hear from that is good God, don't wind up getting locked into a particular database that is only available from one source. So, if you're all-in on a cloud provider, which I'm a fan of, personally—I don't care which one but pick a cloud provider—having a database that is not only going to work in that environment is just a reasonable step as far as how I view things. Trading up that optionality has got to pay serious dividends, and in many database use cases, I've just don't see it.Ed: Yeah, I think you're bringing up a really important point. So, let's unpack it for a minute.Corey: Please.Ed: Because I think you brought up some really prominent specialty database technologies, and I'm not sure there's ever a way out of that intersection and commitment to a single vendor if you pick their specialty database. But underneath this is exactly one of the things that we've worried about here at EDB, which is to make Postgres a more capable, robust database in its entirety. A Postgres superpower is its ability to run a vast array of workloads. Guess what, it's not sexy. It's not sexy not to be that specialty database, but it's incredibly powerful in the hands of an enterprise who can do more.And that really creates an opportunity, so we're trying to make Postgres apply to a much broader set of workloads, from traditional systems of record, like your ERP systems; systems of analysis, where people are doing lightweight analytic workloads or reporting, you can think in the world of data warehouse; and then systems of engagement, where customers are interacting with a website and have a database on the backend. All areas Postgres has done incredibly well in and we have customer experience with. So, when you separate out that core capability and then you look at it on a broader scale like Postgres, you realize that customers who want to make Postgres strategic, by definition need to be able to deploy it wherever they want to deploy it, and not be gated or bound by one cloud vendor. And all the cloud vendors picked up Postgres offerings, and that's been great for Postgres and great for enterprises. But that corresponding lock-in is what people want to get away from, at this point.Corey: There's something to be said for acknowledging that there is a form of lock-in as far as technology selection goes. If you have a team of folks who are terrific at one database engine and suddenly you're switching over to an entirely different database, well, folks who spent their entire career working on one particular database that's still in widespread use are probably not super thrilled to stick around for that. Having something that can migrate from environment to environment is valuable and important. When you say you're launching this as a database as a service offering, how does that actually work? Is that going to be running in your own cloud environment somewhere and people just make queries across the wire through standard connections to the database like they would something locally? Are you running inside of their account or environment? Is it something else?Ed: So, this is a fully-managed database as a service, just like you'd get from any cloud vendor or DBAAS vendor that you've worked with in the past, just being managed and run by EDB. And with that, you get lot of the goodies that we bring, including our compatibility, and all our deep Postgres expertise, but I think one of the other important attributes is we're going to run that service in our clients' account, which gives them a level of isolation and a level of independence that we think is really important. And as different as that is, it's not heroic; it's exactly what our customers told us they wanted.Corey: There's something to be said for building the thing that your customers have said that they want and make sense for you to build as opposed to, “We're going to build this ridiculous thing and we're sure folks are going to love it.” It's nice to see that shaping up in the proper order. And I've fallen victim to that myself; I think most technologists have to some extent. How big is EDB these days?Ed: So, we have over 650 employees. Now, around the world, we have 6000 customers. And of the 650 employees, about 300 of those are focused on Postgres. A subset of that are 30-odd core team members in the Postgres community, committers in the Postgres community, major contributors, and contributors in the Postgres community. So, we have a density of technical depth that is really unparalleled in Postgres.Corey: You're not, for lack of a better term, pulling an Amazon, insofar as you're, “Well, we have three people working on open-source projects, so we're going to go ahead and claim we're an open-source company,” in other words. Conversely, you're also not going down the path of this is a project that you folks have launched, and it claims to be open-source because we love it when people volunteer for for-profit entities, but we exercise total control over the project. You have a lot of contributors, but you're also still a minority, I think the largest minority, but still a minority of people contributing to Postgres.Ed: That's right. And, look, we're all-in on Postgres, and it's been that way since I got here. As I mentioned earlier, I came from Red Hat where I was—I was at Red Hat for a little over six years, so I've been an open-source now for 20 years. So, my orientation is towards really powerful, independent open-source projects. And I think we'll see Postgres really be the most transformative open-source technology since Linux.I think we'll see that as we look forward. And you're right, though, I think what's powerful about Postgres is it's an independent project, which means it's supported by thousands of contributors who aren't tied to single companies, around the world. And it just makes the software—we develop innovation faster, and I think it makes the software better. Now, EDB plays a big part in there. Roughly, a little less than a third of the last res—actually, the 13 release—were contributions that came from contributors who came from EDB.So, that's not a majority, and that's healthy. But it's a big part of what helps move Postgres along and there aren't—you know, the next set of companies are much, much—next set of combined contributors add up to quite small numbers. But the cloud vendors are virtually non-existent in that contribution.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: Something else that does strike me as, I guess, strange, just because I've seen so many companies try to navigate this in different ways with varying levels of success. I always encountered EDB—even back when it was EnterpriseDB, which was, given their love of acronyms, I'm still somewhat partial to. I get it; branding, it's a thing—but the folks that I engaged with were always there in a consulting service's capacity, and they were great at this. Is EDB a services company or a product company?Ed: Yeah, we are unashamedly a product technology company. Our business is over 90% of our revenue is annually recurring subscription revenue that comes from technical products, database server, mostly, but then various adjacent capabilities in replication and other areas that we add around the database server itself. So no, we're a database technology company selling a subscription. Now, we help our customers, so we do have a really talented team of consultants who help our customers with their business strategy for Postgres, but also with migrations and all the things they need to do to get Postgres up and running.Corey: And the screaming, “Help, help, help, fix it, fix it, fix it now,” emergencies as well.Ed: I think we have the best Postgres support operation in the world. It is a global 24/7 organization, and I think a lot of what you likely experienced, Corey, came out of our support organization. So, our support guys, these guys aren't just handling lightweight issues. I mean, they wade into the gnarly questions and challenges that customers face. But that's a support business for us. So, that's part and parcel. You get that, it's included with the subscription.Corey: I would not be remembering this for 11 years later, if it hadn't been an absolutely stellar experience—or a horrible experience, for that matter; one or the other. You remember the superlatives, not the middle of the road ones—and if it hadn't been important. And it was. It also noteworthy; with many vendors that are product-focused, their services may have an asterisk next to it because it's either a, “Buy our product and then we'll support it,” or it's, “Ohh, we're going to sell you a whole thing just to get us on the phone.” And as I recall, there wasn't a single aspect of upsell involved in this.It was, “Let's get you back up and running and solve the problem.” Sure, later in time, there were other conversations, as all good businesses will have, but there was no point during those crisis moments where it felt like, “Oh, if you had gone ahead and bought this thing that we sell, this wouldn't happen,” or, “You need to buy this or we won't help you.” I guess that's why I've contextualized you folks as a services company, first and foremost.Ed: Well, I'm glad you have that [laugh] experience because that's our goal. And I think—look, this is an interesting point where customers want us to bring that capability to their managed DBAAS world. Step back again, go back to what I said about the big cloud vendors; they are, at their core, infrastructure companies. I mean, they're really good at that. They're not particularly well-positioned to take your Postgres call, and I don't think they want that call.We're the other guys; we want to help you run your Postgres, at scale, on-prem, in the cloud, fully managed in the cloud, by EDB, and solve those problems at the same time. And I think that's missing in the market today. And we can step back and look at this overall cloud evolution, and I think some might think, “Gee, we're into the mature phase of cloud adoption.” I would tell you, since the Red Sox have done well this year, I think in a nine-inning baseball game—for those of your listeners who follow American baseball—we're in, like, the top of the second inning, maybe. Maybe the bottom of the second inning. So, we've been able to listen and learn from the experiences our customers have had. I think that's an incredible advantage as we now firmly plant ourselves in the cloud DBAAS market alongside our robust Postgres capabilities that you experienced.Corey: The world isn't generating less data, and it's important that we're able to access that in a bunch of different ways. And the last time I really was playing with relational databases, you can view my understanding of it as Excel with a weirder interface, and you're mostly there. One thing that really struck me since the last time I went deep into database-land over in the Postgres-squeal world has been just the sheer variety of native data types that it winds up supporting. The idea of, “Here's some JSON. Take this and store it that way,” or it's GIS data that it can represent, or the idea of having data types that are beyond just string or var or whatever other somewhat limited boolean values or whatnot. Without having just that traditional list, which is of course all there as well. It also seems to have extensively improved its coverage that just can only hint to my small mind about these things and what sort of use cases people are really putting these things into.Ed: Yeah, I think this is one of Postgres' superpowers. And it started with Mike Stonebraker's original development of Postgres as an object-relational database. Mike is an adviser to EDB, which has been incredibly helpful as we've continued to evolve our thinking about what's possible in Postgres. But I think because of that core technology, or that core—because of that core technical capability within Postgres, we have been able to build a whole host of data types. And so now you see Postgres being used not just as the context of a traditional relational database, but we see it used as a time-series database. You pointed out a geospatial database, more and more is a document-oriented database with JSON and JSONB.These are all the things that make Postgres have much more universal appeal, universal appeal to developers—which is worth talking about in the recent StackOverflow developer survey, but we can come back to that—and I think universal applicability for new applications. This is what's bringing Postgres forward faster, unlike many of the specialty database companies that you mentioned earlier.Corey: Now, this is something that you can use for your traditional CRUD app, the my first hello world app that returns something from a database, yeah, that stuff works. But it also, for example, has [cyter 00:25:09] data types, where you can say, give me the results where the IP range contains this address, and it'll do that. Before that, you're trying to solve a whole bunch of very messy things in application logic that's generally awful. The database now does that for you automatically, and there's something—well, it would if I were smart and used it instead of storing it as strings because I make terrible life choices, but for sensible people, it solves a lot of those problems super well. And it's taken the idea of where logic should live in application versus database, and sort of turn a lot of those assumptions I was starting my career with on their head.Ed: Yeah, I think if you look now at the appeal of Postgres to developers, which we've paid a lot of attention to—one of our stated strategies at EDB is to make Postgres easier. That's been true for many years, so a drive for engineering and development here has been that call to action. And if you measure that, over time, we've been contributing—not alone, but contributing to making Postgres more approachable, easier to use, easier to engage with. Some of those things we do just through edb.com, and the way we handle EDB docs is a great example of that, and our developer advocacy and outreach into adjacent communities that care about Postgres. But here's where that's landed us. If you looked at the last Stack Overflow developer survey—the 2021 Stack Overflow developer survey, which I love because I think it's very independent-oriented—and they surveyed, I think this past year was 80,000 developers.Corey: Oh yeah, if Stack Overflow is captured by any particular constituency, it's got to be ‘Big Copy and Paste' that is really behind them. But yeah, other than the cabal of keyboard manufacturers for those copy-and-paste stories, yeah, they're fairly objective when it comes to stuff like this.Ed: And if you look at that survey, Corey, if you just took and summed it because it's helpful to sum it, most used, most loved, and most wanted database: Postgres wins. And I find it fascinating that if you—having been here, in this company for 13 years and watch the evolution from—you know, 13 years ago, Postgres needed help, both in terms of its awareness in the market and some technical capabilities it just lacked, we've come so far. For that to be the new standard for developers, I think, is a remarkable achievement. And I think it's a representation of why Postgres is doing so well in the market that we've long served, in the cloud market that we are now serving, and I think it speaks to what's ahead as a transformational database for the future.Corey: There really is something to be said for a technology as—please don't take this term the wrong way—old. As a relational database, Postgres has been around for a very long time, but it's also not your grandparents' Postgres. It is continuing to evolve. It continues to be there in a bunch of really interesting ways for developers in a variety of different capacities, and it's not the sort of thing that you're only using in, “Legacy environments,” quote-unquote. Instead, it's something that you'll see all over the place. It is rare that I see an environment that doesn't have Postgres in it somewhere these days.Ed: Yeah, I think quite the contrary to the old-school database, which I love that; I love that shade because when you step away from it, you realize, the Postgres community represents the very best of what's possible with open-source. And that's why Postgres continues to accelerate and move forward at the rate that it does. And obviously, we're proud to be a contributor to that, so we don't just watch that outcome happen; we're actually part of creating it. But I also think that when you see all that Postgres has become and where it's going, you really start to understand why the market is adopting open-source.Corey: It's one of those areas where even if some company comes out with something that is amazing and transformatively better, and you should jump into it with both feet and never look back, yeah, it turns out that it takes a long time to move databases, even when they're terrible. And you can lobby an awful lot of accusations at Postgres—or Postgres-squeal—but you can't call it terrible. It's used in enough interesting applications by enough large-scale companies out there—and small as well—that it's very hard to find a reason not to explore it. It's my default relational database when Route 53 loses steam. It just makes sense in a bunch of ways that other things really didn't for me before.Ed: Yeah, and I think we'll continue to see that. And we're just going to keep making Postgres better. And it gets better because of that intersection, as I mentioned, that intimate intersection between enterprise users, and the project, and the community, and the bridge that a company like EDB provides for that. That's why it'll get better faster; the breadth of use of Postgres will kee