Podcasts about humblepod

Share on
Share on Facebook
Share on Twitter
Share on Reddit
Copy link to clipboard
  • 3PODCASTS
  • 134EPISODES
  • 31mAVG DURATION
  • 5WEEKLY NEW EPISODES
  • Jan 18, 2022LATEST

POPULARITY

20122013201420152016201720182019202020212022


Best podcasts about humblepod

Latest podcast episodes about humblepod

Screaming in the Cloud
The re:Invent Wheel in the Sky Keeps on Turning with Pete Cheslock

Screaming in the Cloud

Play Episode Listen Later Jan 18, 2022 54:52


About PetePete does many startup things at Allma. Links: Last Tweet in AWS: https://lasttweetinaws.com Twitter: https://twitter.com/petecheslock LinkedIn: https://www.linkedin.com/in/petecheslock/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I am joined—as is tradition, for a post re:Invent wrap up, a month or so later, once everything is time to settle—by my friend and yours, Pete Cheslock. Pete, how are you?Pete: Hi, I'm doing fantastic. New year; new me. That's what I'm going with.Corey: That's the problem. I keep hoping for that, but every time I turn around, it's still me. And you know, honestly, I wouldn't wish that on anyone.Pete: Exactly. [laugh]. I wouldn't wish you on me either. But somehow I keep coming back for this.Corey: So, in two-thousand twenty—or twenty-twenty, as the children say—re:Invent was fully virtual. And that felt weird. Then re:Invent 2021 was a hybrid event which, let's be serious here, is not really those things. They had a crappy online thing and then a differently crappy thing in person. But it didn't feel real to me because you weren't there.That is part of the re:Invent tradition. There's a midnight madness thing, there's a keynote where they announce a bunch of nonsense, and then Pete and I go and have brunch on the last day of re:Invent and decompress, and more or less talk smack about everything that crosses our minds. And you weren't there this year. I had to backfill you with Tim Banks. You know, the person that I backfield you with here at The Duckbill Group as a principal cloud economist.Pete: You know, you got a great upgrade in hot takes, I feel like, with Tim.Corey: And other ways, too, but it's rude of me to say that to you directly. So yeah, his hot takes are spectacular. He was going to be doing this with me, except you cannot mess with tradition. You really can't.Pete: Yeah. I'm trying to think how many—is this third year? It's at least three.Corey: Third or fourth.Pete: Yeah, it's at least three. Yeah, it was, I don't want to say I was sad to not be there because, with everything going on, it's still weird out there. But I am always—I'm just that weird person who actually likes re:Invent, but not for I feel like the reasons people think. Again, I'm such an extroverted-type person, that it's so great to have this, like, serendipity to re:Invent. The people that you run into and the conversations that you have, and prior—like in 2019, I think was a great example because that was the last one I had gone to—you know, having so many conversations so quickly because everyone is there, right? It's like this magnet that attracts technologists, and venture capital, and product builders, and all this other stuff. And it's all compressed into, like, you know, that five-day span, I think is the biggest part that makes so great.Corey: The fear in people's eyes when they see me. And it was fun; I had a pair of masks with me. One of them was a standard mask, and no one recognizes anyone because, masks, and the other was a printout of my ridiculous face, which was horrifyingly uncanny, but also made it very easy for people to identify me. And depending upon how social I was feeling, I would wear one or the other, and it worked flawlessly. That was worth doing. They really managed to thread the needle, as well, before Omicron hit, but after the horrors of last year. So, [unintelligible 00:03:00]—Pete: It really—Corey: —if it were going on right now, it would not be going on right now.Pete: Yeah. I talk about really—yeah—really just hitting it timing-wise. Like, not that they could have planned for any of this, but like, as things were kind of not too crazy and before they got all crazy again, it feels like wow, like, you know, they really couldn't have done the event at any other time. And it's like, purely due to luck. I mean, absolute one hundred percent.Corey: That's the amazing power of frugality. Because the reason is then is it's the week after Thanksgiving every year when everything is dirt cheap. And, you know, if there's one thing that I one-point-seve—sorry, their stock's in the toilet—a $1.6 trillion company is very concerned about, it is saving money at every opportunity.Pete: Well, the one thing that was most curious about—so I was at the first re:Invent in-what—2012 I think it was, and there was—it was quaint, right?—there was 4000 people there, I want to say. It was in the thousands of people. Now granted, still a big conference, but it was in the Sands Convention Center. It was in that giant room, the same number of people, were you know, people's booths were like tables, like, eight-by-ten tables, right? [laugh].It had almost a DevOpsDays feel to it. And I was kind of curious if this one had any of those feelings. Like, did it evoke it being more quaint and personable, or was it just as soulless as it probably has been in recent years?Corey: This was fairly soulless because they reduced the footprint of the event. They dropped from two expo halls down to one, they cut the number of venues, but they still had what felt like 20,000 people or something there. It was still crowded, it was still packed. And I've done some diligent follow-ups afterwards, and there have been very few cases of Covid that came out of it. I quarantined for a week in a hotel, so I don't come back and kill my young kids for the wrong reasons.And that went—that was sort of like the worst part of it on some level, where it's like great. Now I could sit alone at a hotel and do some catch-up and all the rest, but all right I'd kind of like to go home. I'm not used to being on the road that much.Pete: Yeah, I think we're all a little bit out of practice. You know, I haven't been on a plane in years. I mean, the travel I've done more recently has been in my car from point A to point B. Like, direct, you know, thing. Actually, a good friend of mine who's not in technology at all had to travel for business, and, you know, he also has young kids who are under five, so he when he got back, he actually hid in a room in their house and quarantine himself in the room. But they—I thought, this is kind of funny—they never told the kids he was home. Because they knew that like—Corey: So, they just thought the house was haunted?Pete: [laugh].Corey: Like, “Don't go in the west wing,” sort of level of nonsense. That is kind of amazing.Pete: Honestly, like, we were hanging out with the family because they're our neighbors. And it was like, “Oh, yeah, like, he's in the guest room right now.” Kids have no idea. [laugh]. I'm like, “Oh, my God.” I'm like, I can't even imagine. Yeah.Corey: So, let's talk a little bit about the releases of re:Invent. And I'm going to lead up with something that may seem uncharitable, but I don't think it necessarily is. There weren't the usual torrent of new releases for ridiculous nonsense in the same way that there have been previously. There was no, this service talks to satellites in space. I mean, sure, there was some IoT stuff to manage fleets of cars, and giant piles of robots, and cool, I don't have those particular problems; I'm trying to run a website over here.So okay, great. There were enhancements to a number of different services that were in many cases appreciated, in other cases, irrelevant. Werner said in his keynote, that it was about focusing on primitives this year. And, “Why do we have so many services? It's because you asked for it… as customers.”Pete: [laugh]. Yeah, you asked for it.Corey: What have you been asking for, Pete? Because I know what I've been asking for and it wasn't that. [laugh].Pete: It's amazing to see a company continually say yes to everything, and somehow, despite their best efforts, be successful at doing it. No other company could do that. Imagine any other software technology business out there that just builds everything the customers ask for. Like from a product management business standpoint, that is, like, rule 101 is, “Listen to your customers, but don't say yes to everything.” Like, you can't do everything.Corey: Most companies can't navigate the transition between offering the same software in the Cloud and on a customer facility. So, it's like, “Ooh, an on-prem version, I don't know, that almost broke the company the last time we tried it.” Whereas you have Amazon whose product strategy is, “Yes,” being able to put together a whole bunch of things. I also will challenge the assertion that it's the primitives that customers want. They don't want to build a data center out of popsicle sticks themselves. They want to get something that solves a problem.And this has been a long-term realization for me. I used to work at Media Temple as a senior systems engineer running WordPress at extremely large scale. My websites now run on WordPress, and I have the good sense to pay WP Engine to handle it for me, instead of doing it myself because it's not the most productive use of my time. I want things higher up the stack. I assure you I pay more to WP Engine than it would cost me to run these things myself from an infrastructure point of view, but not in terms of my time.What I see sometimes as the worst of all worlds is that AWS is trying to charge for that value-added pricing without adding the value that goes along with it because you still got to build a lot of this stuff yourself. It's still a very janky experience, you're reduced to googling random blog posts to figure out how this thing is supposed to work, and the best documentation comes from externally. Whereas with a company that's built around offering solutions like this, great. In the fullness of time, I really suspect that if this doesn't change, their customers are going to just be those people who build solutions out of these things. And let those companies capture the up-the-stack margin. Which I have no problem with. But they do because Amazon is a company that lies awake at night actively worrying that someone, somewhere, who isn't them might possibly be making money somehow.Pete: I think MongoDB is a perfect example of—like, look at their stock price over the last whatever, years. Like, they, I feel like everyone called for the death of MongoDB every time Amazon came out with their new things, yet, they're still a multi-billion dollar company because I can just—give me an API endpoint and you scale the database. There's is—Corey: Look at all the high-profile hires that Mongo was making out of AWS, and I can't shake the feeling they're sitting there going, “Yeah, who's losing important things out of production now?” It's, everyone is exodus-ing there. I did one of those ridiculous graphics of the naming all the people that went over there, and in—with the hurricane evacuation traffic picture, and there's one car going the other way that I just labeled with, “Re:Invent sponsorship check,” because yeah, they have a top tier sponsorship and it was great. I've got to say I've been pretty down on MongoDB for a while, for a variety of excellent reasons based upon, more or less, how they treated customers who were in pain. And I'd mostly written it off.I don't do that anymore. Not because I inherently believe the technology has changed, though I'm told it has, but by the number of people who I deeply respect who are going over there and telling me, no, no, this is good. Congratulations. I have often said you cannot buy authenticity, and I don't think that they are, but the people who are working there, I do not believe that these people are, “Yeah, well, you bought my opinion. You can buy their attention, not their opinion.” If someone changes their opinion, based upon where they work, I kind of question everything they're telling me is, like, “Oh, you're just here to sell something you don't believe in? Welcome aboard.”Pete: Right. Yeah, there's an interview question I like to ask, which is, “What's something that you used to believe in very strongly that you've more recently changed your mind on?” And out of politeness because usually throws people back a little bit, and they're like, “Oh, wow. Like, let me think about that.” And I'm like, “Okay, while you think about that I want to give you mine.”Which is in the past, my strongly held belief was we had to run everything ourselves. “You own your availability,” was the line. “No, I'm not buying Datadog. I can build my own metric stack just fine, thank you very much.” Like, “No, I'm not going to use these outsourced load balancers or databases because I need to own my availability.”And what I realized is that all of those decisions lead to actually delivering and focusing on things that were not the core product. And so now, like, I've really flipped 180, that, if any—anything that you're building that does not directly relate to the core product, i.e. How your business makes money, should one hundred percent be outsourced to an expert that is better than you. Mongo knows how to run Mongo better than you.Corey: “What does your company do?” “Oh, we handle expense reports.” “Oh, what are you working on this month?” “I'm building a load balancer.” It's like that doesn't add the value. Don't do that.Pete: Right. Exactly. And so it's so interesting, I think, to hear Werner say that, you know, we're just building primitives, and you asked for this. And I think that concept maybe would work years ago, when you had a lot of builders who needed tools, but I don't think we have any, like, we don't have as many builders as before. Like, I think we have people who need more complete solutions. And that's probably why all these businesses are being super successful against Amazon.Corey: I'm wondering if it comes down to a cloud economic story, specifically that my cloud bill is always going to be variable and it's difficult to predict, whereas if I just use EC2 instances, and I build load balancers or whatnot, myself, well, yeah, it's a lot more work, but I can predict accurately what my staff compensation costs are more effectively, that I can predict what a CapEx charge would be or what the AWS bill is going to be. I'm wondering if that might in some way shape it?Pete: Well, I feel like the how people get better in managing their costs, right, you'll eventually move to a world where, like, “Yep, okay, first, we turned off waste,” right? Like, step one is waste. Step two is, like, understanding your spend better to optimize but, like, step three, like, the galaxy brain meme of Amazon cost stuff is all, like, unit economics stuff, where trying to better understand the actual cost deliver an actual feature. And yeah, I think that actually gets really hard when you give—kind of spread your product across, like, a slew of services that have varying levels of costs, varying levels of tagging, so you can attribute it. Like, it's really hard. Honestly, it's pretty easy if I have 1000 EC2 servers with very specific tags, I can very easily figure out what it costs to deliver product. But if I have—Corey: Yeah, if I have Corey build it, I know what Corey is going to cost, and I know how many servers he's going to use. Great, if I have Pete it, Pete's good at things, it'll cut that server bill in half because he actually knows how to wind up being efficient with things. Okay, great. You can start calculating things out that way. I don't think that's an intentional choice that companies are making, but I feel like that might be a natural outgrowth of it.Pete: Yeah. And there's still I think a lot of the, like, old school mentality of, like, the, “Not invented here,” the, “We have to own our availability.” You can still own your availability by using these other vendors. And honestly, it's really heartening to see so many companies realize that and realize that I don't need to get everything from Amazon. And honestly, like, in some things, like I look at a cloud Amazon bill, and I think to myself, it would be easier if you just did everything from Amazon versus having these ten other vendors, but those ten other vendors are going to be a lot better at running the product that they build, right, that as a service, then you probably will be running it yourself. Or even Amazon's, like, you know, interpretation of that product.Corey: A few other things that came out that I thought were interesting, at least the direction they're going in. The changes to S3 intelligent tiering are great, with instant retrieval on Glacier. I feel like that honestly was—they talk a good story, but I feel like that was competitive response to Google offering the same thing. That smacks of a large company with its use case saying, “You got two choices here.” And they're like, “Well, okay. Crap. We're going to build it then.”Or alternately, they're looking at the changes that they're making to intelligent tiering, they're now shifting that to being the default that as far as recommendations go. There are a couple of drawbacks to it, but not many, and it's getting easier now to not have the mental overhead of trying to figure out exactly what your lifecycle policies are. Yeah, there are some corner cases where, okay, if I adjust this just so, then I could save 10% on that monitoring fee or whatnot. Yeah, but look how much work that's going to take you to curate and make sure that you're not doing something silly. That feels like it is such an in the margins issue. It's like, “How much data you're storing?” “Four exabytes.” Okay, yeah. You probably want some people doing exactly that, but that's not most of us.Pete: Right. Well, there's absolutely savings to be had. Like, if I had an exabyte of data on S3—which there are a lot of people who have that level of data—then it would make sense for me to have an engineering team whose sole purpose is purely an optimizing our data lifecycle for that data. Until a point, right? Until you've optimized the 80%, basically. You optimize the first 80, that's probably, air-quote, “Easy.” The last 20 is going to be incredibly hard, maybe you never even do that.But at lower levels of scale, I don't think the economics actually work out to have a team managing your data lifecycle of S3. But the fact that now AWS can largely do it for you in the background—now, there's so many things you have to think about and, like, you know, understand even what your data is there because, like, not all data is the same. And since S3 is basically like a big giant database you can query, you got to really think about some of that stuff. But honestly, what I—I don't know if—I have no idea if this is even be worked on, but what I would love to see—you know, hashtag #AWSwishlist—is, now we have countless tiers of EBS volumes, EBS volumes that can be dynamically modified without touching, you know, the physical host. Meaning with an API call, you can change from the gp2 to gp3, or io whatever, right?Corey: Or back again if it doesn't pan out.Pete: Or back again, right? And so for companies with large amounts of spend, you know, economics makes sense that you should have a team that is analyzing your volumes usage and modifying that daily, right? Like, you could modify that daily, and I don't know if there's anyone out there that's actually doing it at that level. And they probably should. Like, if you got millions of dollars in EBS, like, there's legit savings that you're probably leaving on the table without doing that. But that's what I'm waiting for Amazon to do for me, right? I want intelligent tiering for EBS because if you're telling me I can API call and you'll move my data and make that better, make that [crosstalk 00:17:46] better [crosstalk 00:17:47]—Corey: Yeah it could be like their auto-scaling for DynamoDB, for example. Gives you the capacity you need 20 minutes after you needed it. But fine, whatever because if I can schedule stuff like that, great, I know what time of day, the runs are going to kick off that beat up the disks. I know when end-of-month reporting fires off. I know what my usage pattern is going to be, by and large.Yeah, part of the problem too, is that I look at this stuff, and I get excited about it with the intelligent tiering… at The Duckbill Group we've got a few hundred S3 buckets lurking around. I'm thinking, “All right, I've got to go through and do some changes on this and implement all of that.” Our S3 bill's something like 50 bucks a month or something ridiculous like that. It's a no, that really isn't a thing. Like, I have a screenshot bucket that I have an app installed—I think called Dropshare—that hooks up to anytime I drag—I hit a shortcut, I drag with the mouse to select whatever I want and boom, it's up there and the URL is not copied to my clipboard, I can paste that wherever I want.And I'm thinking like, yeah, there's no cleanup on that. There's no lifecycle policy that's turning into anything. I should really go back and age some of it out and do the rest and start doing some lifecycle management. It—I've been using this thing for years and I think it's now a whopping, what, 20 cents a month for that bucket. It's—I just don't—Pete: [laugh].Corey: —I just don't care, other than voice in the back of my mind, “That's an unbounded growth problem.” Cool. When it hits 20 bucks a month, then I'll consider it. But until then I just don't. It does not matter.Pete: Yeah, I think yeah, scale changes everything. Start adding some zeros and percentages turned into meaningful numbers. And honestly, back on the EBS thing, the one thing that really changed my perspective of EBS, in general, is—especially coming from the early days, right? One terabyte volume, it was a hard drive in a thing. It was a virtual LUN on a SAN somewhere, probably.Nowadays, and even, like, many years after those original EBS volumes, like all the limits you get in EBS, those are actually artificial limits, right? If you're like, “My EBS volume is too slow,” it's not because, like, the hard drive it's on is too slow. That's an artificial limit that is likely put in place due to your volume choice. And so, like, once you realize that in your head, then your concept of how you store data on EBS should change dramatically.Corey: Oh, AWS had a blog post recently talking about, like, with io2 and the limits and everything, and there was architecture thinking, okay. “So, let's say this is insufficient and the quarter-million IOPS a second that you're able to get is not there.” And I'm sitting there thinking, “That is just ludicrous data volume and data interactivity model.” And it's one of those, like, I'm sitting here trying to think about, like, I haven't had to deal with a problem like that decade, just because it's, “Huh. Turns out getting these one thing that's super fast is kind of expensive.” If you paralyze it out, that's usually the right answer, and that's how the internet is mostly evolved. But there are use cases for which that doesn't work, and I'm excited to see it. I don't want to pay for it in my view, but it's nice to see it.Pete: Yeah, it's kind of fun to go into the Amazon calculator and price out one of the, like, io2 volumes and, like, maxed out. It's like, I don't know, like $50,000 a month or a hun—like, it's some just absolutely absurd number. But the beauty of it is that if you needed that value for an hour to run some intensive data processing task, you can have it for an hour and then just kill it when you're done, right? Like, that is what is most impressive.Corey: I copied 130 gigs of data to an EFS volume, which was—[unintelligible 00:21:05] EFS has gone from “This is a piece of junk,” to one of my favorite services. It really is, just because of its utility and different ways of doing things. I didn't have the foresight, just use a second EFS volume for this. So, I was unzipping a whole bunch of small files onto it. Great.It took a long time for me to go through it. All right, now that I'm done with that I want to clean all this up. My answer was to ultimately spin up a compute node and wind up running a whole bunch of—like, 400, simultaneous rm-rf on that long thing. And it was just, like, this feels foolish and dumb, but here we are. And I'm looking at the stats on it because the instance was—all right, at that point, the load average [on the instance 00:21:41] was like 200, or something like that, and the EFS volume was like, “Ohh, wow, you're really churning on this. I'm now at, like, 5% of the limit.” Like, okay, great. It turns out I'm really bad at computers.Pete: Yeah, well, that's really the trick is, like, yeah, sure, you can have a quarter-million IOPS per second, but, like, what's going to break before you even hit that limit? Probably many other things.Corey: Oh, yeah. Like, feels like on some level if something gets to that point, it a misconfiguration somewhere. But honestly, that's the thing I find weirdest about the world in which we live is that at a small-scale—if I have a bill in my $5 a month shitposting account, great. If I screw something up and cost myself a couple hundred bucks in misconfiguration it's going to stand out. At large scale, it doesn't matter if—you're spending $50 million a year or $500 million a year on AWS and someone leaks your creds, and someone spins up a whole bunch of Bitcoin miners somewhere else, you're going to see that on your bill until they're mining basically all the Bitcoin. It just gets lost in the background.Pete: I'm waiting for those—I'm actually waiting for the next level of them to get smarter because maybe you have, like, an aggressive tagging system and you're monitoring for untagged instances, but the move here would be, first get the creds and query for, like, the most used tags and start applying those tags to your Bitcoin mining instances. My God, it'll take—Corey: Just clone a bunch of tags. Congratulations, you now have a second BI Elasticsearch cluster that you're running yourself. Good work.Pete: Yeah. Yeah, that people won't find that until someone comes along after the fact that. Like, “Why do we have two have these things?” And you're like—[laugh].Corey: “Must be a DR thing.”Pete: It's maxed-out CPU. Yeah, exactly.Corey: [laugh].Pete: Oh, the terrible ideas—please, please, hackers don't take are terrible ideas.Corey: I had a, kind of, whole thing I did on Twitter years ago, talking about how I would wind up using the AWS Marketplace for an embezzlement scheme. Namely, I would just wind up spinning up something that had, like, a five-cent an hour charge or whatnot on just, like, basically rebadge the CentOS Community AMI or whatnot. Great. And then write a blog post, not attached to me, that explains how to do a thing that I'm going to be doing in production in a week or two anyway. Like, “How to build an auto-scaling group,” and reference that AMI.Then if it ever comes out, like, “Wow, why are we having all these marketplace charges on this?” “I just followed the blog post like it said here.” And it's like, “Oh, okay. You're a dumbass. The end.”That's the way to do it. A month goes by and suddenly it came out that someone had done something similarly. They wound up rebadging these community things on the marketplace and charging big money for it, and I'm sitting there going like that was a joke. It wasn't a how-to. But yeah, every time I make these jokes, I worry someone's going to do it.Pete: “Welcome to large-scale fraud with Corey Quinn.”Corey: Oh, yeah, it's fraud at scale is really the important thing here.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: I still remember a year ago now at re:Invent 2021 was it, or was it 2020? Whatever they came out with, I want to say it wasn't gp3, or maybe it was, regardless, there was a new EBS volume type that came out that you were playing with to see how it worked and you experimented with it—Pete: Oh, yes.Corey: —and the next morning, you looked at the—I checked Slack and you're like well, my experiments yesterday cost us $5,000. And at first, like, the—my response is instructive on this because, first, it was, “Oh, my God. What's going to happen now?” And it's like, first, hang on a second.First off, that seems suspect but assume it's real. I assumed it was real at the outset. It's “Oh, right. This is not my personal $5-a-month toybox account. We are a company; we can absolutely pay that.” Because it's like, I could absolutely reach out, call it a favor. “I made a mistake, and I need a favor on the bill, please,” to AWS.And I would never live it down, let's be clear. For a $7,000 mistake, I would almost certainly eat it. As opposed to having to prostrate myself like that in front of Amazon. I'm like, no, no, no. I want one of those like—if it's like, “Okay, you're going to, like, set back the company roadmap by six months if you have to pay this. Do you want to do it?” Like, [groans] “Fine, I'll eat some crow.”But okay. And then followed immediately by, wow, if Pete of all people can mess this up, customers are going to be doomed here. We should figure out what happened. And I'm doing the math. Like, Pete, “What did you actually do?” And you're sitting there and you're saying, “Well, I had like a 20 gig volume that I did this.” And I'm doing the numbers, and it's like—Pete: Something's wrong.Corey: “How sure are you when you say ‘gigabyte,' that you were—that actually means what you think it did? Like, were you off by a lot? Like, did you mean exabytes?” Like, what's the deal here?Pete: Like, multiple factors.Corey: Yeah. How much—“How many IOPS did you give that thing, buddy?” And it turned out what happened was that when they launched this, they had mispriced it in the system by a factor of a million. So, it was fun. I think by the end of it, all of your experimentation was somewhere between five to seven cents. Which—Pete: Yeah. It was a—Corey: Which is why you don't work here anymore because no one cost me seven cents of money to give to Amazon—Pete: How dare you?Corey: —on my watch. Get out.Pete: How dare you, sir?Corey: Exactly.Pete: Yeah, that [laugh] was amazing to see, as someone who has done—definitely maid screw-ups that have cost real money—you know, S3 list requests are always a fun one at scale—but that one was supremely fun to see the—Corey: That was a scary one because another one they'd done previously was they had messed up Lightsail pricing, where people would log in, and, like, “Okay, so what is my Lightsail instance going to cost?” And I swear to you, this is true, it was saying—this was back in 2017 or so—the answer was, like, “$4.3 billion.” Because when you see that you just start laughing because you know it's a mistake. You know, that they're not going to actually demand that you spend $4.3 billion for a single instance—unless it's running SAP—and great.It's just, it's a laugh. It's clearly a mispriced, and it's clearly a bug that's going to get—it's going to get fixed. I just spun up this new EBS volume that no one fully understands yet and it cost me thousands of dollars. That's the sort of thing that no, no, I could actually see that happening. There are instances now that cost something like 100 bucks an hour or whatnot to run. I can see spinning up the wrong thing by mistake and getting bitten by it. There's a bunch of fun configuration mistakes you can make that will, “Hee, hee, hee. Why can I see that bill spike from orbit?” And that's the scary thing.Pete: Well, it's the original CI and CD problem of the per-hour billing, right? That was super common of, like, yeah, like, an i3, you know, 16XL server is pretty cheap per hour, but if you're charged per hour and you spin up a bunch for five minutes. Like, it—you will be shocked [laugh] by what you see there. So—Corey: Yeah. Mistakes will show. And I get it. It's also people as individuals are very different psychologically than companies are. With companies it's one of those, “Great we're optimizing to bring in more revenue and we don't really care about saving money at all costs.”Whereas people generally have something that looks a lot like a fixed income in the form of a salary or whatnot, so it's it is easier for us to cut spend than it is for us to go out and make more money. Like, I don't want to get a second job, or pitch my boss on stuff, and yeah. So, all and all, routing out the rest of what happened at re:Invent, they—this is the problem is that they have a bunch of minor things like SageMaker Inference Recommender. Yeah, I don't care. Anything—Pete: [laugh].Corey: —[crosstalk 00:28:47] SageMaker I mostly tend to ignore, for safety. I did like the way they described Amplify Studio because they made it sound like a WYSIWYG drag and drop, build a React app. It's not it. It basically—you can do that in Figma and then it can hook it up to some things in some cases. It's not what I want it to be, which is Honeycode, except good. But we'll get there some year. Maybe.Pete: There's a lot of stuff that was—you know, it's the classic, like, preview, which sure, like, from a product standpoint, it's great. You know, they have a level of scale where they can say, “Here's this thing we're building,” which could be just a twinkle in a product managers, call it preview, and get thousands of people who would be happy to test it out and give you feedback, and it's a, it's great that you have that capability. But I often look at so much stuff and, like, that's really cool, but, like, can I, can I have it now? Right? Like—or you can't even get into the preview plan, even though, like, you have that specific problem. And it's largely just because either, like, your scale isn't big enough, or you don't have a good enough relationship with your account manager, or I don't know, countless other reasons.Corey: The thing that really throws me, too, is the pre-announcements that come a year or so in advance, like, the Outpost smaller ones are finally available, but it feels like when they do too many pre-announcements or no big marquee service announcements, as much as they talk about, “We're getting back to fundamentals,” no, you have a bunch of teams that blew the deadline. That's really what it is; let's not call it anything else. Another one that I think is causing trouble for folks—I'm fortunate in that I don't do much work with Oracle databases, or Microsoft SQL databases—but they extended RDS Custom to Microsoft SQL at the [unintelligible 00:30:27] SQL server at re:Invent this year, which means this comes down to things I actually use, we're going to have a problem because historically, the lesson has always been if I want to run my own databases and tweak everything, I do it on top of an EC2 instance. If I want to managed database, relational database service, great, I use RDS. RDS Custom basically gives you root into the RDS instance. Which means among other things, yes, you can now use RDS to run containers.But it lets you do a lot of things that are right in between. So, how do you position this? When should I use RDS Custom? Can you give me an easy answer to that question? And they used a lot of words to say, no, they cannot. It's basically completely blowing apart the messaging and positioning of both of those services in some unfortunate ways. We'll learn as we go.Pete: Yeah. Honestly, it's like why, like, why would I use this? Or how would I use this? And this is I think, fundamentally, what's hard when you just say yes to everything. It's like, they in many cases, I don't think, like, I don't want to say they don't understand why they're doing this, but if it's not like there's a visionary who's like, this fits into this multi-year roadmap.That roadmap is largely—if that roadmap is largely generated by the customers asking for it, then it's not like, oh, we're building towards this Northstar of RDS being whatever. You might say that, but your roadmap's probably getting moved all over the place because, you know, this company that pays you a billion dollars a year is saying, “I would give you $2 billion a year for all of my Oracle databases, but I need this specific thing.” I can't imagine a scenario that they would say, “Oh, well, we're building towards this Northstar, and that's not on the way there.” Right? They'd be like, “New Northstar. Another billion dollars, please.”Corey: Yep. Probably the worst release of re:Invent, from my perspective, is RUM, Real User Monitoring, for CloudWatch. And I, to be clear, I wrote a shitposting Twitter threading client called Last Tweet in AWS. Go to lasttweetinaws.com. You can all use it. It's free; I just built this for my own purposes. And I've instrumented it with RUM. Now, Real User Monitoring is something that a lot of monitoring vendors use, and also CloudWatch now. And what that is, is it embeds a listener into the JavaScript that runs on client load, and it winds up looking at what's going on loading times, et cetera, so you can see when users are unhappy. I have no problem with this. Other than that, you know, liking users? What's up with that?Pete: Crazy.Corey: But then, okay, now, what this does is unlike every other RUM tool out there, which charges per session, meaning I am going to be… doing a web page load, it charges per data item, which includes HTTP errors, or JavaScript errors, et cetera. Which means that if you have a high transaction volume site and suddenly your CDN takes a nap like Fastly did for an hour last year, suddenly your bill is stratospheric for this because errors abound and cascade, and you can have thousands of errors on a single page load for these things, and it is going to be visible from orbit, at least with a per session basis thing, when you start to go viral, you understand that, “Okay, this is probably going to cost me some more on these things, and oops, I guess I should write less compelling content.” Fine. This is one of those one misconfiguration away and you are wailing and gnashing teeth. Now, this is a new service. I believe that they will waive these surprise bills in the event that things like that happen. But it's going to take a while and you're going to be worrying the whole time if you've rolled this out naively. So it's—Pete: Well and—Corey: —I just don't like the pricing.Pete: —how many people will actively avoid that service, right? And honestly, choose a competitor because the competitor could be—the competitor could be five times more expensive, right, on face value, but it's the certainty of it. It's the uncertainty of what Amazon will charge you. Like, no one wants a surprise bill. “Well, a vendor is saying that they'll give us this contract for $10,000. I'm going to pay $10,000, even though RUM might be a fraction of that price.”It's honestly, a lot of these, like, product analytics tools and monitoring tools, you'll often see they price be a, like, you know, MAU, Monthly Active User, you know, or some sort of user-based pricing, like, the number of people coming to your site. You know, and I feel like at least then, if you are trying to optimize for lots of users on your site, and more users means more revenue, then you know, if your spend is going up, but your revenue is also going up, that's a win-win. But if it's like someone—you know, your third-party vendor dies and you're spewing out errors, or someone, you know, upgraded something and it spews out errors. That no one would normally see; that's the thing. Like, unless you're popping open that JavaScript console, you're not seeing any of those errors, yet somehow it's like directly impacting your bottom line? Like that doesn't feel [crosstalk 00:35:06].Corey: Well, there is something vaguely Machiavellian about that. Like, “How do I get my developers to care about errors on consoles?” Like, how about we make it extortionately expensive for them not to. It's, “Oh, all right, then. Here we go.”Pete: And then talk about now you're in a scenario where you're working on things that don't directly impact the product. You're basically just sweeping up the floor and then trying to remove errors that maybe don't actually affect it and they're not actually an error.Corey: Yeah. I really do wonder what the right answer is going to be. We'll find out. Again, we live, we learn. But it's also, how long does it take a service that has bad pricing at launch, or an unfortunate story around it to outrun that reputation?People are still scared of Glacier because of its original restore pricing, which was non-deterministic for any sensible human being, and in some cases lead to I'm used to spending 20 to 30 bucks a month on this. Why was I just charged two grand?Pete: Right.Corey: Scare people like that, they don't come back.Pete: I'm trying to actually remember which service it is that basically gave you an estimate, right? Like, turn it on for a month, and it would give you an estimate of how much this was going to cost you when billing started.Corey: It was either Detective or GuardDuty.Pete: Yeah, it was—yeah, that's exactly right. It was one of those two. And honestly, that was unbelievably refreshing to see. You know, like, listen, you have the data, Amazon. You know what this is going to cost me, so when I, like, don't make me spend all this time to go and figure out the cost. If you have all this data already, just tell me, right?And if I look at it and go, “Yeah, wow. Like, turning this on in my environment is going to cost me X dollars. Like, yeah, that's a trade-off I want to make, I'll spend that.” But you know, with some of the—and that—a little bit of a worry on some of the intelligent tiering on S3 is that the recommendation is likely going to be everything goes to intelligent tiering first, right? It's the gp3 story. Put everything on gp3, then move it to the proper volume, move it to an sc or an st or an io. Like, gp3 is where you start. And I wonder if that's going to be [crosstalk 00:37:08].Corey: Except I went through a wizard yesterday to launch an EC2 instance and its default on the free tier gp2.Pete: Yeah. Interesting.Corey: Which does not thrill me. I also still don't understand for the life of me why in some regions, the free tier is a t2 instance, when t3 is available.Pete: They're uh… my guess is that they've got some free t—they got a bunch of t2s lying around. [laugh].Corey: Well, one of the most notable announcements at re:Invent that most people didn't pay attention to is their ability now to run legacy instance types on top of Nitro, which really speaks to what's going on behind the scenes of we can get rid of all that old hardware and emulate the old m1 on modern equipment. So, because—you can still have that legacy, ancient instance, but now you're going—now we're able to wind up greening our data centers, which is part of their big sustainability push, with their ‘Sustainability Pillar' for the well-architected framework. They're talking more about what the green choices in cloud are. Which is super handy, not just because of the economic impact because we could use this pretty directly to reverse engineer their various margins on a per-service or per-offering basis. Which I'm not sure they're aware of yet, but oh, they're going to be.And that really winds up being a win for the planet, obviously, but also something that is—that I guess puts a little bit of choice on customers. The challenge I've got is, with my serverless stuff that I build out, if I spend—the Google search I make to figure out what the most economic, most sustainable way to do that is, is going to have a bigger carbon impact on the app itself. That seems to be something that is important at scale, but if you're not at scale, it's one of those, don't worry about it. Because let's face it, the cloud providers—all of them—are going to have a better sustainability story than you are running this in your own data centers, or on a Raspberry Pi that's always plugged into the wall.Pete: Yeah, I mean, you got to remember, Amazon builds their own power plants to power their data centers. Like, that's the level they play, right? There, their economies of scale are so entirely—they're so entirely different than anything that you could possibly even imagine. So, it's something that, like, I'm sure people will want to choose for. But, you know, if I would honestly say, like, if we really cared about our computing costs and the carbon footprint of it, I would love to actually know the carbon footprint of all of the JavaScript trackers that when I go to various news sites, and it loads, you know, the whatever thousands of trackers and tracking the all over, like, what is the carbon impact of some of those choices that I actually could control, like, as a either a consumer or business person?Corey: I really hope that it turns into something that makes a meaningful difference, and it's not just greenwashing. But we'll see. In the fullness of time, we're going to figure that out. Oh, they're also launching some mainframe stuff. They—like that's great.Pete: Yeah, those are still a thing.Corey: I don't deal with a lot of customers that are doing things with that in any meaningful sense. There is no AWS/400, so all right.Pete: [laugh]. Yeah, I think honestly, like, I did talk to a friend of mine who's in a big old enterprise and has a mainframe, and they're actually replacing their mainframe with Lambda. Like they're peeling off—which is, like, a great move—taking the monolith, right, and peeling off the individual components of what it can do into these discrete Lambda functions. Which I thought was really fascinating. Again, it's a five-year-long journey to do something like that. And not everyone wants to wait five years, especially if their support's about to run out for that giant box in the, you know, giant warehouse.Corey: The thing that I also noticed—and this is probably the—I guess, one of the—talk about swing and a miss on pricing—they have a—what is it?—there's a VPC IP Address Manager, which tracks the the IP addresses assigned to your VPCs that are allocated versus not, and it's 20 cents a month per IP address. It's like, “Okay. So, you're competing against a Google Sheet or an Excel spreadsheet”—which is what people are using for these things now—“Only you're making it extortionately expensive?”Pete: What kind of value does that provide for 20—I mean, like, again—Corey: I think Infoblox or someone like that offers it where they become more cost-effective as soon as you hit 500 IP addresses. And it's just—like, this is what I'm talking about. I know it does not cost AWS that kind of money to store an IP address. You can store that in a Route 53 TXT record for less money, for God's sake. And that's one of those, like, “Ah, we could extract some value pricing here.”Like, I don't know if it's a good product or not. Given its pricing, I don't give a shit because it's going to be too expensive for anything beyond trivial usage. So, it's a swing and a miss from that perspective. It's just, looking at that, I laugh, and I don't look at it again.Pete: See I feel—Corey: I'm not usually price sensitive. I want to be clear on that. It's just, that is just Looney Tunes, clown shoes pricing.Pete: Yeah. It's honestly, like, in many cases, I think the thing that I have seen, you know, in the past few years is, in many cases, it can honestly feel like Amazon is nickel-and-diming their customers in so many ways. You know, the explosion of making it easy to create multiple Amazon accounts has a direct impact to waste in the cloud because there's a lot of stuff you have to have her account. And the more accounts you have, those costs grow exponentially as you have these different places. Like, you kind of lose out on the economies of scale when you have a smaller number of accounts.And yeah, it's hard to optimize for that. Like, if you're trying to reduce your spend, it's challenging to say, “Well, by making a change here, we'll save, you know, $10,000 in this account.” “That doesn't seem like a lot when we're spending millions.” “Well, hold on a second. You'll save $10,000 per account, and you have 500 accounts,” or, “You have 1000 accounts,” or something like that.Or almost cost avoidance of this cost is growing unbounded in all of your accounts. It's tiny right now. So, like, now would be the time you want to do something with it. But like, again, for a lot of companies that have adopted the practice of endless Amazon accounts, they've almost gone, like, it's the classic, like, you know, I've got 8000 GitHub repositories for my source code. Like, that feels just as bad as having one GitHub repository for your repo. I don't know what the balance is there, but anytime these different types of services come out, it feels like, “Oh, wow. Like, I'm going to get nickeled and dimed for it.”Corey: This ties into the re:Post launch, which is a rebranding of their forums, where, okay, great, it was a little crufty and it need modernize, but it still ties your identity to an IAM account, or the root email address for an Amazon account, which is great. This is completely worthless because as soon as I change jobs, I lose my identity, my history, the rest, on this forum. I'm not using it. It shows that there's a lack of awareness that everyone is going to have multiple accounts with which they interact, and that people are going to deal with the platform longer than any individual account will. It's just a continual swing and a miss on things like that.And it gets back to the billing question of, “Okay. When I spin up an account, do I want them to just continue billing me—because don't turn this off; this is important—or do I want there to be a hard boundary where if you're about to charge me, turn it off. Turn off the thing that's about to cost me money.” And people hem and haw like this is an insurmountable problem, but I think the way to solve it is, let me specify that intent when I provision the account. Where it's, “This is a production account for a bank. I really don't want you turning it off.” Versus, “I'm a student learner who thinks that a Managed NAT Gateway might be a good thing. Yeah, I want you to turn off my demo Hello World app that will teach me what's going on, rather than surprising me with a five-figure bill at the end of the month.”Pete: Yeah. It shouldn't be that hard. I mean, but again, I guess everything's hard at scale.Corey: Oh, yeah. Oh yeah.Pete: But still, I feel like every time I log into Cost Explorer and I look at—and this is years it's still not fixed. Not that it's even possible to fix—but on the first day of the month, you look at Cost Explorer, and look at what Amazon is estimating your monthly bill is going to be. It's like because of your, you know—Corey: Your support fees, and your RI purchases, and savings plans purchases.Pete: [laugh]. All those things happened, right? First of the month, and it's like, yeah, “Your bill's going to be $800,000 this year.” And it's like, “Shouldn't be, like, $1,000?” Like, you know, it's the little things like that, that always—Corey: The one-off charges, like, “Oh, your Route 53 zone,” and all the stuff that gets charged on a monthly cadence, which fine, whatever. I mean, I'm okay with it, but it's also the, like, be careful when that happen—I feel like there's a way to make that user experience less jarring.Pete: Yeah because that problem—I mean, in my scenario, companies that I've worked at, there's been multiple times that a non-technical person will look at that data and go into immediate freakout mode, right? And that's never something that you want to have happen because now that's just adding a lot of stress and anxiety into a company that is—with inaccurate data. Like, the data—like, the answer you're giving someone is just wrong. Perhaps you shouldn't even give it to them if it's that wrong. [laugh].Corey: Yeah, I'm looking forward to seeing what happens this coming year. We're already seeing promising stuff. They—give people a timeline on how long in advance these things record—late last night, AWS released a new console experience. When you log into the AWS console now, there's a new beta thing. And I gave it some grief on Twitter because I'm still me, but like the direction it's going. It lets you customize your view with widgets and whatnot.And until they start selling widgets on marketplace or having sponsored widgets, you can't remove I like it, which is no guarantee at some point. But it shows things like, I can move the cost stuff, I can move the outage stuff up around, I can have the things that are going on in my account—but who I am means I can shift this around. If I'm a finance manager, cool. I can remove all the stuff that's like, “Hey, you want to get started spinning up an EC2 instance?” “Absolutely not. Do I want to get told, like, how to get certified? Probably not. Do I want to know what the current bill is and whether—and my list of favorites that I've pinned, whatever services there? Yeah, absolutely do.” This is starting to get there.Pete: Yeah, I wonder if it really is a way to start almost hedging on organizations having a wider group of people accessing AWS. I mean, in previous companies, I absolutely gave access to the console for tools like QuickSight, for tools like Athena, for the DataBrew stuff, the Glue DataBrew. Giving, you know, non-technical people access to be able to do these, like, you know, UI ETL tasks, you know, a wider group of a company is getting access into Amazon. So, I think anything that Amazon does to improve that experience for, you know, the non-SREs, like the people who would traditionally log in, like, that is an investment definitely worth making.Corey: “Well, what could non-engineering types possibly be doing in the AWS console?” “I don't know, jackhole, maybe paying the bill? Just a thought here.” It's the, there are people who look at these things from a variety of different places, and you have such sprawl in the AWS world that there are different personas by a landslide. If I'm building Twitter for Pets, you probably don't want to be pitching your mainframe migration services to me the same way that you would if I were a 200-year-old insurance company.Pete: Yeah, exactly. And the number of those products are going to grow, the number of personas are going to grow, and, yeah, they'll have to do something that they want to actually, you know, maintain that experience so that every person can have, kind of, the experience that they want, and not be distracted, you know? “Oh, what's this? Let me go test this out.” And it's like, you know, one-time charge for $10,000 because, like, that's how it's charged. You know, that's not an experience that people like.Corey: No. They really don't. Pete, I want to thank you for spending the time to chat with me again, as is our tradition. I'm hoping we can do it in person this year, when we go at the end of 2022, to re:Invent again. Or that no one goes in person. But this hybrid nonsense is for the birds.Pete: Yeah. I very much would love to get back to another one, and yeah, like, I think there could be an interesting kind of merging here of our annual re:Invent recap slash live brunch, you know, stream you know, hot takes after a long week. [laugh].Corey: Oh, yeah. The real way that you know that it's a good joke is when one of us says something, the other one sprays scrambled eggs out of their nose. Yeah, that's the way to do it.Pete: Exactly. Exactly.Corey: Pete, thank you so much. If people want to learn more about what you're up to—hopefully, you know, come back. We miss you, but you're unaffiliated, you're a startup advisor. Where can people find you to learn more, if they for some unforgivable reason don't know who or what a Pete Cheslock is?Pete: Yeah. I think the easiest place to find me is always on Twitter. I'm just at @petecheslock. My DMs are always open and I'm always down to expand my network and chat with folks.And yeah, right, now, I'm just, as I jokingly say, professionally unaffiliated. I do some startup advisory work and have been largely just kind of—honestly checking out the state of the economy. Like, there's a lot of really interesting companies out there, and some interesting problems to solve. And, you know, trying to spend some of my time learning more about what companies are up to nowadays. So yeah, if you got some interesting problems, you know, you can follow my Twitter or go to LinkedIn if you want some great, you know, business hot takes about, you know, shitposting basically.Corey: Same thing. Pete, thanks so much for joining me, I appreciate it.Pete: Thanks for having me.Corey: Pete Cheslock, startup advisor, professionally unaffiliated, and recurring re:Invent analyst pal of mine. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment calling me a jackass because do I know how long it took you personally to price CloudWatch RUM?Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

AWS Morning Brief
CISOs Should Ideally Stay Out of Prison

AWS Morning Brief

Play Episode Listen Later Jan 13, 2022 6:22


Links: Comes with a cryptominer: https://krebsonsecurity.com/2022/01/norton-360-now-comes-with-a-cryptominer/ You could be federally charged with wire fraud for paying off a security researcher: https://www.justice.gov/usao-ndca/pr/former-uber-chief-security-officer-face-wire-fraud-charges-0 A source code leak of its Azure App Service: https://www.theregister.com/2021/12/24/azure_app_service_not_legit_source_code_leak/ “Comprehensive Cyber Security Framework for Primary (Urban) Cooperative Banks (UCBs)”: https://aws.amazon.com/blogs/security/comprehensive-cyber-security-framework-for-primary-urban-cooperative-banks/ “Disabling Security Hub controls in a multi account environment”: https://aws.amazon.com/blogs/security/disabling-security-hub-controls-in-a-multi-account-environment/ Ipv6-ghost-ship: https://github.com/aidansteele/ipv6-ghost-ship TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it's nobody in particular's job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.This episode is sponsored in part by our friends at Rising Cloud, which I hadn't heard of before, but they're doing something vaguely interesting here. They are using AI, which is usually where my eyes glaze over and I lose attention, but they're using it to help developers be more efficient by reducing repetitive tasks. So, the idea being that you can run stateless things without having to worry about scaling, placement, et cetera, and the rest. They claim significant cost savings, and they're able to wind up taking what you're running as it is in AWS with no changes, and run it inside of their data centers that span multiple regions. I'm somewhat skeptical, but their customers seem to really like them, so that's one of those areas where I really have a hard time being too snarky about it because when you solve a customer's problem and they get out there in public and say, “We're solving a problem,” it's very hard to snark about that. Multus Medical, Construx.ai and Stax have seen significant results by using them. And it's worth exploring. So, if you're looking for a smarter, faster, cheaper alternative to EC2, Lambda, or batch, consider checking them out. Visit risingcloud.com/benefits. That's risingcloud.com/benefits, and be sure to tell them that I said you because watching people wince when you mention my name is one of the guilty pleasures of listening to this podcast.Welcome to Last Week in AWS: Security. Let's dive in. Norton 360—which sounds like a prelude to an incredibly dorky attempt at the moonwalk—now comes with a cryptominer. You know, the thing that use tools like this to avoid having on your computer? This is apparently to offset how zippy modern computers have gotten, in a direct affront to Norton's ability to make even maxed-out laptops run like total garbage. Speaking of total garbage, you almost certainly want to use literally any other vendor for this stuff now.“What's the worst that can happen?” Is sometimes a comforting thought when dealing with professional challenges. If you're the former Uber CISO, the answer to that question is apparently, “you could be federally charged with wire fraud for paying off a security researcher.”And lastly, Azure continues to have security woes, this time in the form of a source code leak of its Azure App Service. It's a bad six months and counting to be over in Microsoft-land when it comes to cloud.Let's take a look what AWS has done. “Comprehensive Cyber Security Framework for Primary (Urban) Cooperative Banks (UCBs)”. This is a perfect case study in what's wrong with the way we talk about security. First, clicking the link to the report in the blog post threw an error; I had to navigate to the AWS Artifact console and download the PDF manually. Then, the PDF is all of two pages long, as it apparently has an embedded Excel document within it that Preview on my Mac can't detect. The proper next step is to download Adobe Acrobat for Mac in order to read this, but I've given up by this point. This may be the most remarkable case of AWS truly understanding its customer mentality that we've seen so far this year.Are you building cloud applications with a distributed team? Check out Teleport, an open-source identity-aware access proxy for cloud resources. Teleport provides secure access for anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps, and databases. Teleport gives engineers superpowers. Get access to everything via single sign-on with multi-factor, list and see all of SSH servers, Kubernetes clusters, or databases available to you in one place, and get instant access to them using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility, and ensuring compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That's goteleport.com.“Disabling Security Hub controls in a multi account environment”. I hate that this is a solution instead of a native feature, but it's important. There are some Security Hub controls that are just nonsense. “Oh no, you didn't encrypt your EBS volumes.” “Oh dear, you haven't rotated your IAM credentials in 90 days.” “Holy CRAP, the S3 bucket serving static assets to the world is world-readable.” You get the picture.And a tool I found fun, “Port Knocking” is an old security technique in which you attempt to connect to a host on a predetermined sequence of ports. Get it right and you're now able to connect to the host in question on the port that you want. ipv6-ghost-ship has done something similar yet ever more ridiculous: It takes advantage of the fact that IPv6 means that each EC2 instance gets 281 trillion IP addresses to only accept SSH connections when the last three octets of the IP address on the instance match the time-based authentication code. This is a ridiculous hack, and I love it oh so very much. I'm Chief Cloud Economist at The Duckbill Group, and this has been Last Week in AWS: Security. Thanks for listening.Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
“Cloudash”ing onto Mac with Maciej Winnicki

Screaming in the Cloud

Play Episode Listen Later Jan 13, 2022 34:41


About MaciejMaciej Winnicki is a serverless enthusiast with over 6 years of experience in writing software with no servers whatsoever. Serverless Engineer at Stedi, Cloudash Founder, ex-Engineering Manager, and one of the early employees at Serverless Inc.Links: Cloudash: https://cloudash.dev Maciej Winnicki Twitter: https://twitter.com/mthenw Tomasz Łakomy Twitter: https://twitter.com/tlakomy Cloudash email: hello@cloudash.dev TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: This episode is sponsored in part by our friends at Rising Cloud, which I hadn't heard of before, but they're doing something vaguely interesting here. They are using AI, which is usually where my eyes glaze over and I lose attention, but they're using it to help developers be more efficient by reducing repetitive tasks. So, the idea being that you can run stateless things without having to worry about scaling, placement, et cetera, and the rest. They claim significant cost savings, and they're able to wind up taking what you're running as it is in AWS with no changes, and run it inside of their data centers that span multiple regions. I'm somewhat skeptical, but their customers seem to really like them, so that's one of those areas where I really have a hard time being too snarky about it because when you solve a customer's problem and they get out there in public and say, “We're solving a problem,” it's very hard to snark about that. Multus Medical, Construx.ai and Stax have seen significant results by using them. And it's worth exploring. So, if you're looking for a smarter, faster, cheaper alternative to EC2, Lambda, or batch, consider checking them out. Visit risingcloud.com/benefits. That's risingcloud.com/benefits, and be sure to tell them that I said you because watching people wince when you mention my name is one of the guilty pleasures of listening to this podcast.Corey: Welcome to Screaming in the Cloud. I'm Cloud Economist Corey Quinn. And my guest today is Maciej Winnicki, who is the founder of Cloudash. Now, before I dive into the intricacies of what that is, I'm going to just stake out a position that one of the biggest painful parts of working with AWS in any meaningful sense, particularly in a serverless microservices way, is figuring out what the hell's going on in the environment. There's a bunch of tools offered to do this and they're all—yeee, they aspire to mediocrity. Maciej, thank you for joining me today.Corey: Welcome to Screaming in the Cloud. I'm Cloud Economist Corey Quinn. And my guest today is Maciej Winnicki, who is the founder of Cloudash. Now, before I dive into the intricacies of what that is, I'm going to just stake out a position that one of the biggest painful parts of working with AWS in any meaningful sense, particularly in a serverless microservices way, is figuring out what the hell's going on in the environment. There's a bunch of tools offered to do this and they're all—yeee, they aspire to mediocrity. Maciej, thank you for joining me today.Maciej: Thank you for having me.Corey: So, I turned out to have accidentally blown up Cloudash, sort of before you were really ready for the attention. You, I think, tweeted about it or put it on Hacker News or something; I stumbled over it because it turns out that anything that vaguely touches cloud winds up in my filters because of awesome technology, and personality defects on my part. And I tweeted about it as I set it up and got the thing running, and apparently this led to a surge of attention on this thing that you've built. So, let me start off with an apology. Oops, I didn't realize it was supposed to be a quiet launch.Maciej: I actually thank you for that. Like, that was great. And we get a lot of attention from your tweet thread, actually because at the end, that was the most critical part. At the end of the twitter, you wrote that you're staying as a customer, so we have it on our website and this is perfect. But actually, as you said, that's correct.Our marketing strategy for releasing Cloudash was to post it on LinkedIn. I know this is not, kind of, the best strategy, but that was our plan. Like, it was like, hey, like, me and my friend, Tomasz, who's also working on Cloudash, we thought like, let's just post it on LinkedIn and we'll see how it goes. And accidentally, I'm receiving a notification from Twitter, “Hey, Corey started tweeting about it.” And I was like, “Oh, my God, I'm having a heart attack.” But then I read the, you know—Corey: Oops.Maciej: [laugh]. Yeah. I read the, kind of, conclusion, and I was super happy. And again, thank you for that because this is actually when Cloudash kind of started rolling as a product and as a, kind of, business. So yeah, that was great.Corey: To give a little backstory and context here is, I write a whole bunch of serverless nonsense. I build API's Gateway, I hook them up to Lambda's Function, and then it sort of kind of works. Ish. From there, okay, I would try and track down what was going on because in a microservices land, everything becomes a murder mystery; you're trying to figure out what's broken, and things have exploded. And I became a paying customer of IOpipe. And then New Relic bought them. Well, crap.Then I became a paying customer of Epsagon. And they got acquired by Cisco, at which point I immediately congratulated the founders, who I know on a social basis, and then closed my account because I wanted to get out before Cisco ruins it because, Cisco. Then it was, what am I going to use next? And right around that time is when I stumbled across Cloudash. And it takes a different approach than any other entity in the space that I've seen because you are a native Mac desktop app. I believe your Mac only, but you seem to be Electron, so I could be way off base on that.Maciej: So, we're Linux as well right now and soon we'll be Windows as well. But yeah, so, right now is Mac OS and Linux. Yeah, that's correct. So, our approach is a little bit different.So, let me start by saying what's Cloudash? Like, Cloudash is a desktop app for, kind of, monitoring and troubleshooting serverless architectures services, like, serverless stuff in general. And the approach that we took is a little bit different because we are not web-based, we're desktop-based. And there's a couple of advantages of that approach. The first one is that, like, you don't need to share your data with us because we're not, kind of, downloading your metrics and logs to our back end and to process them, et cetera, et cetera. We are just using the credentials, the AWS profiles that you have defined on your computer, so nothing goes out of your AWS account.And I think this is, like, considering, like, from the security perspective, this is very crucial. You don't need to create a role that you give us access to or anything like that. You just use the stuff that you have on your desktop, and everything stays on your AWS account. So, nothing—we don't download it, we don't process it, we don't do anything from that. And that's one approach—well, that's the one advantage. The other advantage is, like, kind of, onboarding, as I kind of mentioned because we're using the AWS profiles that you have defined in your computer.Corey: Well, you're doing significantly more than that because I have a lot of different accounts configured different ways, and when I go to one of them that uses SSO, it automatically fires me off to the SSO login page if I haven't logged in that day for a 12 hour session—Maciej: Yes.Corey: —for things that have credentials stored locally, it uses those; and for things that are using role-chaining to use assuming roles from the things I have credentials for, and the things that I just do role assumption in, and it works flawlessly. It just works the way that most of my command-line tools do. I've never seen a desktop app that does this.Maciej: Yeah. So, we put a lot of effort into making sure that this works great because we know that, like, no one will use Cloudash if there's—like, not no one, but like, we're targeting, like, serverless teams, maybe, in enterprise companies, or serverless teams working on some startups. And in most cases, those teams or those engineers, they use SSO, or at least MFA, right? So, we have it covered. And as you said, like, it should be the onboarding part is really easy because you just pick your AWS profile, you just pick region, and just pick, right now, a CloudFormation stack because we get the information about your service based on CloudFormation stack. So yeah, we put a lot of effort in making sure that this works without any issues.Corey: There are some challenges to it that I saw while setting it up, and that's also sort of the nature of the fact you are, in fact, integrating with CloudWatch. For example, it's region specific. Well, what if I want to have an app that's multi-region? Well, you're going to have a bad time because doing [laugh] anything multi-region in AWS means you're going to have a bad time that gets particularly obnoxious and EC2 get to when you're doing something like Lambda@Edge, where, oh, where are the logs live; that's going to be in a CloudFront distribution in whatever region it winds up being accessed from. So, it comes down to what distribution endpoint or point of presence did that particular request go through, and it becomes this giant game of whack-a-mole. It's frustrating, and it's obnoxious, and it's also in no way your fault.Maciej: Yeah, I mean, we are at the beginning. Right now, it's the most straightforward, kind of pe—how people think about stacks of serverless. They're think in terms of regions because I think for us, regions, or replicated stacks, or things like that are not really popular yet. Maybe they will become—like, this is how AWS works as a whole, so it's not surprising that we're kind of following this path. I think my point is that our main goal, the ultimate goal, is to make monitoring, as I said, the troubleshooting serverless app as simple as possible.So, once we will hear from our customers, from our users that, “Hey, we would like to get a little bit better experience around regions,” we will definitely implement that because why not, right? And I think the whole point of Cloudash—and maybe we can go more deep into that later—is that we want to bring context into your metrics and logs. If you're seeing a, for example, X-Ray trace ID in your logs, you should be able with one click just see that the trace. It's not yet implemented in Cloudash, but we are having it in the backlog. But my point is that, like, there should be some journey when you're debugging stuff, and you shouldn't be just, like, left alone having, like, 20 tabs, Cloudash tabs open and trying to figure out where I was—like, where's the Lambda? Where's the API Gateway logs? Where are the CloudFront logs? And how I can kind of connect all of that? Because that's—it's an issue right now.Corey: Even what you've done so far is incredibly helpful compared to the baseline experience that folks will often have, where I can define a service that is comprised of a number of different functions—I have one set up right now that has seven functions in it—I grab any one of those things, and I can set how far the lookback is, when I look at that function, ranging from 5 minutes to 30 days. And it shows me at the top the metrics of invocations, the duration that the function runs for, and the number of errors. And then, in the same pane down below it, it shows the CloudWatch logs. So, “Oh, okay, great. I can drag and zoom into a specific timeframe, and I see just the things inside of that.”And I know this sounds like well, what's the hard part here? Yeah, except nothing else does it in an easy-to-use, discoverable way that just sort of hangs out here. Honestly, the biggest win for me is that I don't have to log in to the browser, navigate through some ridiculous other thing to track down what I'm talking about. It hangs out on my desktop all the time, and whether it's open or not, whenever I fire it up, it just works, basically, and I don't have to think about it. It reduces the friction from, “This thing is broken,” to, “Let me see what the logs say.”Very often I can go from not having it open at all to staring at the logs and having to wait a minute because there's some latency before the event happens and it hits CloudWatch logs itself. I'm pretty impressed with it, and I've been keeping an eye on what this thing is costing me. It is effectively nothing in terms of CloudWatch retrieval charges. Because it's not sitting there sucking all this data up all the time, for everything that's running. Like, we've all seen the monitoring system that winds up costing you more than it costs more than they charge you ancillary fees. This doesn't do that.I also—while we're talking about money, I want to make very clear—because disclaiming the direction the money flows in is always important—you haven't paid me a dime, ever, to my understanding. I am a paying customer at full price for this service, and I have been since I discovered it. And that is very much an intentional choice. You did not sponsor this podcast, you are not paying me to say nice things. We're talking because I legitimately adore this thing that you've built, and I want it to exist.Maciej: That's correct. And again, thank you for that. [laugh].Corey: It's true. You can buy my attention, but not my opinion. Now, to be clear, when I did that tweet thread, I did get the sense that this was something that you had built as sort of a side project, as a labor of love. It does not have VC behind it, of which I'm aware, and that's always going to, on some level, shade how I approach a service and how critical I'm going to be on it. Just because it's, yeah, if you've raised a couple 100 million dollars and your user experience is trash, I'm going to call that out.But if this is something where you just soft launched, yeah, I'm not going to be a jerk about weird usability bugs here. I might call it out as “Ooh, this is an area for improvement,” but not, “What jackwagon thought of this?” I am trying to be a kinder, gentler Corey in the new year. But at the same time, I also want to be very clear that there's room for improvement on everything. What surprised me the most about this is how well you nailed the user experience despite not having a full team of people doing UX research.Maciej: That was definitely a priority. So, maybe a little bit of history. So, I started working on Cloudash, I think it was April… 2019. I think? Yeah. It's 2021 right now. Or we're 2022. [unintelligible 00:11:33].Corey: Yeah. 2022, now. I—Maciej: I'm sorry. [laugh].Corey: —I've been screwing that up every time I write the dates myself, I'm with you.Maciej: [laugh]. Okay, so I started working on Cloudash, in 2020, April 2020.Corey: There we go.Maciej: So, after eight months, I released some beta, like, free; you could download it from GitHub. Like, you can still download on GitHub, but at that time, there was no license, you didn't have to buy a license to run it. So, it was, like, very early, like, 0.3 version that was working, but sort of, like, [unintelligible 00:12:00] working. There were some bugs.And that was the first time that I tweeted about it on Twitter. It gets some attention, but, like, some people started using it. I get some feedback, very initial feedback. And I was like, every time I open Cloudash, I get the sense that, like, this is useful. I'm talking about my own tool, but like, [laugh] that's the thing.So, further in the history. So, I'm kind of service engineer by my own. I am a software engineer, I started focusing on serverless, in, like, 2015, 2016. I was working for Serverless Inc. as an early employee.I was then working as an engineering manager for a couple of companies. I work as an engineering manager right now at Stedi; we're also, like, fully serverless. So I, kind of, trying to fix my own issues with serverless, or trying to improve the whole experience around serverless in AWS. So, that's the main purpose why we're building Cloudash: Because we want to improve the experience. And one use case I'm often mentioning is that, let's say that you're kind of on duty. Like, so in the middle of night PagerDuty is calling you, so you need to figure out what's going on with your Lambda or API Gateway.Corey: Yes. PagerDuty, the original [Call of Duty: Nagios 00:13:04]. “It's two in the morning; who is it?” “It's PagerDuty. Wake up, jackass.” Yeah. We all had those moments.Maciej: Exactly. So, the PagerDuty is calling you and you're, kind of, in the middle of night, you're not sure what's going on. So, the kind of thing that we want to optimize is from waking up into understanding what's going on with your serverless stuff should be minimized. And that's the purpose of Cloudash as well. So, you should just run one tool, and you should immediately see what's going on. And that's the purpose.And probably with one or two clicks, you should see the logs responsible, for example, in your Lambda. Again, like that's exactly what we want to cover, that was the initial thing that we want to cover, to kind of minimize the time you spent on troubleshooting serverless apps. Because as we all know, kind of, the longer it's down, the less money you make, et cetera, et cetera, et cetera.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: One of the things that I appreciate about this is that I have something like five different microservices now that power my newsletter production pipeline every week. And periodically, I'll make a change and something breaks because testing is something that I should really get around to one of these days, but when I'm the only customer, cool. Doesn't really matter until suddenly I'm trying to write something and it doesn't work. Great. Time to go diving in, and always I'm never in my best frame of mind for that because I'm thinking about writing for humans not writing for computers. And that becomes a challenge.And okay, how do I get to the figuring out exactly what is broken this time? Regression testing: It really should be a thing more than it has been for me.Maciej: You should write those tests. [laugh].Corey: Yeah. And then I fire this up, and okay, great. Which sub-service is it? Great. Okay, what happened in the last five minutes on that service? Oh, okay, it says it failed successfully in the logs. Okay, that's on me. I can't really blame you for that. But all right.And then it's a matter of adding more [print or 00:14:54] debug statements, and understanding what the hell is going on, mostly that I'm bad at programming. And then it just sort of works from there. It's a lot easier to, I guess, to reason about this from my perspective than it is to go through the CloudWatch dashboards, where it's okay, here's a whole bunch of metrics on different graphs, most of which you don't actually care about—as opposed to unified view that you offer—and then “Oh, you want to look at logs, that's a whole separate sub-service. That's a different service team, obviously, so go open that up in another browser.” And I'm sitting here going, “I don't know who designed this, but are there any windows in their house? My God.”It's just the saddest thing I can possibly experience when I'm in the middle of trying to troubleshoot. Let's be clear, when I'm troubleshooting, I am in no mood to be charitable to anyone or anything, so that's probably unfair to those teams. But by the same token, it's intensely frustrating when I keep smacking into limitations that get in my way while I'm just trying to get the thing up and running again.Maciej: As you mentioned about UX that, like, we've spent a lot of time thinking about the UX, trying different approaches, trying to understand which metrics are the most important. And as we all know, kind of, serverless simplifies a lot of stuff, and there's, like, way less metrics that you need to look into when something is happening, but we want to make sure that the stuff that we show—which is duration errors, and p95—are probably the most important in most cases, so like, covering most of this stuff. So sorry, I didn't mention that before; it was very important from the very beginning. And also, like, literally, I spent a lot of time, like, working on the colors, which sounds funny, [laugh] but I wanted to get them right. We're not yet working on dark mode, but maybe soon.Anyways, the visual part, it's always close to my heart, so we spent a lot of time going back to what just said. So, definitely the experience around using CloudWatch right now, and CloudWatch logs, CloudWatch metrics, is not really tailored for any specific use case because they have to be generic, right? Because AWS has, like, I don't know, like, 300, or whatever number of services, probably half of them producing logs—maybe not half, maybe—Corey: We shouldn't name a number because they'll release five more between now and when this publishes in 20 minutes.Maciej: [laugh]. So, CloudWatch has to be generic. What we want to do with Cloudash is to take those generic tools—because we use, of course, CloudWatch logs, CloudWatch metrics, we fetch data from them—but make the visual part more tailored for specific use case—in our case, it's the serverless use case—and make sure that it's really, kind of—it shows only the stuff that you need to see, not everything else. So again, like that's the main purpose. And then one more thing, we—like this is also some kind of measurement of success, we want to reduce number of tabs that you need to have open in your browser when you're dealing with CloudWatch. So, we tried to put most important stuff in one view so you don't need to flip between tabs, as you usually do when try to under some kind of broader scope, or broader context of your, you know, error in Lambda.Corey: What inspired you to do this as a desktop application? Because a lot of companies are doing similar things, as SaaS, as webapps. And I have to—as someone who yourself—you're a self-described serverless engineer—it seems to me that building a webapp is sort of like the common description use case of a lot of serverless stuff. And you're sitting here saying, “Nope, it's desktop app time.” Which again, I'm super glad you did. It's exactly what I was looking for. How do you get here?Maciej: I'd been thinking about both kinds of types of apps. So like, definitely webapp was the initial idea how to build something, it was the webapp. Because as you said, like, that's the default mode. Like, we are thinking webapp; like, let's build a webapp because I'm an engineer, right? There is some inspiration coming from Dynobase, which was made by a friend [unintelligible 00:18:55] who also lives in Poland—I didn't mention that; we're based in [Poznań 00:18:58], Poland.And when I started thinking about it, there's a lot of benefits of using this approach. The biggest benefit, as I mentioned, is security; and the second benefit is just most, like, cost-effective because we don't need to run in the backend, right? We don't need to download all your metrics, all your logs. We I think, like, let's think about it, like, from the perspective. Listen, so everyone in the company to start working, they have to download all of your stuff from your AWS account. Like, that sounds insane because you don't need all of that stuff elsewhere.Corey: Store multiple copies of it. Yeah I, generally when I'm looking at this, I care about the last five to ten minutes.Maciej: Exactly.Corey: I don't—Maciej: Exactly.Corey: —really care what happened three-and-a-half years ago on this function. Almost always. But occasionally I want to look back at, “Oh, this has been breaking. How long has it been that way?” But I already have that in the AWS environment unless I've done the right thing and turned on, you know, log expiry.Maciej: Exactly. So, this is a lot of, like, I don't want to be, like, you know, mean to anyone but like, that's a lot of waste. Like, that's a lot of waste of compute power because you need to download it; of cost because you need to get this data out of AWS, which you need to pay for, you know, get metric data and stuff like this. So, you need to—Corey: And almost all of its—what is it? Write once, read never. Because it's, you don't generally look at these things.Maciej: Yeah, yeah. Exactly.Corey: And so much of this, too, for every invocation I have, even though it's low traffic stuff, it's the start with a request ID and what version is running, it tells me ‘latest.' Helpful. A single line of comment in this case says ‘200.' Why it says that, I couldn't tell you. And then it says ‘End request ID.' The end.Now, there's no way to turn that off unless you disabled the ability to write to CloudWatch logs in the function, but ingest on that cost 50 cents a gigabyte, so okay, I guess that's AWS's money-making scam of the year. Good for them. But there's so much of that, it's like looking at—like, when things are working, it's like looking at a low traffic site that's behind a load balancer, where there's a whole—you have gigabytes, in some cases, of load balancer—of web server logs on the thing that's sitting in your auto-scaling group. And those logs are just load balancer health checks. 98% of it is just that.Same type of problem here, I don't care about that, I don't want to pay to store it, I certainly don't want to pay to store it twice. I get it, that makes an awful lot of sense. It also makes your security job a hell of a lot easier because you're not sitting on a whole bunch of confidential data from other people. Because, “Well, it's just logs. What could possibly be confidential in there?” “Oh, my sweet summer child, have you seen some of the crap people put in logs?”Maciej: I've seen many things in logs. I don't want to mention them. But anyways—and also, you know, like, usually when you gave access to your AWS account, it can ruin you. You know, like, there might be a lot of—like, you need to really trust the company to give access to your AWS account. Of course, in most cases, the roles are scoped to, you know, only CloudWatch stuff, actions, et cetera, et cetera, but you know, like, there are some situations in which something may not be properly provisioned. And then you give access to everything.Corey: And you can get an awful lot of data you wouldn't necessarily want out of that stuff. Give me just the PDF printout of last month's bill for a lot of environments, and I can tell you disturbing levels of detail about what your architecture is, just because when you—you can infer an awful lot.Maciej: Yeah.Corey: Yeah, I hear you. It makes your security story super straightforward.Maciej: Yeah, exactly. So, I think just repeat my, like, the some inspiration. And then when I started thinking about Cloudash, like, definitely one of the inspiration was Dynobase, from the, kind of, GUI for, like, more powerful UI for DynamoDB. So, if you're interested in that stuff, you can also check this out.Corey: Oh, yeah, I've been a big fan of that, too. That'll be a separate discussion on a different episode, for sure.Maciej: [laugh]. Yeah.Corey: But looking at all of this, looking at the approach of, the only real concern—well, not even a concern. The only real challenge I have with it for my use case is that when I'm on the road, the only thing that I bring with me for a computer is my iPad Pro. I'm not suggesting by any means that you should build this as a new an iPad app; that strikes me as, like, 15 levels of obnoxious. But it does mean that sometimes I still have to go diving into the CloudWatch console when I'm not home. Which, you know, without this, without Cloudash, that's what I was doing originally anyway.Maciej: You're the only person that requested that. And we will put that into backlog, and we will get to that at some point. [laugh].Corey: No, no, no. Smart question is to offer me a specific enterprise tier pricing—.Maciej: Oh, okay. [laugh].Corey: —that is eye-poppingly high. It's like, “Hey, if you want a subsidize feature development, we're thrilled to empower that.” But—Maciej: [laugh]. Yeah, yeah. To be honest, I like that would be hard to write [unintelligible 00:23:33] implement as iPad app, or iPhone app, or whatever because then, like, what's the story behind? Like, how can I get the credentials, right? It's not possible.Corey: Yeah, you'd have to have some fun with that. There are a couple of ways I can think of offhand, but then that turns into a sandboxing issue, and it becomes something where you have to store credentials locally, regardless, even if they're ephemeral. And that's not great. Maybe turn it into a webapp someday or something. Who knows.What I also appreciate is that we had a conversation when you first launched, and I wound up basically going on a Zoom call with you and more or less tearing apart everything you've built—and ideally constructive way—but looking at a lot of the things you've changed in your website, you listened to an awful lot of feedback. You doubled your pricing, for example. Used to be ten bucks a month; now you're twenty. Great. I'm a big believer in charging more.You absolutely add that kind of value because it's, “Well, twenty bucks a month for a desktop app. That sounds crappy.” It's, “Yeah, jackwagon, what's your time worth?” I was spending seven bucks a month in serverless charges, and 120 or 130 a month for Epsagon, and I was thrilled to pieces to be doing it because the value I got from being able to quickly diagnose what the hell was going on far outstripped what the actual cost of doing these things. Don't fall into the trap of assuming that well, I shouldn't pay for software. I can just do it myself. Your time is never free. People think it is, but it's not.Maciej: That's true. The original price of $9.99, I think that was the price was the launch promo. After some time, we've decided—and after adding more features: API Gateway support—we've decided that this is, like, solving way more problems, so like, you should probably pay a little bit more for that. But you're kind of lucky because you subscribed to it when it was 9.99, and this will be your kind of prize for the end of, you know—Corey: Well, I'm going to argue with you after the show to raise the price on mine, just because it's true. It's the—you want to support the things that you want to exist in the world. I also like the fact that you offered an annual plan because I will go weeks without ever opening the app. And that doesn't mean it isn't adding value. It's that oh, yeah, I will need that now that I'm hitting these issues again.And if I'm paying on a monthly basis, and it shows up with a, “Oh, you got charged again.” “Well, I didn't use it this month; I should cancel.” And [unintelligible 00:25:44] to an awful lot of subscriber churn. But in the course of a year, if I don't have at least one instance in which case, wow, that ten minute span justified the entire $200 annual price tag, then, yeah, you built the wrong thing or it's not for me, but I can think of three incidents so far since I started using it in the past four months that have led to that being worth everything you will charge me a year, and then some, just because it made it so clear what was breaking.Maciej: So, in that regard, we are also thinking about the team licenses, that's definitely on the roadmap. There will be some changes to that. And we definitely working on more and more features. And if we're—like, the roadmap is mostly about supporting more and more AWS services, so right now it's Lambda, API Gateway, we're definitely thinking about SQS, SNS, to get some sense how your messages are going through, probably something, like, DynamoDB metrics. And this is all kind of serverless, but why not going wider? Like, why not going to Fargate? Like, Fargate is theoretically serverless, but you know, like, it's serverless on—Corey: It's serverless with a giant asterisk next to it.Maciej: Yeah, [laugh] exactly. So, but why not? Like, it's exactly the same thing in terms of, there is some user flow, there is some user journey, when you want to debug something. You want to go from API Gateway, maybe to the container to see, I don't know, like, DynamoDB metric or something like that, so it should be all easy. And this is definitely something.Later, why not EC2 metrics? Like, it would be a little bit harder. But I'm just saying, like, first thing here is that you are not, like, at this point, we are serverless, but once we cover serverless, why not going wider? Why not supporting more and more services and just making sure that all those use cases are correctly modeled with the UI and UX, et cetera?Corey: That's going to be an interesting challenge, just because that feels like what a lot of the SaaS monitoring and observability tooling is done. And then you fire this thing up, and it looks an awful lot like the AWS console. And it's, “Yeah, I just want to look at this one application that doesn't use any of the rest of those things.” Again, I have full faith and confidence in your ability to pull this off. You clearly have done that well based upon what we've seen so far. I just wonder how you're going to wind up tackling that challenge when you get there.Maciej: And maybe not EC2. Maybe I went too far. [laugh].Corey: Yeah, honestly, even EC2-land, it feels like that is more or less a solved problem. If you want to treat it as a bunch of EC2, you can use Nagios. It's fine.Maciej: Yeah, totally.Corey: There are tools that have solved that problem. But not much that I've seen has solved the serverless piece the way that I want it solved. You have.Maciej: So, it's definitely a long road to make sure that the serverless—and by serverless, I mean serverless how AWS understands serverless, so including Fargate, for example. So, there's a lot of stuff that we can improve. It's a lot of stuff that can make easier with Cloudash than it is with CloudWatch, just staying inside serverless, it will take us a lot of time to make sure that is all correct. And correctly modeled, correctly designed, et cetera. So yeah, I went too far with EC2 sorry.Corey: Exactly. That's okay. We all go too far with EC2, I assure you.Maciej: Sorry everyone using EC2 instances. [laugh].Corey: If people want to kick the tires on it, where can they find it?Maciej: They can find it on cloudash.dev.Corey: One D in the middle. That one throws me sometimes.Maciej: One D. Actually, after talking to you, we have a double-D domain as well, so we can also try ‘Clouddash' with double-D. [laugh].Corey: Excellent, excellent. Okay, that is fantastic. Because I keep trying to put the double-D in when I'm typing it in my search tool on my desktop, and it doesn't show up. And it's like, “What the—oh, right.” But yeah, we'll get there one of these days.Maciej: Only the domain. It's only the domain. You will be redirected to single-D.Corey: Exactly.Maciej: [laugh].Corey: We'll have to expand later; I'll finance the feature request there. It'll go well. If people want to learn more about what you have to think about these things, where else can they find you?Maciej: On Twitter, and my Twitter handle is @mthenw. M-then-W, which is M-T-H—mthenw. And my co-founder @tlakomy. You can probably add that to [show notes 00:29:35]. [laugh].Corey: Oh, I certainly will. It's fine, yeah. Here's a whole bunch of letters. I hear you. My Twitter handle used to be my amateur radio callsign. It turns out most people don't think like that. And yeah, it's become an iterative learning process. Thank you so much for taking the time to speak with me today and for building this thing. I really appreciate both of them.Maciej: Thank you for having me here. I encourage everyone to visit cloudash.dev, if you have any feature requests, any questions just send us an email at hello@cloudash.dev, or just go to GitHub repository in the issues; just create an issue, describe what you want and we can talk about it.We are always happy to help. The main purpose, the ultimate goal of Cloudash is to make the serverless engineer's life easier, on very high level. And on a little bit lower level, just to make, you know, troubleshooting and debugging serverless apps easier.Corey: Well, from my perspective, you've succeeded.Maciej: Thank you.Corey: Thank you. Maciej Winnicki, founder of Cloudash. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment telling me exactly why I'm wrong for using an iPad do these things, but not being able to send it because you didn't find a good way to store the credentials.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Slinging CDK Knowledge with Matt Coulter

Screaming in the Cloud

Play Episode Listen Later Jan 12, 2022 37:37


About MattMatt is an AWS DevTools Hero, Serverless Architect, Author and conference speaker. He is focused on creating the right environment for empowered teams to rapidly deliver business value in a well-architected, sustainable and serverless-first way.You can usually find him sharing reusable, well architected, serverless patterns over at cdkpatterns.com or behind the scenes bringing CDK Day to life.Links: AWS CDK Patterns: https://cdkpatterns.com The CDK Book: https://thecdkbook.com CDK Day: https://www.cdkday.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com.Corey: This episode is sponsored in part by our friends at Rising Cloud, which I hadn't heard of before, but they're doing something vaguely interesting here. They are using AI, which is usually where my eyes glaze over and I lose attention, but they're using it to help developers be more efficient by reducing repetitive tasks. So, the idea being that you can run stateless things without having to worry about scaling, placement, et cetera, and the rest. They claim significant cost savings, and they're able to wind up taking what you're running as it is in AWS with no changes, and run it inside of their data centers that span multiple regions. I'm somewhat skeptical, but their customers seem to really like them, so that's one of those areas where I really have a hard time being too snarky about it because when you solve a customer's problem and they get out there in public and say, “We're solving a problem,” it's very hard to snark about that. Multus Medical, Construx.ai and Stax have seen significant results by using them. And it's worth exploring. So, if you're looking for a smarter, faster, cheaper alternative to EC2, Lambda, or batch, consider checking them out. Visit risingcloud.com/benefits. That's risingcloud.com/benefits, and be sure to tell them that I said you because watching people wince when you mention my name is one of the guilty pleasures of listening to this podcast.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined today by Matt Coulter, who is a Technical Architect at Liberty Mutual. You may have had the privilege of seeing him on the keynote stage at re:Invent last year—in Las Vegas or remotely—that last year of course being 2021. But if you make better choices than the two of us did, and found yourself not there, take the chance to go and watch that keynote. It's really worth seeing.Matt, first, thank you for joining me. I'm sorry, I don't have 20,000 people here in the audience to clap this time. They're here, but they're all remote as opposed to sitting in the room behind me because you know, social distancing.Matt: And this left earphone, I just have some applause going, just permanently, just to keep me going. [laugh].Corey: That's sort of my own internal laugh track going on. It's basically whatever I say is hilarious, to that. So yeah, doesn't really matter what I say, how I say it, my jokes are all for me. It's fine. So, what was it like being on stage in front of that many people? It's always been a wild experience to watch and for folks who haven't spent time on the speaking circuit, I don't think that there's any real conception of what that's like. Is this like giving a talk at work, where I just walk on stage randomly, whatever I happened to be wearing? And, oh, here's a microphone, I'm going to say words. What is the process there?Matt: It's completely different. For context for everyone, before the pandemic, I would have pretty regularly talked in front of, I don't know, maybe one, two hundred people in Liberty, in Belfast. So, I used to be able to just, sort of, walk in front of them, and lean against the pillar, and use my clicker, and click through, but the process for actually presenting something as big as a keynote and re:Invent is so different. For starters, you think that when you walk onto the stage, you'll actually be able to see the audience, but the way the lights are set up, you can pretty much see about one row of people, and they're not the front row, so anybody I knew, I couldn't actually see.And yeah, you can only see, sort of like, the from the void, and then you have your screens, so you've six sets of screens that tell you your notes as well as what slides you're on, you know, so you can pivot. But other than that, I mean, it feels like you're just talking to yourself outside of whenever people, thankfully, applause. It's such a long process to get there.Corey: I've always said that there are a few different transition stages as the audience size increases, but for me, the final stage is more or less anything above 750 people. Because as you say, you aren't able to see that many beyond that point, and it doesn't really change anything meaningfully. The most common example that you see in the wild is jokes that work super well with a small group of people fall completely flat to large audiences. It's why so much corporate numerous cheesy because yeah, everyone in the rehearsals is sitting there laughing and the joke kills, but now you've got 5000 people sitting in a room and that joke just sounds strained and forced because there's no longer a conversation, and no one has the shared context that—the humor has to change. So, in some cases when you're telling a story about what you're going to say on stage, during a rehearsal, they're going to say, “Well, that joke sounds really corny and lame.” It's, “Yeah, wait until you see it in front of an audience. It will land very differently.” And I'm usually right on that.I would also advise, you know, doing what you do and having something important and useful to say, as opposed to just going up there to tell jokes the whole time. I wanted to talk about that because you talked about how you're using various CDK and other serverless style patterns in your work at Liberty Mutual.Matt: Yeah. So, we've been using CDK pretty extensively since it was, sort of, Q3 2019. At that point, it was new. Like, it had just gone GA at the time, just came out of dev preview. And we've been using CDK from the perspective of we want to be building serverless-first, well-architected apps, and ideally we want to be building them on AWS.Now, the thing is, we have 5000 people in our IT organization, so there's sort of a couple of ways you can take to try and get those people onto the cloud: You can either go the route of being, like, there is one true path to architecture, this is our architecture and everything you want to build can fit into that square box; or you can go the other approach and try and have the golden path where you say this is the paved road that is really easy to do, but if you want to differentiate from that route, that's okay. But what you need to do is feed back into the golden path if that works. Then everybody can improve. And that's where we've started been using CDK. So, what you heard me talk about was the software accelerator, and it's sort of a different approach.It's where anybody can build a pattern and then share it so that everybody else can rapidly, you know, just reuse it. And what that means is effectively you can, instead of having to have hundreds of people on a central team, you can actually just crowdsource, and sort of decentralize the function. And if things are good, then a small team can actually come in and audit them, so to speak, and check that it's well-architected, and doesn't have flaws, and drive things that way.Corey: I have to confess that I view the CDK as sort of a third stage automation approach, and it's one that I haven't done much work with myself. The first stage is clicking around in the console; the second is using CloudFormation or Terraform; the third stage is what we're talking about here is CDK or Pulumi, or something like that. And then you ascend to the final fourth stage, which is what I use, which is clicking around in the AWS console, but then you lie to people about it. ClickOps is poised to take over the world. But that's okay. You haven't gotten that far yet. Instead, you're on the CDK side. What advantages does CDK offer that effectively CloudFormation or something like it doesn't?Matt: So, first off, for ClickOps in Liberty, we actually have the AWS console as read-only in all of our accounts, except for sandbox. So, you can ClickOps in sandbox to learn, but if you want to do something real, unfortunately, it's going to fail you. So.—Corey: I love that pattern. I think I might steal that.Matt: [laugh]. So, originally, we went heavy on CloudFormation, which is why CDK worked well for us. And because we've actually—it's been a long journey. I mean, we've been deploying—2014, I think it was, we first started deploying to AWS, and we've used everything from Terraform, to you name it. We've built our own tools, believe it or not, that are basically CDK.And the thing about CloudFormation is, it's brilliant, but it's also incredibly verbose and long because you need to specify absolutely everything that you want to deploy, and every piece of configuration. And that's fine if you're just deploying a side project, but if you're in an enterprise that has responsibilities to protect user data, and you can't just deploy anything, they end up thousands and thousands and thousands of lines long. And then we have amazing guardrails, so if you tried to deploy a CloudFormation template with a flaw in it, we can either just fix it, or reject the deploy. But CloudFormation is not known to be the fastest to deploy, so you end up in this developer cycle, where you build this template by hand, and then it goes through that CloudFormation deploy, and then you get the failure message that it didn't deploy because of some compliance thing, and developers just got frustrated, and were like, sod this. [laugh].I'm not deploying to AWS. Back the on-prem. And that's where CDK was a bit different because it allowed us to actually build abstractions with all of our guardrails baked in, so that it just looked like a standard class, for developers, like, developers already know Java, Python, TypeScript, the languages off CDK, and so we were able to just make it easy by saying, “You want API Gateway? There's an API Gateway class. You want, I don't know, an EC2 instance? There you go.” And that way, developers could focus on the thing they wanted, instead of all of the compliance stuff that they needed to care about every time they wanted to deploy.Corey: Personally, I keep lobbying AWS to add my preferred language, which is crappy shell scripting, but for some reason they haven't really been quick to add that one in. The thing that I think surprises me, on some level—though, perhaps it shouldn't—is not just the adoption of serverless that you're driving at Liberty Mutual, but the way that you're interacting with that feels very futuristic, for lack of a better term. And please don't think that I'm in any way describing this in a way that's designed to be insulting, but I do a bunch of serverless nonsense on Twitter for Pets. That's not an exaggeration. twitterforpets.com has a bunch of serverless stuff behind it because you know, I have personality defects.But no one cares about that static site that's been a slide dump a couple of times for me, and a running joke. You're at Liberty Mutual; you're an insurance company. When people wind up talking about big enterprise institutions, you're sort of a shorthand example of exactly what they're talking about. It's easy to contextualize or think of that as being very risk averse—for obvious reasons; you are an insurance company—as well as wanting to move relatively slowly with respect to technological advancement because mistakes are going to have drastic consequences to all of your customers, people's lives, et cetera, as opposed to tweets or—barks—not showing up appropriately at the right time. How did you get to the, I guess, advanced architectural philosophy that you clearly have been embracing as a company, while having to be respectful of the risk inherent that comes with change, especially in large, complex environments?Matt: Yeah, it's funny because so for everyone, we were talking before this recording started about, I've been with Liberty since 2011. So, I've seen a lot of change in the length of time I've been here. And I've built everything from IBM applications right the way through to the modern serverless apps. But the interesting thing is, the journey to where we are today definitely started eight or nine years ago, at a minimum because there was something identified in the leadership that they said, “Listen, we're all about our customers. And that means we don't want to be wasting millions of dollars, and thousands of hours, and big trains of people to build software that does stuff. We want to focus on why are we building a piece of software, and how quickly can we get there? If you focus on those two things you're doing all right.”And that's why starting from the early days, we focused on things like, okay, everything needs to go through CI/CD pipelines. You need to have your infrastructure as code. And even if you're deploying on-prem, you're still going to be using the same standards that we use to deploy to AWS today. So, we had years and years and years of just baking good development practices into the company. And then whenever we started to move to AWS, the question became, do we want to just deploy the same thing or do we want to take full advantage of what the cloud has to offer? And I think because we were primed and because the leadership had the right direction, you know, we were just sitting there ready to say, “Okay, serverless seems like a way we can rapidly help our customers.” And that's what we've done.Corey: A lot of the arguments against serverless—and let's be clear, they rhyme with the previous arguments against cloud that lots of people used to make; including me, let's be clear here. I'm usually wrong when I try to predict the future. “Well, you're putting your availability in someone else's hands,” was the argument about cloud. Yeah, it turns out the clouds are better at keeping things up than we are as individual companies.Then with serverless, it's the, “Well, if they're handling all that stuff for you on their side, when they're down, you're down. That's an unacceptable business risk, so we're going to be cloud-agnostic and multi-cloud, and that means everything we build serverlessly needs to work in multiple environments, including in our on-prem environment.” And from the way that we're talking about servers and things that you're building, I don't believe that is technically possible, unless some of the stuff you're building is ridiculous. How did you come to accept that risk organizationally?Matt: These are the conversations that we're all having. Sort of, I'd say once a week, we all have a multi-cloud discussion—and I really liked the article you wrote, it was maybe last year, maybe the year before—but multi-cloud to me is about taking the best capabilities that are out there and bringing them together. So, you know, like, Azure [ID 00:12:47] or whatever, things from the other clouds that they're good at, and using those rather than thinking, “Can I build a workload that I can simultaneously pay all of the price to run across all of the clouds, all of the time, so that if one's down, theoretically, I might have an outage?” So, the way we've looked at it is we embraced really early the well-architected framework from AWS. And it talks about things like you need to have multi-region availability, you need to have your backups in place, you need to have things like circuit breakers in place for if third-party goes down, and we've just tried to build really resilient architectures as best as we can on AWS. And do you know what I think, if [laugh] it AWS is not—I know at re:Invent, there it went down extraordinarily often compared to normal, but in general—Corey: We were all tired of re:Invent; their us-east-1 was feeling the exact same way.Matt: Yeah, so that's—it deserved a break. But, like, if somebody can't buy insurance for an hour, once a year, [laugh] I think we're okay with it versus spending millions to protect that one hour.Corey: And people make assumptions based on this where, okay, we had this problem with us-east-1 that froze things like the global Route 53 control planes; you couldn't change DNS for seven hours. And I highlighted that as, yeah, this is a problem, and it's something to severely consider, but I will bet you anything you'd care to name that there is an incredibly motivated team at AWS, actively fixing that as we speak. And by—I don't know how long it takes to untangle all of those dependencies, but I promise they're going to be untangled in relatively short order versus running data centers myself, when I discover a key underlying dependency I didn't realize was there, well, we need to break that. That's never going to happen because we're trying to do things as a company, and it's just not the most important thing for us as a going concern. With AWS, their durability and reliability is the most important thing, arguably compared to security.Would you rather be down or insecure? I feel like they pick down—I would hope in most cases they would pick down—but they don't want to do either one. That is something they are drastically incentivized to fix. And I'm never going to be able to fix things like that and I don't imagine that you folks would be able to either.Matt: Yeah, so, two things. The first thing is the important stuff, like, for us, that's claims. We want to make sure at any point in time, if you need to make a claim you can because that is why we're here. And we can do that with people whether or not the machines are up or down. So, that's why, like, you always have a process—a manual process—that the business can operate, irrespective of whether the cloud is still working.And that's why we're able to say if you can't buy insurance in that hour, it's okay. But the other thing is, we did used to have a lot of data centers, and I have to say, the people who ran those were amazing—I think half the staff now work for AWS—but there was this story that I heard where there was an app that used to go down at the same time every day, and nobody could work out why. And it was because someone was coming in to clean the room at that time, and they unplugged the server to plug in a vacuum, and then we're cleaning the room, and then plugging it back in again. And that's the kind of thing that just happens when you manage people, and you manage a building, and manage a premises. Whereas if you've heard that happened that AWS, I mean, that would be front page news.Corey: Oh, it absolutely would. There's also—as you say, if it's the sales function, if people aren't able to buy insurance for an hour, when us-east-1 went down, the headlines were all screaming about AWS taking an outage, and some of the more notable customers were listed as examples of this, but the story was that, “AWS has massive outage,” not, “Your particular company is bad at technology.” There's sort of a reputational risk mitigation by going with one of these centralized things. And again, as you're alluding to, what you're doing is not life-critical as far as the sales process and getting people to sign up. If an outage meant that suddenly a bunch of customers were no longer insured, that's a very different problem. But that's not your failure mode.Matt: Exactly. And that's where, like, you got to look at what your business is, and what you're specifically doing, but for 99.99999% of businesses out there, I'm pretty sure you can be down for the tiny window that AWS is down per year, and it will be okay, as long as you plan for it.Corey: So, one thing that really surprised me about the entirety of what you've done at Liberty Mutual is that you're a big enterprise company, and you can take a look at any enterprise company, and say that they have dueling mottos, which is, “I am not going to comment on that,” or, “That's not funny.” Like, the safe mode for any large concern is to say nothing at all. But a lot of folks—not just you—at Liberty have been extremely vocal about the work that you're doing, how you view these things, and I almost want to call it advocacy or evangelism for the CDK. I'm slightly embarrassed to admit that for a little while there, I thought you were an AWS employee in their DevRel program because you were such an advocate in such strong ways for the CDK itself.And that is not something I expected. Usually you see the most vocal folks working in environments that, let's be honest, tend to play a little bit fast and loose with things like formal corporate communications. Liberty doesn't and yet, there you folks are telling these great stories. Was that hard to win over as a culture, or am I just misunderstanding how corporate life is these days?Matt: No, I mean, so it was different, right? There was a point in time where, I think, we all just sort of decided that—I mean, we're really good at what we do from an engineering perspective, and we wanted to make sure that, given the messaging we were given, those 5000 teck employees in Liberty Mutual, if you consider the difference in broadcasting to 5000 versus going external, it may sound like there's millions, billions of people in the world, but in reality, the difference in messaging is not that much. So, to me what I thought, like, whenever I started anyway—it's not, like, we had a meeting and all decided at the same time—but whenever I started, it was a case of, instead of me just posting on all the internal channels—because I've been doing this for years—it's just at that moment, I thought, I could just start saying these things externally and still bring them internally because all you've done is widened the audience; you haven't actually made it shallower. And that meant that whenever I was having the internal conversations, nothing actually changed except for it meant external people, like all their Heroes—like Jeremy Daly—could comment on these things, and then I could bring that in internally. So, it almost helped the reverse takeover of the enterprise to change the culture because I didn't change that much except for change the audience of who I was talking to.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: One thing that you've done that I want to say is admirable, and I stumbled across it when I was doing some work myself over the break, and only right before this recording did I discover that it was you is the cdkpatterns.com website. Specifically what I love about it is that it publishes a bunch of different patterns of ways to do things. This deviates from a lot of tutorials on, “Here's how to build this one very specific thing,” and instead talks about, “Here's the architecture design; here's what the baseline pattern for that looks like.” It's more than a template, but less than a, “Oh, this is a messaging app for dogs and I'm trying to build a messaging app for cats.” It's very generalized, but very direct, and I really, really like that model of demo.Matt: Thank you. So, watching some of your Twitter threads where you experiment with new—Corey: Uh oh. People read those. That's a problem.Matt: I know. So, whatever you experiment with a new piece of AWS to you, I've always wondered what it would be like to be your enabling architect. Because technically, my job in Liberty is, I meant to try and stay ahead of everybody and try and ease the on-ramp to these things. So, if I was your enabling architect, I would be looking at it going, “I should really have a pattern for this.” So that whenever you want to pick up that new service the patterns in cdkpatterns.com, there's 24, 25 of them right there, but internally, there's way more than dozens now.The goal is, the pattern is the least amount to code for you to learn a concept. And then that way, you can not only see how something works, but you can maybe pick up one of the pieces of the well-architected framework while you're there: All of it's unit tested, all of it is proper, you know, like, commented code. The idea is to not be crap, but not be gold-plated either. I'm currently in the process of upgrading that all to V2 as well. So, that [unintelligible 00:21:32].Corey: You mentioned a phrase just now: “Enabling architect.” I have to say this one that has not crossed my desk before. Is that an internal term you use? Is that an enterprise concept I've somehow managed to avoid? Is that an AWS job role? What is that?Matt: I've just started saying [laugh] it's my job over the past couple of years. That—I don't know, patent pending? But the idea to me is—Corey: No, it's evocative. I love the term, I'd love to learn more.Matt: Yeah, because you can sort of take two approaches to your architecture: You can take the traditional approach, which is the ‘house of no' almost, where it's like, “This is the architecture. How dare you want to deviate. This is what we have decided. If you want to change it, here's the Architecture Council and go through enterprise architecture as people imagine it.” But as people might work out quite quickly, whenever they meet me, the whole, like, long conversational meetings are not for me. What I want to do is teach engineers how to help themselves, so that's why I see myself as enabling.And what I've been doing is using techniques like Wardley Mapping, which is where you can go out and you can actually take all the components of people's architecture and you can draw them on a map for—it's a map of how close they are to the customer, as well as how cutting edge the tech is, or how aligned to our strategic direction it is. So, you can actually map out all of the teams, and—there's 160, 170 engineers in Belfast and Dublin, and I can actually go in and say, “Oh, that piece of your architecture would be better if it was evolved to this. Well, I have a pattern for that,” or, “I don't have a pattern for that, but you know what? I'll build one and let's talk about it next week.” And that's always trying to be ahead, instead of people coming to me and I have to say no.Corey: AWS Proton was designed to do something vaguely similar, where you could set out architectural patterns of—like, the two examples that they gave—I don't know if it's in general availability yet or still in public preview, but the ones that they gave were to build a REST API with Lambda, and building something-or-other with Fargate. And the idea was that you could basically fork those, or publish them inside of your own environment of, “Oh, you want a REST API; go ahead and do this.” It feels like their vision is a lot more prescriptive than what yours is.Matt: Yeah. I talked to them quite a lot about Proton, actually because, as always, there's different methodologies and different ways of doing things. And as I showed externally, we have our software accelerator, which is kind of our take on Proton, and it's very open. Anybody can contribute; anybody can consume. And then that way, it means that you don't necessarily have one central team, you can have—think of it more like an SRE function for all of the patterns, rather than… the Proton way is you've separate teams that are your DevOps teams that set up your patterns and then separate team that's consumer, and they have different permissions, different rights to do different things. If you use a Proton pattern, anytime an update is made to that pattern, it auto-deploys your infrastructure.Corey: I can see that breaking an awful lot.Matt: [laugh]. Yeah. So, the idea is sort of if you're a consumer, I assume you [unintelligible 00:24:35] be going to change that infrastructure. You can, they've built in an escape hatch, but the whole concept of it is there's a central team that looks to what the best configuration for that is. So, I think Proton has so much potential, I just think they need to loosen some of the boundaries for it to work for us, and that's the feedback I've given them directly as well.Corey: One thing that I want to take a step beyond this is, you care about this? More than most do. I mean, people will work with computers, yes. We get paid for that. Then they'll go and give talks about things. You're doing that as well. They'll launch a website occasionally, like, cdkpatterns.com, which you have. And then you just sort of decide to go for the absolute hardest thing in the world, and you're one of four authors of a book on this. Tell me more.Matt: Yeah. So, this is something that there's a few of us have been talking since one of the first CDK Days, where we're friends, so there's AWS Heroes. There's Thorsten Höger, Matt Bonig, Sathyajith Bhat, and myself, came together—it was sometime in the summer last year—and said, “Okay. We want to write a book, but how do we do this?” Because, you know, we weren't authors before this point; we'd never done it before. We weren't even sure if we should go to a publisher, or if we should self-publish.Corey: I argue that no one wants to write a book. They want to have written a book, and every first-time author I've ever spoken to at the end has said, “Why on earth would anyone want to do this a second time?” But people do it.Matt: Yeah. And that's we talked to Alex DeBrie, actually, about his book, the amazing Dynamodb Book. And it was his advice, told us to self-publish. And he gave us his starter template that he used for his book, which took so much of the pain out because all we had to do was then work out how we were going to work together. And I will say, I write quite a lot of stuff in general for people, but writing a book is completely different because once it's out there, it's out there. And if it's wrong, it's wrong. You got to release a new version and be like, “Listen, I got that wrong.” So, it did take quite a lot of effort from the group to pull it together. But now that we have it, I want to—I don't have a printed copy because it's only PDF at the minute, but I want a copy just put here [laugh] in, like, the frame. Because it's… it's what we all want.Corey: Yeah, I want you to do that through almost a traditional publisher, selfishly, because O'Reilly just released the AWS Cookbook, and I had a great review quote on the back talking about the value added. I would love to argue that they use one of mine for The CDK Book—and then of course they would reject it immediately—of, “I don't know why you do all this. Using the console and lying about it is way easier.” But yeah, obviously not the direction you're trying to take the book in. But again, the industry is not quite ready for the lying version of ClickOps.It's really neat to just see how willing you are to—how to frame this?—to give of yourself and your time and what you've done so freely. I sometimes make a joke—that arguably isn't that funny—that, “Oh, AWS Hero. That means that you basically volunteer for a $1.6 trillion company.”But that's not actually what you're doing. What you're doing is having figured out all the sharp edges and hacked your way through the jungle to get to something that is functional, you're a trailblazer. You're trying to save other people who are working with that same thing from difficult experiences on their own, having to all thrash and find our own way. And not everyone is diligent and as willing to continue to persist on these things. Is that a somewhat fair assessment how you see the Hero role?Matt: Yeah. I mean, no two Heroes are the same, from what I've judged, I haven't met every Hero yet because pandemic, so Vegas was the first time [I met most 00:28:12], but from my perspective, I mean, in the past, whatever number of years I've been coding, I've always been doing the same thing. Somebody always has to go out and be the first person to try the thing and work out what the value is, and where it'll work for us more work for us. The only difference with the external and public piece is that last 5%, which it's a very different thing to do, but I personally, I like even having conversations like this where I get to meet people that I've never met before.Corey: You sort of discovered the entire secret of why I have an interview podcast.Matt: [laugh]. Yeah because this is what I get out of it, just getting to meet other people and have new experiences. But I will say there's Heroes out there doing very different things. You've got, like, Hiro—as in Hiro, H-I-R-O—actually started AWS Newbies and she's taught—ah, it's hundreds of thousands of people how to actually just start with AWS, through a course designed for people who weren't coders before. That kind of thing is next-level compared to anything I've ever done because you know, they have actually built a product and just given it away. I think that's amazing.Corey: At some level, building a product and giving it away sounds like, “You know, I want to never be lonely again.” Well, that'll work because you're always going to get support tickets. There's an interesting narrative around how to wind up effectively managing the community, and users, and demands, based on open-source maintainers, that we're all wrestling with as an industry, particularly in the wake of that whole log4j nonsense that we've been tilting at that windmill, and that's going to be with us for a while. One last thing I want to talk about before we wind up calling this an episode is, you are one of the organizers of CDK Day. What is that?Matt: Yeah, so CDK Day, it's a complete community-organized conference. The past two have been worldwide, fully virtual just because of the situation we're in. And I mean, they've been pretty popular. I think we had about 5000 people attended the last one, and the idea is, it's a full day of the community just telling their stories of how they liked or disliked using the CDK. So, it's not a marketing event; it's not a sales event; we actually run the whole event on a budget of exactly $0. But yeah, it's just a day of fun to bring the community together and learn a few things. And, you know, if you leave it thinking CDK is not for you, I'm okay with that as much as if you just make a few friends while you're there.Corey: This is the first time I'd realized that it wasn't a formal AWS event. I almost feel like that's the tagline that you should have under it. It's—because it sounds like the CDK Day, again, like, it's this evangelism pure, “This is why it's great and why you should use it.” But I love conferences that embrace critical views. I built one of the first talks I ever built out that did anything beyond small user groups was “Heresy in the Church of Docker.”Then they asked me to give that at ContainerCon, which was incredibly flattering. And I don't think they made that mistake a second time, but it was great to just be willing to see some group of folks that are deeply invested in the technology, but also very open to hearing criticism. I think that's the difference between someone who is writing a nuanced critique versus someone who's just [pure-on 00:31:18] zealotry. “But the CDK is the answer to every technical problem you've got.” Well, I start to question the wisdom of how applicable it really is, and how objective you are. I've never gotten that vibe from you.Matt: No, and that's the thing. So, I mean, as we've worked out in this conversation, I don't work for AWS, so it's not my product. I mean, if it succeeds or if it fails, it doesn't impact my livelihood. I mean, there are people on the team who would be sad for, but the point is, my end goal is always the same. I want people to be enabled to rapidly deliver their software to help their customers.If that's CDK, perfect, but CDK is not for everyone. I mean, there are other options available in the market. And if, even, ClickOps is the way to go for you, I am happy for you. But if it's a case of we can have a conversation, and I can help you get closer to where you need to be with some other tool, that's where I want to be. I just want to help people.Corey: And if I can do anything to help along that axis, please don't hesitate to let me know. I really want to thank you for taking the time to speak with me and being so generous, not just with your time for this podcast, but all the time you spend helping the rest of us figure out which end is up, as we continue to find that the way we manage environments evolves.Matt: Yeah. And, listen, just thank you for having me on today because I've been reading your tweets for two years, so I'm just starstruck at this moment to even be talking to you. So, thank you.Corey: No, no. I understand that, but don't worry, I put my pants on two legs at a time, just like everyone else. That's right, the thought leader on Twitter, you have to jump into your pants. That's the rule. Thanks again so much. I look forward to having a further conversation with you about this stuff as I continue to explore, well honestly, what feels like a brand new paradigm for how we manage code.Matt: Yeah. Reach out if you need any help.Corey: I certainly will. You'll regret asking. Matt [Coulter 00:33:06], Technical Architect at Liberty Mutual. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, write an angry comment, then click the submit button, but lie and say you hit the submit button via an API call.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
GCP's Many Profundities with Miles Ward

Screaming in the Cloud

Play Episode Listen Later Jan 11, 2022 42:06


About MilesAs Chief Technology Officer at SADA, Miles Ward leads SADA's cloud strategy and solutions capabilities. His remit includes delivering next-generation solutions to challenges in big data and analytics, application migration, infrastructure automation, and cost optimization; reinforcing our engineering culture; and engaging with customers on their most complex and ambitious plans around Google Cloud.Previously, Miles served as Director and Global Lead for Solutions at Google Cloud. He founded the Google Cloud's Solutions Architecture practice, launched hundreds of solutions, built Style-Detection and Hummus AI APIs, built CloudHero, designed the pricing and TCO calculators, and helped thousands of customers like Twitter who migrated the world's largest Hadoop cluster to public cloud and Audi USA who re-platformed to k8s before it was out of alpha, and helped Banco Itau design the intercloud architecture for the bank of the future.Before Google, Miles helped build the AWS Solutions Architecture team. He wrote the first AWS Well-Architected framework, proposed Trusted Advisor and the Snowmobile, invented GameDay, worked as a core part of the Obama for America 2012 “tech” team, helped NASA stream the Curiosity Mars Rover landing, and rebooted Skype in a pinch.Earning his Bachelor of Science in Rhetoric and Media Studies from Willamette University, Miles is a three-time technology startup entrepreneur who also plays a mean electric sousaphone.Links: SADA.com: https://sada.com Twitter: https://twitter.com/milesward Email: miles@sada.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I am joined today, once again by my friend and yours, Miles Ward, who's the CTO at SADA. However, he is, as I think of him, the closest thing the Google Cloud world has to Corey Quinn. Now, let's be clear, not the music and dancing part that is Forrest Brazeal, but Forrest works at Google Cloud, whereas Miles is a reasonably salty third-party. Miles, thank you for coming back and letting me subject you to that introduction.Miles: Corey, I appreciate that introduction. I am happy to provide substantial salt. It is easy, as I play brass instruments that produce my spit in high volumes. It's the most disgusting part of any possible introduction. For the folks in the audience, I am surrounded by a collection of giant sousaphones, tubas, trombones, baritones, marching baritones, trumpets, and pocket trumpets.So, Forrest threw down the gauntlet and was like, I can play a keyboard, and sing, and look cute at the same time. And so I decided to fail at all three. We put out a new song just a bit ago that's, like, us thanking all of our customers and partners, covering Kool & the Gang “Celebration,” and I neither look good, [laugh] play piano, or smiling, or [capturing 00:01:46] any of the notes; I just play the bass part, it's all I got to do.Corey: So, one thing that I didn't get to talk a lot about because it's not quite in my universe, for one, and for another, it is during the pre re:Invent—pre:Invent, my nonsense thing—run up, which is Google Cloud Next.Miles: Yes.Corey: And my gag a few years ago is that I'm not saying that Google is more interested in what they're building and what they're shipping, but even their conference is called Next. Buh dum, hiss.Miles: [laugh].Corey: So, I didn't really get to spend a lot of attention on the Google Cloud releases that came out this year, but given that SADA is in fact the, I believe, largest Google Cloud partner on the internet, and thus the world—Miles: [unintelligible 00:02:27] new year, three years in a row back, baby.Corey: Fantastic. I assume someone's watch got stuck or something. But good work. So, you have that bias in the way that I have a bias, which is your business is focused around Google Cloud the way that mine is focused on AWS, but neither of us is particularly beholden to that given company. I mean, you do have the not getting fired as partner, but that's a bit of a heavy lift; I don't think I can mouth off well enough to get you there.So, we have a position of relative independence. So, you were tracking Google Next, the same way that I track re:Invent. Well, not quite the same way I track re:Invent; there are some significant differences. What happened at Cloud Next 2021, that the worst of us should be paying attention to?Miles: Sure. I presented 10% of the material at the first re:Invent. There are 55 sessions; I did six. And so I have been at Cloud events for a really long time and really excited about Google's willingness to dive into demos in a way that I think they have been a little shy about. Kelsey Hightower is the kind of notable deep exception to that. Historically, he's been ready to dive into the, kind of, heavy hands-on piece but—Corey: Wait, those were demos? [Thought 00:03:39] was just playing Tetris on stage for the love of it.Miles: [laugh]. No. And he really codes all that stuff up, him and the whole team.Corey: Oh, absol—I'm sorry. If I ever grow up, I wish to be Kelsey Hightower.Miles: [laugh]. You and me both. So, he had kind of led the charge. We did a couple of fun little demos while I was there, but they've really gotten a lot further into that, and I think are doing a better job of packaging the benefits to not just developers, but also operators and data scientists and the broader roles in the cloud ecosystem from the new features that are being launched. And I think, different than the in-person events where there's 10, 20,000, 40,000 people in the audience paying attention, I think they have to work double-hard to capture attention and get engineers to tune in to what's being launched.But if you squint and look close, there are some, I think, very interesting trends that sit in the back of some of the very first launches in what I think are going to be whole veins of launches from Google over the course of the next several years that we are working really hard to track along with and make sure we're extracting maximum value from for our customers.Corey: So, what was it that they announced that is worth paying attention to? Now, through the cacophony of noise, one announcement that [I want to note 00:04:49] was tied to Next was the announcement that GME group, I believe, is going to be putting their futures exchange core trading systems on Google Cloud. At which point that to me—and I know people are going to yell at me, and I don't even slightly care—that is the last nail in the coffin of the idea that well, Google is going to turn this off in a couple years. Sorry, no. That is not a thing that's going to happen. Worst case, they might just stop investing it as aggressively as they are now, but even that would be just a clown-shoes move that I have a hard time envisioning.Miles: Yeah, you're talking now over a dozen, over ten year, over a billion-dollar commitments. So, you've got to just really, really hate your stock price if you're going to decide to vaporize that much shareholder value, right? I mean, we think that, in Google, stock price is a material fraction of the recognition of the growth trajectory for cloud, which is now basically just third place behind YouTube. And I think you can do the curve math, it's not like it's going to take long.Corey: Right. That requires effectively ejecting Thomas Kurian as the head of Google Cloud and replacing him with the former SVP of Bad Decisions at Yahoo.Miles: [laugh]. Sure. Google has no shyness about continuing to rotate leadership. I was there through three heads of Google Cloud, so I don't expect that Thomas will be the last although I think he may well go down in history as having been the best. The level of rotation to the focuses that I think are most critical, getting enterprise customers happy, successful, committed, building macroscale systems, in systems that are critical to the core of the business on GCP has grown at an incredible rate under his stewardship. So, I think he's doing a great job.Corey: He gets a lot of criticism—often from Googlers—when I wind up getting the real talk from them, which is, “Can you tell me what you really think?” Their answer is, “No,” I'm like, “Okay, next question. Can I go out and buy you eight beers and then”— and it's like, “Yeah.” And the answer that I get pretty commonly is that he's brought too much Oracle into Google. And okay, that sounds like a bad thing because, you know, Oracle, but let's be clear here, but what are you talking about specifically? And what they say distills down to engineers are no longer the end-all be-all of everything that Google Cloud. Engineers don't get to make sales decisions, or marketing decisions, or in some cases, product decisions. And that is not how Google has historically been run, and they don't like the change. I get it, but engineering is not the only hard thing in the world and it's not the only business area that builds value, let's be clear on this. So, I think that the things that they don't like are in fact, what Google absolutely needs.Miles: I think, one, the man is exceptionally intimidating and intentionally just hyper, hyper attentive to his business. So, one of my best employees, Brad [Svee 00:07:44], he worked together with me to lay out what was the book of our whole department, my team of 86 people there. What are we about? What do we do? And like I wanted this as like a memoriam to teach new hires as got brought in. So, this is, like, 38 pages of detail about our process, our hiring method, our promotional approach, all of it. I showed that to my new boss who had come in at the time, and he thought some of the pictures looked good. When we showed it to TK, he read every paragraph. I watched him highlight the paragraphs as he went through, and he read it twice as fast as I can read the thing. I think he does that to everybody's documents, everywhere. So, there's a level of just manual rigor that he's brought to the practice that was certainly not there before that. So, that alone, it can be intimidating for folks, but I think people that are high performance find that very attractive.Corey: Well, from my perspective, he is clearly head and shoulders above Adam Selipsky, and Scott Guthrie—the respective heads of AWS and Azure—for one key reason: He is the only one of those three people who follows me on Twitter. And—Miles: [laugh].Corey: —honestly, that is how I evaluate vendors.Miles: That's the thing. That's the only measure, yep. I've worked on for a long time with Selipsky, and I think that it will be interesting to see whether Adam's approach to capital allocation—where he really, I think, thinks of himself as the manager of thousands of startups, as opposed to a manager of a global business—whether that's a more efficient process for creating value for customers, then, where I think TK is absolutely trying to build a much more unified, much more singular platform. And a bunch of the launches really speak to that, right? So, one of the product announcements that I think is critical is this idea of the global distributed cloud, Google Distributed Cloud.We started with Kubernetes. And then you layer on to that, okay, we'll take care of Kubernetes for you; we call that Anthos. We'll build a bunch of structural controls and features into Anthos to make it so that you can really deal with stuff in a global way. Okay, what does that look like further? How do we get out into edge environments? Out into diverse hardware? How do we partner up with everybody to make sure that, kind of like comparing Apple's approach to Google's approach, you have an Android ecosystem of Kubernetes providers instead of just one place you can buy an outpost. That's generally the idea of GDC. I think that's a spot where you're going to watch Google actually leverage the muscle that it already built in understanding open-source dynamics and understanding collaboration between companies as opposed to feeling like it's got to be built here. We've got to sell it here. It's got to have our brand on it.Corey: I think that there's a stupendous and extreme story that is still unfolding over at Google Cloud. Now, re:Invent this year, they wound up talking all about how what they were rolling out was a focus on improving primitives. And they're right. I love their managed database service that they launched because it didn't exist.Miles: Yeah Werner's slide, “It's primitives, not frameworks.” I was like, I think customers want solutions, not frameworks or primitives. [laugh]. What's your plan?Corey: Yeah. However, I take a different perspective on all of this, which is that is a terrific spin on the big headline launches all missed the re:Invent timeline, and… oops, so now we're just going to talk about these other things instead. And that's great, but then they start talking about industrial IOT, and mainframe migrations, and the idea of private 5G, and running fleets of robots. And it's—Miles: Yeah, that's a cool product.Corey: Which one? I'm sorry, they're all very different things.Miles: Private 5G.Corey: Yeah, if someone someday will explain to me how it differs from Wavelength, but that's neither here nor there. You're right, they're all interesting, but none of them are actually doing the thing that I do, which is build websites, [unintelligible 00:11:31] looking for web services, it kind of says it in the name. And it feels like it's very much broadening into everything, and it's very difficult for me to identify—and if I have trouble that I guarantee you customers do—of, which services are for me and which are very much not? In some cases, the only answer to that is to check the pricing. I thought Kendra, their corporate information search thing was for me, then it's 7500 bucks a month to get started with that thing, and that is, “I can hire an internal corporate librarian to just go and hunt through our Google Drive.” Great.Miles: Yeah.Corey: So, there are—or our Dropbox, or our Slack. We have, like, five different information repositories, and this is how corporate nonsense starts, let me assure you.Miles: Yes. We call that luxury SaaS, you must enjoy your dozens of overlapping bills for, you know, what Workspace gives you as a single flat rate.Corey: Well, we have [unintelligible 00:12:22] a lot of this stuff, too. Google Drive is great, but we use Dropbox for holding anything that touches our customer's billing information, just because I—to be clear, I do not distrust Google, but it also seems a little weird to put the confidential billing information for one of their competitors on there to thing if a customer were to ask about it. So, it's the, like, I don't believe anyone's doing anything nefarious, but let's go ahead and just make sure, in this case.Miles: Go further man. Vimeo runs on GCP. You think YouTube doesn't want to look at Vimeo stats? Like they run everything on GCP, so they have to have arrived at a position of trust somehow. Oh, I know how it's called encryption. You've heard of encryption before? It's the best.Corey: Oh, yes. I love these rumors that crop up every now and again that Amazon is going to start scanning all of its customer content, somehow. It's first, do you have any idea how many compute resources that would take and to if they can actually do that and access something you're storing in there, against their attestations to the contrary, then that's your story because one of them just makes them look bad, the other one utterly destroys their entire business.Miles: Yeah.Corey: I think that that's the one that gets the better clicks. So no, they're not doing that.Miles: No, they're not doing that. Another product launch that I thought was super interesting that describes, let's call it second place—the third place will be the one where we get off into the technical deep end—but there's a whole set of coordinated work they're calling Cortex. So, let's imagine you go to a customer, they say, “I want to understand what's happening with my business.” You go, “Great.” So, you use SAP, right? So, you're a big corporate shop, and that's your infrastructure of choice. There are a bunch of different options at that layer.When you set up SAP, one of the advantages that something like that has is they have, kind of, pre-built configurations for roughly your business, but whatever behaviors SAP doesn't do, right, say, data warehousing, advanced analytics, regression and projection and stuff like that, maybe that's somewhat outside of the core wheelhouse for SAP, you would expect like, oh okay, I'll bolt on BigQuery. I'll build that stuff over there. We'll stream the data between the two. Yeah, I'm off to the races, but the BigQuery side of the house doesn't have this like bitching menu that says, “You're a retailer, and so you probably want to see these 75 KPIs, and you probably want to chew up your SKUs in exactly this way. And here's some presets that make it so that this is operable out of the box.”So, they are doing the three way combination: Consultancies plus ISVs plus Google products, and doing all the pre-work configuration to go out to a customer and go I know what you probably just want. Why don't I just give you the whole thing so that it does the stuff that you want? That I think—if that's the very first one, this little triangle between SAP, and Big Query, and a bunch of consultancies like mine, you have to imagine they go a lot further with that a lot faster, right? I mean, what does that look like when they do it with Epic, when they go do it with Go just generally, when they go do it with Apache? I've heard of that software, right? Like, there's no reason not to bundle up what the obvious choices are for a bunch of these combinations.Corey: The idea of moving up the stack and offering full on solutions, that's what customers actually want. “Well, here's a bunch of things you can do to wind up wiring together to build a solution,” is, “Cool. Then I'm going to go hire a company who's already done that is going to sell it to me at a significant markup because I just don't care.” I pay way more to WP Engine than I would to just run WordPress myself on top of AWS or Google Cloud. In fact, it is on Google Cloud, but okay.Miles: You and me both, man. WP Engine is the best. I—Corey: It's great because—Miles: You're welcome. I designed a bunch of the hosting on the back of that.Corey: Oh, yeah. But it's also the—I—well, it costs a little bit more that way. Yeah, but guess what's not—guess what's more expensive than that bill, is my time spent doing the care and feeding of this stuff. I like giving money to experts and making it their problem.Miles: Yeah. I heard it said best, Lego is an incredible business. I love their product, and you can build almost any toy with it. And they have not displaced all other plastic toy makers.Corey: Right.Miles: Some kids just want to buy a little car. [laugh].Corey: Oh, yeah, you can build anything you want out of Lego bricks, which are great, which absolutely explains why they are a reference AWS customer.Miles: Yeah, they're great. But they didn't beat all other toy companies worldwide, and eliminate the rest of that market because they had the better primitive, right? These other solutions are just as valuable, just as interesting, tend to have much bigger markets. Lego is not the largest toy manufacturer in the world. They are not in the top five of toy manufacturers in the world, right?Like, so chasing that thread, and getting all the way down into the spots where I think many of the cloud providers on their own, internally, had been very uncomfortable. Like, you got to go all the way to building this stuff that they need for that division, inside of that company, in that geo, in that industry? That's maybe, like, a little too far afield. I think Google has a natural advantage in its more partner-oriented approach to create these combinations that lower the cost to them and to customers to getting out of that solution quick.Corey: So, getting into the weeds of Google Next, I suppose, rather than a whole bunch of things that don't seem to apply to anyone except the four or five companies that really could use it, what things did Google release that make the lives of people building, you know, web apps better?Miles: This is the one. So, I'm at Amazon, hanging out as a part of the team that built up the infrastructure for the Obama campaign in 2012, and there are a bunch of Googlers there, and we are fighting with databases. We are fighting so hard, in fact, with RDS that I think we are the only ones that [Raju 00:17:51] has ever allowed to SSH into our RDS instances to screw with them.Corey: Until now, with the advent of RDS Custom, meaning that you can actually get in as root; where that hell that lands between RDS and EC2 is ridiculous. I just know that RDS can now run containers.Miles: Yeah. I know how many things we did in there that were good for us, and how many things we did in there that were bad for us. And I have to imagine, this is not a feature that they really ought to let everybody have, myself included. But I will say that what all of the Googlers that I talk to, you know, at the first blush, were I'm the evil Amazon guy in to, sort of, distract them and make them build a system that, you know, was very reliable and ended up winning an election was that they had a better database, and they had Spanner, and they didn't understand why this whole thing wasn't sitting on Spanner. So, we looked, and I read the white paper, and then I got all drooly, and I was like, yes, that is a much better database than everybody else's database, and I don't understand why everybody else isn't on it. Oh, there's that one reason, but you've heard of it: No other software works with it, anywhere in the world, right? It's utterly proprietary to Google. Yes, they were kind—Corey: Oh, you want to migrate it off somewhere else, or a fraction of it? Great. Step one, redo your data architecture.Miles: Yeah, take all of my software everywhere, rewrite every bit of it. And, oh all those commercial applications? Yeah, forget all those, you got, too. Right? It was very much where Google was eight years ago. So, for me, it was immensely meaningful to see the launch at Next where they described what they are building—and have now built; we have alpha access to it—a Postgres layer for Spanner.Corey: Is that effectively you have to treat it as Postgres at all times, or is it multimodal access?Miles: You can get in and tickle it like Spanner, if you want to tickle it like Spanner. And in reality, Spanner is ANSI SQL compliant; you're still writing SQL, you just don't have to talk to it like a REST endpoint, or a GRPC endpoint, or something; you can, you know, have like a—Corey: So, similar to Azure's Cosmos DB, on some level, except for the part where you can apparently look at other customers' data in that thing?Miles: [laugh]. Exactly. Yeah, you will not have a sweeping discovery of incredible security violations in the structure Spanner, in that it is the control system that Google uses to place every ad, and so it does not suck. You can't put a trillion-dollar business on top of a database and not have it be safe. That's kind of a thing.Corey: The thing that I find is the most interesting area of tech right now is there's been this rise of distributed databases. Yugabyte—or You-ji-byte—Pla-netScale—or PlanetScale, depending on how you pronounce these things.Miles: [laugh]. Yeah, why, why is G such an adversarial consonant? I don't understand why we've all gotten to this place.Corey: Oh, yeah. But at the same time, it's—so you take a look at all these—and they all are speaking Postgres; it is pretty clear that ‘Postgres-squeal' is the thing that is taking over the world as far as databases go. If I were building something from scratch that used—Miles: For folks in the back, that's PostgreSQL, for the rest of us, it's okay, it's going to be, all right.Corey: Same difference. But yeah, it's the thing that is eating the world. Although recently, I've got to say, MongoDB is absolutely stepping up in a bunch of really interesting ways.Miles: I mean, I think the 4.0 release, I'm the guy who wrote the MongoDB on AWS Best Practices white paper, and I would grab a lot of customer's and—Corey: They have to change it since then of, step one: Do not use DocumentDB; if you want to use Mongo, use Mongo.Miles: Yeah, that's right. No, there were a lot of customers I was on the phone with where Mongo had summarily vaporized their data, and I think they have made huge strides in structural reliability over the course of—you know, especially this 4.0 launch, but the last couple of years, for sure.Corey: And with all the people they've been hiring from AWS, it's one of those, “Well, we'll look at this now who's losing important things from production?”Miles: [laugh]. Right? So, maybe there's only actually five humans who know how to do operations, and we just sort of keep moving around these different companies.Corey: That's sort of my assumption on these things. But Postgres, for those who are not looking to depart from the relational model, is eating the world. And—Miles: There's this, like, basic emotional thing. My buddy Martin, who set up MySQL, and took it public, and then promptly got it gobbled up by the Oracle people, like, there was a bet there that said, hey, there's going to be a real open database, and then squish, like, the man came and got it. And so like, if you're going to be an independent, open-source software developer, I think you're probably not pushing your pull requests to our friends at Oracle, that seems weird. So instead, I think Postgres has gobbled up the best minds on that stuff.And it works. It's reliable, it's consistent, and it's functional in all these different, sort of, reapplications and subdivisions, right? I mean, you have to sort of squint real hard, but down there in the guts of Redshift, that's Postgres, right? Like, there's Postgres behind all sorts of stuff. So, as an interface layer, I'm not as interested about how it manages to be successful at bossing around hardware and getting people the zeros and ones that they ask for back in a timely manner.I'm interested in it as a compatibility standard, right? If I have software that says, “I need to have Postgres under here and then it all will work,” that creates this layer of interop that a bunch of other products can use. So, folks like PlanetScale, and Yugabyte can say, “No, no, no, it's cool. We talk Postgres; that'll make it so your application works right. You can bring a SQL alchemy and plug it into this, or whatever your interface layer looks like.”That's the spot where, if I can trade what is a fairly limited global distribution, global transactional management on literally ridiculously unlimited scalability and zero operations, I can handle the hard parts of running a database over to somebody else, but I get my layer, and my software talks to it, I think that's a huge step.Corey: This episode is sponsored in part by my friends at Cloud Academy. Something special just for you folks. If you missed their offer on Black Friday or Cyber Monday or whatever day of the week doing sales it is—good news! They've opened up their Black Friday promotion for a very limited time. Same deal, $100 off a yearly plan, $249 a year for the highest quality cloud and tech skills content. Nobody else can get this because they have a assured me this not going to last for much longer. Go to CloudAcademy.com, hit the "start free trial" button on the homepage, and use the Promo code cloud at checkout. That's c-l-o-u-d, like loud, what I am, with a “C” in front of it. It's a free trial, so you'll get 7 days to try it out to make sure it's really a good fit for you, nothing to lose except your ignorance about cloud. My thanks again for sponsoring my ridiculous nonsense.Corey: I think that there's a strong movement toward building out on something like this. If it works, just because—well, I'm not multiregion today, but I can easily see a world in which I'd want to be. So, great. How do you approach the decision between—once this comes out of alpha; let's be clear. Let's turn this into something that actually ships, and no, Google that does not mean slapping a beta label on it for five years is the answer here; you actually have to stand behind this thing—but once it goes GA—Miles: GA is a good thing.Corey: Yeah. How do you decide between using that, or PlanetScale? Or Yugabyte?Miles: Or Cockroach or or SingleStore, right? I mean, there's a zillion of them that sit in this market. I think the core of the decision making for me is in every team you're looking at what skills do you bring to bear and what problem that you're off to go solve for customers? Do the nuances of these products make it easier to solve? So, I think there are some products that the nature of what you're building isn't all that dependent on one part of the application talking to another one, or an event happening someplace else mattering to an event over here. But some applications, that's, like, utterly critical, like, totally, totally necessary.So, we worked with a bunch of like Forex exchange trading desks that literally turn off 12 hours out of the day because they can only keep it consistent in one geographical location right near the main exchanges in New York. So, that's a place where I go, “Would you like to trade all day?” And they go, “Yes, but I can't because databases.” So, “Awesome. Let's call the folks on the Spanner side. They can solve that problem.”I go, “Would you like to trade all day and rewrite all your software?” And they go, “No.” And I go, “Oh, okay. What about trade all day, but not rewrite all your software?” There we go. Now, we've got a solution to that kind of problem.So like, we built this crazy game, like, totally other end of the ecosystem with the Dragon Ball Z people, hysterical; your like—you literally play like Rock, Paper, Scissors with your phone, and if you get a rock, I throw a fireball, and you get a paper, then I throw a punch, and we figure out who wins. But they can play these games like Europe versus Japan, thousands of people on each side, real-time, and it works.Corey: So, let's be clear, I have lobbied a consistent criticism at Google for a while now, which is the Google Cloud global control plane. So, you wind up with things like global service outages from time to time, you wind up with this thing is now broken for everyone everywhere. And that, for a lot of these use cases, is a problem. And I said that AWS's approach to regional isolation is the right way to do it. And I do stand by that assessment, except for the part where it turns out there's a lot of control plane stuff that winds up single tracking through us-east-1, as we learned in the great us-east-1 outage of 2021.Miles: Yeah, when I see customers move from data center to AWS, what they expect is a higher count of outages that lasts less time. That's the trade off, right? There's going to be more weird spurious stuff, and maybe—maybe—if they're lucky, that outage will be over there at some other region they're not using. I see almost exactly the same promise happening to folks that come from AWS—and in particular from Azure—over onto GCP, which is, there will be probably a higher frequency of outages at a per product level, right? So, like sometimes, like, some weird product takes a screw sideways, where there is structural interdependence between quite a few products—we actually published a whole internal structural map of like, you know, it turns out that Cloud SQL runs on top of GCE not on GKE, so you can expect if GKE goes sideways, Cloud SQL is probably not going to go sideways; the two aren't dependent on each other.Corey: You take the status page and Amazon FreeRTOS in a region is having an outage today or something like that. You're like, “Oh, no. That's terrible. First, let me go look up what the hell that is.” And I'm not using it? Absolutely not. Great. As hyperscalers, well, hyperscale, they're always things that are broken in different ways, in different locations, and if you had a truly accurate status page, it would all be red all the time, or varying shades of red, which is not helpful. So, I understand the challenge there, but very often, it's a partition that is you are not exposed to, or the way that you've architected things, ideally, means it doesn't really matter. And that is a good thing. So, raw outage counts don't solve that. I also maintain that if I were to run in a single region of AWS or even a single AZ, in all likelihood, I will have a significantly better uptime across the board than I would if I ran it myself. Because—Miles: Oh, for sure.Corey: —it is—Miles: For sure they're way better at ops than you are. Me, right?Corey: Of course.Miles: Right? Like, ridiculous.Corey: And they got that way, by learning. Like, I think in 2022, it is unlikely that there's going to be an outage in an AWS availability zone by someone tripping over a power cable, whereas I have actually done that. So, there's a—to be clear in a data center, not an AWS facility; that would not have flown. So, there is the better idea of of going in that direction. But the things like Route 53 is control plane single-tracking through the us-east-1, if you can't make DNS changes in an outage scenario, you may as well not have a DR plan, for most use cases.Miles: To be really clear, it was a part of the internal documentation on the AWS side that we would share with customers to be absolutely explicit with them. It's not just that there are mistakes and accidents which we try to limit to AZs, but no, go further, that we may intentionally cause outages to AZs if that's what allows us to keep broader service health higher, right? They are not just a blast radius because you, oops, pulled the pin on the grenade; they can actually intentionally step on the off button. And that's different than the way Google operates. They think of each of the AZs, and each of the regions, and the global system as an always-on, all the time environment, and they do not have systems where one gets, sort of, sacrificed for the benefit of the rest, right, or they will intentionally plan to take a system offline.There is no planned downtime in the SLA, where the SLAs from my friends at Amazon and Azure are explicit to, if they choose to, they decide to take it offline, they can. Now, that's—I don't know, I kind of want the contract that has the other thing where you don't get that.Corey: I don't know what the right answer is for a lot of these things. I think multi-cloud is dumb. I think that the idea of having this workload that you're going to seamlessly deploy to two providers in case of an outage, well guess what? The orchestration between those two providers is going to cause you more outages than you would take just sticking on one. And in most cases, unless you are able to have complete duplication of not just functionality but capacity between those two, congratulations, you've now just doubled your number of single points of failure, you made the problem actively worse and more expensive. Good job.Miles: I wrote an article about this, and I think it's important to differentiate between dumb and terrifyingly shockingly expensive, right? So, I have a bunch of customers who I would characterize as rich, as like, shockingly rich, as producing businesses that have 80-plus percent gross margins. And for them, the costs associated with this stuff are utterly rational, and they take on that work, and they are seeing benefits, or they wouldn't be doing it.Corey: Of course.Miles: So, I think their trajectory in technology—you know, this is a quote from a Google engineer—it's just like, “Oh, you want to see what the future looks like? Hang out with rich people.” I went into houses when I was a little kid that had whole-home automation. I couldn't afford them; my mom was cleaning house there, but now my house, I can use my phone to turn on the lights. Like—Corey: You know, unless us-east-1 is having a problem.Miles: Hey, and then no Roomba for you, right? Like utterly offline. So—Corey: Roomba has now failed to room.Miles: Conveniently, my lights are Philips Hue, and that's on Google, so that baby works. But it is definitely a spot where the barrier of entry and the level of complexity required is going down over time. And it is definitely a horrible choice for 99% of the companies that are out there right now. But next year, it'll be 98. And the year after that, it'll probably be 97. [laugh].And if I go inside of Amazon's data centers, there's not one manufacturer of hard drives, there's a bunch. So, that got so easy that now, of course you use more than one; you got to do—that's just like, sort of, a natural thing, right? These technologies, it'll move over time. We just aren't there yet for the vast, vast majority of workloads.Corey: I hope that in the future, this stuff becomes easier, but data transfer fees are going to continue to be a concern—Miles: Just—[makes explosion noise]—Corey: Oh, man—Miles: —like, right in the face.Corey: —especially with the Cambrian explosion of data because the data science folks have successfully convinced the entire industry that there's value in those mode balancer logs in 2012. Okay, great. We're never deleting anything again, but now you've got to replicate all of that stuff because no one has a decent handle on lifecycle management and won't for the foreseeable future. Great, to multiple providers so that you can work on these things? Like, that is incredibly expensive.Miles: Yeah. Cool tech, from this announcement at Next that I think is very applicable, and recognized the level of like, utter technical mastery—and security mastery to our earlier conversation—that something like this requires, the product is called BigQuery Omni, what Omni allows you to do is go into the Google Cloud Console, go to BigQuery, say I want to do analysis on this data that's in S3, or in Azure Blob Storage, Google will spin up an account on your behalf on Amazon and Azure, and run the compute there for you, bring the result back. So, just transfer the answers, not the raw data that you just scanned, and no work on your part, no management, no crapola. So, there's like—that's multi-cloud. If I've got—I can do a join between a bunch of rows that are in real BigQuery over on GCP side and rows that are over there in S3. The cross-eyedness of getting something like that to work is mind blowing.Corey: To give this a little more context, just because it gets difficult to reason about these things, I can either have data that is in a private subnet in AWS that traverses their horribly priced Managed NAT Gateways, and then goes out to the internet and sent there once, for the same cost as I could take that same data and store it in S3 in their standard tier for just shy of six full months. That's a little imbalanced, if we're being direct here. And then when you add in things like intelligent tiering and archive access classes, that becomes something that… there's no contest there. It's, if we're talking about things that are now approaching exabyte scale, that's one of those, “Yeah, do you want us to pay by a credit card?”—get serious. You can't at that scale anyway—“Invoice billing, or do we just, like, drive a dump truck full of gold bricks and drop them off in Seattle?”Miles: Sure. Same trajectory, on the multi-cloud thing. So, like a partner of ours, PacketFabric, you know, if you're a big, big company, you go out and you call Amazon and you buy 100 gigabit interconnect on—I think they call theirs Direct Connect, and then you hook that up to the Google one that's called Dedicated Interconnect. And voila, the price goes from twelve cents a gig down to two cents a gig; everybody's much happier. But Jesus, you pay the upfront for that, you got to set the thing up, it takes days to get deployed, and now you're culpable for the whole pipe if you don't use it up. Like, there are charges that are static over the course of the month.So, PacketFabric just buys one of those and lets you rent a slice of it you need. And I think they've got an incredible product. We're working with them on a whole bunch of different projects. But I also expect—like, there's no reason the cloud providers shouldn't be working hard to vend that kind of solution over time. If a hundred gigabit is where it is now, what does it look like when I get to ten gigabit? When I get to one gigabit? When I get to half gigabit? You know, utility price that for us so that we get to rational pricing.I think there's a bunch of baked-in business and cost logic that is a part of the pricing system, where egress is the source of all of the funding at Amazon for internal networking, right? I don't pay anything for the switches that connect to this machine to that machine, in region. It's not like those things are cheap or free; they have to be there. But the funding for that comes from egress. So, I think you're going to end up seeing a different model where you'll maybe have different approaches to egress pricing, but you'll be paying like an in-system networking fee.And I think folks will be surprised at how big that fee likely is because of the cost of the level of networking infrastructure that the providers deploy, right? I mean, like, I don't know, if you've gone and tried to buy a 40 port, 40 gig switch anytime recently. It's not like they're those little, you know, blue Netgear ones for 90 bucks.Corey: Exactly. It becomes this, [sigh] I don't know, I keep thinking that's not the right answer, but part of it also is like, well, you know, for things that I really need local and don't want to worry about if the internet's melting today, I kind of just want to get, like, some kind of Raspberry Pi shoved under my desk for some reason.Miles: Yeah. I think there is a lot where as more and more businesses bet bigger and bigger slices of the farm on this kind of thing, I think it's Jassy's line that you're, you know, the fat in the margin in your business is my opportunity. Like, there's a whole ecosystem of partners and competitors that are hunting all of those opportunities. I think that pressure can only be good for customers.Corey: Miles, thank you for taking the time to speak with me. If people want to learn more about you, what you're up to, your bad opinions, your ridiculous company, et cetera—Miles: [laugh].Corey: —where can they find you?Miles: Well, it's really easy to spell: SADA.com, S-A-D-A dot com. I'm Miles Ward, it's @milesward on Twitter; you don't have to do too hard of a math. It's miles@sada.com, if you want to send me an email. It's real straightforward. So, eager to reach out, happy to help. We've got a bunch of engineers that like helping people move from Amazon to GCP. So, let us know.Corey: Excellent. And we will, of course, put links to this in the [show notes 00:37:17] because that's how we roll.Miles: Yay.Corey: Thanks so much for being so generous with your time, and I look forward to seeing what comes out next year from these various cloud companies.Miles: Oh, I know some of them already, and they're good. Oh, they're super good.Corey: This is why I don't do predictions because like, the stuff that I know about, like, for example, I was I was aware of the Graviton 3 was coming—Miles: Sure.Corey: —and it turns out that if your—guess what's going to come up and you don't name Graviton 3, it's like, “Are you simple? Did you not see that one coming?” It's like—or if I don't know it's coming and I make that guess—which is not the hardest thing in the world—someone would think I knew and leaked. There's no benefit to doing predictions.Miles: No. It's very tough, very happy to do predictions in private, for customers. [laugh].Corey: Absolutely. Thanks again for your time. I appreciate it.Miles: Cheers.Corey: Myles Ward, CTO at SADA. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and be very angry in your opinion when you write that obnoxious comment, but then it's going to get lost because it's using MySQL instead of Postgres.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
An Enterprise Level View of Cloud Architecture with Levi McCormick

Screaming in the Cloud

Play Episode Listen Later Jan 6, 2022 33:52


About LeviLevi's passion lies in helping others learn to cloud better.Links: Jamf: https://www.jamf.com Twitter: https://twitter.com/levi_mccormick TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open-source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers, and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com.Corey: This episode is sponsored in part by our friends at Rising Cloud, which I hadn't heard of before, but they're doing something vaguely interesting here. They are using AI, which is usually where my eyes glaze over and I lose attention, but they're using it to help developers be more efficient by reducing repetitive tasks. So, the idea being that you can run stateless things without having to worry about scaling, placement, et cetera, and the rest. They claim significant cost savings, and they're able to wind up taking what you're running as it is in AWS with no changes, and run it inside of their data centers that span multiple regions. I'm somewhat skeptical, but their customers seem to really like them, so that's one of those areas where I really have a hard time being too snarky about it because when you solve a customer's problem and they get out there in public and say, “We're solving a problem,” it's very hard to snark about that. Multus Medical, Construx.ai and Stax have seen significant results by using them. And it's worth exploring. So, if you're looking for a smarter, faster, cheaper alternative to EC2, Lambda, or batch, consider checking them out. Visit risingcloud.com/benefits. That's risingcloud.com/benefits, and be sure to tell them that I said you because watching people wince when you mention my name is one of the guilty pleasures of listening to this podcast.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I am known-slash-renowned-slash-reviled for my creative pronunciations of various technologies, company names, et cetera. Kubernetes, for example, and other things that get people angry on the internet. The nice thing about today's guest is that he works at a company where there is no possible way for me to make it more ridiculous than it sounds because Levi McCormick is a cloud architect at Jamf. I know Jamf sounds like I'm trying to pronounce letters that are designed to be silent, but no, no, it's four letters: J-A-M-F. Jamf. Levi, thanks for joining me.Levi: Thanks for having me. I'm super excited.Corey: Exactly. Also professional advice for anyone listening: Making fun of company names is hilarious; making fun of people's names makes you a jerk. Try and remember that. People sometimes blur that distinction.So, very high level, you're a cloud architect. Now, I remember the days of enterprise architects where their IDEs were basically whiteboards, and it was a whole bunch of people sitting in a room. They call it an ivory tower, but I've been in those rooms; I assure you there is nothing elevated about this. It's usually a dank sub-basement somewhere. What do you do, exactly?Levi: Well, I am part of the enterprise architecture team at Jamf. My roles include looking at our use of cloud; making sure that we're using our resources to the greatest efficacy possible; coordinating between many teams, many products, many architectures; trying to make sure that we're using best practices; bringing them from the teams that develop them and learn them, socializing them to other teams; and just trying to keep a handle on this wild ride that we're on.Corey: So, what I find fun is that Jamf has been around for a long time. I believe it is not your first name. I want to say Casper was originally?Levi: I believe so, yeah.Corey: We're Jamf customers. You're not sponsoring this episode or anything, to the best of my knowledge. So, this is not something I'm trying to shill the company, but we're a customer; we use you to basically ensure that all of our company MacBooks, and laptops, et cetera, et cetera, are basically ensured that there's disk encryption turned on, that people have a password, and that screensaver is turned on, basically to mean that if someone gets their laptop stolen, it's a, “Oh, I have to spend more money with Apple,” and not, “Time to sound the data breach alarm,” for reasons that should be blindingly obvious. And it's great not just at the box check, but also fixing the real problem of I [laugh] don't want to lose data that is sensitive for obvious reasons. I always thought of this is sort of a thing that worked on the laptops. Why do you have a cloud team?Levi: Many reasons. First of all, we started in the business of providing the software that customers would run in their own data centers, in their own locations. Sometime in about 2015, we decided that we are properly equipped to run this better than other people, and we started to provide that as a service. People would move in, migrate their services into the cloud, or we would bring people into the cloud to start with.Device management isn't the only thing that we do. We provide some SSO-type services, we recently acquired a company called Wandera, which does endpoint security and a VPN-like experience for traffic. So, there's a lot of cloud powering all of those things.Corey: Are you able to disclose whether you're focusing mostly on AWS, on Azure, on Google Cloud, or are you pretending a cloud with something like IBM?Levi: All of the above, I believe.Corey: Excellent. That tells you it's a real enterprise, in seriousness. It's the—we talk about the idea of going all in on one providers being a general best practice of good place to start. I believe that. And then there are exceptions, and as companies grow and accumulate technical debt, that also is load-bearing and generates money, you wind up with this weird architectural series of anti-patterns, and when you draw it on a whiteboard of, “Here's our architecture,” the junior consultant comes in and says, “What moron built this?” Usually two said quote-unquote, “Moron,” and then they've just pooched the entire engagement.Yeah, most people don't show up in the morning hoping to do a terrible job today, unless they work at Facebook. So, there are reasons things are the way they are; they're constraints that shape these things. Yeah, if people were going to be able to shut down the company for two years and rebuild everything from scratch from the ground up, it would look wildly different. But you can't do that most of the time.Levi: Yeah. Those things are load bearing, right? You can't just stop traffic one day, and re-architect it with the golden image of what it should have been. We've gone through a series of acquisitions, and those architectures are disparate across the different acquired products. So, you have to be able to leverage lessons from all of them, bring them together and try and just slowly, incrementally march towards a better future state.Corey: As we take a look at the challenges we see The Duckbill Group over on my side of the world, where we talk to customers, it's I think it is surprising to folks to learn that cloud economics as I see it is—well, first, cost and architecture the same thing, which inherently makes sense, but there's a lot more psychology that goes into it than math. People often assume I spend most of my time staring into spreadsheets. I assure you that would not go super well. But it has to do with the psychological elements of what it is that people are wrestling with, of their understanding of the environment has not kept pace with reality, and APIs tend to, you know, tell truths.It's always interesting to me to see the lies that customers tell, not intentionally, but the reality of it of, “Okay, what about those big instances you're running in Australia?” “Oh, we don't have any instances in Australia.” “Look, I understand that you are saying that in good faith, however…” and now we're in a security incident mode and it becomes a whole different story. People's understanding always trails. What do you spend the bulk of your time doing? Is it building things? Is it talking to people? Is it trying to more or less herd cats in certain directions? What's the day-to-day?Levi: I would say it varies week-to-week. Depends on if we have a new product rolling out. I spend a lot of my time looking at architectural diagrams, reference architectures from AWS. The majority of the work I do is in AWS and that's where my expertise lies. I haven't found it financially incentivized to really branch out into any of the other clouds in terms of expertise, but I spend a lot of my time developing solutions, socializing them, getting them in front of teams, and then educating.We have a wide range of skills internally in terms of what people know or what they've been exposed to. I'd say a lot of engineers want to learn the cloud and they want to get opportunities to work on it, and their day-to-day work may not bring them those opportunities as often as they'd like. So, a good portion of my time is spent educating, guiding, joining people's sprints, joining in their stand-ups, and just kind of talking through, like, how they should approach a problem.Corey: Whenever you work at a big company, you invariably wind up with—well, microservices becomes the right answer, not because of the technical reasons; because of the people reason, the way that you get a whole bunch of people moving in roughly the same direction. You are a large scale company; who owns services in your idealized view of the world? Is it, “Well, I wrote something and it's five o'clock. Off to production with it. Talk to you in two days, if everything—if we still have a company left because I didn't double-check what I just wrote.”Do you think that the people who are building services necessarily should be the ones supporting it? Like, in other words, Amazon's approach of having the software engineers being responsible for the ones running it in production from an ops perspective. Is that the direction you trend towards, or do you tend to be from my side of the world—which is grumpy sysadmin—where people—developers hurl applications into your yard for you to worry about?Levi: I would say, I'm an extremist in the view of supporting the Amazon perspective. I really like you build it, you run it, you own it, you architect it, all of it. I think the other teams in the organization should exist to support and enable those paths. So, if you have platform teams are a really common thing you see hired right now, I think those platforms should be built to enable the company's perspective on operating infrastructure or services, and then those service teams on top of that should be enabled to—and empowered to make the decisions on how they want to build a service, how they want to provide it. Ultimately, the buck should stop with them.You can get into other operational teams, you could have a systems operation team, but I think there should be an explicit contract between a service team, what they build, and what they hand off, you know, you could hand off, like, a tier one level response, you know, you can do playbooks, you could do, you know, minimal alert, response, routing, that kind of stuff with a team, but I think that even that team should have a really strong contract with, like, here's what our team provides, here's how you engage with our team, here's how you will transition services to our team.Corey: The challenge with doing that, in some shops, has been that if you decide to roll out a, you build it, you own it, approach that has not been there since the beginning, you wind up with a lot of pushback from engineers who until now really enjoyed their 5:30 p.m. quitting time, or whenever it was they wound up knocking off work. And they started pushing back, like, “Working out of hours? That's inhumane.” And the DevOps team would be sitting there going, “We're right here. How dare you? Like, what do you think our job is?” And it's a, “Yes, but you're not people.” And then it leads to this whole back and forth acrimonious—we'll charitably call it a debate. How do you drive that philosophy?Levi: It's a challenge. I've seen many teams fracture, fall apart, disperse, if you will, under the transition of going through, like, an extreme service ownership. I think you balance it out with the carrot of you also get to determine your own future, right? You get to determine the programming language you use, you get to determine the underlying technologies that you use. Again, there's a contract: You have to meet this list of security concerns, you need to meet these operational concerns, and how you do that is up to you.Corey: When you take a look across various teams—let's bound this to the industry because I don't necessarily want you to wind up answering tough questions at work the day this episode airs—what do you see the biggest blockers to achieving, I guess, a functional cultural service ownership?Levi: It comes down to people's identity. They've established their own identity, “As I am X,” right? I'm a operations engineer. I'm a developer, I'm an engineer. And getting people to kind of branch out of that really fixed mindset is hard, and that, to me, is the major blocker to people assuming ownership.I've seen people make the transition from, “I'm just an engineer. I just want to write code.” I hate those lines. That frustrates me so much: “I just want to write code.” Transitioning into that, like, ownership of, “I had an idea. I built the platform or the service. It's a huge hit.” Or you know, “Lots of people are using it.” Like, seeing people go through that transformation become empowered, become fulfilled, I think is great.Corey: I didn't really expect to get called out quite like this, but you're absolutely right. I was against the idea, back when I was a sysadmin type because I didn't know how to code. And if you have developers supporting all of the stuff that they've built, then what does that mean for me? It feels like my job is evaporating. I don't know how to write code.Well, then I started learning how to write code incredibly badly. And then wow, it turns out, everyone does this. And here we are. But it's—I don't build applications, for obvious reasons. I'm bad at it, but I found another way to proceed in the wide world that we live in of high technology.But yeah, it was hard because this idea of my sense of identity being tied to the thing that I did, it really was an evolve-or-die dinosaur kind of moment because I started seeing this philosophy across the board. You take a look, even now at modern SRE is, or modern DevOps folks, or modern sysadmins, what they're doing looks a lot less like logging into Linux systems and tinkering on the command line a lot more like running and building distributed applications. Sure, this application that you're rolling out is the one that orchestrates everything there, but you're still running this in the same way the software engineers do, which is, interestingly.Levi: And that doesn't mean a team has to be only software engineers. Your service team can be multiple disciplines. It should be multiple disciplines. I've seen a traditional ops team broken apart, and those individuals distributed into the services that they were chiefly skilled in supporting in the past, as the ops team, as we transitioned those roles from one of the worst on-call rotations I've ever seen—you know, 13 to 14 alerts a night—transitioning those out to those service teams, training them up on the operations, building the playbooks. That was their role. Their role wasn't necessarily to write software, day one.Corey: I quit a job after six weeks because of that style of, I guess, mismanagement. Their approach was that, oh, we're going to have our monitoring system live in AWS because one of our VPs really likes AWS—let's be clear, this was 2008, 2009 era—latency was a little challenging there. And [unintelligible 00:17:04] he really liked Big Brother, which was—not to—now before that became a TV show and at rest, it was a monitoring system—but network latency was always a weird thing in AWS in those days, so instead, he insisted we set up three of them. And whenever—if we just got one page, it was fine. But if we got three, then we had to jump in. And two was always undefined.And they turned this off from I think, 10 p.m. to 6 a.m. every night, just so the person I call could sleep. And I'm looking at this, like, this might be the worst thing I've ever seen in my life. This was before they released the Managed NAT Gateway, so possibly it was.Levi: And then the flood, right, when you would get—Corey: Oh, God this was the days, too—Levi: Yeah.Corey: —when you were—if you weren't careful, you'd set this up to page you on the phone with a text message and great, now it takes time for my cell provider to wind up funneling out the sudden onslaught of 4000 text messages. No thanks.Levi: If your monitoring system doesn't have the ability to say, you know, the alert flood, funnel them into one alert, or just pause all alerts, while—because we know there's an incident; you know, us-east-1 is down, right? We know this; we don't need to get 500 text messages to each engineer that's on call.Corey: Well, my philosophy at that point was no, I'm going to instead take a step beyond. If I'm not empowered to fix this thing that is waking me up—and sometimes that's the monitoring system, and sometimes it's the underlying application—I'm not on call.Levi: Yes, exactly. And that's why I like the model of extre—you know, the service ownership: Because those alerts should go to the people—the pain should be felt by the people who are empowered to fix it. It should not land anywhere else. Otherwise, that creates misaligned incentives and nothing gets better.Corey: Yeah. But in large distributed systems, very often the person is on call more or less turns into a traffic router.Levi: Right. That's unfair to them.Corey: That's never fun—yeah, that's unfair, and it's not fun, either, and there's no great answer when you've all these different contributory factors.Levi: And how hard is it to keep the team staffed up?Corey: Oh, yeah. It's a, “Hey, you want a really miserable job one week out of every however many there are in the cycle?” Eh, people don't like that.Levi: Exactly.Corey: This episode is sponsored by our friends at Oracle HeatWave, a new high-performance accelerator for the Oracle MySQL Database Service, although I insist on calling it, “My squirrel.” While MySQL has long been the world's most popular open source database, shifting from transacting to analytics required way too much overhead and, you know, work. With HeatWave you can run your OLAP and OLTP—don't ask me to ever say those acronyms again—workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: So, I've been tracking what you're up to for little while now—you're always a blast to talk with—what is this whole Cloud Builder thing that you were talking about for a bit, and then I haven't seen much about it.Levi: Ah, so at the beginning of the pandemic, our mutual friend, Forrest Brazeal, released the Cloud Resume Challenge. I looked at that, and I thought, this is a fantastic idea. I've seen lots of people going through it. I recommend the people I mentor go through it. Great way to pick up a couple cloud skills here and there, tell an interesting story in an interview, right? It's a great prep.I intended the Cloud Builder Challenge to be a natural kind of progression from that Resume Challenge to the Builder Challenge where you get operational experience. Again, back to that, kind of, extreme service ownership mentality, here's a project where you can build, really modeled on the Amazon GameDays from re:Invent, you build a service, we'll send you traffic, you process those payloads, do some matching, some sorting, some really light processing on these payloads, and then send it back to us, score some points, we'll build a public dashboard, people can high five each other, they can razz each other, kind of competition they want to do. Really low, low pressure, but just a fun way to get more operational experience in an area where there is really no downside. You know, playing like that at work, bad idea, right?Corey: Generally, yes. [crosstalk 00:21:28] production, we used to have one of those environments; oops-a-doozy.Levi: Yeah. I don't see enough opportunities for people to gain that experience in a way that reflects a real workload. You can go out and you can find all kinds of Hello Worlds, you can find all kinds of—like, for front end development, there are tons of activity activities and things you can do to learn the skills, but for the middleware, the back end engineers, there's just not enough playgrounds out there. Now, standing up a Hello World app, you know, you've got your infrastructures code template, you've got your pre-written code, you deploy it, congratulations. But now what, right?And I intended this challenge to be kind of a series of increasingly more difficult waves, if you will, or levels. I really had a whole gamification aspect to it. So, it would get harder, it would get bigger, more traffic, you know, all of those things, to really put people through what it would be like to receive your, “Post got slash-dotted today,” or those kinds of things where people don't get an opportunity to deal with large amounts of traffic, or variable payloads, that kind of stuff.Corey: I love the idea. Where is it?Levi: It is sitting in a bunch of repos, and I am afraid to deploy it. [laugh].Corey: What is it that scares you about it specifically?Levi: The thing that specifically scares me is encouraging early career developers to go out there, deploy this thing, start playing with it, and then incur a huge cloud bill.Corey: Because they failed to secure something or other reasons behind that?Levi: There are many ways that this could happen, yeah. You could accidentally push your access key, secret key up into a public repo. Now, you've got, you know, Bitcoin miners or Monero miners running in your environment. You forget to shut things off, right? That's a really common thing.I went through a SageMaker demo from AWS a couple years ago. Half the room of intelligent, skilled engineers forgot to shut off the SageMaker instances. And everybody ran out of the $25 of credit they had from the demo—Corey: In about ten minutes. Yeah.Levi: In about ten minutes, yeah. And we had to issue all kinds of requests for credits and back and forth. But granted, AWS was accommodating to all of those people, but it was still a lot of stress.Corey: But it was also slow. They're very slow on that, which is fair. Like, if someone's production environment is down, I can see why you care more about that than you do about someone with, “Ah, I did something wrong and lost money.” The counterpoint to that is that for early career folks, that money is everything. We remember earlier this year, that tragic story from the Robinhood customer who committed suicide after getting a notification that he was $730,000 in debt. Turns out it wasn't even accurate; he didn't owe anything when all was said and done.I can see a scenario in which that happens in the AWS world because of their lack of firm price controls on a free tier account. I don't know what the answer on this is. I'm even okay with a, “Cool you will—this is a special kind of account that we will turn you off at above certain levels.” Fine. Even if you hard cap at the 20 or 50 bucks, yeah, it's going to annoy some people, but no one is going to do something truly tragic over that. And I can't believe that Oracle Cloud of all companies is the best shining example of this because you have to affirmatively upgrade your account before they'll charge you a dime. It's the right answer.Levi: It is. And I don't know if you've ever looked at—well, I'm sure you'd have. You've probably looked at the solutions provided by AWS for monitoring costs in your accounts, preventing additional spend. Like, the automation to shut things down, right, it's oftentimes more engineering work to make it so that your systems will shut down automatically when you reach a certain billing threshold than the actual applications that are in place there.Corey: And I don't for the life of me understand why things are the way that they are. But here we go. It's a—[sigh] it just becomes this perpetual strange world. I wish things were better than they are, but they're not.Levi: It makes me terribly sad. I mean, I think AWS is an incredible product, I think the ecosystem is great, and the community is phenomenal; everyone is super supportive, and it makes me really sad to be hesitant to recommend people dive into it on their own dime.Corey: Yeah. And that is a—[sigh] I don't know how you fix that or square that circle. Because I don't want to wind up, I really do not want to wind up, I guess, having to give people all these caveats, and then someone posts about a big bill problem on the internet, and all the comments are, “Oh, you should have set up budgets on that.” Yeah, that's thing still a day behind. So okay, great, instead of having an enormous bill at the end of the month, you just have a really big one two days later.I don't think that's the right answer. I really don't. And I don't know how to fix this, but, you know, I'm not the one here who's a $1.7 trillion company, either, that can probably find a way to fix this. I assure you, the bulk of that money is not coming from a bunch of small accounts that forgot to turn something off or got exploited.Levi: I haven't done my 2021 taxes yet, but I'm pretty sure I'm not there either.Corey: The world in which we live.Levi: [laugh]. I would love this challenge. I would love to put it out there. If I could, on behalf of, you know, early career people who want to learn—if I could issue credits, if I could spin up sandboxes and say, like, “Here's an account, I know you're going to be safe. I have put in a $50 limit.” Right?Corey: Yeah.Levi: “You can't spend more than $50,” like, if I had that control or that power, I would do this in a heartbeat. I'm passionate about getting people these opportunities to play, you know, especially if it's fun, right? If we can make this thing enjoyable, if we can gamify it, we can play around, I think that'd be great. The experience, though, would be a significant amount of engineering on my side, and then a huge amount of outreach, and that to me makes me really sad.Corey: I would love to be able to do something like that myself with a, “Look, if you get a bill, they will waive it, or I will cover it.” But then you wind up with the whole problem of people not operating in good faith as well. Like, “All right, I'm going to mine a bunch of Bitcoin and claim someone else did it.” Or whatnot. And it's just… like, there are problems with doing this, and the whole structure doesn't lend itself to that working super well.Levi: Exactly. I often say, you know, I face a lot of people who want to talk about mining cryptocurrency in the cloud because I'm a cloud architect, right? That's a really common conversation I have with people. And I remind them, like, it's not economical unless you're not paying for it.Corey: Yeah, it's perfectly economical on someone else's account.Levi: Exactly.Corey: I don't know why people do things the way that they do, but here we are. So, re:Invent. What did you find that was interesting, promising there, promising but not there yet, et cetera? What was your takeaway from it? Since you had the good sense not to be there in person?Levi: [laugh]. To me, the biggest letdown was Amplify Studio.Corey: I thought it was just me. Thank you. I just assumed it was something I wasn't getting from the explanation that they gave. Because what I heard was, “You can drag and drop, basically, a front end web app together and then tie it together with APIs on the back end.” Which is exactly what I want, like Retool does; that's what I want only I want it to be native. I don't think it's that.Levi: Right. I want the experience I already have of operating the cloud, knowing the security posture, knowing the way that my users access it, knowing that it's backed by Amazon, and all of their progressively improving services, right? You say it all the time. Your service running on Amazon is better today than it was two years ago. It was better than it was five years ago. I want that experience. But I don't think Amplify Studio delivered.Corey: I wish it had. And maybe it will, in the fullness of time. Again, AWS services do not get worse as they age they get better.Levi: Some gets stale, though.Corey: Yeah. The worst case scenario is they sit there and don't ever improve.Levi: Right. I thought the releases from S3 in terms of, like, the intelligent tiering, were phenomenal. I would love to see everybody turn on intelligent tiering with instant access. Those things to me were showing me that they're thinking about the problem the right way. I think we're missing a story of, like, how do we go from where we're at today—you know, if I've got trillions of objects in storage, how do I transition into that new world where I get the tiering automatically? I'm sure we'll see blog posts about people telling us; that's what the community is great for.Corey: Yeah, they explain these things in a way that the official docs for some reason fail to.Levi: Right. And why don't—Corey: Then again, it's also—I think—I think it's because the people that are building these things are too close to the thing themselves. They don't know what it's like to look at it through fresh eyes.Levi: Exactly. They're often starting from a blank slate, or from a greenfield perspective. There's not enough thought—or maybe there's a lot of thought to it, but there's not enough communication coming out of Amazon, like, here's how you transition. We saw that with Control Tower, we saw that with some of the releases around API Gateway. There's no story for transitioning from existing services to these new offerings. And I would love to see—and maybe Amazon needs a re:Invent Echo, where it's like, okay, here's all the new releases from re:Invent and here's how you apply them to existing infrastructure, existing environments.Corey: So, what's next for you? What are you looking at that's exciting and fun, and something that you want to spend your time chasing?Levi: I spend a lot of my time following AWS releases, looking at the new things coming out. I spend a lot of energy thinking about how do we bring new engineers into the space. I've worked with a lot of operations teams—those people who run playbooks, they hop on machines, they do the old sysadmin work, right—I want to bring those people into the modern world of cloud. I want them to have the skills, the empowerment to know what's available in terms of services and in terms of capabilities, and then start to ask, “Why are we not doing it that way?” Or start looking at making plans for how do we get there.Corey: Levi, I really want to thank you for taking the time to speak with me. If people want to learn more. Where can they find you?Levi: I'm on Twitter. My Twitter handle is @levi_mccormick. Reach out, I'm always willing to help people. I mentor people, I guide people, so if you reach out, I will respond. That's a passion of mine, and I truly love it.Corey: And we'll of course, include a link to that in the [show notes 00:32:28]. Thank you so much for being so generous with your time. I appreciate it.Levi: Thanks, Corey. It's been awesome.Corey: Levi McCormick, cloud architect at Jamf. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me that service ownership is overrated because you are the storage person, and by God, you will die as that storage person, potentially in poverty.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

AWS Morning Brief
Time to Give LastPass the Heave

AWS Morning Brief

Play Episode Listen Later Jan 6, 2022 5:16


Links: “Tokyo police lose 2 floppy disks containing personal info on 38 public housing applicants”: https://mainichi.jp/english/articles/20211227/p2a/00m/0na/072000c LastPass may have suffered a breach: https://news.ycombinator.com/item?id=29705957 “Worst AWS Data Breaches of 2021”: https://securityboulevard.com/2021/12/worst-aws-data-breaches-of-2021/ D.W. Morgan: https://www.hackread.com/logistics-giant-d-w-morgan-exposed-clients-data/ SEGA Europe: https://vpnoverview.com/news/sega-europe-suffers-major-security-breach/ “Identity Guide–Preventive controls with AWS Identity–SCPs”: https://aws.amazon.com/blogs/mt/identity-guide-preventive-controls-with-aws-identity-scps/ Log4j scanner: https://github.com/google/log4jscanner TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it's nobody in particular's job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: The first security round-up of the year in Last Week in AWS: Security. This is relatively light, just because it covers the last week of the year, where people didn't really “Work” so much as “Get into fights on Twitter.” Onward.So, from the community, ever see a data breach announcement that raises oh so very many more questions than it answers? I swear this headline is from a week or so ago, not 1998: “Tokyo police lose 2 floppy disks containing personal info on 38 public housing applicants”. Yes, I said floppy disks.The terrible orange website, also known as Hacker News, reports that LastPass may have suffered a breach. At the time I write this, the official LastPass blog has a, “No, it's just people reusing passwords.” Enough people I trust have seen this behavior that I'd be astounded if that were true. If you can't trust your password manager, ditch them immediately.Security Boulevard had a roundup of the “Worst AWS Data Breaches of 2021”, and it's the usual run-of-the-mill S3 bucket problems, but my personal favorite's the Twitch breach because it's particularly embarrassing, given that it is, in fact, an Amazon subsidiary.First one goes to D.W. Morgan by leaking 100GB of client data. And they're a logistics company that serves giant enterprises, so these are companies with zero sense of humor, so I would not want to be in D.W. Morgan's position this week.And the other is a little funnier. It goes to SEGA Europe, after Sonic the Hedgehog forgets to perform due diligence on his AWS environment.Corey: Are you building cloud applications with a distributed team? Check out Teleport, an open-source identity-aware access proxy for cloud resources. Teleport provides secure access for anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps, and databases. Teleport gives engineers superpowers. Get access to everything via single sign-on with multi-factor, list and see all of SSH servers, Kubernetes clusters, or databases available to you in one place, and get instant access to them using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility, and ensuring compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That's goteleport.com.AWS had only a single thing that I found interesting: “Identity Guide–Preventive controls with AWS Identity–SCPs”. I've been waiting for a while for a good explainer on SCPs to come out for a while, and this looks like it actually is a thing that I want. I've been playing around with SCPs a lot more for the past couple of weeks. If you're unfamiliar, it's a way to override what the root user can do in an organization's member accounts. It's super handy to constrain people from doing things that are otherwise foolhardy.And lastly, an interesting tool came out from Google—which I should not have to explain what that is to you folks; they turn things off, like Reader—they also released a log4j scanner. This one scans files on disk to detect the bad versions of log4j—which is most of them—and can replace them with the good version—which is, of course, print statements. And that's what happened last week in AWS security. Hopefully next week will be… well, I don't want to say less contentful, but I do want to say it's at least not as exciting as the last month has been. Thanks for listening.Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Fear and Loathing on the re:Invent Show Floor of ‘21 with Aaron Booth

Screaming in the Cloud

Play Episode Listen Later Jan 5, 2022 33:30


About AaronI am a Cloud Focused Product Management and Technical Product Ownership Consultant. I have worked on several Cloud Products & Services including resale, management & governance, cost optimisation, platform management, SaaS, PaaS. I am also recognised as a AWS Community Builder due to my work building cloud communities cross-government in the UK over the last 3 years. I have extensive commercial experience dealing with Cloud Service Providers including AWS, Azure, GCP & UKCloud. I was the Single Point of Contact for Cloud at the UK Home Office and was the business representative for the Home Office's £120m contract with AWS. I have been involved in contract negotiation, supplier relationship management & financial planning such as business cases & cost management.I run a IT Consultancy called Embue, specialising in Agile, Cloud & DevOps consulting, coaching and training. Links: Twitter: https://twitter.com/AaronBoothUK LinkedIn: https://www.linkedin.com/in/aaronboothuk/ Embue: https://embue.co.uk Publicgood.cloud: https://publicgood.cloud TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open-source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers, and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com.Corey: This episode is sponsored in part by our friends at Rising Cloud, which I hadn't heard of before, but they're doing something vaguely interesting here. They are using AI, which is usually where my eyes glaze over and I lose attention, but they're using it to help developers be more efficient by reducing repetitive tasks. So, the idea being that you can run stateless things without having to worry about scaling, placement, et cetera, and the rest. They claim significant cost savings, and they're able to wind up taking what you're running as it is in AWS with no changes, and run it inside of their data centers that span multiple regions. I'm somewhat skeptical, but their customers seem to really like them, so that's one of those areas where I really have a hard time being too snarky about it because when you solve a customer's problem and they get out there in public and say, “We're solving a problem,” it's very hard to snark about that. Multus Medical, Construx.ai and Stax have seen significant results by using them. And it's worth exploring. So, if you're looking for a smarter, faster, cheaper alternative to EC2, Lambda, or batch, consider checking them out. Visit risingcloud.com/benefits. That's risingcloud.com/benefits, and be sure to tell them that I said you because watching people wince when you mention my name is one of the guilty pleasures of listening to this podcast.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. So, when I went to re:Invent last year, I discovered a whole bunch of things I honestly was a little surprised to discover. One of those things is my guest today, Aaron Booth, who's a cloud consultant with an emphasis on sustainability. Now, you see a number of consultants at things like re:Invent, but what made Aaron interesting was that this was apparently his first time visiting the United States, and he started with not just Las Vegas, but Las Vegas to attend re:Invent. Aaron, thank you for joining me, and honestly, I'm a little surprised you survived.Aaron: Yeah, I think one of the things about going to Las Vegas or Nevada is no one really prepared me for how dry it was. I ended up walking out of re:Invent with my fingers, like, bleeding, and everything else. And there was so much about America that I didn't expect, but that was one thing I wish somebody had warned me about. But yeah, it was my first time in the US, first time at re:Invent, and I really enjoyed it. It was probably the best investment in myself and my business that I think I've done so far.Corey: It's always strange to look at a place that you live and realize, oh, yeah, this is far away for someone else. What would their experience be of coming and learning about the culture we have here? And then you go to Las Vegas, and it's easy to forget there are people who live there. And even the people who live there do not live on the strip, in the casinos, at loud, obnoxious cloud conferences. So, it feels like it's one of those ideas of oh, I'm going to go to a movie for the first time and then watching something surreal, like Memento or whatnot, that leaves everyone very confused. Like, “Is this what movies are like?” “Well, this one, but no others are quite like that.” And I feel that way about Las Vegas and re:Invent, simultaneously.Aaron: I mean, talking about movies, before it came to the US and before I came to Vegas, I was like, “Oh, how can I prepare myself for this trip?” I ended up watching Fear and Loathing in Las Vegas. And I don't know if you ever seen it, with Johnny Depp, but it's probably not the best representation, or the most modern representation what Vegas would be like. And I think halfway through the conference, went down to Fremont Street in the old downtown. And they have this massive, kind of, free block screen in the sky that is lit up and doing all these animations. And you're just thinking, “What world am I on?” And it kind of is interesting as well, from a point of view of, we're at this tech conference; it's in Vegas; what is the reason for that? And there's obviously lots of different things. We want people to have fun, but you know, it is an interesting place to put 30,000 people, especially during a pandemic.Corey: It really is. I imagine it's going to have to stay there because in a couple more years, you're going to need a three block long screen just to list all of the various services that AWS offers because they don't believe in turning anything off. Now, it would be remiss for me not to ask you, what was announced at re:Invent that got you the most, let's call it excited, I guess? What got you enthusiastic? What are you happy to start working with more?Aaron: I think from my perspective, there's a few different announcements. The first one that comes to mind is the stuff of AWS Amplify Studio, and that's taken this, kind of, no-code Figma designs and turn into a working front end. And it's really interesting for me to think about, okay, what is the point of cloud? Why are we moving forward in the world, especially in technology? And, you know, abstracting a lot of stuff we worry about today to simple drag-and-drop tools is probably going to be the next big thing for most of the world.You know, we've come from a privileged position in the West where we follow technology along the whole of the journey, where now we have an opportunity to open this out to many more regions, and many more AWS customers, for example. But for me, as a small business owner—I've run multiple businesses—there's a lot of effort you put into, okay, I need to set up a business, and a website, and newsletter, or whatever else. But the more you can just turn that into, “I've got an idea, and I can give it to people with one click,” you'll enable a lot more business and a lot more future customers as well.Corey: I was very excited about that one, too, just from a perspective of I want to drag and drop something together to make a fairly crappy web app, that sounds like the thing that I could use to do that. No, that feels a lot more like what Honeycode is trying to be, as opposed to the Amplify side of the world, which is still very focused on React. Which, okay, that makes sense. There's a lot of front end developers out there, and if you're trying to get into tech today and are asking what language should I learn, I would be very hard-pressed to advise you pick anything that isn't JavaScript because it is front end, it is back end, it runs slash eats the world. And I've just never understood it. It does not work the way that I think about computers because I'm old and grumpy. I have high hopes of where it might go, but so far I'm looking at it's [sigh] it's not what I want it to be, yet. And maybe that's just because I'm weird.Aaron: Well, I mean, you know, you mentioned part of the problem really is two different competing AWS services themselves, which with a business like AWS and their product strategy being the word, “Yes,” you know, you're never really going to get a lot of focus or forward direction with certain products. And hopefully, there'll be the next, no-code tool announced in re:Invent in a few years' time, which is exactly what we're looking for, and gives startup founders or small businesses drag-and-drop tools. But for now, there's going to be a lot of competing services.Corey: There's so much out there that it's almost impossible to wind up contextualizing re:Invent as a single event. It feels like it's too easy to step back and say, “Oh, okay. I'm here to build websites”—is what we're talking about now in the context of Amplify—and then they start talking about mainframes. And then they start talking about RoboRunner to control 10,000 robots at once. And I'm looking around going, “I don't have problems that feel a lot like that. What's the deal?”Aaron: I think even just, like you said in perspective of re:Invent is like, when you go to an event like this, that you can't experience everything and you probably have a very specific focus of, you know, what am I here to do. And I was really surprised—again, my first time at a big tech conference, as well as Vegas and the US is, how important it was just to meet people and how valuable that was. First time I met you, and you know, going from somebody who's probably very likely interacted with you on Twitter before the event to being on this podcast and having a great conversation now is kind of crazy to think that the value you can get out of it. I mean, in terms of over services, and areas of re:Invent that I found interesting was the announcement of the new sustainability pillar, as part of the well-architected framework. You know, I've tried to use that before in previous workplaces, and it has been useful. You know, I'm hoping it is more useful in the future, and the cynical part of me worries about whether the whole point of putting this as part of a well-architected framework review where the customer is supposed to do it is Amazon passing the buck for sustainability. But it's an interesting way forward for what we care about.Corey: An interesting quirk of re:Invent—to me—has always been that despite there being tens of thousands of people there are always a few folks that you wind up running into again and again and again throughout the week. One year for me it was Ben Kehoe; this trip it was you where we kept finding ourselves at the same events, we kept finding ourselves at the same restaurants, and we had three or four meals together as a result, and it was a blast talking to you. And I was definitely noticing that sustainability was a topic that you kept going back to a bunch of different ways. I mean previously, before starting your current consulting company, you did a lot of work in the government—specifically the UK Government, for those who are having trouble connecting the fact this is the first time in America to the other thing. Like, “Wow, you can be far away and work for the government?” It's like, we have more than one on this planet, as it turns out.Yes, it was a fun series of conversations, and I am honestly a little less cynical about the idea of the sustainability pillar, in no small part due to the conversations that we had together. I initially had the cynical perspective of here's how to make your cloud infrastructure more sustainable. It's, isn't that really a “you” problem? You're the cloud provider. I can't control how you get energy on the markets, how you wind up handling heat issues, how you address water issues from your data center outflows, et cetera. It seems to me that the only thing I can really do is use the services you give me, and then it becomes a “you” problem. You have a more nuanced take on it.Aaron: I think there's a log of different things to think about when it comes to sustainability. One of the main ones is, from my perspective, you know, I worked at the UK Home Office in the UK, and we'd been using cloud for about six or seven years. And just looking at how we use clouds as an enterprise organization, one of the things I really started to see was these different generations of cloud and you've got aspects of legacy infrastructure, almost, that we lifted-and-shifted in the early days, versus maybe stuff would run on serverless now. And you know, that's one element, from a customer is how you control your energy usage is actually the use of servers, how efficient your code is, and there's definitely a difference between stringing together EC2 and S3 buckets compared to using serverless or Lambda functions.Corey: There's also a question of scale. When I'm trying to build something out of Lambda functions, and okay, which region is the most cost effective way to run this thing? The Google search for that will have a larger climate impact than any decision I can make at the scale that I operate at. Whereas if you're a company running tens of thousands of instances at any given point in time and your massive scale, then yeah, the choices you make are going to have significant impact. I think that a problem AWS has always struggled with has been articulating who needs to care about what, when.If you go down the best practices for security and governance and follow the white papers, they put out as a one-person startup trying to build an idea this evening, just to see if it's viable, you're never going to get anywhere. If you ignore all those things, and now you're about to go public as a bank, you're going to have a bad time, but at what point do you have to start caring about these different things in different ways? And I don't think we know the answer yet, from a sustainability perspective.Aaron: I think it's interesting in some senses, that sustainability is only just enter the conversation when it comes to stuff we care about in businesses and enterprises. You know, we all know about risk registers, and security reviews, and all those things, but sustainability, while we've, kind of, maybe said nice public statements, and put things on our website, it's not really been a thing that's, okay, this is how we're going to run our business, and the thing we care about as number one. You know, Amazon always says security is job zero, but maybe one day someone will be saying sustainability is our job zero. And especially when it comes down to, sort of, you know, the ethics of running a business and how you want that to be run, whether it is going to be a capitalistic VC-funded venture to extract wealth from citizens and become a billionaire versus creating something that's a bit more circular, and gives back as sustainability might be a key element of what you care about when you make decisions.Corey: The challenge that I find as well is, I don't know how you can talk about the relative sustainability impact of various cloud services within the AWS umbrella without, effectively, AWS explaining to you what their margins are on different services, in many respects. Power usage is the primary driver of this and that determines the cost of running things. It is very clear that it is less expensive and more efficient to run more modern hardware than older hardware, so we start seeing, okay, wow, if I start seeing those breakdowns, what does that say about the margin on some of these products and services? And I don't think they want to give that level of transparency into their business, just because as soon as someone finds out just how profitable Managed NAT gateways are, my God, everything explodes.Aaron: I think it's interesting from a cloud provider or hyperscaler perspective, as well, is, you know, what is your USP? And I think Amazon is definitely not saying sustainability is their USP right now, and I think you know, there are other cloud providers, like Azure for example, who basically can provide you a Power BI plugin; if you just log in with your Cloud account details, it will show you a sustainability dashboard and give you more of this information that you might be looking for, whereas Amazon currently doesn't offer anything like that automated. And even having conversations with your account team or trying to get hold of the right person, Amazon isn't going to go anywhere at the moment, just because maybe that's the reason why we don't want to talk about it: It's too sensitive. I'm sure that'll change because of the public statements they've made at re:Invent now and previously of, you know, where they're going in terms of energy usage. They want to be carbon neutral by 2025, so maybe it'll change to next re:Invent, we'll get the AWS Sustainability Explorer add-on for [unintelligible 00:15:23] or 12—Corey: Oh no.Aaron: —tools to do the same thing [laugh].Corey: In the Google Cloud Console, you click around, and there are green leafs next to some services and some regions, and it's, on the one hand, okay, I appreciate the attention that is coming from. On the other hand, it feels like you're shaming me for putting things in a region that I've already built things out in when there weren't these green leafs here, and I don't know that I necessarily want to have that conversation with my entire team because we can't necessarily migrate at this point. And let's also be clear, here, I cannot fathom a scenario in which running your own data centers is ever going to be more climate-friendly than picking a hyperscaler.Aaron: And I think that's sort of, you know, we all might think about is, at the end of the day, if your sustainability strategy for your business is to go all-in-on cloud, and bet horse on AWS or another cloud provider, then, at the end of the day, that's going to be viable. I know, from the, sort of, hands-on stuff I've done with our own data centers, you can never get it as efficient as what some of these cloud providers are doing. And I mean, look at Microsoft. The fact that they're putting some of their data centers under the sea to use that as a cooling mechanism, and kind of all the interesting things that they're able to do because they can invest at scale, you're never going to be able to do that with the cupboard beyond the desks in your local office to make it more efficient or sustainable.Corey: There are definite parallels between Cloud economics and sustainability because as mentioned, I worship at the altar of Our Lady of Turn that Shit Off because that's important. If you don't have a workload running and it doesn't exist, it has no climate impact. Mostly. I'm sure there are corner cases. But that does lead to the question then of okay, what is the climate sustainability impact, for example, of storing a petabyte of data and EBS versus in S3?And that has architectural impact as well, and there's also questions of how often does it move because when you move it, Lord knows there is nothing more dear than the price of data transfer for data movement. And in order to answer those questions, they're going to start talking a lot more about their architecture. I believe that is why Peter DeSantis's keynote talked so much about—finally—the admission of what we sort of known for ages now that they use erasure coding to make S3 as durable yet inexpensive, as it is. That was super interesting. Without that disclosure, it would have been pretty clear as soon as they start publishing sustainability numbers around things like that.Aaron: And I think is really interesting, you know, when you look at your business and make decisions like that. I think the first thing to start with is do you need that data at all? What's a petabyte of data are going to do? Unless it's for serious compliance reasons for, you know, the sector or the business that you're doing, the rest of it is, you know, you've got to wonder how long is that relevant for. And you know, even as individuals, we could delete junk mail and take things off our internal emails, it's the same thing of businesses, what you're doing with this data.But it is interesting, when you look at some of the specific services, even just the tiering of S3, for example, put that into Glacier instead of keeping it on S3 general. And I think you've talked about this before, I think cost the same to transfer something in and out of Glacier as just to hold it for a month. So, at the end of the day, you've got to make these decisions in the right way, and you know, with the right goals in mind, and if you're not able to make these decisions or you need help, then that's where, you know, people like us come in to help you do this.Corey: There's also the idea of—when I was growing up, the thing they always told us about being responsible was, “Oh, turn out the lights when you're not in the room.” Great. Well, cloud economics starts to get in that direction, too. If you have a job that fires off once a day at two in the morning and it stops at four in the morning, you should not be running those instances the other 22 hours of the day. What's the deal here?And that becomes an interesting expiratory area just as far as starting to wonder, okay, so you're telling me that if I'm environmentally friendly, I'm also going to save money? Let's be clear people, in many cases—in a corporate sense—care about sustainability only insofar as that don't get yelled out about it. But when it comes to saving money, well, now you've got the power of self-interest working for you. And if you can dress them both up and do the exact same things and have two reasons to do it. That feels like it could in some respects, be an accelerator towards achieving both outcomes.Aaron: Definitely. I think, you know, at the end of the day, we all want to work on things that are going to hopefully make the world a better place. And if you use that as a way of motivating, not just yourself as a business, but the workforce and the people that you want to work for you, then that is a really great goal as well. And I think you just got to look at companies that are in this world and not doing very great things that maybe they end up paying more for engineers. I think I read an interesting article the other day about Facebook is basically offering almost double or 150 percent of over salaries because it feels like a black mark on the soul to work for that company. And if there is anything—maybe it's not greenwashing per se, but if you can just make your business a better place, then that could be something that you can hopefully attract other like-minded people with.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of, “Hello World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And let me be clear here, it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications, or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: One would really like to hope that the challenge, of course, is getting there in such a way that it, well, I guess makes sense, is probably the best way to frame it. These are still early days, and we don't know how things are going to wind up… I guess, it playing out. I have hopes, I have theories, but I just don't know.Aaron: I mean, even looking at Cloud as a concept, how long we've all worked with this now ranges probably from fifteen to five, and for me the last six years, but you got to think looking at the outages at the end of last year at Amazon, that [unintelligible 00:21:57], very close to re:Invent, that impacted a lot of different workloads, not just if you were hosted in us-west or east-1, but actually for a lot of the regional services that actually were [laugh]… discovered to be kind of integral to these regions. You know, one AZ going down can impact single-sign-on logins around the world. And let's see what Amazon looks like in ten years' time as well because it could be very different.Corey: Do you find that as you talk to folks, both in government and in private sector, that there is a legitimate interest in the sustainability story? Or is it the self-serving cynical perspective that I've painted?Aaron: I mean, a lot of my experience is biased towards the public sector, so I'll start with that. In terms of the public sector, over the last few years, especially in the UK, there's been a lot more focus on sustainability as part of your business cases and your project plans for when you're making new services or building new things. And one of the things they've recently asked every government department in the UK to do is come up with a sustainability strategy for their technology. And that's been something that a lot of people have been working on as part of something called the One Gov Cloud Strategy Working Groups—which in the UK, we do love an abbreviation, so [laugh] a bit of a long name—but I think there's definitely more of an interest in it.In terms of the private sector, I'm not too sure if that's something that people are prioritizing. A lot of the focus I kind of come across as either, we want to focus on enterprise customers, so we're going to offer migration professional services, or you're a new business and you're starting to go up and already spending a couple a hundred pounds, or thousands of pounds a month. And at that scale, it's probably not going to be something you need to worry about right now.Corey: I want to talk a little bit about how you got into tech in the first place because you told me elements of this story, and I generally find them to be—how do I put this?—they strain the bounds of credulity. So, how did you wind up in this ridiculous industry?Aaron: I mean, hoping as I explain them, you don't just think I'm a liar. I have got a Scouse accent, so you're probably predisposed towards it. But my journey into tech was quite weird, I guess, in the sense that when I was 16—I was, again, like I said, born in Liverpool and didn't really know what I wanted to do in the world, and had no idea what the hell to do. So, I was at college, and kind of what happened to me there is I joined, like, an entrepreneurship club and was like, “Okay, I'll start my own business and do something interesting.” And I went to a conference at college, and there was a panel with Richard Branson and other few of business leaders, and I stood up and asked the question said, you know, “I'm 16. I want to start a business. Where can I get money to start a business?”And the panel answered with kind of a couple of different things, but one of them was, “Get a job.” The other one was, “Get money off your parents.” And I was kind of like, “Oh, a bit weird. I've got a job already. You know, I would ask my parents put their own benefits.”And asked the woman with the microphone, “Can I say something back?” And she said, “No.” So, being… a young person, I guess, and just I stood back up and said, you know, “You're in Liverpool. You've kind of come to one of the poorest cities in some sense in the UK, and you kind of—I've already got a job. What can I really do?”And that's when Richard Branson turned round and said, “Well, what is it you want to do?” And I said, “I make really good cheesecakes and I want to sell them to people.” And after that sort of exchange, he said he'd give me the money. So, he gave me 200 pounds to start my own business. And that was just, kind of like, this whirlwind of what the hell's going on here?But for me, it's one of those moments in my life, which I think back on, and honestly, it's like one of these ten [left 00:25:15] moments of, you know, I didn't stand back up and say something, if I didn't join the entrepreneurship club, like, I just wouldn't be in the position I am right now. And it was also weird in the sense that I said at the start of the story, I didn't know what I wanted to do in my life. This was the first time that anyone had ever said to me, “I trust you to do something, and here's 200 pounds to do it.” And it was such a small thing, and a small moment that basically got me to where I am today. And kind of a condensed version of that is, you know, after that event, I started volunteering for a charity who—a, sort of, magazine launch, and then applied for the civil service and progressed through six to eight years of the civil service.And it was because of that moment, and that experience, and that confidence boost, where I was like, “Oh, I actually can do something with my life.” And I think tech, and I think a lot of people talk about this is, it can be a bit of a crazy whirlwind, and to go from that background into, you know, working with great people and earning great money is a bit of a crazy thing sometimes.Corey: Is there another path that you might have gone down instead and completely missed out on, for lack of a better term—and not missed out. You probably would have been far happier not working in tech; I know I would have been—but as far as trying to figure out, like, what does the road not taken look like for you?Aaron: I'm not too sure, really. And at the time, I was working in a club. I was like 16, 17 years old, working in a nightclub in Liverpool for five pounds an hour, and was doing that while I was studying, and that was almost like, what was in my mind at the time. When it came to the end of college, I was applying for universities, I got in on, like, a second backup course, and that was the only thing to do was food science. And it was like, I can't imagine coming out of university three years after that, studying something that's not really that relevant to a lot of industries, and trying to find a good job. It could have just been that I was working in a supermarket for minimum wage after I came out for uni trying to find what I wanted to do in the world. And, yeah, I'm really glad that I kind of ended up where I am now.Corey: As you take a look at what you want your career to be about in the broad sweep of things, what is it that drives you? What is it that makes you, for example, decide to spend the previous portion of career working in public service? That is a very, shall we say, atypical path—I say, as someone who lives in San Francisco and is surrounded by people who want to make the world a better place, but all those paths just coincidentally would result in them also becoming billionaires along the way.Aaron: I mean, it is interesting. You know, one of the things that worked for the civil service for so long, is the fact that I did want to do more than just make somebody else more money. And you know, there are not really a lot of ways you can do that and make a good wage for yourself. And I think early on in your career, working for somewhere like the civil service or federal government can be a little bit of that opportunity. And especially with some of the government's focus on tech these days, and investments—you know, I joined through an apprenticeship scheme and then progressed on to a digital leadership scheme, you know, they were guided schemes to help me become a better leader and improve my skills.And I think I would have probably not gone to the same position if I just got the tech job or my first engineering job somewhere else. I think, if I was to look at the future and where do I want to go, what do I care about? And, you know, you ask me, sort of, this question at re:Invent, and it took me a few days to really figure out, but one of the things when I talk about making the world a better place is thinking about how you can start businesses that give back to people in local areas, or kind of solve problems and kind of keep itself running a bit like a trust does, [laugh], if only that keeping rich people running. And a lot of the time, like, you've highlighted is coincidentally these things that we try and solve whether it's, like, a new app or a new thing that does something seems to either be making money for VCs, reinventing things that we already have, or just trying to make people billionaires rather than trying to make everyone rise up and—high tide rise all ships, is the saying. And there are a few people that do this, a few CEOs who take salaries the same as everyone else in the business. And I think that's hopefully you know, as I grow my own business and work on different things in the future, is how can I just help people live better lives?Corey: It's a big question, and it's odd in that I don't find that most people asking it tend to find themselves going toward government work so much as they do NGOs, and nonprofits, and things that are very focused on specific things.Aaron: And it can be frustrating in some sense is that, you know, you look at the landscape of NGOs, and charities, and go, “Why are they involved in solving this problem?” You know, one of the big problems we have in the UK is the use of food banks where people who don't have enough money, whether they receive benefits or not, have to go and get food which is donated just by people of the UK and people who donate to these charities. You know, at the end of the day, I'm really interested in government, and public sector work, and potentially one day, being a bit more involved in policy elements of that, is how can we solve these problems with broad brushstrokes, whether it's technology advancements, or kind of policy decisions? And one of the interesting things that I got close to a few times, but I don't think we've ever really solved is stuff like how can we use Agile to build policy?How can we iterate on what that policy might look like, get customers or citizens of countries involved in those conversations, and measure outcomes, and see whether it's successful afterwards. And a lot of the time, policies and decisions are just things that come out of politicians minds, and it'd be interesting to see how we can solve some of these problems in the world with stuff like Agile methodologies or tech practices.Corey: So, it's easy to sit and talk about these things in the grand sweep of how the world could be or how it should look, but for those of us who think in more, I guess, tactical terms, what's a good first step?Aaron: I think from my point of view, and you know, meeting so many people at re:Invent, and just have my eyes opened of these great conversations we can have a great people and get things changed, one of the things that I'm looking at starting next year is a podcast and a newsletter, around the use of public cloud for public good. And when I say that, it does cover elements of sustainability, but it is other stuff like how do we use Cloud to deliver things in the public sector and NGOs and charities? And I think having more conversations like that would be really interesting. Obviously, that's just the start of a conversation, and I'm sure when I speak to more people in the future, more opportunities and more things might come out of it. But I'd just love to speak to more people about stuff like this.Corey: I want to thank you for spending so much time to speak with me today about… well, the wide variety of things, and of course, spending as much time as you did chatting with me at re:Invent in person. If people want to learn more, where can they find you?Aaron: So yep, got a few social media handles on Twitter, I'm @AaronBoothUK. On LinkedIn is the same, forward slash aaronboothuk, and I've also got the website for my consultancy, which is embue.co.uk—E-M-B-U-E dot co dot uk. And for the newsletter, it's publicgood.cloud.Corey: And we will, of course, include links to that in the [show notes 00:32:11]. Thank you so much for taking the time to speak with me. I really do appreciate it.Aaron: Thank you so much for having me.Corey: Aaron Booth, cloud consultant with an emphasis on sustainability. I'm Cloud Economist Corey Quinn with an emphasis on optimizing bills. And this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that you will then kickstart the coal-burning generator under your desk to wind up posting.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Security Can Be More than Hues of Blue with Ell Marquez

Screaming in the Cloud

Play Episode Listen Later Jan 4, 2022 40:08


About EllEll, former SysAdmin, cloud builder, podcaster, and container advocate, has always been a security enthusiast. This enthusiasm and driven curiosity have helped her become an active member of the InfoSec community, leading her to explore the exciting world of Genetic Software Mapping at Intezer.Links: Intezer: https://www.intezer.com Twitter: https://twitter.com/Ell_o_Punk TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. If there's one thing we love doing in the world of cloud, it's forgetting security until the very end, going back and bolting it on as if we intended to do it that way all along. That's why AWS says security is job zero because they didn't want to remember all of their slides once they realized they forgot security. Here to talk with me about that today is Ell Marquez, security research advocate at Intezer. Ell, thank you for joining me.Ell: Of course.Corey: So, what does a security research advocate do, for lack of a better question, I suppose? Because honestly, you look at that, it's like, security research advocate, it seems, would advocate for doing security research. That seems like a good thing to do. I agree, but there's probably a bit more nuance to it, then I can pick up just by the [unintelligible 00:01:17] reading of the title.Ell: You know, we have all of these white papers that you end up getting, the pen test reports that are dropped on your desk that nobody ever gets to, they become low priority, my job is to actually advocate that you do something with the information that you get. And part of that just involves translating that into plain English, so anyone can go with it.Corey: I've got to say, if you want to give the secrets of the universe and make sure that no one ever reads them, make sure that it has a whole bunch of academic-style citations at the beginning, and ideally put it behind some academic paywall, and it feels like people will claim to have read it but never actually read the thing.Ell: Don't forget charts.Corey: Oh yes, with the charts. In varying shades of blue. Apparently that's the only color you're allowed to do some of these charts in; despite having a full universe of color palettes out there, we're just going to put it in varying shades of corporate blue and hope that people read it.Ell: Yep, that sounds about security there. [laugh].Corey: So, how much of, I guess, modern security research these days is coming out of academia versus coming out of industry?Ell: In my experience in, you know, research I've done in researching researchers, it all really revolves around actual practitioners these days, people who are on the front lines, you know, monitoring their honey pots, and actually reporting back on what they're seeing, not just theoretical.Corey: Which I guess brings us to the question of, I wind up watching all of the keynotes that all the big cloud providers put on and they simultaneously pat me on the head and tell me that their side of security is just fine with their shared responsibility model and the rest, whereas all of the breaches I'm ever going to deal with and the only way anyone can ever see my data is if I make a mistake in configuring something. And honestly, does that really sound like something I would do? Probably not, but let's face it, they claim that they are more or less infallible. How accurate is that?Ell: I wish that I could find the original person that said this, but I've heard it so many times. And it's actually the ‘cloud irresponsibility model.' We have this blind faith that if we're paying somebody for it, it's going to be done correctly. I think you may have seen this with billing. How many people are paying for redundant security services with a cloud provider?Corey: I've once—well, more than once have noticed that if you were to configure every AWS security service that they have and enable it in your account, that the resulting bill would be larger than the cost of the data breach it was preventing. So, on some level, there is a point at which it just becomes ridiculous and it's not necessarily worth pursuing further. I honestly used to think that the shared responsibility model story was a sales pitch, and then I grew ever more cynical. And now my position on it is that it's because if you get breached, it's your fault is what they're trying to say. But if you say it outright to someone who just got breached, they're probably not going to give you money anymore. So, you need to wrap that in this whole involved 45-minute presentation with slides, and charts, and images and the rest because people can't refute one of those quite the way that they can a—it's in a tweet sentence of, “It's your fault.”Ell: I kind of have to agree with them in the end that it is your fault. Like, the buck stops with you, regardless. You are the one that chose to trust that cloud provider was going to do everything because your security team might make a mistake, but the cloud provider is made up of humans as well who can make just as many mistakes. At the end of the day, I don't care what cloud provider you used; I care that my data was compromised.Corey: One of the things that irks me the most is when I read about a data breach from a vendor that I had either trusted knowingly with my data or worse, never trusted but they somehow scraped it somewhere and then lost it, and they said, “Oh, a third-party contractor that we hired.” It's, “Yeah, look, I'm doing business with you, ideally, not the people that you choose to do business with in turn. I didn't select that contractor. You did, you can pass out the work and delegate that. You cannot delegate the responsibility.” So no, Verizon, when you talk about having a third-party contractor have a data breach of customer data, you lost the data by not vetting your contractors appropriately.Ell: Let's go back in time to hopefully something everybody remembers: Target. Target being compromised because of their HVAC provider. Yet how many people—you know this is being recorded in the holiday season—are still shopping at Target right now? I don't know if people forget or they just don't care.Corey: A year later, their stock price was higher than it was before the breach. Sure they had a complete turnover of their C-suite at that point; their CSO and CEO were forced out as a result, but life went on. And they continue to remain a going concern despite quite literally having a bull's eye painted on the building. You'd think that would be a metaphor for security issues. But no, no, that is something they actually do.Ell: You know, when you talk about, you know, the CEO being let go or, you know, being run out, but what part did he honestly have to do with it? They're talking about, oh, well, they made the decisions and they were responsible. What because they got that, you know, list of just 8000 papers with the charts on it?Corey: As I take a look at a lot of the previous issues that we've seen with I've been doing my whole S3 Bucket Negligence Awards for a while, but once I actually had a bucket engraved and sent to a company years ago, the Pokémon Company, based upon a story that I read in the Wall Street Journal, how they declined to do business with a prospective vendor because going through their onboarding process, they noticed among other things, insufficient security controls around a whole bunch of things including S3 buckets, and it's holy crap, a company actually making a meaningful decision based upon security. And say what you will about the Pokémon Company, their audience is—at least theoretically—children and occasionally adults who believe they're children—great, not here to shame—but they understand that this is not something you can afford to be lax in and they kiboshed the entire deal. They didn't name the vendor, obviously, but that really took me aback. It was such a rarity to see that, and it's why I unfortunately haven't had to make a bucket like that since. I wish I did. I wish more companies did things like this. But no it's just a matter of, well, we claim to do the right thing, and we checked all the boxes and called it good, and oops, these things happen.Ell: Yes, but even when it goes that way, who actually remembers what happened, and did you ever follow up if there were any consequences to not going, “Okay, third-party. You screwed up, we're out. We're not using you.” I can't name a single time that happened.Corey: Over at The Duckbill Group, we have large enterprise customers. We have to be respectful and careful with their data, let's be very clear here. We have all of their AWS billing data going back for some fixed period of time. And it worries me what happens if that data gets breached. Now, sure, I've done the standard PR crisis comms thing, I have statements and actions prepared to go in the event that it happens, but I'm also taking great pains to make sure it doesn't.It's the idea of okay, let's make sure that we wind up keeping these things not just distinct from the outside world, but distinct from individual clients so we're not mixing and matching any of this stuff. It's one of those areas where if we wind up having a breach, it's not because we didn't follow the baseline building blocks of doing this right. It's something that goes far beyond what we would typically expect to see in an environment like this. This, of course, sets aside the fact that while a breach like that would be embarrassing, it isn't actually material to anyone's business. This is not to say that I'm not taking it seriously because we have contractual provisions that we will not disclose a lot of this stuff, but it does not mean the end of someone's business if this stuff were to go public in the same way that, for example, back when I worked at Grindr many years ago, in the event that someone's data had been leaked there, people could theoretically been killed. There's a spectrum of consequences here, but it still seems like you just do the basic block-and-tackling to make sure that this stuff isn't publicly exposed, then you start worrying about the more advanced stuff. But with all these breaches, it seems like people don't even do that.Ell: You have Tesla, right, who's working on going to Mars, sending people there who had their S3 buckets compromised. At that point, if we've got this technology, just giant there, I think we're safe to do that whole, “Hey, assume breach, assume compromise.” But when I say that, it drives me up the wall how many people just go, “Okay, well, there's nothing we can do. We should just assume that there's going to be an issue,” and just have this mentality where they give up. No, that gives you a starting point to work from, but that's not the way it's being seen.Corey: One of the things that I've started doing as I built up my new laptop recently has been all right, how do I work with this in such a way that I don't have credentials that are going to grant access to things in any long-lived way ever residing on disk? And so that meant with AWS, I started using SSO to log into a bunch of things. It goes through a website, and then it gives a token and the rest that lasts for 12 hours. Great.Okay, SSH keys, how do I handle that? Historically, I would have them encrypted with a passphrase, but then I found for Mac OS an app called Secretive that stores it in the Secure Enclave. I have to either type in a password or prove it with a biometric Touch ID nonsense every time something tries to access the key. It's slightly annoying when I'm checking out five or six Git repos at once, but it also means that nothing that I happen to have compromised in a browser or whatnot is going to be able to just grab the keys, send it off somewhere, and then I'll never realize that I've been compromised throughout. It's the idea of at least theoretically defense in depth because it's me, it's my personal electronics, in all likelihood, that are going to be compromised, more so than it is configured, locked-down S3 buckets, managed properly. And if not me, someone else in my company who has access to these things.Ell: I'm going to give you the best advice you're ever going to get, and people are going to go, “Duh,” but it's happening right now: Don't get complacent, don't get lazy, how many of us are, “Okay, we're just going to put the key over here for a second.” Or, “We're just going to do this for a minute,” and then we forget. I recently, you know, did some research into Emotet and—you know, the new virus and the group behind it—you know how they got caught? When they were raided, everything was in plain text. They forgot to use their VPN for a while, all the files that they'd gotten no encryption. These were the people that that's what they were looking for, but you get lazy.Corey: I've started treating at least the security credential side of doing weird things, even one off bash scripts, as if they were in production. I stuff the credentials into something like AWS's parameter store, and then just have a one line snippet of code that retrieves them at runtime to wind up retrieving those. Would it be easier to just slap it in there in the code? Absolutely, of course it would. But I also look at my newsletter production pipeline, and I count the number of DynamoDB tables that are in active use that are labeled Test or Dev, and I realized, huh, I'm actually kind of bad at taking something that was in Dev and getting it ready for production. Very often, I just throw a load at it and call it good. So, if I never get complacent around things like that, it's a lot harder for me to get yelled at for checking secrets into Git, for example.Ell: Probably not the first time that you've heard this but, Corey, I'm going to have to go with you're abnormal because that is not what we're seeing in a day-to-day production environment.Corey: Oh, of course not. And the reason I do this is because I was a grumpy old sysadmin for so long, and have gotten burned in so many weird ways of messing things up. And once it's in Git, it's eternal—we all know that—and I don't ever want to be in a scenario where I open-source something and surprise, surprise, come to find out in the first two days of doing something, I had something on disk. It's just better not to go down that path if at all possible.Ell: Being a former sysad as well, I must say, what you're able to do within your environment, your computer is almost impossible within a corporate environment. Because as a sysad, I'm looking at, “What did the devs do again? Oh, man, what's the security team going to do?” And you're stuck in the middle trying to figure out how to solve a problem and then manage it through that entire environment.Corey: I never really understood intrinsically the value of things like single-sign-on, until I wound up starting this company. Because first, it was just me for a few years. And yeah, I can manage my developer environments and my AWS environments in such a way that if they get compromised, it's not going to be through basic, “Oops, I forgot that's how computers work,” type of moment. It's going to be at least something a little bit more difficult, I would imagine. Because if you—all right, if you managed to wind up getting my keys and the passphrase, and in some cases, the MFA device, great, good, congratulations, you've done something novel and probably deserve the data.Whereas as soon as I started bringing other people in who themselves were engineers, I sort of still felt the same way. Okay, we're all responsible adults here, and by and large, since I wasn't working with junior people, that held true. And then I started bringing in people who did not come from a deeply computer-y technical background, doing things like finance, and doing things like sales, and doing things like marketing, all of which are themselves deeply technical in their own way, but data privacy and data security are not really something that aligns with that. So, it got into the weeds of, “How do I make sure that people are doing responsible things on their work computers like turning on disk encryption, and forcing a screensaver, and a password and the rest.” And forcing them to at least do some responsible things like having 1Password for everyone was great until I realized a couple people weren't even using it for something, and oh dear. It becomes a much more difficult problem at scale when you have to deal with people who, you know, have actual work to do rather than sitting around trying to defend the technology against any threat they can imagine.Ell: In what you just said though, there is one flaw is we tend to focus on, like you said, marketing and finance and all these organizations who—don't get phished, don't click on this link. But we kind of give the just the openness that your security team, your sysads, your developers, they're going to know best practices. And then we focus on Windows because that's what the researchers are doing. And then we focus on Windows because that's what marketing is using, that's what finance is using. So, what there's no way to compromise a Mac or Linux box? That's a huge, huge open area that you're allowing for attackers.Corey: Let's be very clear here. We don't have any Windows boxes—of which I'm aware—in the company. And yeah, the technical folk we have brought in, most of them I'd worked—or at least the early folks—I'd worked with previously. And we had a shared understanding of security. At least we all said the right things.But yeah, as you—right, as you grow, as you scale, this becomes a big deal. And it's, I also think there's something intrinsically flawed about a model where the entire instruction set is, it all falls on you to not click the link or you're going to doom us all. Maybe if someone can click a link and doom us all, the problem is not with them; it's the fact that we suck at building secure systems that respect defense in depth.Ell: Something that we do wrong, though, is we split it up. We have endpoint protection when we're talking about, you know, our Windows boxes, our Linux boxes, our Mac boxes. And then we have server-side and cloud security. Those connect. Think about, there's a piece of malware called EvilGNOME. You go in on a Linux box, you have access to my camera, keylogging, and watching exactly what I'm doing. I'm your sysad. I then cat out your SSH keys, I go into your box, they now have the password, but we don't look for that. We just assume that those two aren't really that connected, and if we monitor our network and we monitor these devices, we'll be fine. But we don't connect the two pieces.Corey: One thing that I did at a consulting client back in 2012, or so that really raised eyebrows whenever I told people about it was that we wound up going to some considerable trouble building a allow list within Squid—a proxy server that those of us in Linux-land are all too familiar with in some cases—so everything in production could only talk to the outside world via that proxy; it was not allowed to establish any outbound connections other than through that proxy. So, it was at that point only allowed to talk to specify update servers, specified third-party APIs and the rest, so at least in theory, I haven't checked back on them since, I don't imagine that the log4yay nonsense that we've seen recently would necessarily work there. I mean, sure, you have the arbitrary execution of code—that's bad—but reaching out to random endpoints on the internet would not have worked from within that environment. And I liked that model, but oh my God, was it a pain in the butt to set up properly because it turns out, even in 2012, just to update a Linux system reasonably, there's a fair number of things it needs to connect to, from time-to-time, once you have all the things like New Relic instrumentation in, and the app repository you're talking to, and whatever container source you're using, and, and, and. Then you wind up looking at challenges like, oh, I don't know, if you're looking at an AWS-style environment, like most modern things are, okay, we're only going to allow it to talk to AWS endpoints. Well, that's kind of the entire internet now. The goalposts move, the rules change, the game marches on.Ell: On an even simpler point, with that you're assuming only outbound traffic through those devices. Are they not connected to anything within the internal network? Is there no way for an attacker to pivot between systems? I pivot over to that, I get the information, and I make an outbound connection on something that's not configured that way.Corey: We had—you're allowed to talk outbound to the management subnet, which was on its own VLAN, and that could make established connections into other things, but nothing else was allowed to connect into that. There was some defense in depth and some thought put into this. I didn't come up with most of this to be clear, it was—this was smart people sitting around. And yeah, if I sit here and think about this for a while, of course there's going to be ways to do it. This was also back in the days of doing it in physical data centers, so you could have a pretty good idea of what was connect to the outside world just by looking at where the cables went. But there was also always the question of how does this–does this do what I think it's doing or what have I overlooked? Security's job is never done.Ell: Or what was misconfigured in the last update. It's an assumption that everything goes correctly.Corey: Oh, there is that. I want to talk though, about the things I had to worry about back then, it seems like in many cases get kicked upstairs to the cloud providers that we're using these days. But then we see things like Azurescape where security researchers were able to gain access to the Azure control plane where customers using Cosmos DB—Azure's managed database service, one of them—could suddenly have their data accessed by another customer. And Azure is doing its clam up thing and not talking about this publicly other than a brief disclosure, but how is this even possible from security architecture point of view? It makes me wonder if it hadn't been disclosed publicly by the researcher, would they have ever said something? Most assuredly not.Ell: I've worked with several researchers, in Intezer and outside of Intezer, and the amount of frustration that I see within reasonable disclosure, it just blows my mind. You have somebody threatening to sue the researcher if they bring it out. You have a company going, “Okay, well, we've only had six weeks. Give us three more weeks.” And next thing we know, it's six months.There is just this pushback about what we can actually bring out to the public on why they're vulnerable in organizations. So, we're put in this catch-22 as researchers. At what point is my responsibility to the public, and at what point is my responsibility to protect myself, to keep myself from getting sued personally, to keep my company from going down? How can we win when we have small research groups and these massive cloud providers?Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: For a while, I was relatively confident that we had things like Google's Project Zero, but then they started softening their disclosure timelines and the rest, and it was, we had the full disclosure security distribution list that has been shuttered to my understanding. Increasingly, it's become risky to—yourself—to wind up publishing something that has not been patched and blessed by the providers and the rest. For better or worse, I don't have those problems, just because I'm posting about funny implications of the bill. Yeah, worst case, AWS is temporarily embarrassed, and they can wind up giving credits to people who were affected and be mad at me for a while, but there's no lasting harm in the way that there is with well, people were just able to look at your data for six months, and that's our bad oops-a-doozy. Especially given the assertions that all of these providers have made to governments, to banks, to tax authorities, to all kinds of environments where security really, really matters.Ell: The last statistic that I heard, and it was earlier this year, that it takes over 200 days for compromise even to be detected. How long is it going to take for them to backtrack, figure out how it got in, have they already patched those systems and that vulnerability is gone, but they managed to establish persistence somehow, the layers that go into actually doing your digital forensics only delay the amount of time that any of that is going to come out where that they have some information to present to you. We keep going, “Oh, we found this vulnerability. We're working on patches. We have it fixed.” But does every single vendor already have it pitched? Do they know how it actually interacted within one customer's environment that allowed that breach to happen? It's just ridiculous to think that's actually occurring, and every company is now protected because that patch came out.Corey: As I take a look at how companies respond to these things, you're right, the number one concern most of them have is image control, if I'm being honest with you. It's the reputational management of we are still good at security, even though we've had a lapse here. Like, every breach notification starts out with, “Your security is important to us.” Well, clearly not that important because look at the email you had to send. And it's almost taken on aspects of a comedy piece where it [grips 00:23:10] with corporate insincerity. On some level, when you tell a company that they have a massive security vulnerability, their first questions are not about the data privacy; it's about how do we spend this to make ourselves come out of this with the least damage possible. And I understand it, but it's still crappy.Ell: Us tech folk talk to each other. When we have security and developers speaking to each other, we're a lot more honest than when we're talking to the public, right? We don't try to hold that PR umbrella over ourselves. I was recently on a panel speaking with developers, head SRE folk—what was there? I think there was a CISO on there—and one of the developers just honestly came out and said, “At the end, my job is to say, ‘How much is that breach going to cost, versus how much money will the company lose if I don't make that deployment?'” The first thing that you notice there is that whole how much money you'll lose? The second part is why is the developer the one looking at the breach?Corey: Yeah. The work flows downward. One of the most depressing aspects to me of the CISO role is that it seems like the job is to delegate everything, sign binding contracts in your name, and eventually get fired when there's a breach and your replacement comes in to sign different papers. All the work gets delegated, none of the responsibility does, ideally—unless you're SolarWinds and try and blame it on an intern; I mean, I wish I had an ablative intern or two around here to wind up a casting blame they don't deserve on them. But that's a separate argument—there is no responsibility-taking as I look at this. And that's really a depressing commentary on the state of the world.Ell: You say there's no responsibility taken, but there is a lot of blame assigned. I love the concept of post-mortems to why that breach happened, but the only people in the room are the security team because they had that much control over anything. Companies as a whole need a scapegoat, and more and more, security teams are being blamed for every single compromised as more and more responsibility, more and more privileges, and visibility into what's going on is being taken away from them. Those two just don't balance. And I think it's causing a lot of just complacency and almost giving up from our security teams.Corey: To be clear, when we talk about blameless post-mortems for things like this, I agree with it wholeheartedly within the walls of a company. However, externally as someone whose data has been taken in some of these breaches, oh, I absolutely blame the company. As I should, especially when it's something like well, we have inadvertently leaked your browsing history. Why were you collecting that in the first place? Is sort of the next logical question.I don't believe that my ISP needs that to serve me better. But now you have Verizon sending out emails recently—as of this recording—saying that unless anyone opts out, all the lines in our cell account are going to wind up being data mined effectively, so they can better target advertisements and understand us better. It's no, I absolutely do not want you to be doing that on my phone. Are you out of your mind? There are a few things in this world that we consider more private than our browsing histories. We ask the internet things we wouldn't ask our doctors in many cases, and that is no small thing as far as the level of trust that we place in our ISPs that they are now apparently playing fast and loose with.Ell: I'm going to take this step back because you do a lot of work with cloud providers. Do you think that we actually know what information is being collected about our companies and what we have configured internally and externally by the cloud provider?Corey: That's a good question. I've seen this before, where people will give me the PDF exploded view of last month's AWS bill, and they'll laugh because what information can I possibly get out of that. It just shows spend on services. But I could do that to start sketching out a pretty good idea of what their architecture looks like from that alone. There's an awful lot of value in the metadata.Now, I want to be clear, I do not believe on any provider—except possibly Azure because who knows at this point—that if you encrypt the data, using their encryption facilities—with AWS, I know it's KMS, for example—I do not believe that they can arbitrarily decrypt it and then scan for whatever it is they're looking for. I do not believe that they are doing that because as soon as something like that comes out, it puts the lie to a whole bunch of different audit attestations that they've made and brings the entire empire crumbling down. I don't think they're going to get any useful data from that. However, if I'm trying to build something like Amazon Prime Video, and I can just look at the bill from the Netflix account. Well, that tells me an awful lot about things that they might be doing internally; it's highly suggestive. Could that be used to give them an unfair advantage? Absolutely.I had a tweet a while back that I don't believe that Google's Gmail division is scanning inboxes for things that look like AWS invoices to target their sales teams, but I sure would feel better if they would assure me that was the case. No one was able to ever assure me of that. It's I don't mean to be sitting here slinging mud, but at the same time, it's given that when you don't explicitly say you're not doing something as a company, there's a great chance you might be doing it, that's the sort of stuff that worries me, it's a bunch of unfair dirty trick style stuff.Ell: Maybe I'm just cynical, or maybe I just focus on these topics too much, but after giving a presentation on cloud security, I had two groups, both, you know, from three letter government agencies, come up to me and say, “How do I have these conversations with the cloud provider?” In the conversation, they say, “We've contacted them several times; we want to look at this data; we want to see what they've collected, and we get ghosted, or we end up talking to attorneys. And despite over a year of communication, we've yet to be able to sit down with them.”Corey: Now, that's an interesting story. I would love to have someone come to me with that problem. I don't know how I would solve that yet. But I have a couple ideas.Ell: Hey, maybe they're listening, and they'll reach out to you. But—Corey: You know, if you're having that problem of trying to understand what your cloud provider is doing, please talk to me. I would love to go a little more in depth on that conversation, under an NDA or six.Ell: I was at a loss because the presentation that I was giving was literally about the compromise of managed service providers, whether that be an outsourced security group, whether that be your cloud provider, we're seeing attack groups going after these tar—think about how juicy they are. Why do I need to compromise your account or your company if I can compromise that managed service provider and have access to 15 companies?Corey: Oh, yeah. It's why would someone spend time trying to break into my NetApp when they could break into S3 and get access to everyone's data, theoretically? It's a centralization of security model risk.Ell: Yeah, it seems to so many people as just this crazy idea. It's so far out there. We don't need to worry about it. I mean, we've talked about how Azure Functions has been compromised. We talked about all of these cloud services that people are specifically going after and being able to make traction in these attacks.It's not just this crazy idea. It's something that's happening now, and with the progress that attackers are making, criminal groups are making, this is going to happen pretty soon.Corey: Sometimes when I'm out for a meal with someone who works with AWS in the security org, there'll be an appetizer where, “Oh, there's two of you. I'm going to bring three of them,” because I guess waitstaff love to watch people fight like that. And whenever I want the third one, all I have to do is say, “Can you imagine a day in which, just imagine hypothetically, IAM failed open and allowed every request to go through regardless of everything else?” Suddenly, they look sick, lose their appetite, and I get the third one. But it's at least reassuring to know that even the idea of that is that disgusting to them, and it's not the, “Oh, that happened three weeks ago, but don't tell anyone.” Like, there's none of that going on.I do believe that the people working on these systems at the cloud providers are doing amazingly good work. I believe they are doing far better than I would be able to do in trying to manage all those things myself, by a landslide. But nothing is ever perfect. And it makes me wonder that if and when there are vulnerabilities, as we've already seen—clearly—with Azure, how forthcoming and transparent would they really be? And that's the thing that keeps me up at night.Ell: I keep going back during this talk, but just the interaction with the people there and the crowd was just so eye-opening. And I don't want to be that person, but I keep getting to these moments of, “I told you so.” And I'm not going to go into SolarWinds. Lord, that has been covered, but shortly after that, we saw the same group going through and trying to—I'm not sure if they successfully did it, but they were targeting networks for cloud computing providers. How many companies focused outside of that compromise at that moment to see what it was going to build out to?Corey: That's the terrifying thing is if you can compromise a cloud service provider at this point, it's well, you could sell that exploit on the dark web to someone. Yeah, that is a—if you can get a remote code execution be able to look into any random Cloud account, there's almost no amount of money that is enough for something like that. You could think of the insider trading potential of just compromising Slack. A single company, but everyone talks about everything there, and Slack retains data in perpetuity. Think at the sheer M&A discussions you could come up with? Think of what you could figure out with a sort of a God's eye view of something like that, and then realize that they run on AWS, as do an awful lot of other companies. The damage would be incalculable.Ell: I am not an attacker, nor do I play one on TV, but let's just, kind of, build this out. If I was to compromise a cloud provider, the first thing I would do is lay low. I don't want them to know that I'm there. The next thing I would do is start getting into company environments and scanning them. That way I can see where the vulnerabilities are, I can compromise them that way, and not give out the fact that I came in through that cloud provider. Look, I'm just me sitting here. I'm not a nation state. I'm not somebody who is paid to do this from nine to five, I can only imagine what they would come up with.Corey: It really feels like this is no longer a concern just for those folks who manage have gotten on the bad side of some country's secret service. It seems like APTs, Advanced Persistent Threats, are now theoretically something almost anyone has to worry about.Ell: Let me just set the record straight right now on what I think we need to move away from: The whole APTs are nation states. Not anymore. And APT is anyone who has advanced tactics, anyone who's going to be persistent—because you know what, it's not that they're targeting you, it's that they know that they eventually can get in. And of course, they're a threat to you. When I was researching my work into Advanced Persistent Threats, we had a group named TNT that said, “Okay, you know what? We're done.”So, I contacted them and I said, “Here's what I'm presenting on you. Would you mind reviewing it and tell me if I'm right?” They came back and said, “You know what? We're not in APT because we target open Docker API ports. That's how easy it is.” So, these big attack groups are not even having to rely on advanced methods anymore. The line onto what that is just completely blurring.Corey: That's the scariest part to me is we take a look at this across the board. And the things I have to worry about are no longer things that are solely within my arena of control. They used to be, back when it was in my data center, but now increasingly, I have to extend trust to a whole bunch of different places. Because we're not building anything ourselves. We have all kinds of third-party dependencies, and we have to trust that they're doing the right things as they go, too, and making sure that they're bound so that the monitoring agent that I'm using can't compromise my entire environment. It's really a good time to be professionally paranoid.Ell: And who is actually responsible for all this? Did you know that 70% of the vulnerabilities on our systems right now are on the application level? Yet security teams have to protect it? That doesn't make sense to me at all. And yet, developers can pull in any third-party repository that they need in order to make that application work because hey, we're on a deadline. That function needs to come out.Corey: Ell, I want to thank you for taking the time to speak with me. If people want to learn more about how you see the world and what kind of security research you're advocating for, where can they find you?Ell: I live on Twitter to the point where I'm almost embarrassed to say, but you can find me at @Ell_o_Punk.Corey: Excellent. And we will wind up putting a link to that in the [show notes 00:35:37], as we always do. Thanks so much again for your time. I appreciate it.Ell: Always. I'd be happy to come again. [laugh].Corey: Ell Marquez, security research advocate at Intezer. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that ends in a link that begs me to click it that somehow it looks simultaneously suspicious and frightening.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Spreading the Networking Vibes with Serena (@shenetworks)

Screaming in the Cloud

Play Episode Listen Later Dec 30, 2021 38:43


About Serena Serena is a Network Engineer who specializes in Data Center Compute and Virtualization. She has degrees in Computer Information Systems with a concentration on networking and information security and is currently pursuing a master's in Data Center Systems Engineering. She is most known for her content on TikTok and Twitter as Shenetworks. Serena's content focuses on networking and security for beginners which has included popular videos on bug bounties, switch spoofing, VLAN hoping, and passing the Security+ certification in 24 hours.Links: TikTok: https://www.tiktok.com/@shenetworks Twitter: https://twitter.com/notshenetworks?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Once upon a time, I was a grumpy Unix systems administrator—because it's not like there's a second kind of Unix systems administrator—then I decided it was time to get better at the networking piece, so I got a CCNA one year. Did this make me a competent network engineer? Absolutely not. But it made me a slightly better systems person.My guest today is coming from the other side of the world, specifically someone who is, in fact, good at the networking things. Serena—or @SheNetworks as you might know her from TikTok or @notshenetworks from the Twitters—thank you for joining me, I appreciate your time.Serena: Yeah, thanks for inviting me on.Corey: So, at a very high level, you are a network engineer, and you specialize in data center compute and virtualization, which is fun because I remember doing a lot of that once upon a time before I went basically all in on Cloud consulting, and then sort of forgot that data centers existed. That's still a thing that's still going well, and there are computers out there that don't belong to what are the three biggest tech companies in the world?Serena: Yeah. Shockingly, there's still a ton of data centers out there, still a lot of private hosting, and a lot of the environments that we see are mixed environment; they will have some cloud, some on-prem. But yes, data centers are still relevant. [laugh].Corey: On some level, it feels like once you get into the world of cloud, you don't have to really think about networking anymore. You know, until there's a big outage, and suddenly everyone had think about the networks. But it also feels like it is abstractions piled upon abstractions in the cloud infrastructure space. How much of what happens in data centers these days maps to what happens in these hyperscaler provider environments?Serena: That's a good question. I think—so I have two CCNAs; I'm very familiar with networking, I'm very familiar with virtualization, and I went and got my AWS certification because as we're talking about a lot of cloud things happening now, it's big, it's good to know about it. And underlying infrastructure under the cloud is all the data centers that I work with, all the networking things that I work with. So, it maps very well to me. I thought I had, like, a really easy time studying for my AWS certification because a lot of the concepts just had, like, a different fancy name for AWS versus just what you know, as, like, NAT, or, you know, DNS, different things like that.Corey: Of course, NAT used to be a thing that was—everyone would yell at you, “It's not security,” even though there are—I would argue there are security elements tied into it. But honestly, that feels like one of the best ways to pick fights with people who are way better at this than I am. Nowadays, of course, I just view NAT through a lens of, “Yeah, I totally want to pay an extra four-and-a-half cents per gigabyte passing through a managed NAT gateway,” which remains, of course, my nemesis. The intersection of security, networking, and billing leads to basically just being very angry all the time.Serena: Yeah. You come into the field, like, so ready to go, and then sometimes you do get beat down. But it's worth it, I think. I really like what I do.Corey: And what you do is something of an anomaly because most people who focus on this world of data center networking and the security aspects thereof, and the virtualization stuff, are all—how do I put it politely?—old, grumpy and unpleasant. I mean, I guess I'm not going to put it politely because I'm just going to be honest with it. Because I'm one of those people, let's be clear here. Instead, you are creating a whole bunch of content on Twitter and on TikTok, where I've got to say that the union set in the Venn diagram between TikTok and deep-dive networking and cybersecurity is basically you. How did you get there?Serena: That's a really good question. To your first point, the, you know, old grumpy, kind of, stereotype, those are honestly some of my favorite people, truly, because I don't know what it is, but I just vibe with them in a work environment so well. And it's funny, you know, when I got my first job out of college, I was definitely the youngest person on my team by far. And we would all go out to lunch, I would mess with all of them, we'd all play pranks on each other. Just integrating into the teams was always super easy for me, which I'm really lucky that—not everybody has that experience, especially in their first job; things are a little rough.But it's always great. Like, I love the diversity in tech. And to your second point, how did I end up here, right, with this kind of intersection from this networking world to TikTok? People are always confused. Like, how did that happen? How are you finding followers on TikTok that are interested in networking?And I'm just as shocked honestly. [laugh]. I started making this content this time last year, and… you know, at first I was like, nobody wants to learn about DNS on TikTok. This is where people dance and play pranks and all this stuff.Corey: And if there's dancing when it comes to DNS, at some point, something has gone other hilarious or terrifyingly. That again, I use it as a database, so who am I to talk?Serena: [laugh]. Yeah, but it's been fun. I am shocked. But there's such a wide variety of people now using TikTok and it's growing so quickly. Early on in my TikTok career, I had messages and emails from people who are vice presidents at major Fortune 100 companies asking me, you know, if I'd be interested in working there or, you know, something like that, and I was just—I was so shocked because there was a company that was a Fortune 100, and one of their VPs joined one of my Lives, and was asking me questions, just about, like, my background career, and then they sent me a follow up email [laugh] to be like, “Hey.”So, I was like, “Did I just get interviewed on my Live on TikTok?” And that they always, like, cracked me up. And at that point, I knew I was like, okay, this is something different; like, this is interesting. Because, you know, at the end of the day, you see the views and the numbers and the followers, but you don't have, really, faces to put to them or names, and you don't really know where a lot of these people are from, so you don't know who's seeing it. And a lot of times, I think I made the assumption that they are younger kids. Which is true, but there are also a lot of very seasoned professionals that have been in this field for a very long time that also follow me, and comment on my videos, and add great input and things like that.Corey: There's a giant misunderstanding, I think across the industry, that the executives at the big serious companies, you know, the ones whose mottos may as well be, “That's not funny,” have no personality themselves as people and that they live their entire lives in this corporate bubble where they talk to their kids primarily via I don't know, Microsoft Teams, or WebEx, or something else equally sad. And in practice, that just doesn't work that way. They're human beings, too. And granted, you have to present in certain ways in certain rooms, but the idea that, oh, you're only going to reach developers with attitude problems by having a personality of being on modern platforms. I mean, it's an easy mistake to make.I know this because I spent years making it myself with the nonsense that I do until suddenly people are reaching out and it's, “Huh. You sure did use a lot of high-level strategic terms for a developer.” And you start digging into it, and it's like, “Oh, you're your chief operating officer to giant company. I bet your code is terrible.” Is it? It's like, “Yeah. Turns out, maybe I'm not looking at that through the right lens.” Meeting people where they are with engaging content is important, and I think that a lot of folks completely miss that bus.Serena: Yeah, I agree. And this is a small field, right, so it gets kind of nerve wracking sometimes because sometimes you say things and it's so easy to be like, this is how I joke with my friends. But I'm still somewhat in a professional capacity because of me associating with my career, right? And then when my videos reach a million, half-a-million views, when we think about how many people are actually in this field that would be interested in viewing that content, you realize, oh, wow. Like, this is a huge mixed bag of people, which does include very high level executives, all the way to people that are in high school that are just interested in learning more. So, it's definitely been interesting to figure that out along the way. [laugh]. But yeah, they will have regular personalities. They all like TikTok too. If they don't, they're lying. [laugh].Corey: I used to be very down on the whole TikTok thing, but I started experimenting with it. And yeah, it turns out I have a face for radio and, you know, the social graces for Twitter. So, it's not really my cup of tea, but I enjoy watching it. I found that I'm not really a video person, but something about the TikTok format means I'm just going to start scrolling. And oh, dear, it's been six hours and my phone battery died. Thank God, or I'd still be there. There's something very captivating about it and I really like the format.The problem I always had with looking at a lot of the deeply technical content out there is so many companies are out there producing this and selling this. And that's fine. Like, money is not the end all, be all [of this 00:09:40]. I'm about to spend weeks of my life on something, the fact that it cost me 30 or 50 bucks or whatnot is really not economic thing I should be concerning myself with. But it all feels like it's classroom stuff. It's if you give people an option, are you going to go to a college lecture or are you going to go to a comedy show? Does the idea of, I want to be entertained. If you can teach me something while entertaining me, that feels like the winning combination, and you've absolutely nailed that.Serena: I think a lot of these companies that are producing content, hold themselves back a lot. And that is why they're not successful, right? Because there's so many stipulations, and there's teams of people, and boardrooms of approvals, and all these things, and me, all I'm doing—I record all my TikToks on my iPhone, and I just use in-app editing. I spend a lot of time kind of researching, right, maybe I will experiment with different formats, but the best format that's worked for me is just being authentic, kind of, not having that corporate vibe, right? And also not really expecting anything in return.So, a lot of times, corporations are putting out content because they obviously want to drive traffic to their websites, and different things like that, but the companies that do the best are the ones that are just putting out content for free, and really not necessarily expecting anything in return. And they also give themselves so much more leeway into the type of content that they create because they're not thinking about the numbers at the end of it, right? You just got to put stuff out there and people will see it. For me, I just put stuff out there, I don't need to wait for someone to approve my TikTok for me to push it out and have this content there. So, that is a big difference.And I've learned that through working with sponsors where they'll send you a giant list of talking points they want you to say and I'm like, “You guys know this is a 60-second video, right?” It needs to be really small. You need to, like, really learn how to get the really important stuff out there because the rest of the smaller stuff doesn't matter as much. Like, sell them on one big thing, and that really makes a difference.Corey: Oh, very much so. I see that sometimes with this show where people will reach out and ask about sponsoring, and they'll want to have a URL that I read into the microphone, and it's with UTM tracking parameters and the rest. And it's, like, “I appreciate where you're coming from and your intention here, however, that is not generally how this format works, so let's talk about this and the outcome.” And again, it's a brave new world out there. Yeah, if you're used to buying display ads in various places, that is exactly what you do.For some reason, there's this corporate mentality toward we're going to spend $25 million on a billboard saturation campaign, and not really give any thought about what we're actually going to say now that we have all of that visual real estate to get people's attention with. It's, there's not enough focus on the message itself, and I think that is a giant lost opportunity. Enterprise marketing doesn't have to be boring, it can be a lot of fun.Serena: I agree. And I think podcasting was the last, probably, big area that people budgeted for marketing, right? So, you have your traditional TV commercials and there was YouTube, and—you know, TV commercials, billboards, newspapers, then there's YouTube, and then podcasts, I would say, probably came a little bit later, as far as these companies look at for marketing potential. And now TikTok is so new and a lot of these marketing companies have no idea how to be successful on it because it's just so different. It's Gen Z, the humor is different.It's kind of like [laugh] the wild west on social media where things are just, like, crazy, and you have to fight the algorithm because on TikTok it's, if you don't like it, you just scroll within three seconds. The attention span is so short. So, you really have to capture people's attention within those first three seconds. Versus a podcast, you have the whole, let's say, first 20 minutes to get people, kind of, interested before you can be like, oh, hey, and here's my sponsor. So, it's very different versus TikTok, they'll just, like, oh, scroll. So, [laugh] you have to get creative and think differently.Corey: Many moons ago, when I was getting my CCNA, I worked at a company where we wound up getting a core switches for the data center, which was at the time, something like 65 grand. Great. And then we rented—because we had configured it in our office—and then a couple of us had to rent a commercial van, which I think ran something like $30,000 itself to transport this thing 20 miles to the data center, and I'm sitting there going, like, “Wow, the switch is worth way more than the van that's sitting within. Also were really shitty movers and that doesn't seem like the best idea for anything.” But I just think they remember that, and it left an impression on me.What I like about cloud with what I do is I can take a credit card and then spend less than $10 on AWS—or theoretically, Azure, or Google Cloud or, you know, $2 million on IBM because oops-a-doozy, but fine—and I wind up coming out the other side of that with having done some interesting disaster stuff. You are teaching people about how this stuff works, but in a data center world, it seems to me that the startup costs of, “Oh, I'm going to buy this random router or switch to wind up doing some demonstration stuff for,” it feels like the startup costs of getting hands on that equipment would be out of reach for an awful lot of people. Am I just completely out of touch with how that world works?Serena: No, you're right, you're one hundred percent, right. It is difficult. So, in college, my undergraduate degree is computer information systems, and they had a Cisco Networking Academy. And so we had old switches, old layer 3 switches, and then we had some routers, and this is all stuff that was EOL, donated equipment, right? And this is going to—Corey: It breaks down you're bidding against very faraway places with no budget on eBay for replacements. Oh, yes.Serena: Yeah, exactly. And it was a lot of IOS stuff, right? And so when I was in college, I had no idea that NX-OS existed, which is the data center Nexus version operating system for their switches and things. And so when I got to my first job and saw NX-OS, I was like, “Oh, crap, [laugh] like, what is this?” Right?Because I honestly didn't even know. I graduated and did not know that existed. And I didn't know a lot of the stuff that I was working on at my first shop existed. And I really had to rely on, kind of, the fundamentals. And they are transferable, right? That's why it's good to kind of get into—like, I know what these routing protocols are. I know, layer 2, I know this cabling, so let me just learn these command differences and things like that.And once you get into a production environment in general, out of a lab, it hits the fan. Like, everything you feel like you've learned is gone almost because there's so many layers and now all of a sudden, you have these firewalls, when before you were just trying to get, like, your routing neighborships to establish [laugh] and you weren't worried about rules on a firewall somewhere. And [crosstalk 00:16:39]—Corey: “Oh, and by the way, in this environment, that link that you're working on goes down, every minute it's down, here is the number of commas in the amount of money that we're losing, and yes, that's a plural.” It's, “Okay, so I guess I'm going to double-check everything I run first.” Yeah, it's that caution that gives people a bit of credence there. [unintelligible 00:16:58] do these things in a, more or less, cowboy style in these environments, at least not for very long. Because you can break individual servers; that's fine, but if you break the network suddenly, you may as well not have the computers.Serena: Yeah. It can be paralyzing, truly. It can be very overwhelming your first networking job. Especially for me, I was just dealing with outages constantly because I worked for a vendor, and I was [laugh] like, I was just scared, you know? Because I would get these cases and it would be a hospital outage.And I'm like, “I just graduated college. Like, what do you want from me?” You know, and back to your original point, it is difficult in a data center space because the equipment's so expensive. So, a lot of people ask, “Do you have a home lab?” And one—there's a couple of reasons I don't really have a significant home lab. One, I move so much.Corey: Oh, and in the spare room basically is always 90 degrees and sounds like a jet engine taking off.Serena: Yeah.Corey: Yeah, it's one of those, I should probably find a different place where I don't live, to have that equipment. Yeah.Serena: Yeah. And I have access, like, remotely to all the lab equipment that I really need. So, I don't personally have one, but a lot of things that I do work with are so expensive, that I'm like, I can't afford to put this data center equipment in my house. That doesn't make any sense.And there is luckily now a lot of virtual labs that you can do. There's some sandboxes by Cisco and other vendors, where you can kind of get a little bit of hands-on experience. A lot of it relates to their certifications. You can rent racks, but that gets pretty pricey, too. So, it is difficult, and sometimes that's why a lot of these jobs, I think I have a lot of people who are looking for entry-level work, and it's hard to get into a specifically a data center space.And aside from racking, stacking, working in a data center—maybe a NOC—if you want to get into the actual,s I'm configuring Nexus switches, I'm configuring, you know, Palo Alto firewalls, it can be difficult because it's hard to get to that point, there's not a clear path.Corey: What is the entry path these days? I entered tech by working on a help desk, and those aren't really the jobs that they once were, in a lot of different ways. So, I've stopped talking to entry-level folks with the position of, “Oh, yeah, this is what you should do because that's what I did.” It turns into, like, “Okay, Boomer. Great job. Tell me a little bit more, though, about what the Great War was like, first.” No, we aren't going to go down that path. It's just I don't know what the entry-level point is for someone who's legitimately interested in these things these days.Serena: Nobody does. It's crazy. And you're right at the, “Okay, Boomer,” thing. See, networking was one of those… things that just got pushed onto people in, just, a general IT department, right? So, that's when everything was like, “Okay, we need to get on the internet, so, you know, hey, you handle some of the computer stuff. It's your job now. Good luck. Figure it out.”And so, people started doing that and they kind of just got pushed into it, and then as the internet grew, as our capabilities grew, then the job became, like, a little bit more specialized. And now we have, you know, dedicated network engineers, we have people running data centers. But that's not necessarily a viable path now for people just because there's so much to it now. There's cloud, there's security risks, there's data center, wireless, pho—I mean, you can be an engineer just for phones, right? So, it's a little bit difficult for, especially, the younger people coming in, and the people that I talk to, and figuring out, well, how do I get to what you're doing?And the way that I did is I went and got a four-year degree and then joined a new college graduate program at a Fortune 100 company. Which is a great path, I highly recommend it to anybody that can do it, but it's also not available for everybody, right, because not everybody has the means to get a four-year education, nor do you necessarily need one to do what I do. So, everybody's kind of has this different path, and it's very confusing for people who are aspiring network engineers, or aspiring cloud engineers, even.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: The narrative the cloud companies have been pushing for a while—like, and I'm in that space deeply enough that I haven't really thought to go super deep into questioning this—is that well, the future is all cloud, the data center is basically this legacy thing that the tide is slowly eroding, in the fullness of time, because everything will one day be cloud. Do you think that's accurate?Serena: I don't. I really don't think that's accurate. Don't get me wrong, I think that the cloud is here to stay, and a lot of people are going to be using it. And it's going to be—and it currently is a huge part of our lives. Like, as we've seen recently with a few of the AWS outages, when it goes down and goes down hard because everything's so centralized.And people like to think, like, oh, you know, we have all this redundancy, yadda, yadda. That has not protected us so far, [laugh] like, from these major outages, right? And a lot of places that I see—especially when you're looking at public sector—is a hybrid, where you do have data center on-prem and you have cloud. And I think that, personally, is the best way to go. Unless, you know, maybe you're a fast growing startup and AWS or Azure makes a lot of sense to you.And it does. There's great use cases for that, right? But they're—not only aside from the whole cloud shift, there's another shift of, you know, making our data centers eco-friendly, too, and workload optimization. So, maybe the price point that you're looking for, what's going to save your business the most money, is doing that hybrid. So, I'm going to store a lot of my private documents on site, I'm going to have this as a backup disaster recovery, but we're also going to operate in the cloud. I don't think that the data centers as we know them are going to go extinct. [laugh]. I think they will be around.Corey: Well, AWS finally made their Outpost—the smaller ones; read as servers that run AWS services on in your facility—available a year after announcing them. And I looked at it like, oh, wow, these things are 600 bucks a month. Which is not nothing, but certainly something I could afford to wind up exploring and doing some content. But okay, first, it's a three-year commitment. So, that's 20 grand or so. Okay, not ideal, but fine.That would effectively almost double my AWS bill, but that's not the hardest part because, oh, and to get one of these, you have to have enterprise support. And when I pointed this out to some Amazonian friends, their response was, “Well, what's the problem on this?” Yeah, enterprise support starts at $15,000 a month minimum, and that means that people aren't going to pick these up to do proof of concept work. They're going to do it when they already have a significant infrastructure out there, and I think that's leaving an awful lot of money on the table by making people jump through sales hoops, and getting proof of concept credits, and doing all the other stuff for this. It's just ship me a box for a few weeks and let me kick the tires on in my environment and see if it works or doesn't work.Worst case, I'll ship it back to you. Worst, worst case, I lose the thing, and then you charge me whatever it costs to replace this. But it still feels like they are really doing the whole, “Oh, it's only big legacy companies that have on-premises stuff.” I don't like that narrative.Serena: I don't either. And I honestly think it's a bad idea, right, because if you do put all of your eggs in the AWS basket and they have all the power, that's not going to give us a lot of bargaining, right? That's not going to give people a lot of—because they'll know. They know how hard it is to get off of AWS at that point: They know it's costly, it takes manpower, it takes knowledge, right? And I think that it is in people's best interest to kind of have that mixed environment. Just for long-term, I'm just very wary of centralizing everything in one area. I think it's a bad idea. [laugh]. I think that we need to be prepared for ourselves, and that means also relying a little bit on ourselves. We can't just, in my opinion, put everything in the AWS basket. [laugh].Corey: Not very long anyway. It just doesn't seem to work.Serena: Right. And it's a great product.Corey: Oh, it absolutely is, but—Serena: There's so many positive things about using cloud. Because I'm not the type of person that likes to, kind of, talk crap about any vendor. I think everybody has their pros, cons, flaws, whatever. It's really about what works best for your environment, and that's part of being a network engineer or an architect is evaluating your environment and figuring out what is going to be the best for you, right? There's no one size fits all, unfortunately.Corey: Yeah. And AWS is uniformly excellent, let's be very clear. Okay, not—maybe not uniformly. Some services are significantly better than others, but I have an opinion piece in the information—paywalled, unfortunately, but I'm working on i—the general thesis that AWS has gotten too big to fail, in that when it's not—like, first, they are going to have better uptime than you or I will running our own data centers, across the board.They are very good at keeping things up, but when they do go down, it's not just your company or my company anymore having an outage, it is a significant portion of, you know, the global economy, and that is an awful lot of systemic concentrated risk. I'm not suggesting they did anything wrong, as far as how they sold these things—though, some people will want to argue with that—but it's the, “What does this mean?” Are we ready to reckon with that as a society that whenever us-east-1 has a bad day, so does the stock market? Is that something we're really prepared to accept or wrangle with? Or worse than that, there are life-critical services now. Does that mean that we're going to accept there is some number of people who will die when there's an outage of a data center? And that's new territory for me. I have not worked in environments where it was life or death consequential. At least not directly.Serena: Yeah, I have. So, I have definitely worked in those environments, right, and it's very scary, and especially when it's outside of your control. So, if you are relying, or just waiting on AWS to get back up, you don't have the control to get in there and start fixing things yourself, which is my instinct, right? Like, I immediately want to get hands-on. I put my troubleshooting hat on, like, let's figure this out, let me look through logs, let me do this.And you don't have that option with AWS when it's a significant outage that's impacting multiple people, it's not some configuration internally to you, right?And that's scary. It's a scary place to be. And I think that we need to really consider the cascading effects that will happen, which a lot of these outages that are kind of starting to show us, right? And luckily, there hasn't been anything major catastrophic, but we do need to really consider life when we're talking about, you know, hospitals, 911 systems, all of these critical infrastructures that are going to be cloud managed, and out of our control, and centralized.So, you know, you lose one 911 system, okay, well, you can do a backup, right? You may be able to route all your calls to the city over because their 911 systems are up and running. Well, what if there's are out now, too, because you're both hosted on AWS?Corey: Or you're, “Ah, we're going to diversify and we're going to have this other one on a different cloud provider.” That's great, but there's a critical third-party dependency that's right back to the thing you're trying to avoid. And there you go again.Serena: Yep. And that's dependency hell, right? [laugh].Corey: Oh, yeah. And I don't know how we get away from that.Serena: Yeah.Corey: Like, we don't want everyone writing all their own stuff from scratch, like starting with assembly, move up the stack. But here we are.Serena: Right. And it's funny because these AWS outages specifically effects—or cloud outages, right? I feel like I'm picking on them. I'm not trying to—sorry, AWS, but [laugh] don't come for me.But you know, explaining to my mom, why her Ring doorbell is not working and her Roomba stopped working when that outage happened, right, she's like, “Why is this not—it won't connect.” Like, “I don't understand.” She's like, “What's AWS?” And then to tell my mom that the company that she buys her socks from, like, that she goes online and, like, buys on Amazon is the company that also is hosting her Roomba, you know, services, her Ring services, it's so interesting to have those conversations. And a lot of people who aren't in our field don't understand that. They don't understand cloud, they don't understand on-prem versus, you know, hosted by a third-party. So, it's interesting to watch that kind of unfold now because it's very new. It's very new territory.Corey: And one last question before we wind up calling it an episode. It is remarkably clear in talking to you that you are in no way, shape, or form, junior. You are not a beginner. You know exactly how this stuff works in significant depth. Your content that you put out is aimed at beginners. I do something very similar. So, to be very clear, this is not a criticism in the slightest, but I am curious as to why that's the direction you went in.Serena: I think there's a few reasons. Well, I might have this knowledge, right? I still consider myself very junior in my career, very early in my career. There's so many things that I don't know and I recognize that. When you're first starting out, you might have this kind of inflated sense of knowledge where you're like—like, me, I was like, “Oh, yeah. I know all about OSPF and running on IOS and the command line,” until I figured out there was an NX-OS and I'm like, “Oh crap, what else do I not know about?” Right? [laugh].Corey: Oh, by the way, that never goes away. I feel exactly the same way 20 years into my career, now. I still have absolutely no idea what I'm doing. So smile, nod, and get used to it is the only insight I've got there. But please, go on.Serena: And even on Twitter sometimes, I'm reading people's stuff, and I'm like, “How did you get into these obscure protocols and all these things?” And, you know, I just kind of dive deeper into there. But I think the big reason that I create a lot of my content for beginners is because I remember so well how it was at the beginning, learning about subnetting, and that IOS—[laugh]—[unintelligible 00:30:52] learning about subnetting, and all of the different models that we have, right? And I was overwhelmed, and I was stressed out, and it just seems so… just, like, a giant mountain to climb. It seems so daunting in the beginning, for me it did because there's so much, right?And it felt like everybody was so far ahead of me. And I don't want other people to really feel like that. Like, I don't want people to be turned off from networking because they feel like the bar is too high, that we're not letting enough new people enter because we're discouraging them from the beginning by saying, “Oh, well, you're going to have to know all this. And let me throw this certification book at you.” And they're big. Like, my certification books—and these are massive. And this is for one half of the CCNA.Corey: For those who aren't, like, on the video call—it's not being recorded video-wise—she's holding a book that you could use to kill a mid-sized dog by accident if it falls off a table. It looks like a phonebook with a hardcover on it.Serena: Yeah. [laugh]. It's huge, right? And there are thousands of pages, and we just give this to somebody and say, like, “Here you go. Make sure you remember all this.” And this is all new information.Corey: And does it still cover things like EIGRP? Like Cisco's proprietary routing protocols that I've never once seen in the wild?Serena: Yeah. So, sometimes you will have to learn that, and they've changed it recently, too. They update their certification exam. So, you will learn about some legacy protocols because sometimes you do run into them.Corey: Oh, yes. That's when I have the good sense to pay professionals who know what they're doing.Serena: [laugh]. Yeah. Exactly. So yeah, you do run into those sometimes. But it feels so daunting for new people, and I totally recognize that. And by nature of TikTok I, especially when I first start making content, I assume that most of the people on there are going to be people who are younger, who are interested in this career.And as you know, in tech in general, especially networking, security, cloud, there's a massive shortage of people, and how are we solving that, right? And my contribution to helping solve that is by getting people interested. And now I have people that DM me and say, “I passed my [Network+ 00:33:01],” or, “I just took the CCNA,” or, “This has been helping me with my class so much.” And that is like, okay, this is great.Like, that's exactly what I want. I want to help the pipeline, I want to get more people interested and help a diverse group of people get interested in tech and say, “Hey, like, this is, you know, where I came from. And I did it; you can do it; let's do it together,” type situation.Corey: I really want to thank you for being so generous with your time. If people want to learn more, as they absolutely should, where can they find you?Serena: I am on TikTok as @SheNetworks. I am on Twitter as @notshenetworks because somebody else—Corey: That is very confusing.Serena: [laugh]. I know. Well, my initial thing was like, I didn't really use Twitter that much, and I would just like—I kind of used it as, like, a backchannel to my TikTok, right, where I would just, like, “Hey, I'm going to go live,” or do this. And then my Twitter, kind of, got a little out of control [laugh] and out of my hands. And so—Corey: It does that sometimes.Serena: Yeah. I had no idea there would be so much interest. And it surprises me every day. So, it's exciting though. I really love all the people that I've met, and I feel like I fit in, and I've met so many good friends that it's been great. But yeah, so @notshenetworks on Twitter because somebody had shenetworks and it was a joke. And [laugh] so if you want to find me there, you could also find me there.Corey: And we will, of course, put links to that in the [show notes 00:34:20]. Thank you so much for taking the time to speak with me today. I really do appreciate it.Serena: Thank you for having me. This has been great. [laugh].Corey: Serena, also known as @SheNetworks, networking content creator to the stars. I'm cloud economist, Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and then a long, angry, rambling comment about how the network isn't that important that you're then not going to be able to submit because the network isn't working.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

AWS Morning Brief
Self-Disclosure Heals Many Wounds

AWS Morning Brief

Play Episode Listen Later Dec 30, 2021 6:01


Links: “Cloud Security Breaches and Vulnerabilities”: https://blog.christophetd.fr/cloud-security-breaches-and-vulnerabilities-2021-in-review/ S3 Bucket Negligence Award: https://mytechdecisions.com/audio/sennheiser-responds-after-customer-data-from-2018-was-exposed-online/ Granted the role its support teams use to access customer accounts access to S3 objects: https://Twitter.com/0xdabbad00/status/1473448889948598275?s=12 S3 Bucket Negligence Award: https://www.modernghana.com/news/1127205/report-ghana-government-agency-exposes-100000s.html “Simplify setup of Amazon Detective with AWS Organizations”: https://aws.amazon.com/blogs/security/simplify-setup-of-amazon-detective-with-aws-organizations/ “AWSSupportServiceRolePolicy Informational Update”: https://aws.amazon.com/security/security-bulletins/AWS-2021-007/ aws-sso-cli: https://github.com/synfinatic/aws-sso-cli TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it's nobody in particular's job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: Are you building cloud applications with a distributed team? Check out Teleport, an open-source identity-aware access proxy for cloud resources. Teleport provides secure access for anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps, and databases. Teleport gives engineers superpowers. Get access to everything via single sign-on with multi-factor, list and see all of SSH servers, Kubernetes clusters, or databases available to you in one place, and get instant access to them using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility, and ensuring compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That's goteleport.com.Corey: Well, we're certainly ending 2021 with a whirlwind in the security space. Log4J continues to haunt us, while AWS took not only an outage but also a bit of a security blunder that they managed to turn into a messaging win. Listen on.But first, the Community. A depressing review of 2021's “Cloud Security Breaches and Vulnerabilities.” Honestly, it seems like there are just so damned many ways for bad security to set the things we care about on fire. The takeaways are actionable though. Stop using static long-lived credentials and start with the basics before you get fancy.Sennheiser scores itself an S3 Bucket Negligence Award, and of all the countries in which to suffer a data breach, I've got to say that Germany is at the bottom of the list. They do not mess around with data protection there.And, Holy hell, AWS inadvertently granted the role its support teams use to access customer accounts access to S3 objects. It lasted for ten hours, and while there are mitigations out there, this is far from the first time that AWS has biffed it with regard to an unreviewed change making it into a managed IAM policy. This needs to be addressed. If you've got specific questions about how those things are handled, reach out to your account team; but it's a terrible look. But there's more to come in a second here.Corey: This episode is sponsored in part by my friends at Cloud Academy. Something special for you folks: If you missed their offer on Black Friday or Cyber Monday or whatever day of the week doing sales it is, good news, they've opened up their Black Friday promotion for a very limited time. Same deal: $100 off a yearly plan, 249 bucks a year for the highest quality cloud and tech skills content. Nobody else is going to get this, and you have to act now because they have assured me this is not going to last for much longer. Go to cloudacademy.com, hit the ‘Start Free Trial' button on the homepage and use the promo code, ‘CLOUD' when checking out. That's C-L-O-U-D. Like loud—what I am—with a C in front of it. They've got a free trial, too, so you'll get seven days to try it out to make sure it really is a good fit. You've got nothing to lose except your ignorance about cloud. My thanks to Cloud Academy once again for sponsoring my ridiculous nonsense.A bit off the beaten path, this week's S3 Bucket Negligence Award goes to the government of Ghana. This one is pretty bad. I mean, you can't exactly opt out of doing business with your government, you know?Now, AWS has two things I want to talk about. The first is that they offer a way to “Simplify setup of Amazon Detective with AWS Organizations.” I'm actually enthusiastic about this one because there's a significant lack of security tooling available to folks at the lower end of the market. A bunch of companies seem to start off targeting this segment, but soon realize that there's a better future in selling things to bigger companies for $200,000 a month instead of $20.Now, “AWSSupportServiceRolePolicy Informational Update.” Now, you heard a minute ago, I was initially extremely unhappy about this mistake. That said, I am such a fan of this notification that I can't even articulate it without sounding like I'm fanboying. Because mistakes happen and talking about those mistakes and why defense in depth mitigates the harm of those mistakes goes a long way. This affirms my trust in AWS rather than harming it. Meanwhile Azure has absolutely nothing to say about why their tenant separation is aspirational at best.And lastly a bit of tooling story here. To end up the year, I've been kicking the tires on aws-sso-cli over on GitHub, which is a tool for using AWS SSO for both the CLI and web console. I don't know why the native SSO tooling is quite as trash as it is, but it's a problem. There's a lot of value to using SSO but AWS hides it as if the entire thing were under NDA. Thank you for listening. It's been a heck of a year as we've launched the security portion of this weekly nonsense. I'll talk to you more in 2022. Stay safe.Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Breaching the Coding Gates with Anil Dash

Screaming in the Cloud

Play Episode Listen Later Dec 29, 2021 39:03


About AnilAnil Dash is the CEO of Glitch, the friendly developer community where coders collaborate to create and share millions of web apps. He is a recognized advocate for more ethical tech through his work as an entrepreneur and writer. He serves as a board member for organizations like the Electronic Frontier Foundation, the leading nonprofit defending digital privacy and expression, Data & Society Research Institute, which researches the cutting edge of tech's impact on society, and The Markup, the nonprofit investigative newsroom that pushes for tech accountability. Dash was an advisor to the Obama White House's Office of Digital Strategy, served for a decade on the board of Stack Overflow, the world's largest community for coders, and today advises key startups and non-profits including the Lower East Side Girls Club, Medium, The Human Utility, DonorsChoose and Project Include.As a writer and artist, Dash has been a contributing editor and monthly columnist for Wired, written for publications like The Atlantic and Businessweek, co-created one of the first implementations of the blockchain technology now known as NFTs, had his works exhibited in the New Museum of Contemporary Art, and collaborated with Hamilton creator Lin-Manuel Miranda on one of the most popular Spotify playlists of 2018. Dash has also been a keynote speaker and guest in a broad range of media ranging from the Obama Foundation Summit to SXSW to Desus and Mero's late-night show.Links: Glitch: https://glitch.com Web.dev: https://web.dev Glitch Twitter: https://twitter.com/glitch Anil Dash Twitter: https://twitter.com/anildash TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today's guest is a little bit off the beaten path from the cloud infrastructure types I generally drag, kicking and screaming, onto the show. If we take a look at the ecosystem and where it's going, it's clear that in the future, not everyone who wants to build a business, or a tool, or even an application is going to necessarily spring fully-formed into the world from the forehead of some God, knowing how to code. And oh, “I'm going to go to a boot camp for four months to learn how to do it first,” is increasingly untenable. I don't know if you would call it low-code or not. But that's how it feels. My guest today is Anil Dash, CEO of Glitch. Anil, thank you for joining me.Anil: Thanks so much for having me.Corey: So, let's get the important stuff out of the way first, since I have a long-standing history of mispronouncing the company Twitch as ‘Twetch,' I should probably do the same thing here. So, what is Gletch? And what does it do?Anil: Glitch is, at its simplest, a tool that lets you build a full-stack app in your web browser in about 30 seconds. And, you know, for your community, your audience, it's also this ability to create and deploy code instantly on a full-stack server with no concern for deploy, or DevOps, or provisioning a container, or any of those sort of concerns. And what it is for the users is, honestly, a community. They're like, “I looked at this app that was on Glitch; I thought it was cool; I could do what we call [remixing 00:02:03].” Which is to kind of fork that app, a running app, make a couple edits, and all of a sudden live at a real URL on the web, my app is running with exactly what I built. And that's something that has been—I think, just captured a lot of people's imagination to now where they've built over 12 or 15 million apps on the platform.Corey: You describe it somewhat differently than I would, and given that I tend to assume that people who create and run successful businesses don't generally tend to do it without thought, I'm not quite, I guess, insufferable enough to figure out, “Oh, well, I thought about this for ten seconds, therefore I've solved a business problem that you have been needling at for years.” But when I look at Glitch, I would describe it as something different than the way that you describe it. I would call it a web-based IDE for low-code applications and whatnot, and you never talk about it that way. Everything I can see there describes it talks about friendly creators, and community tied to it. Why is that?Anil: You're not wrong from the conventional technologist's point of view. I—sufficient vintage; I was coding in Visual Basic back in the '90s and if you squint, you can see that influence on Glitch today. And so I don't reject that description, but part of it is about the audience we're speaking to, which is sort of a next generation of creators. And I think importantly, that's not just age, right, but that could be demographic, that can be just sort of culturally, wherever you're at. And what we look at is who's making the most interesting stuff on the internet and in the industry, and they tend to be grounded in broader culture, whether they're on, you know, Instagram, or TikTok, or, you know, whatever kind of influencer, you want to point at—YouTube.And those folks, they think of themselves as creators first and they think of themselves as participating in the community first and then the tool sort of follow. And I think one of the things that's really striking is, if you look at—we'll take YouTube as an example because everyone's pretty familiar with it—they have a YouTube Creator Studio. And it is a very rich and deep tool. It does more than, you know, you would have had iMovie, or Final Cut Pro doing, you know, 10 or 15 years ago, incredibly advanced stuff. And those [unintelligible 00:04:07] use it every day, but nobody goes to YouTube and says, “This is a cloud-based nonlinear editor for video production, and we target cinematographers.” And if they did, they would actually narrow their audience and they would limit what their impact is on the world.And so similarly, I think we look at that for Glitch where the social object, the central thing that people organize around a Glitch is an app, not code. And that's this really kind of deep and profound idea, which is that everybody can understand an app. Everybody has an idea for an app. You know, even the person who's, “Ah, I'm not technical,” or, “I'm not really into technology,” they're like, “But you know what? If I could make an app, I would make this.”And so we think a lot about that creative impulse. And the funny thing is, that is a common thread between somebody that literally just got on the internet for the first time and somebody who has been doing cloud deploys for as long as there's been a cloud to deploy to, or somebody has been coding for decades. No matter who you are, you have that place that is starting from what's the experience I want to build, the app I want to build? And so I think that's where there's that framing. But it's also been really useful, in that if you're trying to make a better IDE in the cloud and a better text editor, and there are multiple trillion-dollar companies that [laugh] are creating products in that category, I don't think you're going to win. On the other hand, if you say, “This is more fun, and cooler, and has a better design, and feels better,” I think we could absolutely win in a walk away compared to trillion-dollar companies trying to be cool.Corey: I think that this is an area that has a few players in it could definitely stand to benefit by having more there. My big fear is not that AWS is going to launch stuff in your space and drive you out of business; I think that is a somewhat naive approach. I'm more concerned that they're going to try to launch something in your space, give it a dumb name, fail that market and appropriately, not understand who it's for and set the entire idea back five years. That is, in some cases, it seems like their modus operandi for an awful lot of new markets.Anil: Yeah, I mean, that's not an uncommon problem in any category that's sort of community driven. So, you know, back in the day, I worked on building blogging tools at the beginning of this, sort of, social media era, and we worried about that a lot. We had built some of the first early tools, Movable Type, and TypePad, and these were what were used to launch, like, Gawker and Huffington Post and all the, sort of, big early sites. And we had been doing it a couple years—and then at that time, major player—AOL came in, and they launched their own AOL blog service, and we were, you know, quaking in our boots. I remember just being kind of like, pit in your stomach, “Oh, my gosh. This is going to devastate the category.”And as it turns out, people were smart, and they have taste, and they can tell. And the domain that we're in is not one that is about raw computing power or raw resources that you can bring to bear so much as it is about can you get people to connect together, collaborate together, and feel like they're in a place where they want to make something and they want to share it with other people? And I mean, we've never done a single bit of advertising for Glitch. There's never been any paid acquisition. There's never done any of those things. And we go up against, broadly in the space, people that have billboards and they buy out all the ads of the airport and, you know, all the other kind of things we see—Corey: And they do the typical enterprise thing where they spend untold millions in acquiring the real estate to advertise on, and then about 50 cents on the message, from the looks of it. It's, wow, you go to all this trouble and expense to get something in front of me, and after all of that to get my attention, you don't have anything interesting to say?Anil: Right.Corey: [crosstalk 00:07:40] inverse of that.Anil: [crosstalk 00:07:41] it doesn't work.Corey: Yeah. Oh, yeah. It's brand awareness. I love that game. Ugh.Anil: I was a CIO, and not once in my life did I ever make a purchasing decision based on who was sponsoring a golf tournament. It never happened, right? Like, I never made a call on a database platform because of a poster that was up at, you know, San Jose Airport. And so I think that's this thing that developers in particular, have really good BS filters, and you can sort of see through.Corey: What I have heard about the airport advertising space—and I but a humble cloud economist; I don't know if this is necessarily accurate or not—but if you have a company like Accenture, for example, that advertises on airport billboards, they don't even bother to list their website. If you go to their website, it turns out that there's no shopping cart function. I cannot add ‘one consulting' to my cart and make a purchase.Anil: “Ten pounds of consult, please.”Corey: Right? I feel like the primary purpose there might very well be that when someone presents to your board and says, “All right, we've had this conversation with Accenture.” The response is not, “Who?” It's a brand awareness play, on some level. That said, you say you don't do a bunch traditional advertising, but honestly, I feel like you advertise—more successfully—than I do at The Duckbill Group, just by virtue of having a personality running the company, in your case.Now, your platform is for the moment, slightly larger than mine, but that's okay,k I have ambition and a tenuous grasp of reality and I'm absolutely going to get there one of these days. But there is something to be said for someone who has a track record of doing interesting things and saying interesting things, pulling a, “This is what I do and this is how I do it.” It almost becomes a personality-led marketing effort to some degree, doesn't it?Anil: I'm a little mindful of that, right, where I think—so a little bit of context and history: Glitch as a company is actually 20 years old. The product is only a few years old, but we were formerly called Fog Creek Software, co-founded by Joel Spolsky who a lot of folks will know from back in the day as Joel on Software blog, was extremely influential. And that company, under leadership of Joel and his co-founder Michael Pryor spun out Stack Overflow, they spun out Trello. He had created, you know, countless products over the years so, like, their technical and business acumen is off the charts.And you know, I was on the board of Stack Overflow from, really, those first days and until just recently when they sold, and you know, you get this insight into not just how do you build a developer community that is incredibly valuable, but also has a place in the ecosystem that is unique and persists over time. And I think that's something that was very, very instructive. And so when it came in to lead Glitch I, we had already been a company with a, sort of, visible founder. Joel was as well known as a programmer as it got in the world?Corey: Oh, yes.Anil: And my public visibility is different, right? I, you know, I was a working coder for many years, but I don't think that's what people see me on social media has. And so I think, I've been very mindful where, like, I'm thrilled to use the platform I have to amplify what was created on a Glitch. But what I note is it's always, “This person made this thing. This person made this app and it had this impact, and it got these results, or made this difference for them.”And that's such a different thing than—I don't ever talk about, “We added syntax highlighting in the IDE and the editor in the browser.” It's just never it right. And I think there are people that—I love that work. I mean, I love having that conversation with our team, but I think that's sort of the difference is my enthusiasm is, like, people are making stuff and it's cool. And that sort of is my lens on the whole world.You know, somebody makes whatever a great song, a great film, like, these are all things that are exciting. And the Glitch community's creations sort of feel that way. And also, we have other visible people on the team. I think of our sort of Head of Community, Jenn Schiffer, who's a very well known developer and her right. And you know, tons of people have read her writing and seen her talks over the years.And she and I talk about this stuff; I think she sort of feels the same way, which is, she's like, “If I were, you know, being hired by some cloud platform to show the latest primitives that they've deployed behind an API,” she's like, “I'd be miserable. Like, I don't want to do that in the world.” And I sort of feel the same way. But if you say, “This person who never imagined they would make an app that would have this kind of impact.” And they're going to, I think of just, like, the last couple of weeks, some of the apps we've seen where people are—it could be [unintelligible 00:11:53]. It could be like, “We made a Slack bot that finally gets this reporting into the right channel [laugh] inside our company, but it was easy enough that I could do it myself without asking somebody to create it even though I'm not technically an engineer.” Like, that's incredible.The other extreme, we have people that are PhDs working on machine learning that are like, “At the end of the day, I don't want to be responsible for managing and deploying. [laugh]. I go home, and so the fact that I can do this in create is really great.” I think that energy, I mean, I feel the same way. I still build stuff all the time, and I think that's something where, like, you can't fake that and also, it's bigger than any one person or one public persona or social media profile, or whatever. I think there's this bigger idea. And I mean, to that point, there are millions of developers on Glitch and they've created well over ten million apps. I am not a humble person, but very clearly, that's not me, you know? [laugh].Corey: I have the same challenge to it's, effectively, I have now a 12 employee company and about that again contractors for various specialized functions, and the common perception, I think, is that mostly I do all the stuff that we talk about in public, and the other 11 folks sort of sit around and clap as I do it. Yeah, that is only four of those people's jobs as it turns out. There are more people doing work here. It's challenging, on some level, to get away from the myth of the founder who is the person who has the grand vision and does all the work and sees all these things.Anil: This industry loves the myth of the great man, or the solo legend, or the person in their bedroom is a genius, the lone genius, and it's a lie. It's a lie every time. And I think one of the things that we can do, especially in the work at Glitch, but I think just in my work overall with my whole career is to dismantle that myth. I think that would be incredibly valuable. It just would do a service for everybody.But I mean, that's why Glitch is the way it is. It's a collaboration platform. Our reference points are, you know, we look at Visual Studio and what have you, but we also look at Google Docs. Why is it that people love to just send a link to somebody and say, “Let's edit this thing together and knock out a, you know, a memo together or whatever.” I think that idea we're going to collaborate together, you know, we saw that—like, I think of Figma, which is a tool that I love. You know, I knew Dylan when he was a teenager and watching him build that company has been so inspiring, not least because design was always supposed to be collaborative.And then you think about we're all collaborating together in design every day. We're all collaborating together and writing in Google Docs—or whatever we use—every day. And then coding is still this kind of single-player game. Maybe at best, you throw something over the wall with a pull request, but for the most part, it doesn't feel like you're in there with somebody. Certainly doesn't feel like you're creating together in the same way that when you're jamming on these other creative tools does. And so I think that's what's been liberating for a lot of people is to feel like it's nice to have company when you're making something.Corey: Periodically, I'll talk to people in the AWS ecosystem who for some reason appear to believe that Jeff Barr builds a lot of these services himself then writes blog posts about them. And it's, Amazon does not break out how many of its 1.2 million or so employees work at AWS, but I'm guessing it's more than five people. So yeah, Jeff probably only wrote a dozen of those services himself; the rest are—Anil: That's right. Yeah.Corey: —done by service teams and the rest. It's easy to condense this stuff and I'm as guilty of it as anyone. To my mind, a big company is one that has 200 people in it. That is not apparently something the world agrees with.Anil: Yeah, it's impossible to fathom an organization of hundreds of thousands or a million-plus people, right? Like, our brains just aren't wired to do it. And I think so we reduce things to any given Jeff, whether that's Barr or Bezos, whoever you want to point to.Corey: At one point, I think they had something like more men named Jeff on their board than they did women, which—Anil: Yeah. Mm-hm.Corey: —all right, cool. They've fixed that and now they have a Dave problem.Anil: Yeah [unintelligible 00:15:37] say that my entire career has been trying to weave out of that dynamic, whether it was a Dave, a Mike, or a Jeff. But I think that broader sort of challenge is this—that is related to the idea of there being this lone genius. And I think if we can sort of say, well, creation always happens in community. It always happens influenced by other things. It is always—I mean, this is why we talk about it in Glitch.When you make an app, you don't start from a blank slate, you start from a working app that's already on the platform and you're remix it. And there was a little bit of a ego resistance by some devs years ago when they first encountered that because [unintelligible 00:16:14] like, “No, no, no, I need a blank page, you know, because I have this brilliant idea that nobody's ever thought of before.” And I'm like, “You know, the odds are you'll probably start from something pretty close to something that's built before.” And that enabler of, “There's nothing new under the sun, and you're probably remixing somebody else's thoughts,” I think that sort of changed the tenor of the community. And I think that's something where like, I just see that across the industry.When people are open, collaborative, like even today, a great example is web browsers. The folks making web browsers at Google, Apple, Mozilla are pretty collaborative. They actually do share ideas together. I mean, I get a window into that because they actually all use Glitch to do test cases on different bugs and stuff for them, but you see, one Glitch project will add in folks from Mozilla and folks from Apple and folks from the Chrome team and Google, and they're like working together and you're, like—you kind of let down the pretense of there being this secret genius that's only in this one organization, this one group of people, and you're able to make something great, and the web is greater than all of them. And the proof, you know, for us is that Glitch is not a new idea. Heroku wanted to do what we're doing, you know, a dozen years ago.Corey: Yeah, everyone wants to build Heroku except the company that acquired Heroku, and here we are. And now it's—I was waiting for the next step and it just seemed like it never happened.Anil: But you know when I talked to those folks, they were like, “Well, we didn't have Docker, and we didn't have containerization, and on the client side, we didn't have modern browsers that could do this kind of editing experience, all this kind of thing.” So, they let their editor go by the wayside and became mostly deploy platform. And—but people forget, for the first year or two Heroku had an in-browser editor, and an IDE and, you know, was constrained by the tech at the time. And I think that's something where I'm like, we look at that history, we look at, also, like I said, these browser manufacturers working together were able to get us to a point where we can make something better.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: I do have a question for you about the nuts and bolts behind the scenes of Glitch and how it works. If I want to remix something on Glitch, I click the button, a couple seconds later it's there and ready for me to start kicking the tires on, which tells me a few things. One, it is certainly not using CloudFormation to provision it because I didn't have time to go and grab a quick snack and take a six hour nap. So, it apparently is running on computers somewhere. I have it on good authority that this is not just run by people who are very fast at assembling packets by hand. What does the infrastructure look like?Anil: It's on AWS. Our first year-plus of prototyping while we were sort of in beta and early stages of Glitch was getting that time to remix to be acceptable. We still wish it were faster; I mean, that's always the way but, you know, when we started, it was like, yeah, you did sit there for a minute and watch your cursor spin. I mean, what's happening behind the scenes, we're provisioning a new container, standing up a full stack, bringing over the code from the Git repo on the previous project, like, we're doing a lot of work, lift behind the scenes, and we went through every possible permutation of what could make that experience be good enough. So, when we start talking about prototyping, we're at five-plus, almost six years ago when we started building the early versions of what became Glitch, and at that time, we were fairly far along in maturity with Docker, but there was not a clear answer about the use case that we're building for.So, we experimented with Docker Swarm. We went pretty far down that road; we spent a good bit of time there, it failed in ways that were both painful and slow to fix. So, that was great. I don't recommend that. In fairness, we have a very unusual use case, right? So, Glitch now, if you talk about ten million containers on Glitch, no two of those apps are the same and nobody builds an orchestration infrastructure assuming that every single machine is a unique snowflake.Corey: Yeah, massively multi-tenant is not really a thing that people know.Anil: No. And also from a security posture Glitch—if you look at it as a security expert—it is a platform allowing anonymous users to execute arbitrary code at scale. That's what we do. That's our job. And so [laugh], you know, so your threat model is very different. It's very different.I mean, literally, like, you can go to Glitch and build an app, running a full-stack app, without even logging in. And the reason we enable that is because we see kids in classrooms, they're learning to code for the first time, they want to be able to remix a project and they don't even have an email address. And so that was about enabling something different, right? And then, similarly, you know, we explored Kubernetes—because of course you do; it's the default choice here—and some of the optimizations, again, if you go back several years ago, being able to suspend a project and then quickly sort of rehydrate it off disk into a running app was not a common use case, and so it was not optimized. And so we couldn't offer that experience because what we do with Glitch is, if you haven't used an app in five minutes, and you're not a paid member, who put that app to sleep. And that's just a reasonable—Corey: Uh, “Put the app to sleep,” as in toddler, or, “Put the app to sleep,” as an ill puppy.Anil: [laugh]. Hopefully, the former, but when we were at our worst and scaling the ladder. But that is that thing; it's like we had that moment that everybody does, which is that, “Oh, no. This worked.” That was a really scary moment where we started seeing app creation ramping up, and number of edits that people were making in those apps, you know, ramping up, which meant deploys for us ramping up because we automatically deploy as you edit on Glitch. And so, you know, we had that moment where just—well, as a startup, you always hope things go up into the right, and then they do and then you're not sleeping for a long time. And we've been able to get it back under control.Corey: Like, “Oh, no, I'm not succeeding.” Followed immediately by, “Oh, no, I'm succeeding.” And it's a good problem to have.Anil: Exactly. Right, right, right. The only thing worse than failing is succeeding sometimes, in terms of stress levels. And organizationally, you go through so much; technically, you go through so much. You know, we were very fortunate to have such thoughtful technical staff to navigate these things.But it was not obvious, and it was not a sort of this is what you do off the shelf. And our architecture was very different because people had looked at—like, I look at one of our inspirations was CodePen, which is a great platform and the community love them. And their front end developers are, you know, always showing off, “Here's this cool CSS thing I figured out, and it's there.” But for the most part, they're publishing static content, so architecturally, they look almost more like a content management system than an app-running platform. And so we couldn't learn anything from them about our scaling our architecture.We could learn from them on community, and they've been an inspiration there, but I think that's been very, very different. And then, conversely, if we looked at the Herokus of the world, or all those sort of easy deploy, I think Amazon has half a dozen different, like, “This will be easier,” kind of deploy tools. And we looked at those, and they were code-centric not app-centric. And that led to fundamentally different assumptions in user experience and optimization.And so, you know, we had to chart our own path and I think it was really only the last year or so that we were able to sort of turn the corner and have high degree of confidence about, we know what people build on Glitch and we know how to support and scale it. And that unlocked this, sort of, wave of creativity where there are things that people want to create on the internet but it had become too hard to do so. And the canonical example I think I was—those of us are old enough to remember FTPing up a website—Corey: Oh, yes.Anil: —right—to Geocities, or whatever your shared web host was, we remember how easy that was and how much creativity was enabled by that.Corey: Yes, “How easy it was,” quote-unquote, for those of us who spent years trying to figure out passive versus active versus ‘what is going on?' As far as FTP transfers. And it turns out that we found ways to solve for that, mostly, but it became something a bit different and a bit weird. But here we are.Anil: Yeah, there was definitely an adjustment period, but at some point, if you'd made an HTML page in notepad on your computer, and you could, you know, hurl it at a server somewhere, it would kind of run. And when you realize, you look at the coding boot camps, or even just to, like, teach kids to code efforts, and they're like, “Day three. Now, you've gotten VS Code and GitHub configured. We can start to make something.” And you're like, “The whole magic of this thing getting it to light up. You put it in your web browser, you're like, ‘That's me. I made this.'” you know, north star for us was almost, like, you go from zero to hello world in a minute. That's huge.Corey: I started participating one of those boot camps a while back to help. Like, the first thing I changed about the curriculum was, “Yeah, we're not spending time teaching people how to use VI in, at that point, the 2010s.” It was, that was a fun bit of hazing for those of us who were becoming Unix admins and knew that wherever we'd go, we'd find VI on a server, but here in the real world, there are better options for that.Anil: This is rank cruelty.Corey: Yeah, I mean, I still use it because 20 years of muscle memory doesn't go away overnight, but I don't inflict that on others.Anil: Yeah. Well, we saw the contrast. Like, we worked with, there's a group called Mouse here in New York City that creates the computer science curriculum for the public schools in the City of New York. And there's a million kids in public school in New York City, right, and they all go through at least some of this CS education. [unintelligible 00:24:49] saw a lot of work, a lot of folks in the tech community here did. It was fantastic.And yet they were still doing this sort of very conceptual, theoretical. Here's how a professional developer would set up their environment. Quote-unquote, “Professional.” And I'm like, you know what really sparks kids' interests? If you tell them, “You can make a page and it'll be live and you can send it to your friend. And you can do it right now.”And once you've sparked that creative impulse, you can't stop them from doing the rest. And I think what was wild was kids followed down that path. Some of the more advanced kids got to high school and realized they want to experiment with, like, AI and ML, right? And they started playing with TensorFlow. And, you know, there's collaboration features in Glitch where you can do real-time editing and a code with this. And they went in the forum and they were asking questions, that kind of stuff. And the people answering their questions were the TensorFlow team at Google. [laugh]. Right?Corey: I remember those days back when everything seemed smaller and more compact, [unintelligible 00:25:42] but almost felt like a balkanization of community—Anil: Yeah.Corey: —where now it's oh, have you joined that Slack team, and I'm looking at this and my machine is screaming for more RAM. It's, like, well, it has 128 gigs in it. Shouldn't that be enough? Not for Slack.Anil: Not for chat. No, no, no. Chat is demanding.Corey: Oh, yeah, that and Chrome are basically trying to out-ram each other. But if you remember the days of volunteering as network staff on Freenode when you could basically gather everyone for a given project in the entire stack on the same IRC network. And that doesn't happen anymore.Anil: And there's something magic about that, right? It's like now the conversations are closed off in a Slack or Discord or what have you, but to have a sort of open forum where people can talk about this stuff, what's wild about that is, for a beginner, a teenage creator who's learning this stuff, the idea that the people who made the AI, I can talk to, they're alive still, you know what I mean? Like, yeah, they're not even that old. But [laugh]. They think of this is something that's been carved in stone for 100 years.And so it's so inspiring to them. And then conversely, talking to the TensorFlow team, they made these JavaScript examples, like, tensorflow.js was so accessible, you know? And they're like, “This is the most heartwarming thing. Like, we think about all these enterprise use cases or whatever. But like, kids wanting to make stuff, like recognize their friends' photo, and all the vision stuff they're doing around [unintelligible 00:26:54] out there,” like, “We didn't know this is why we do it until we saw this is why we do it.”And that part about connecting the creative impulse from both, like, the most experienced, advanced coders at the most august tech companies that exist, as well as the most rank beginners in public schools, who might not even have a computer at home, saying that's there—if you put those two things together, and both of those are saying, “I'm a coder; I'm able to create; I can make something on the internet, and I can share it with somebody and be inspired by it,” like, that is… that's as good as it gets.Corey: There's something magic in being able to reach out to people who built this stuff. And honestly—you shouldn't feel this way, but you do—when I was talking to the folks who wrote the things I was working on, it really inspires you to ask better questions. Like when I'm talking to Dr. Venema, the author of Postfix and I'm trying to figure out how this thing works, well, I know for a fact that I will not be smarter than he is at basically anything in that entire universe, and maybe most beyond that, as well, however, I still want to ask a question in such a way that doesn't make me sound like a colossal dumbass. So, it really inspires you—Anil: It motivates you.Corey: Oh, yeah. It inspires you to raise your question bar up a bit, of, “I am trying to do x. I expect y to happen. Instead, z is happening as opposed to what I find the documentation that”—oh, as I read the documentation, discover exactly what I messed up, and then I delete the whole email. It's amazing how many of those things you never send because when constructing a question the right way, you can help yourself.Anil: Rubber ducking against your heroes.Corey: Exactly.Anil: I mean, early in my career, I'd gone through sort of licensing mishap on a project that later became open-source, and sort of stepped it in and as you do, and unprompted, I got an advice email from Dan Bricklin, who invented the spreadsheet, he invented VisiCalc, and he had advice and he was right. And it was… it was unreal. I was like, this guy's one of my heroes. I grew up reading about his work, and not only is he, like, a living, breathing person, he's somebody that can have the kindness to reach out and say, “Yeah, you know, have you tried this? This might work.”And it's, this isn't, like, a guy who made an app. This is the guy who made the app for which the phrase killer app was invented, right? And, you know, we've since become friends and I think a lot of his inspiration and his work. And I think it's one of the things it's like, again, if you tell somebody starting out, the people who invented the fundamental tools of the digital era, are still active, still building stuff, still have advice to share, and you can connect with them, it feels like a cheat code. It feels like a superpower, right? It feels like this impossible thing.And I think about like, even for me, the early days of the web, view source, which is still buried in our browser somewhere. And you can see the code that makes the page, it felt like getting away with something. “You mean, I can just look under the hood and see how they made this page and then I can do it too?” I think we forget how radical that is—[unintelligible 00:29:48] radical open-source in general is—and you see it when, like, you talk to young creators. I think—you know, I mean, Glitch obviously is used every day by, like, people at Microsoft and Google and the New York Timesor whatever, like, you know, the most down-the-road, enterprise developers, but I think a lot about the new creators and the people who are learning, and what they tell me a lot is the, like, “Oh, so I made this app, but what do I have to do to put it on the internet?”I'm like, “It already is.” Like, as soon as you create it, that URL was live, it all works. And their, like, “But isn't there, like, an app store I have to ask? Isn't there somebody I have to get permission to publish this from? Doesn't somebody have to approve it?”And you realize they've grown up with whether it was the app stores on their phones, or the cartridges in their Nintendo or, you know, whatever it was, they had always had this constraint on technology. It wasn't something you make; it's something that is given to you, you know, handed down from on high. And I think that's the part that animates me and the whole team, the community, is this idea of, like, I geek out about our infrastructure. I love that we're doing deploys constantly, so fast, all the time, and I love that we've taken the complexity away, but the end of the day, the reason why we do it, is you can have somebody just sort of saying, I didn't realize there was a place I could just make something put it in front of, maybe, millions of people all over the world and I don't have to ask anybody permission and my idea can matter as much as the thing that's made by the trillion-dollar company.Corey: It's really neat to see, I guess, the sense of spirit and soul that arises from a smaller, more, shall we say, soulful company. No disparagement meant toward my friends at AWS and other places. It's just, there's something that you lose when you get to a certain point of scale. Like, I don't ever have to have a meeting internally and discuss things, like, “Well, does this thing that we're toying with doing violate antitrust law?” That is never been on my roadmap of things I have to even give the slightest crap about.Anil: Right, right? You know, “What does the investor relations person at a retirement fund think about the feature that we shipped?” Is not a question that we have to answer. There's this joy in also having community that sort of has come along with us, right? So, we talk a lot internally about, like, how do we make sure Glitch stays weird? And, you know, the community sort of supports that.Like, there's no reason logically that our logo should be the emoji of two fish. But that kind of stuff of just, like, it just is. We don't question it anymore. I think that we're very lucky. But also that we are part of an ecosystem. I also am very grateful where, like… yeah, that folks at Google use Glitch as part of their daily work when they're explaining a new feature in Chrome.Like, if you go to web.dev and their dev portal teaches devs how to code, all the embedded examples go to these Glitch apps that are running, showing running code is incredible. When we see the Stripe team building examples of, like, “Do you want to use this new payment API that we made? Well, we have a Glitch for you.” And literally every day, they ship one that sort of goes and says, “Well, if you just want to use this new Stripe feature, you just remix this thing and it's instantly running on Glitch.”I mean, those things are incredible. So like, I'm very grateful that the biggest companies and most influential companies in the industry have embraced it. So, I don't—yeah, I don't disparage them at all, but I think that ability to connect to the person who'd be like, “I just want to do payments. I've never heard of Stripe.”Corey: Oh yeah.Anil: And we have this every day. They come into Glitch, and they're just like, I just wanted to take credit cards. I didn't know there's a tool to do that.Corey: “I was going to build it myself,” and everyone shrieks, “No, no. Don't do that. My God.” Yeah. Use one of their competitors, fine,k but building it yourself is something a lunatic would do.Anil: Exactly. Right, right. And I think we forget that there's only so much attention people can pay, there's only so much knowledge they have.Corey: Everything we say is new to someone. That's why I always go back to assuming no one's ever heard of me, and explain the basics of what I do and how I do it, periodically. It's, no one has done all the mandatory reading. Who knew?Anil: And it's such a healthy exercise to, right, because I think we always have that kind of beginner's mindset about what Glitch is. And in fairness, I understand why. Like, there have been very experienced developers that have said, “Well, Glitch looks too colorful. It looks like a toy.” And that we made a very intentional choice at masking—like, we're doing the work under the hood.And you can drop down into a terminal and you can do—you can run whatever build script you want. You can do all that stuff on Glitch, but that's not what we put up front and I think that's this philosophy about the role of the technology versus the people in the ecosystem.Corey: I want to thank you for taking so much time out of your day to, I guess, explain what Glitch is and how you view it. If people want to learn more about it, about your opinions, et cetera. Where can they find you?Anil: Sure. glitch.com is easiest place, and hopefully that's a something you can go and a minute later, you'll have a new app that you built that you want to share. And, you know, we're pretty active on all social media, you know, Twitter especially with Glitch: @glitch. I'm on as @anildash.And one of the things I love is I get to talk to folks like you and learn from the community, and as often as not, that's where most of the inspiration comes from is just sort of being out in all the various channels, talking to people. It's wild to be 20-plus years into this and still never get tired of that.Corey: It's why I love this podcast. Every time I talk to someone, I learn something new. It's hard to remain too ignorant after you have enough people who've shared wisdom with you as long as you can retain it.Anil: That's right.Corey: Thank you so much for taking the time to speak with me.Anil: So, glad to be here.Corey: Anil Dash, CEO of Gletch—or Glitch as he insists on calling it. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment telling me how your small team at AWS is going to crush Glitch into the dirt just as soon as they find a name that's dumb enough for the service.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
President Biden's Advice in Action with Dan Woods

Screaming in the Cloud

Play Episode Listen Later Dec 28, 2021 39:28


About DanDan is CISO and VP of Cybersecurity for Shipt, a Target subsidiary. He worked previously as a Distinguished Engineer on Target's cloud infrastructure. He served as CTO for Joe Biden's 2020 Presidential campaign. Prior to that Dan worked with the Hillary for America tech team through the Groundwork, and contributed as a founding developer on Spinnaker while at Netflix. Dan is an O'Reilly published author and avid public speaker.  Links: Shipt: https://www.shipt.com/ Twitter: https://twitter.com/danveloper LinkedIn: https://www.linkedin.com/in/danveloper TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com.Corey: Writing ad copy to fit into a 30 second slot is hard, but if anyone can do it the folks at Quali can. Just like their Torque infrastructure automation platform can deliver complex application environments anytime, anywhere, in just seconds instead of hours, days or weeks. Visit Qtorque.io today and learn how you can spin up application environments in about the same amount of time it took you to listen to this ad.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. Sometimes I talk to people who are involved in working on the nonprofit slash political side of the world. Other times I talk to folks who are deep in the throes of commercial businesses, and I obviously personally spend more of my time on one of those sides of the world than I do the other. But today's guest is a little bit different, Dan Woods is the CISO and VP of Cybersecurity at Shipt, a division of Target where he's worked for a fair number of years, but took some time off for his side project, the side hustle as the kids call it, as the CTO for the Biden campaign. Dan, thank you for joining me.Dan: Yeah. Thank you, Corey. Happy to be here.Corey: So, you have an interesting track record as far as your career goes, you've been at Target for a long time. You were a distinguished engineer—not to be confused with ‘extinguished engineer,' which is just someone who is finally—the fire has gone out. And from there you went from being a distinguished engineer to a VP slash CISO, which generally looks a lot less engineer-like, and a lot more, at least in my experience, of sitting in a whole lot of executive-level meetings, managing teams, et cetera. Was that, in fact, an individual contributor—or IC—move into a management track, or am I just misunderstanding this because these are commonly overloaded terms in our industry?Dan: Yeah, yeah, no, that's exactly right. So, IC to leadership, two distinct tracks, distinct career paths. It was something that I've spent a number of years thinking about and more or less working toward and making sure that it was the right path for me to go. The interesting thing about the break that I took in the middle of Target when I was CTO for the campaign is that that was a leadership role, right. I led the team. I managed the team.I did performance reviews and all of that kind of managerial stuff, but I also sat down and did a lot of tech. So, it was kind of like a mix of being a senior executive, but also still continuing to be a distinguished engineer. So, then the natural path out of that for me was to make a decision about do I continue to be an individual contributor or do I go into a leadership track? And I felt like for a number of reasons that my interests more aligned with being on the leadership side of the world, and so that's how I've ended up where I am.Corey: And correct me if I'm wrong because generally speaking political campaigns are not usually my target customers given the fact that they're turning the entire AWS environment off in a few months—win or lose—and yeah, that is, in fact, remains the best way to save money on your AWS bill; it's hard for me to beat that. But at that point most of the people you're working with are in large part volunteers I would imagine.So, managing in a traditional sense of, “Well, we're going to have your next quarterly review.” Well, your candidate might not be in the race then, and what we're going to put you on a PIP, and what exactly you're going to stop letting me volunteer here? You're going to dock them pay—you're not paying me for this. It becomes an interesting management challenge I would imagine just because the people you're working with are passionate and volunteering, and a lot of traditional management and career advice doesn't necessarily map one-to-one I would have to assume.Dan: That is the best way that I've heard it described yet. I try to explain this to folks sometimes and it's kind of difficult to get that message across that like there is sort of a base level organization that exists, right. There were full-time employees who were a part of the tech team, really great group of folks especially from very early on willing to join the campaign and be a part of what it was that we were doing.And then there was this whole ecosystem of folks who just wanted to volunteer, folks who wanted to be a part of it but didn't want to leave their 9:00 to 5:00 who wanted to come in. One of the most difficult things about—we rely on volunteers very heavily in the political space, and very grateful for all the folks who step up and volunteer with organizations that they feel passionate about. In fact, one of the best little tidbits of wisdom the President imparted to me at one point, we were having dinner at his house very early on in the campaign, and he said, “The greatest gift that you can give somebody is your time.” And I think that's so incredibly true. So, the folks who volunteer, it's really important, really grateful that they're all there.In particular, how it becomes difficult, is that you need somebody to manage the volunteers, right, who are there. You need somebody to come up with work and check in that work is getting done because while it's great that folks want to volunteer five, ten hours a week, or whatever it is that they can put in, we also have very real things that need to get done, and they need to get done in a timely manner.So, we had a lot of difficulty especially early on in the campaign utilizing the volunteers to the extent that we could because we were such a small and scrappy team and because everybody who was working on the campaign at the time had a lot of responsibilities that they needed to see through on their own. And so getting into this, it's quite literally a full-time job having to sit down and follow up with volunteers and make sure that they have the appropriate amount of work and make sure that we've set up our environment appropriately so that volunteers can come and go and all of that kind of stuff, so yeah.Corey: It's always an interesting joy looking at the swath of architectural decisions and how they came to be. I talked on a previous episode with Jackie Singh, who was, I believe, after your tenure as CISO, she was involved on the InfoSec side of things, and she was curious as to your thought process or rationale with a lot of the initial architectural decisions that she talked about on her episode which I'm sure she didn't intend it this way, but I am going to blatantly miscategorize as, “Justify yourself. What were you thinking?” Usually it takes years for that kind of, “I don't understand what's going on here so I'm playing data center archeologist or cloud spelunker.” This was a very short window. How did decisions get made architecturally as far as what you're going to run things on? It's been disclosed that you were on AWS, for example. Was that a hard decision?Dan: No, not at all. Not at all. We started out the campaign—I in particular I was one of the first employees hired onto the campaign and the idea all along was that we're not going to be clever, right? We're basically just going to develop what needs to be developed. And the idea with that was that a lot of the code that we were going to sit down and write or a lot of the infrastructure that we were going to build was going to be glue, it not AWS Glue, right, ideally, but just glue that would bind data streams together, right?So, data movement, vendor A produces a CSV file for you and it needs to end up in a bucket somewhere. So, somebody needs to write the code to make that happen, or you need to find a sufficient vendor who can make that happen. There's a lot more vendors today believe it or not than there were two years ago that are doing much better in that kind of space, but two years ago we had the constraints of time and money.Our idea was that the code that we were going to write was going to be for those purposes. What it actually turned into is that in other areas of the business—and I will call it a business because we had formalized roadmaps and different departments working on different things—but in other areas of the business where we didn't have enough money to purchase a solution, we had the ability to go and write software.The interesting thing about this group of technologists who came together especially early on in the campaign to build out the tech team most of them came from an enterprise software development background, right? So, we had the know-how of how to build things at scale and how to do continuous delivery and continuous deployment, and how to operate a cloud-native environment, and how to build applications for that world.So, we ended up doing things like writing an API for managing our donor vetting pipeline, right? And that turned into a complex system of Lambda functions and continuous delivery for a variety of different services that facilitated that pipeline. We also built an architecture for our mobile app which there were plenty of companies that wanted to sell us a mobile app and we just couldn't afford it so we ended up writing the mobile app ourselves.So, after some point in time, what we said was we actually have a fairly robust and complex software infrastructure. We have a number of microservices that are doing various things to facilitate the operation of the business, and something that we need to do is we need to spend a little bit of time and make sure that we're building this in a cohesive way, right? And what part of that means was that, for example, we had to take a step back and say, “Okay, we need to have a unified identity service.” We can't have a different identity—or we can't have every single individual service creating its own identity. We need to have—Corey: I really wish you could pass that lesson out on some of the AWS service teams.Dan: [laugh]. Yes, I know. I know. Yeah. So, we went through—Corey: So, there were some questionable choices you made in there, like you started that with the beginning of, “Well, we had no time which is fine and no budget. So, we chose AWS.” It's like, “Oh, that looks like the exact opposite direction of a great decision, given, you know, my view on it.” Stepping past that entirely, you are also dealing with challenges that I don't think map very well to things that exist in the corporate world. For example, you said you had to build a donor vetting pipeline.It's in the corporate world I didn't have it. It's one of those, “Why in the world would I get in the way of people trying to give me money?” And the obvious answer in your case is, federal law, and it turns out that the best outcome generally does not involve serving prison time. So, you have to address these things in ways that don't necessarily have a one-to-one analog in other spaces.Dan: That's true. That's true. Yes, correct to the federal law thing. Our more pressing reason to do this kind of thing was that we made a commitment very early on in the campaign that we wouldn't take money from executives of the gas and oil industry, for example. There were another bunch of other commitments that were made, but it was inconceivable for us to have enough people that could possibly go manually through those filings. So, for us to be able to build an automated system for doing that meant that we were literally saving thousands of human hours and still getting a beneficial result out of it.Corey: And everything you do is subject to intense scrutiny by folks who are willing to make hay out of anything. If it had leaked at the time, I would have absolutely done some ridiculous nonsense thing about, “Ah, clearly looking at this AWS bill. Joe Biden's supports managed NAT gateway data processing pricing.” And it's absolutely not, but that doesn't stop people from making hay about this because headlines are going to be headlines.And do you have to also deal with the interesting aspect—industrial espionage is always kind of a thing, but by and large most companies don't have to worry that effectively half of the population is diametrically opposed to the thing it is that they're trying to do to the point where they might very well try to get insiders there to start leaking things out. Everything you do has to be built with optics in mind, working under tight constraints, and it seems like an almost insurmountable challenge except for the fact where you actually pulled it off.Dan: Yeah. Yeah. Yeah. We kept saying that the tech was not the story, right, and we wanted to do everything within our power to keep the conversation on the candidate and not on emails or AWS bills or any of that kind of stuff. And so we were very intentional about a lot of the decisions that we ended up making with the idea that if the optics are bad, we pull away from the primary mission of what it is that we're trying to do.Corey: So, what was it that qualified you to be the CTO of a—at the time very fledgling and uncertain campaign, given that you were coming from a role where you were a distinguished engineer, which is not nothing, let's be clear, but it's an executive-level of role rather than a hands-on level of role as CTO. And then if we go back in time, you were one of the founding developers of Spinnaker over at Netflix.And I have a lot of thoughts about Netflix technology and a lot of thoughts about Spinnaker as well, and none of those thoughts are, “This seems like a reasonable architecture I should roll out for a presidential campaign.” So, please, don't take this as the insult that probably sounds like, but why were you the CTO that got tapped?Dan: Great question. And I think in some ways, right place, right time. But in other ways probably needs to speak a little bit to the journey of how I've gotten anywhere in my career. So, going back to Netflix, yeah, so I worked in Netflix. I had the opportunity to work with a lot of incredibly bright and talented folks there. One of the people in particular who I met there and became friends with was Corey Bertram who worked on the core SRE team.Corey left Netflix to go off and at the time he was just like, “I'm going to go do a political startup.” The interesting thing about Netflix at the time—this was 2013, so, this was just after the Obama for America '12 campaign. And a bunch of folks from OFA world came and worked at Netflix and a variety of other organizations in the Bay Area. Corey was not one of those people but we were very well-connected with folks in that world, and Corey said he was going off to do a political startup, and so after my non-mutual departure from Netflix, I was talking to Corey and he said, “Hey, why don't you come over and help us figure out how to do continuous delivery over on the political startup.” That political startup turned into the groundwork which turned into essentially the tech platform for the Hillary for America campaign.So, I had the opportunity working for the groundwork to work very closely with the folks in the technology organization at HFA. And that got me more exposure to what that world is and more connections into that space. And the groundwork was run by Corey, but was the CEO or head—I don't even know what he called himself, was Michael Slaby, who was President Obama's CTO in 2008 and had a bigger technical role in the 2012 campaign.And so, for his involvement in HFA '16 meant that he was a person who was very well connected for the 2020 campaign. And when we were out at a political conference in late 2018 and he said, “Hey, I think that Vice President Biden is going to run. Do you have any interest in talking with his team?” And I said, “Yes, absolutely. Please introduce me.”And I had a couple of conversations with Greg Schultz who was the campaign manager and we just hit it off. And it was a really great fit. Greg was an excellent leader. He was a real visionary, exactly the person that President Biden needed. And he brought me in to set up the tech operation and get everything to where we ultimately won the primary and won the election after that.Corey: And then, as all things do, it ended and the question then becomes, “Great, what's next?” And the answer for you was apparently, “Okay, I'm going to go back to Target-ish.” Although now you're the CISO of a Target subsidiary, Shipt and Target's relationship is—again, I imagine I have that correct as far as you are in fact a subsidiary of Target, so it wasn't exactly a new company, but rather a transition into the previous organization you were in a different role.Dan: Yeah, correct. Yeah, it's a different department inside of Target, but my paycheck still come from Target. [laugh].Corey: So, what was it that inspired you to go into the CISO role? Because obviously security is everyone's job, which is what everyone says, which is why we get away with treating it like it's nobody's job because shared responsibilities tend to work out that way.Dan: Yeah.Corey: And you've done an awful lot of stuff that was not historically deeply security-centric although there's always an element passing through it. Now, going into a CISO role as someone without a deep InfoSec background that I'm aware of, what drove that? How did that work?Dan: You know, I think the most correct answer is that security has always been in my blood. I think like most people who started out—Corey: There are medications for that now.Dan: Yeah, [laugh] good. I might need them. [laugh]. I think like most folks who are kind of my era who started seriously getting into software development and computer system administration in the late ‘90s, early thousands, cybersecurity it wasn't called cybersecurity at the time. It wasn't even called InfoSec, right, it was just called, I don't know, dabbling or something. But that was a gateway for getting into Linux system administration, network engineering, so forth and so on.And for a short period of time I became—when I was getting my RHCE certification way back in the day, I became pretty entrenched in network security and that was a really big focus area that I spent a lot of time on and I got whatever the supplemental network security certification from Red Hat was at the time. And then I realized pretty quickly that the world isn't going to need box operators for very long, and this was just before the DevOps revolution had really come around and more and more things were automated.So, we were still doing hand deployments. I was still dropping WAR files onto a file system and restarting Apache. That was our deployment process. And I saw the writing on the wall and I said, “If I don't dedicate myself to becoming first and foremost a software engineer, then I'm not going to have a very good time in technology here.” So, I jumped out of that and I got into software development, and so that's where my software engineering career evolved out of.So, when I was CTO for the campaign, I like to tell people that I was a hundred percent of CTO, I was a hundred percent a CIO, and I was a hundred percent of CISO for the first 514 days of the campaign or whatever it was. So, I was 300 percent doing all of the top-level technology jobs for the campaign, but cybersecurity was without a doubt the one that we would drop everything for every single time.And that was by necessity; we were constantly under attack on the campaign. And a lot of my headspace during that period of time was dedicated to how do we make sure that we're doing things in the most secure way? So, when I left—when I came back into Target and I came back in as a distinguished engineer there were some areas that they were hoping that I could contribute positively and help move a couple of things along.The idea always the whole time was going to be for me to jump into a leadership position. And I got a call one day from Rich Agostino who's the CISO for Target and he said, “Hey, Shipt needs a cybersecurity operation built out and you're looking for a leadership role. Would you be interested in doing this?” And believe it or not, I had missed the world of cybersecurity so much that when the opportunity came up I said, “Yes, absolutely. I'll dive in head first.” And so that was the path for getting there.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: My take to cybersecurity space is, a little, I think, different than most people's journeys through it. The reason I started a Thursday edition of the Last Week in AWS newsletter is the security happenings in the AWS ecosystem for folks who don't have the word security in their job titles because I used to dabble in that space a fair bit. The problem I found is that is as you move up the ladder to executives that our directors, VPs, and CISOs, the language changes significantly.And it almost becomes a dialect of corporate-speak that I find borderline impenetrable, versus the real world terminology we're talking about when, “Okay, let's make sure that we rotate credentials on a reasonable expected basis where it makes sense,” et cetera et cetera. It almost becomes much more of a box-checking compliance exercise slash layering on as much as you possibly can that for plausible deniability for the inevitable breach that one day hits and instead of actually driving towards better outcomes.And I understand that's a cynical, strange perspective, but I started talking to people about this, and I'm very far from alone in that, which is why people are subscribing to that newsletter and that's the corner of the market I wanted to start speaking to. So, given that you've been an engineer practitioner trying to build things and now a security executive as well, is my assessment of the further higher up you go the entire messaging and purpose change, or is that just someone who's been in the trenches for too long and hasn't been on that side of the world, and I have a certain lack of perspective that would make this all very clear. Which I freely accept, if that's the case.Dan: No, I think that you're right for a lot of organizations. I think that that's a hundred percent true, and it is exactly as you described: a box-checking exercise for a lot of organizations. Something that's important to remember about Target is—Target was the subject of a data breach in 2012, and that was before there were data breaches every single day, right.Now, we look at a data breach and we say that's just going to happen, right, that's the cost of doing business. But back in 2012 it was really a very big story and it was a very big deal, and there was quite a bit of activity in the Target technology world after that breach. So, it reshaped the culture quite literally, new executives were brought in, but there's this whole world of folks inside of Target who have never forgotten that, right, and work day-in and day-out to make sure that we don't have another breach.So, security at Target is a main centrally thought about kind of thing. So, it's very much something that is a part of the way that people operate inside of Target. So, coming over to Shipt, obviously, Shipt is—it is a subsidiary. It is a part of Target, but it doesn't have that long history and hasn't had that same kind of experience. The biggest thing that we really needed at Shipt is first and foremost to get the program established, right. So, I'm three or four months onto the job now and we've tripled the team size. I've been—Corey: And you've stayed out of the headlines, which is basically the biggest and most accurate breach indicator I've found so far.Dan: So far so good. Well, but the thing that we want to do though is to be able to bring that same kind of focus of importance that Target has on cybersecurity into the world of engineering at Shipt. And it's not just a compliance game, and it's not just a thing where we're just trying to say that we have it. We're actually trying to make sure that as we go forward we've got all these best practices from an organization that's been through the bad stuff that we can adopt into our day-to-day and kind of get it done.When we talk about it at an executive level, obviously we're not talking about the penetration tests done by the red team the earlier day, right. We're not calling any of that stuff out in particular. But we do try to summarize it in a way that makes it clear that the thing that we're trying to do is build a security-minded culture and not just check some boxes and make sure that we have the appropriate titles in the appropriate places so that our insurance rates go down, right. We're actually trying to keep people safe.Corey: There's a lot to be said for that. With the Target breach back in—I want to say 2012, was it?Dan: 2012. Yep.Corey: Again, it was a wake-up call and the argument that I've always seen is that everyone is vulnerable—just depends on how much work it's going to take to get there. And for, credit where due, there was a complete rotation in the executive levels which whether that's fair or not, I—people have different opinions on it; my belief has always been you own the responsibility, regardless of who's doing the work.And there's no one as fanatical as a convert, on some level, and you've clearly been doing a lot of things in the right direction. The thing that always surprises me is that when I wind up seeing these surveys in the industry that—what is it? 65% of companies say that they would be vulnerable to a breach, and everybody said, “Oh, we should definitely look at those companies.” My argument is, “Hang on a sec. I want to talk to the 35% who say, ‘oh, we're impenetrable.'” because, spoiler, you are not.No one is. Just the question of how heavy is the lift and how much work is it going to take to get there? I do know that mouthing off in public about how perfect the security of anything is, is the best way to more or less climb to the top of a mountain during a thunderstorm, a hold up a giant metal rod, and curse the name of God. It doesn't lead to positive outcomes, basically ever. In turn, this also leads to companies not talking about security openly.I find that in many cases it is easier for me to get people to talk about their AWS bills than their InfoSec posture. And I do believe, incidentally, those two things are not entirely unrelated, but how do you view it? It was surprisingly easy to get Shipt's CISO to have a conversation with me here on this podcast. It is significantly more challenging in most other companies.Dan: Well, in fairness, you've been asking me for about two-and-a-half years pretty regularly [laugh] to come.Corey: And I always say I will stop bothering you if you want. You said, “No, no. Ask me again in a few months. Ask me again, after the election. Ask me again after—I don't know, like, the one-day delivery thing gets sorted out.” Whatever it happens to be. And that's fine. I follow up religiously, and eventually I can wear people down by being polite yet persistent.Dan: So, persistence on you is actually to credit here. No, I think to your question though, I think that there's a good balance. There's a good balance in being open about what it is that you're trying to do versus over-sharing areas that maybe you're less proficient in, right. So, it wouldn't make a lot of sense for me to come on here and tell you the areas that we need to develop into security. But on the other side of things, I am very happy to come in and talk to you about how our incident response plan is evolving, right, and what our plan looks like for doing all of that kind of stuff.Some of the best security practitioners who I've worked with in the world will tell you that you're not going to prevent a breach from a motivated attacker, and your job as CISO is to make sure that your response is appropriate, right, more so than anything. So, our incident response areas where today we're dedicating quite a bit of effort to build up our proficiency, and that's a very important aspect of the cybersecurity program that we're trying to build here.Corey: And unlike the early days of a campaign, you still have to be ultra-conscious about security, but now you have the luxury of actually being able to hire security staff because it turns out that, “Please come volunteer here,” is not presumably Shipt's hiring pitch.Dan: That's correct. Yeah, exactly. We have a lot of buy-in from the rest of leadership to build out this program. Shipt's history with cybersecurity is one where there were a couple of folks who did a remarkably good job for just being two or three of them for a really long period of time who ran the cybersecurity operation very much was not a part of the engineering culture at Shipt, but there still was coverage.Those folks left earlier in the year, all of them, simultaneously, unfortunately. And that's sort of how the position became open to me in the first place. But it also meant that I was quite literally starting with next to nothing, right. And from that standpoint it made it feel a lot like the early days of the campaign because I was having to build a team from scratch and having to get people motivated to come and work on this thing that had kind of an unknown future roadmap associated with it and all of that kind of stuff.But we've been very privileged to—because we have that leadership support we're able to pay market rates and actually hire qualified and capable and competent engineers and engineering leaders to help build out the aspects of this program that we need. And like I said, we've managed to—we weren't exactly at zero when I walked in the door. So, when I say we were able to quadruple the team, it doesn't mean that we just added four zeros there, [laugh] but we've got a little bit over a dozen people focusing on all areas of security for the business that we can think of. And that's just going to continue to grow. So, it's exciting; it's a challenge. But having the support of the entire organization behind something like this really, really helps a lot.Corey: I know we're running out of time for a lot of the interview, but one more question I want to ask you about is, when you're the CISO for a nationally known politician who is running for the highest office, the risk inherent to getting it wrong is massive. This is one of those mistakes will show indelibly for the rest of, well, one would argue US history, you could arguably say that there will be consequences that go that far out.On the other side of it, once you're done on the campaign you're now the CISO at Shipt. And I am not in any way insinuating that the security of your customers, and your partners, and your data across the board is important. But it does not seem to me from the outside that it has the same, “If we get this wrong there are repercussions that will extend into my grandchildren's time.” How do you find that your ability to care as deeply about this has changed, if it has?Dan: My stress levels are a lot lower I'll say that, but—Corey: You can always spot the veterans on an SRE team because—when I say veterans I mean veterans from the armed forces because, “No one's shooting at me. We can't serve ads right now. I'm really not going to run around and scream like, ‘My hair's on fire,' because this is nothing compared to what stress can look like.” And yeah there's always a worst stressor, but, on some level, it feels like it would be an asset. And again this is not to suggest you don't take security seriously. I want to be very clear on that point.Dan: Yeah, yeah, no. The important challenge of the role is building this out in a way that we have coverage over all the areas that we really need, right, and that is actually the kind of stuff that I enjoy quite a bit. I enjoy starting a program. I enjoy seeing a program come to fruition. I enjoy helping other people build their careers out, and so I have a number of folks who are at earlier at points in their career who I'm very happy that we have them on our team because I can see them grow and I can see them understand and set up what the next thing for them to do is.And so when I look at the day-to-day here, I was motivated on the campaign by that reality of like there is some quite literal life or death stuff that is going to happen here. And that's a really strong presser to make sure that you're doing all the right stuff at the right time. In this case, my motivation is different because I actually enjoy building this kind of stuff out and making sure that we're doing all the right stuff and not having the stress of, like, this could be the end of the world if we get this wrong.Means that I can spend time focusing on making sure that the program is coming together as it should, and getting joy from seeing the program come together is where a lot of that motivation is coming from today. So, it's just different, right? It's a different thing, but at the end of the day it's very rewarding and I'm enjoying it and can see this continuing on for quite some time.Corey: And I look forward to ideally getting you back in another two-and-a-half years after I began badgering you in two hours in order to come back on the show. If—Dan: [laugh].Corey: —people want to hear more about what you're up to, how you view about these things, potentially consider working with you, where can they find you?Dan: Best place although I've not been as active because it has been very busy the last couple of months, but find me on Twitter, @danveloper, find me on LinkedIn. Those—you know, I posted a couple of blog posts about the technology choices that we made on the campaign that I think folks find interesting, and periodically I'll share out my thoughts on Twitter about whatever the most current thing is, Kubernetes or AWS about to go down or something along those lines. So, yeah, that's the best way. And I tweet out all the jobs and post all the jobs that we're hiring for on LinkedIn and all of that kind of stuff. So, usual social channels. Just not Facebook.Corey: Amen to that. And I will of course include links to those things in the [show notes 00:37:29]. Thank you so much for taking the time to speak with me. I appreciate it.Dan: Thank you, Corey.Corey: Dan Woods, CISO and VP of Cybersecurity at Shipt, also formerly of the Biden campaign because wherever he goes he clearly paints a target on his back. I'm Cloud Economist, Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast please leave a five-star review on your podcast platform of choice along with an incoherent rant that is no doubt tied to either politics or the alternate form of politics: Spinnaker.Dan: [laugh].Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

AWS Morning Brief

Links: Has its own vulnerability that's actively under exploit: https://arstechnica.com/information-technology/2021/12/patch-fixing-critical-log4j-0-day-has-its-own-vulnerability-thats-under-exploit/ Google Project Zero deep dive into the NSO group's iMessage exploit: https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-into-nso-zero-click.html Three flaws: https://thehackernews.com/2021/12/hackers-begin-exploiting-second-log4j.html How to customize behavior of AWS Managed Rules for WAF: https://aws.amazon.com/blogs/security/how-to-customize-behavior-of-aws-managed-rules-for-aws-waf/ Using AWS security services to protect against, detect, and respond to the Log4j vulnerability: https://aws.amazon.com/blogs/security/using-aws-security-services-to-protect-against-detect-and-respond-to-the-log4j-vulnerability/ Update for Apache Log4j2 Issue: https://aws.amazon.com/security/security-bulletins/AWS-2021-006/ An innocent question: https://Twitter.com/QuinnyPig/status/1473382549535662082?s=20 TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it's nobody in particular's job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Announcer: Are you building cloud applications with a distributed team? Check out Teleport, an open-source identity-aware access proxy for cloud resources. Teleport provides secure access for anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps, and databases. Teleport gives engineers superpowers. Get access to everything via single sign-on with multi-factor, list and see all of SSH servers, Kubernetes clusters, or databases available to you in one place, and get instant access to them using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility, and ensuring compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That's goteleport.com.Corey: The burning yule log that is the log4j exploit and its downstream issues continues to burn fiercely. Meanwhile the year winds down, and it's certainly been an eventful one. I'll talk to you next week because that is what I do.Now, let's see from the community what happened. The patch to fix the log4j vulnerability apparently has its own vulnerability that's actively under exploit. Find your nearest InfoSec friend and buy them a beer or forty because this is going to suck for a long time and basically ruin everyone's holiday.Also, I've seen the most hair-raising thing I can remember in InfoSec-land, which is the Google Project Zero deep dive into the NSO group's iMessage exploit. Seriously, this thing requires no clicks on the part of the victim, the exploit uses a bug in the GIF processing inherent to iMessage to build a virtual CPU and assembly instruction set. There is no realistic defense against this short of hurling your phone into the sea, which I heartily recommend at this point as a best practice.Oh, and everything is on fire and somehow worse. There are now at least three flaws in the log4j library that we're counting, so far. Everything is terrible and we clearly should never log anything again.Corey: This episode is sponsored in part by my friends at Cloud Academy. Something special for you folks: If you missed their offer on Black Friday or Cyber Monday or whatever day of the week doing sales it is, good news, they've opened up their Black Friday promotion for a very limited time. Same deal: $100 off a yearly plan, 249 bucks a year for the highest quality cloud and tech skills content. Nobody else is going to get this, and you have to act now because they have assured me this is not going to last for much longer. Go to cloudacademy.com, hit the ‘Start Free Trial' button on the homepage and use the promo code, ‘CLOUD' when checking out. That's C-L-O-U-D. Like loud—what I am—with a C in front of it. They've got a free trial, too, so you'll get seven days to try it out to make sure it really is a good fit. You've got nothing to lose except your ignorance about cloud. My thanks to Cloud Academy once again for sponsoring my ridiculous nonsense.Now, AWS had a few things to say. The most relevant of them are How to customize behavior of AWS Managed Rules for WAF. So, if you're a WAF vendor and you don't link to this blog post as part of your, “Why should I pay you?” sales material, you're missing a golden opportunity. Every time I dig into AWS's Web Application Firewall offering, I end up regretting it, and with a headache.There was also a post on Using AWS security services to protect against, detect, and respond to the Log4j vulnerability. I'm disappointed to see AWS starting to use the log4nonsense stuff to pitch a dizzying array of expensive security services that require customers to do an awful lot of independent work to get stuff configured properly. This kind of isn't the time for that.And they have an update page that they continue to update called Update for Apache Log4j2 Issue, and this post has more frequent updates than AWS's “What's new” RSS feed. It really drives home the sheer scope of the issue, how pervasive it is, and just how much empathy we should have for the AWS security team. Their job has pretty clearly been not fun for the last couple of weeks.And lastly, the tip of the week is more of a request for help, honestly. I asked what I thought was an innocent question on Twitter: “What are people using to read and consume CloudTrail logs?” The answers made it clear that the answer was basically, “A bunch of very expensive enterprise grade things,” or, “Nothing.” This feels like a missed opportunity for some enterprising company out there. If you've got a better answer here, please whack reply and let me know. You know where to find me. Thanks for listening. That's what happened last week in AWS security. Enjoy the time off if you're lucky enough to get any, and I'll talk to you next week.Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Into the Great Wide Open Source with Julia Ferraioli

Screaming in the Cloud

Play Episode Listen Later Dec 23, 2021 40:19


About JuliaJulia Ferraioli calls herself an Open Source Archaeologist, focusing on sustainability, tooling, and research. Her background includes research in machine learning, robotics, HCI, and accessibility. Julia finds energy in developing creative demos, creating beautiful documents, and rainbow sprinkles. She's also a fierce supporter of LaTeX, the Oxford comma, and small pull requests.Links:Open Source Stories: https://www.opensourcestories.org TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com. Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is someone I have been very politely badgering to come on the show for a while, ever since I saw her speak a couple years ago in the Before Times, at Monktoberfest. As I've said before, anytime the RedMonk folks are involved in something, it is something you probably want to be involved in. That is my new guiding star philosophy when it comes to conferences, Twitter threads, opinions, breakfast cereals, you name it. Please welcome Julia Ferraioli, the co-founder of Open Source Stories, Julia, thank you for joining me today.Julia: Thank you for having me. And I definitely agree on the RedMonk side of things. They are fantastic folk.Corey: They're a small company, which is sort of interesting to me from a perspective of just how outsized their impact on this entire industry is. But it's, I've had as many of them as they will let me have on the show. They are welcome to come back whatever they want, just because they—every single one of them, though they're very different from one another, make everyone around them better with their presence. And that's just a hard thing to see. I didn't mean to turn this into a love letter to RedMonk, but here we are.Julia: I don't mind it. They have the ability to amplify the goodness that they see, anything from their survey designs to just how they interact online. It's wonderful to see.Corey: Speaking of amplifications, you are the co-founder of Open Source Stories, the idea of telling the—to my understanding—the stories behind open source. Like this is sort of like—what is it, Behind the Music, only in this case it's Behind the Code? I mean, how do you envision this?Julia: Oh, I like that framing. So, Open Source Stories is a project that myself and Amanda Casari founded not that terribly long ago because when we were doing research about how to model open source and open source ecosystems, we realized that a lot of the research papers that have been published about open source are pulled mostly from GitHub Archive, which is this repository of GitHub data. It could be the actual Git commit history as well as the activity streams from GitHub as well, but that doesn't capture a lot of the nuances behind open source, things like the narratives, how communities interact, where communication is happening, et cetera. All of these things can happen outside of the hosting platform. So, we launched this project to help tell these stories of the people and events and scenarios behind the open source projects that really power our industry.Corey: I'm going to get letters for this one, I'm sure of it, but I've been involved in the open source ecosystem for a while and I've noticed that there's been a recurring theme among various projects, particularly the more passionate folks working on them, where they talk an awful lot but they aren't very good at telling stories at the same time. And nowhere is this more evident than when we look at what passes for a lot of these projects' documentation. One of the transformative talks that I went to was Jordan Sissel's years and years ago, at the Southern California Linux Expo. And it was a talk about LogStash, which doesn't actually matter because the part that he'd said that really resonated with me, that his whole theme of his talk was around, was if a new user has a bad time, it's a bug. And the idea that, “Oh, you didn't read the documentation properly.”When about I started working with Linux, in some IRC chat rooms, the standard response to someone asked for help was to assume that they're an idiot, begin immediately accosting them with RTFM, for Read the Frickin' Manual, and then look for ways that you could turn this back around on them and make it their fault. And I looked at this and at the time, it's like, “Wow, these are people that are mean to other people,” and I was a small, angry teenager; it's like, “This is my jam. Here I am.” And yeah, many decades later, I'm looking at this and I feel a sense of shame because that's not the energy I want to put into the world. A lot of those communities have evolved and grown and what used to be the area and arena for hobbyists is now powering trillion-dollar companies.Julia: Absolutely. I like the whole, “If the user has a bad experience, that's a bug,” because it absolutely is. And I feel like a lot of these projects haven't invested nearly as much into the user experience as they have into polishing the code. And the attitude that that kind of perpetuates throughout the project about how you treat your users, it's pervasive and it really sets up the types of features that you develop, the contributors that you encouraged to commit to the project, and it just creates a—to put it minorly—less than welcoming environment for users, contributors, maintainers alike. And we don't really need that sort of hostility, especially when we're talking about projects that underpin the foundations, in some cases, of the internet.Corey: When we look at what open source is, I mean, I shortcut to thinking in terms of the context through which I've always approached it, which was generally code, or in my sad, particular story, back in the olden days on good freenode, when that was where a lot of this discourse happened, I was network staff and helping a bunch of different communities get channels set up through a Byzantine process. Because of course there was a Byzantine process; it was an open source community, and if there's one thing we love in open source, it is pretending to be lawyers when we're not. And we're sort of cargo-culting what we think process and procedure often look like. So yeah, there was a bunch of nonsensical paperwork happening there, but it was mostly about helping folks collaborate and communicate. But I've first and foremost, think in terms of code and in terms of community. What is open source to you?Julia: Well, I entered open source in the Sourceforge days, when all you had to do was go and download some code from the internet and hit the right download button, make sure not to hit one of the extraneous ones. And all you need for that is for the code to be under the right license. And to an extent that's what's true today for open source. At the heart of it, this minimum criteria for what constitutes open source is, “Okay, does it comply with the open source definition that the Open Source Initiative puts forth?” Now, I understand that not everybody necessarily agrees with the Open Source definition, but it's useful as a shortcut for how we think about the basic requirements. But what I find when people are talking about open source online is that they have these very different models. You'll hear from people that, “Okay, well, if it doesn't have a standard governance model, it's not really open source.”Corey: The ‘No True Scotsman' argument.Julia: Yeah. So, I find that we've got these different expectations for what open source is, and that leads to us talking past each other or discounting different types of open source when what we really need to do is come up with better language, a better vocabulary, for how to talk about these things. So, for example, I used to work in developer relations, and in developer relations one of the big things that you do is release sample code. Now, oftentimes, I'm not looking for that sample code to be picked up by a bunch of different developers and incorporated as a library into their project—Corey: [laugh]. Well, that's your error in that case because congratulations, that's running in production at a bank somewhere, now.Julia: Oh, I know. And that has definitely happened with my code, and I'm ashamed to say that. [laugh]. But generally speaking, you're not looking to build a huge community around sample code, right?Corey: You say that, but that again, Stack Overflow, it was—Julia: Okay.Corey: —[unintelligible 00:09:22] done rather well. So, there's that.Julia: Well yes, that is true, but when you release code on Stack Overflow, or GitHub, or in a Jest, or just on your blog, the thing that allows the bank to come in and incorporate that into their own application, or to even just learn from it, is the fact that it is open source. Now, it doesn't have a lot of the things that a community like Python or Kubernetes has, but it is still open source; it just has a different purpose than those communities and those ecosystems.Corey: So, I think it is challenging right now to talk about open source as if it were the same type of thing that it was back in the '90s, and the naughts—and even the teens—where it's a bunch of, more or less either hobbyists or people are perceived to be hobbyists. Sure, an awful lot of them are making commits from their redhat.com email address, but okay. And some of these people are increasingly being paid to work places, but then you see almost—I don't necessarily agree with the framing of The New York Times article by Daisuke Wakabayashi—who's a previous guest on the show—of Amazon strip-mining open source, but they definitely are in there—and other companies as well—are sort of appropriating it, or subverting it, or turning it into something that it was not previously, for lack of a better term. What's your take on that?Julia: Oh, that's a hard one. From a fundamentals perspective, that is absolutely within their rights under the definition of open source, and in some cases, the spirit of open source as well.Corey: Oh, and I would argue with someone who said that they should be constrained from doing this as far as a matter of legalities, or rights, or ridiculous Looney Tunes license changes.Julia: Well, there are definitely folks who are trying to make that the case.Corey: Yeah. Oh, yeah. I'm on the position of, they're within their rights to do it, but it's time for a good old fashioned public shunning as a result.Julia: I'm not sure I agree. I think that it is a natural consequence of how open source has gained in popularity and, in some cases, it's a testament to open source's success. Now, does it pose some serious challenges for the open source community and open source ecosystem? Absolutely because this is a new way of using open source that was unanticipated, and in fact, could be characterized as a Black Swan event in [open source-ware 00:12:18].Corey: The fundamental attribution error that I see, back at the very beginning, was that what we wrote the software, therefore, we are the best in the world at running it, therefore, if there's going to be a managed service, clearly ours will be the best. Amazon's core strength has apparently been operational excellence as they like to call it; my position on that is a little bit less of tying into the mystery, a little bit more of they're really fast and getting paged and fixing things in a hurry before customers notice. So okay, great, but it's column A, column B, whatever. The bigger concern I have with Amazon as its product strategy is, “Yes.” If it were just a way to run EC2 instances or virtual machines, then sure, that's great.And every open source project should, on some level, see some validation of its market through a lens of, “Oh, we're getting some competition. That's great.” The challenge I see is that in the line of competitors, Amazon is at or near the front all the time on basically everything. And it's if they would pick a lane to stay in, great.Google is a good example of this. There are things that Google very strongly considers in its wheelhouse, but for other things, they partner with the open source-based company in question to create a managed service partner offering and that's great. Amazon pulls a, “Nope. We're just going to build this out as first-party. The end.”And they compete with everyone, including themselves on almost every axis. And that's where it just gets into a, “Leave some oxygen for the rest of us.” I mean, it feels like they lie awake at night worrying that someone who isn't them somehow making money somewhere. That is, I think, on some level, more of the Black Swan event than someone else deciding that they can host a particular open source project more effectively. But that's where I stand. And again, this is just me as an enthusiastic and obnoxious observer. You're operating in this space. What do you think? That's the important part of the story.Julia: Well, I mean, you definitely have a point, Amazon—or AWS, maybe not necessarily Amazon—takes on different technologies far and wide, so they're not limiting themselves to a space. But that said, I think it comes down less to what is possible with open source and what is okay under the guise of open source, and what is good for the open source ecosystem. And when you fork a project, you do have to understand that you are bifurcating the open source ecosystem. And that can lead to sustainability problems down the road. So, I think the jury is still out on whether forking a project, running it as a managed service—as Amazon is doing with some of the open source projects—if that's going to come back to bite them just from a developer community standpoint because you're going to have people committing to one or the other, but possibly not both.Corey: I think this is why Amazon—I know, they're very annoyed by their perception in the open source ecosystem, but you take a look at other large tech companies, and almost all of them have a few notable open source projects that started life there. For example, we have—I think Cassandra came out of Facebook, but don't quote me on that; Kubernetes came out of Google, a fact for which they steadfastly refused to apologize, so far; and so on, and so forth. But Amazon's open source initiatives have been, “We've open sourced this thing that is basically only used at Amazon.” Or, my personal favorite, we've put all of our documentation up on GitHub so that you can write a corrections to it yourself from the community, which I'm hearing as, “Please, volunteer for a $1.6 trillion company so that they don't have to improve their documentation by hiring expensive people internally.”You can sort of guess my position on that. It seems like they have not launched anything that has a deep heart within Amazon that is broadly adopted outside of their walls. My question for you is, do you believe that having that level of adoption externally is required for a healthy open source project?Julia: Again, I think it goes back to the goals of why you're open-sourcing something. I don't believe that it's necessarily required for the open source project to be quality and be usable, but if your goal is adoption or if your goal is to get ideas and best practices out there, then yeah, you do need that engagement by the broader community, you do need the contributors. But there are a lot of cases where open-sourcing technology is more for the validation, rather than the adoption of the tech. So, it really depends.Corey: I'd say the most cynical reason I've seen to open source things comes from Netflix, where they have a recurring pattern of open-sourcing something, there are two or three commits, and then it basically sits there unattended. What I firmly believe is happening is that a senior engineer at Netflix is working on the thing and they're about to change jobs, so they open source the project so that they can change jobs and then pick up where they left off with an internal fork, I view it as a game of, basically, they're passing themselves a football as they run across the street. And people laugh when I say that, but I've also had people over drinks say, “You are closer than you might think, sometimes.” Which on some level is terrifying. Feels like life is imitating art, but here we go.Julia: That definitely happens, and I have seen it [laugh] as well. People want to essentially use open source to exfiltrate IP.Corey: Yeah. Only doing it legitimate way as opposed to the, “Please don't—hope they don't find that USB stick I've hidden in my sock on my last day.”Julia: Yes. And this is why open source offices have a challenging job in helping facilitate the release of open source software. So, it is hard to ascertain when that is happening.Corey: Yeah, no company is ever going to have a big statement that is going to be anything other than, honestly, marketing speak when it comes time to explain why they're doing a certain thing. It's, “Oh, yeah, we're open-sourcing this so we don't get sued in three years by this other company that might prove to be a competitive threat.” Or, “We're open-sourcing this as a hiring and recruiting technique.” I mean, I would argue, it wasn't open source, but one of the best approaches that I've seen from that perspective came out of Google, I'm firmly convinced to this day that App Engine was run not by their SRE team, but by their recruiting arm, “Because if you can build a great app on App Engine, well, this is, kind of like, how we think about things inside of Google; come and work here,” either via acqui-hiring or a just outright interview funnel. Maybe that's too cynical, too, but again, that leads to the question of is it really open source when it has these deep ties to specific platforms?Here's an open source tool that presumes you're running on top of AWS. Well, great, sure it's built by the community and anyone can access these things, but without paying per second to a cloud provider, probably the referenced cloud provider they're developing this against, it's not going to get very far. So, it's a nuanced argument, and there are shades of that nuance to every aspect of it. And if there's one thing that Twitter is terrible at is capturing nuance in 280 characters. And even in the, “All right, this is my nuanced take on open source in this thread, I will tweet, one of 5,712.” Great. That's not really the forum for that either. And people lose sight of nuance. It's a sticky, delicate thing, and it feels like a lot of the open source community has been enthusiastically agreeing with each other—sometimes violently so—but they're not sharing a common language in which to do it.Julia: Yeah. And in terms of the purposes of open source projects, it is okay for them to have different ones as long as they're telegraphing those purposes to their users and the people who are looking at the projects for their own use. But whether it's open source? I think it's okay for that to be the baseline and then build out the vocabulary of the types of projects that you want from there, based on those expectations. Yes, this particular technology only works with this cloud provider. That's open source that facilitates and accelerates development with that cloud provider.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: I always try and stay away from explicit value judgments on a lot of these things because it's nuanced, and no one who doesn't work at Facebook wakes up expecting to do terrible things today. We're all trying to do the best we can with the constraints are operating within. The challenge is that when you're at a company like an AWS, or a Google, or a Microsoft, or one of these giant companies, the same pressures that the rest of the quote-unquote “mere mortals” in ecosystem have to contend with are very different. But talking to people who work at these big companies, they have meetings and review processes that here at my twelve-person company, I don't even have to consider.Easy example of that: Never once have I put something out into the world and had a single discussion about is this going to get us in trouble with respect to antitrust? That has never been on my radar as far as things I have to care about. Even at my previous job at a highly regulated financial company, where you could argue that they are approaching monopoly status in some areas of the market organically, with passive investing being what it is, great, their open source discussions were always much more aligned with what licenses are we willing to accept legal risk for using internally? Because there are things that are—like IP is why we have a business in many respects, so anything that touches that theoretically means we'd have to disclose how the entire system, how the rest of it works, is not allowed to be used here. And there are reviews and processes and compliance requirements for that.I get that concern, and at a certain point of scale, you're negligent if you don't have a function that looks at it through that lens. But I look back to the early days of just puttering around with, “I want to do a thing and I found this project somewhere that people are excited about,” in the pre-GitHub days, I can download it off as Sourceforge or whatnot and I can make it work. And but it doesn't do this one thing I want to do, “Hey, the code's available. Can I fix it myself? Absolutely not. I'm crap at writing code. But I can talk to people and piece it together from wisdom that they offer.” And it turns into something awful until finally it gets enough traction that someone who knows what they're doing looks at it and refactors and it makes it good.And that's the open source community I recognize and that I see from my early developmental period. I don't recognize what we see in ecosystem today through that same lens of, “Okay, go online. Be nice to people”—well, that's new—“See how this thing works. And oh, if I'm having a problem, I'm probably not the only person who's having a problem like this.” You have to get really good at using Google more than you do at writing code in some respects. But at that point, it's almost entirely a copy-and-paste, except that's not technical enough for the open source world. So instead, we have to learn the 500 arcane subcommands to Git in order to get it out there. But it works. Ish.Julia: I think that community is still out there. I really do. I think that it is harder to find and it's not necessarily where you might tend to look, but those projects are still there. They're still running. They might be a little less high-profile than a lot of the ones that are getting a lot of attention right now, but they are still there.Corey: On some level, it feels like the blame for this lies—at least partially—at the feat of Slack and its success because it used to be that you had IRC, that was how folks communicated. And I remember the early days of that and things like Jabber or internal servers, grea—or internal IRC servers at companies—great, you'd have engineering all talking on that, and oh, you want to have someone in finance or marketing join that thing? Yeah, the short answer is, that won't be happening. But you can try and delude yourself and set it up with a special client and the rest.Slack removed all of that friction, but it's balkanized to the point where every once in a while, I have to go through and remove a bunch of Slack channels slash workspaces slash whatever we're calling them this week from my desktop client because it's basically eating all the RAM like it's trying to be Google Chrome. And then it's great, but there's no universal federated thing the way that there was with IRC where I just pop in a different channel for a different project. And IRC is still there and it comes back to life whenever Slack takes an outage. And then Slack gets fixed, it sort of bleeds off again. But I don't want to be in 500 different Slack workspaces, one for every open source project that I'm using, and there's no coherent sense of identity and community anymore the way there once was. And I feel like I'm old man yelling at the passing of time at this. But you're right, open source to me was always much more about community than it was about code.Julia: Yeah, and I think that we do not talk about the impact of tools for open source that we use. Because you're right; with IRC, it was unified. You could pretty much guarantee that projects of a certain size were present there. And with Slack, you have to sign up for yet another account, not quite yet sure why I can't find the right channels that I need to join in Slack. So, there's a lot of navigation and a lot of prerequisite knowledge that you need to have in order to be productive.And then you've got other tools being used for communication by other communities like, I believe Gitter is a major one as well. Then you have to make sure that you're up-to-date with all of these different interfaces, Discord, everything. And the sociological implication of that shouldn't be underestimated. What are you going to do if you find a project that uses a communication tool that you just really don't want to use or don't want to sign up for yet another account? Maybe you pass on by and you find one that works within your existing set of tools. There aren't a lack of open source projects to join right now. You can be choosy. And we don't yet know what the impact is of that.Corey: It's challenging. There's no good answer that I found that solves all of these things. It's become so balkanized, on some level, that every project out there that I see—and there are some small ones that are incredibly foundational to, basically, civilization as we know it, but it's not working right because it's you have to figure out where they are and what the community norms are because they change from project to project, and there are so many different things. And, like, you can go into NPM and install some relatively trivial thing that does command-line string processing, or whatnot, and it installs 40 different dependencies. And there's a problem and you want to figure out exactly how that works, and et cetera, et cetera, et cetera.Julia: Absolutely. With NPM specifically, or Node specifically, it is interesting that the development model kind of encourages this obscurity, an obfuscation of a functionality. So, it is hard to go in, debug an issue, go to the specific community, understand how they work, contribute a patch, just to fix something that is, you know, five levels up. It gets confusing for developers. It can contribute to longer-term bugs that we see propagate throughout the system. It is not an easy problem to solve, and I have a lot of sympathy for newcomers to the open source ecosystem because it is so hard to navigate. And I think that's an as yet unsolved problem that we need to address.Corey: So, what was it that inspired you to create Open Source Stories? I mean, I love the direction you're taking this in; I love the way you're thinking about [audio break 00:29:38]. Where did it come from? What started this?Julia: Well, when Amanda and I were going back and doing research around—you know, aside from the code for an open source project, where are the different entry points? Where are the different interaction points between projects, ecosystems, and the industry? And we did a couple of interviews, just very organic interviews, with some subject matter experts in Node, in Python, in Go. And there was a point where we stopped—or at least I stopped taking notes because I was just so fascinated by the narrative that our interviewee was putting forth and was talking about. And what we wanted was for it to not just be this meeting between a few people, we wanted to be able to share that with anyone. And so one of the things that really inspired us was StoryCorps, which allows you to record, much like we're doing today, 40 minutes worth of interactions between one to three people.Corey: Oh, we're going to cut it down to five minutes at most. Like, one question; one answer. Boom, we're done.Julia: [laugh].Corey: I kid, I kid.Julia: But it's really about facilitating the sharing of knowledge and sharing of these oral histories. Because as you're doing research into interactions in specific open source communities, you'll get articles, you'll get changelogs, all of that good stuff, but you won't get the nuance that we've been talking about over the course of this podcast. You lose the story behind the story, right? How are decisions made? How are people thinking about the interactions with their users? What are the turning points for a project? What are those conversations between the maintainers that changed the entire game?Those are the sorts of stories that we're hoping to capture because they're important for history, for knowledge sharing, for learning from our past, and making decisions for the future. And so that's really what we wanted to capture. And we wanted to capture the narratives behind the people that don't necessarily show up in the codebase, too: Talking about the designers, the product managers, the marketers behind open source that make it successful. Because there's so much more than code.Corey: Oh, my God, yes. It's… how do I put this politely without getting letters? Well, I guess I'll take a stab at it and see how it plays out. I look at so much of the brilliant code that has been written, and the documentation is abhorrent, and the design of the site, and the icon, and the interface, it looks like a joke that I put on Twitter trying to be funny. It's, the code is important, don't get me wrong, but there's so much more to it than that.And we see this in the industry, too, where companies have gone out of business, trying to get their codebase just right. It's, yeah, you can launch code that is really, really bad, but if you have product-market fit, it is survivable. I've heard stories in the early days of Twitter that we saw the fail whale all the time because it was an abhorrent monstrosity, to the point it became a running joke. But it turns out, when you hit product-market fit, you can afford really good engineers to come in and fix a lot of that stuff. That stuff is more important than the quality of the code, and that is something that I think that we have a collective industry-wide delusion about. And it's a blind spot for us.Julia: Yeah. I think we get wrapped up in the cleverness of the tech, and I've fallen prey to this, too. I get so involved in how I'm solving the problem and forget about the actual problem that I'm trying to solve, right? It's not necessarily about the how, but about the what. And without your fantastic tech writers, designers, usability experts, your open source project is going to be your open source project. It's not going to necessarily get that wide adoption, if that is indeed your goal for the technology that you're releasing.So, it really is about making sure that as we're launching and working on these open source projects and ecosystems, that we are inviting people to the table that have these other unique skills that goes beyond that code and speaks to what makes the project different and unique.Corey: I really want to say how much I appreciate your taking the time to talk to me about this. If people want to get involved themselves, how do they do that? Because I have a hard time accepting that you're doing something called Open Source Stories that eschews community involvement.Julia: Yeah. So, we absolutely would love more folks to get involved. I have been primarily the person working on the site, so we can always use contributors to the site itself, but we also want more storytellers and facilitators. And so if you go to opensourcestories.org, we've got a page specifically designed to facilitate contributions. So, check that out, and we look forward to hearing from anyone who wants to participate.Corey: And we will, of course, include links to that in the show notes. Thank you so much for taking the time to speak with me today. I really appreciate it.Julia: Thanks for having me.Corey: Julia Ferraioli, co-founder of Open Source Stories. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment, calling me a fool because I did not bother to RTFM first.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Working the Weather in the Cloud with Jake Hendy

Screaming in the Cloud

Play Episode Listen Later Dec 22, 2021 32:59


About JakeTechnical Lead by day at the Met Office in the UK, leading a team of software developers delivering services for the UK. By night, gamer and fitness instructor, attempting to get a home cinema and gaming setup whilst coralling 3 cats, 2 rabbits, 2 fish tanks, and my wonderful girlfriend.Links: Met Office: https://www.metoffice.gov.uk Twitter: https://twitter.com/jakehendy TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com. Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It's often said that the sun never sets on the British Empire, but it's often very cloudy and hard to see the sun because many parts of it are dreary and overcast. Here to talk today about how we can predict those things in advance—in theory—is Jake Hendy, Tech Lead at the Met Office. Jake, thanks for joining me.Jake: Hey, Corey, it's lovely to be here. Thanks for inviting me on.Corey: There's a common misconception that its startups in San Francisco or the culture thereof, if you can even elevate it to being a culture above something you'd find in a petri dish, that is where cloud stuff happens, where the computer stuff is done. And I've always liked cutting against that. There are governments that are doing interesting things with Cloud; there are large companies and ‘move fast and break things' is the exact opposite of what you generally want from institutions that date back centuries. What's it like working on Cloud, something that for all intents and purposes didn't exist 20 years ago, in the context of a government office?Jake: As you can imagine, it was a bit of a foray into cloud for us when it first came around. We weren't one of the first people to jump. The Met Office, we've got our own data centers, which we've proudly sit on that contains supercomputers and mainframes as well as a plethora of x86 hardware. So, we didn't move fast at the start, but nowadays, we don't move at breakneck speeds, but we like to take advantage of those managed services. It gets out of the way of managing things for us.Corey: Let's back up a second because I tend to be stereotypically American in many ways. What is the Met Office?Jake: What is the Met Office? The Met Office is the UK's National Meteorological Service. And what does that mean? We do a lot of things though with meteorology, from weather forecasting and climate research from our Hadley Centre—which is world-renowned—down to observations, collections, and partnerships around the world. So, if you've been on a plane over Europe, the Middle East, Africa, over parts of Asia, that plane took off because the Met Office provided a forecast for that plane. There's a whole range of things we can talk about there, if you want Corey, of what the Met Office actually does.Corey: Well, let's ask some of the baseline questions. You think of a weather office in a particular country as, oh okay, it tracks the weather in the area of operations for that particular country. Are you looking at weather on a global basis, on a somewhat local basis, or—as mentioned—since due to a long many-century history it turns out that there are UK Commonwealth territories scattered around the globe, where do you start? Where do you stop?Jake: We don't start and we don't stop. The Met Office is very much a 24/7 operation. So, we've got a 24/7 operation center with staff constantly manning it, doing all sorts of things. So, we've got a defense, we work heavily with our defense colleagues from UK armed forces to NATO partners; we've got aviation, as mentioned; we've got marine shipping from—most of the listeners in the UK will have heard of the shipping forecast at one point or another. And we've got private sector as well, from transport, to energy, supermarkets, and more. We have a very heavy UK focus, for obvious reasons, but our remit goes wide. You can actually go and see some of our model data is actually on Amazon Open Data. We've got MOGREPS, which is our ensemble forecast, as well as global models and UK models, with a 24-hour time lag, but feel free to go and have a play. And you can see the wide variety of data that we produce in just those few models.Corey: Yeah, just pulling up your website now; looking at where I am here in San Francisco, it gives me a detailed hour-by-hour forecast. There are only two problems I see with it. The first is that it's using Celsius units, which I—Jake: [laugh].Corey: —as a matter of policy, don't believe in because in this country, we don't really use things that make sense in measuring context. And also, I don't believe it's a real weather site because it's not absolutely festooned with advertisements for nonsense, which is apparently—I wasn't aware—a thing that you could have on the internet. I thought that showing weather data automatically meant that you had to attempt to cater to the lowest common denominator at all times.Jake: That's an interesting point there. So, the Met Office is owned and operated by Her Majesty's Government. We are a Trading Fund with the Department for Business, Energy and Industrial Strategy. But what does that mean it's a Trading Fund?k it means that we're funded by public money. So, that's called the Public Weather Service.But we also offer a more commercial venture. So, depending on what extensions you've got going on in your browser, there are actually adverts that do run on our website, and we do this to help recover some of the cost. So, the Public Weather Service has to recover some of that. And then lots of things are funded by the Public Weather Service, from observations, to public forecasting. But then there are more those commercial ventures such as the energy markets that have more paid products, and things like that as well. So, maybe not that many adverts, but definitely more usable.Corey: Yeah, I disabled the ad blocker, and I'm reloading it and I'm not seeing any here. Maybe I'm just considered to be such a poor ad targeting prospect at this point that people have just given up in despair. Honestly, people giving up on me in despair is kind of my entire shtick.Jake: We focus heavily on user-centered design, so I was fortunate in their previous team to work in our digital area, consumer digital, which looked after our web and mobile channels. And I can heartily say that there are a lot of changes, had a lot of heavy research into them. Not just internal, getting [unintelligible 00:06:09] and having a look at it, but what does this is actually mean for members of the? Public sending people out doing guerrilla public testing, standing outside Tescos—which is one of our large superstores here—and saying, “Hey, what do you think of this?” And then you'd get a variety of opinions, and then features would be adjusted, tweaked, and so on.Corey: So, you folks have been a relatively early adopter, especially in an institutional context. And by institution, I mean, one of those things that feels like it is as permanent as the stones in a castle, on some level, something that's lasted more than 20 years here in California, what a concept. And part of me wonders, were you one of the first UK government offices to use the cloud, and is that because you do weather and someone was very confused by what Cloud meant?Jake: [laugh]. I think we were possibly one of the first; I couldn't say if we were the first. Over in the UK, we've got a very capable network of government agencies doing some wonderful, and very cloud things. And the Government Digital Service was an initiative set up—uh, I can't remember, and I—unfortunately I can't remember the name of the report that caused its creation, but they had a big hand in doing design and cloud-first deployments. In the Met Office, we didn't take a, “Ah, screw it. Let's jump in,” we took a measured step into the cloud waters.Like I said, we've been running supercomputers since the '50s, and mainframes as well, and x86. I mean, we've been around for 100 years, so we constantly adapt, and engage, and iterate, and improve. But we don't just jump in and take a risk because like you said, we are an institution; we have to provide services for the public. It's not something that you can just ignore. These are services that protect life and property, both at home and abroad.Corey: You have provided a case study historically to AWS, about your use cases of what you use, back in 2014. It was, oh, you're a heavy user of EC2, and looking at the clock, and oh, it's 2014. Surprise. But you've also focused on other services as well. I believe you personally provided a bit of a case study slash story of round your use of Pinpoint of all things, which is a wrapper around SES, their email service, in the hopes of making it a little bit more, I guess, understandable slash fully-featured for contacting people, but in my experience is a great sales device to drive business to its competitors.What's it been like working, I guess, both simultaneously with the tried and true, tested yadda, yadda, yadda, EC2 RDS style stuff, but then looking at what else you're deep into Lambda, and DynamoDB, and SQS sort of stands between both worlds give it was the first service in beta, but it also is a very modern way of thinking about services. How do you contextualize all of that? Because AWS has product strategies, clearly, “Yes.” And they build anything for anyone is more or less what it seems. How do you think about the ecosystem of services that are available and apply it to problems that you're working on?Jake: So, in my personal opinion, I think the Met Office is one of a very small handfuls of companies around the world that could use every Amazon service that's offered, even things like Ground Station. But on my first day in the office, I went and sat at my desk and was talking to my new colleagues, and I looked to the left and he said, “Oh, yeah, that's a satellite dish collecting data from a satellite passing overhead.” So, we very much pick the best tool for the job. So, we have systems which do heavy number crunching, and very intense things, we'll go for EC2.We have systems that store data that needs relationships and all sorts of things. Fine, we'll go RDS. In my space, we have over a billion observations a year coming through the system I lead on SurfaceNet. So, do we need RDS? No. What about if we use something like S3 and Glue and Athena to run queries against this?We're very fortunate that we can pick the best tool for the job, and we pride ourselves on getting the most out of our tools and getting the most value for money. Because like I said, we're funded by the taxpayer; the taxpayer wants value for money, and we are taxpayers ourselves. We don't want to see our money being wasted when we got a hundred size auto-scaling group, when we could do it with Lambda instead.Corey: It's fascinating talking about some of the forward-looking stuff, and oh, serverless and throw everything at Cloud and be all in on cloud. Cloud, cloud, cloud. Cloud is the future. But earlier this year, there was a press release where the Met Office and Microsoft are going to be joining forces to build the world's, and I quote, “Most powerful weather and climate forecasting supercomputer.” The government—your government, to be clear—is investing over a billion pounds in the project.It is slated to be online and running by the middle of next year, 2022, which for a government project as I contextualize them feels like it's underwear-on-outside-the-pants superhero speed. But that, I guess, is what happens when you start looking at these public-private partnerships in some respects. How do you contextualize that? What is the story behind, oh, we're—you're clearly investing heavily in cloud, but you're also building your own custom enormous supercomputer rather than just waiting for AWS to drop one at re:Invent. What is the decision-making process look like? What is the strategy behind it?Jake: Oh. [laugh]. So—I'll have to be careful here—supercomputing is something that we've been doing for a long time, since the '50s, and we've grown with that. When the Met Office moved offices from Bracknell in 2002, 2003, we run two supercomputers for operational resilience, at that point [unintelligible 00:12:06] building in the new building; it was ready, and they were like, “Okay, let's move a supercomputer.” So, it came hurtling down the motorway, plugged in, and congrats, we've now got two supercomputers running again. We're very fortunate—Corey: We had one. It got lonely. We wanted to make it a friend. Yeah, I get it.Jake: Yeah. It's long distance; it works. And the Met Office is actually very good at running projects. We've done many supercomputers over the years, and supercomputing our models, we run some very intense models, and we have more demands. We know we can do better.We know there's the observations in my group we collect, there's the science that's continually improving and iterating and getting better, and our limit isn't poor optimizations or poorly written code. They're scientists running some fantastic code; we have a team who go and optimize these models, and you know, in one release, they may knock down a model runtime by four minutes. And you think, okay, that's four minutes, but for example, if that's four minutes across 400 nodes, all of a sudden you've now got 400 nodes that have then got four minutes more of compute. That could be more research, that could be a different model run. You know, we're very good at running these things, and we're very fortunate with very technically capable to understand the difference between a workload that belongs on AWS, a workload that belongs on a supercomputer.And you know, a supercomputer has many benefits, which the cloud providers… are getting into, you know, we have a high performance clusters on Amazon and Azure, or with, you know, InfiniBand networking. But sometimes you really can't beat a hunking great big ton of metal and super water-cooling, sat in a data center somewhere, backed by—we're very fortunate to have one hundred percent renewable energy for the supercomputer, which is—if you look at any of the power requirements for a supercomputer is phenomenal, so we're throwing that credentials behind it for climate change as well. You can't beat a supercomputer sometimes.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense. Corey: I'm somewhat fortunate in the despite living in a world of web apps, these days, my business partner used to work at the Department of Energy at Oak Ridge National Lab, helping with the care and feeding of the supercomputer clusters that they had out there. And you're absolutely right; that matches my understanding with the idea that there are certain workloads you're not going to be able to beat just having this enormous purpose-built cluster sitting there ready to go. Or even if you can, certainly not economically. I have friends who are in the batch side of the world, the HPC side of the world over in the AWS organizations, and they keep—“Hey, look at this. This thing's amazing.”But so much of what they're talking about seems to distill down to, “I have this one-off giant compute task that needs to get done.” Yes, you're right. If I need to calculate the weather one time, then okay, I can make an argument for going with cloud but you're doing this on what appears to be a pretty consistent basis. You're not just assuming—as best I can tell that, “And starting next Wednesday, it will be sunny forever. The end.”Jake: I'm sure many people would love it if we could do weather on-demand.Corey: Oh, yes. [unintelligible 00:15:09] going to reserved instance weather. That would be great. Like, “All right. I'd like to schedule some rain, please.” It really seems like it's one of those areas that is one of the most commonly accepted in science fiction without any real understanding of just what it would take to do something like that. Even understanding and predicting the weather is something that is beyond an awful lot of our current capabilities.Jake: This is exactly it. So, the Met Office is world-renowned for its research capabilities and those really in-depth, very powerful models that we run. So, I mentioned earlier, something called MOGREPS, which is the Met Office's ensemble-based models. And what do we mean by ensembles? You may see in the documentation it's got 18 members.What does that mean? It means that we actually run a simulation 18 times, and we tweak the starting parameters based on these real world inputs. And then you have a number of members that iterate through and supercomputer runs all of them. And we have deterministic models, which have one set of inputs. And you know, it's not just, as you say, one time; these models must run.There are a number of models we do, models on sea state as well, and they've all got to run, so we generally tend to run our supercomputers at top capacity. It's not often you get to go on a supercomputer and there'll be some space for your job to execute right this minute. And there's all the setup as well, so it's not just okay, the supercomputer is ready to go, but there's all the things that go into it, like, those observations, whether it's from the surface, whether it's from satellite data passing overhead, we have our own lightning network, as well. We have many things, like a radar network that we own, and operate. We collaborate with the environment agency for rainfall. And all these things they feed into these models.Okay, now we produce a model, and now it's got to go out. So, it's got to come off the supercomputer, it's got to be processed, maybe the grid that we run the models on needs to be reprojected because different people feed maps in different ways. Then there's got to be cut up because not every customer wants to know what the weather is everywhere. They've got a bit they care about. And of course, these models aren't small; you know, they can be terabytes, so there's also a case of customers might not want to download terabytes; that might cost them a lot. They might only be able to process gigabytes an hour.But then there's other products that we do processing on, so weather models, it might take 40 minutes to over an hour for a model to run. Okay, that's great. You might have missed the first step. Okay, well, we can enrich it with other data that's come in, things like nowcasting, where we do very short runs for the next six-hour forecast. There's a whole number of things that run in the office. And we don't have a choice; they run operationally 24/7, around the clock.I mentioned to you before we started recording, we had an incident of ‘Beast from the East' a number of years back. Some of your listeners may remember this; in the UK, we had a front come in from the east and the UK was blanketed with snow. It was a real severe event. We pretty much kept most of our services running. We worked really hard to make sure that they continued working.And personally I say, perhaps when you go shopping for Black Friday, you might go to a retailer and it's got a queue system up because, you know, it mimics that queue thing when you're outside a store, like in Times Square, and it's raining, be like oh, I might get a deal a minute. I think possibly in the Met Office, we have almost the inverse problem. If the weather's benign, we're still there. People rely on us to go, “Yeah, okay. I can go out and have fun.” When the weather's bad, we don't have a choice. We have to be there because everybody wants us to be there, but we need to be there. It's not a case of this is an optional service.Corey: People often forget that yeah, we are living in a world in which, especially with climate change doing what it's doing, if you get this wrong, people can very easily die. That is not something to take lightly. It's not just about can I go outside and play a pickup game of basketball today?Jake: Exactly. So, you know, operationally, we have something called the National Severe Weather Warning Service, where we issue guidance and alerts across the UK, based on severe weather. And there's a number of different weather types that we issued guidance for. And the severity of that goes from yellow to amber to red. And these are manually generated products, so there's the chief meteorologist who's on shift, and he approves these.And these warnings don't just go out to the members of the public. They go out to Cabinet Office, they go out to first responders, they go out to a number of people who are interested in the weather and have a responsibility. But the other side is that we don't issue a weather warning willy-nilly. It's a measured, calculated decision by our very capable operations team. And once that weather system has passed, the weather story has changed, we'll review it. We go back and we say what could we have done differently?Could the models have predicted this earlier? Could we have new data which would have picked up on this? Some of our next generation products that are in beta, would they have spotted this earlier? There's a lot of service review that continually goes on because like I said, we are the best, and we need to stay the best. People rely on us.Corey: So, here's a question that probably betrays my own ignorance, and that's okay, that's what I'm here to do. When I was a kid, I distinctly remember—first, this is not the era wish the world was black and white; I'm a child of the '80s, let's be clear here, so this is not old-timey nonsense quite as much, but distinctly remember that it was a running gag how unreliable the weather report always was, and it was a bit hit or miss, like, “Well, the paper says it's going to be sunny today, but we're going to pack an umbrella because we know how this works.” It feels, and I could be way off base on this, but it really feels like weather forecasting has gotten significantly more accurate since I was a kid. Is that just nostalgia, and I remember my parents complaining about it, or has there been a qualitative improvement in the accuracy of weather forecasting?Jake: I wish I could tell you all the scientific improvements that we've made, but there's many groups of scientists in the office who I would more than happily shift that responsibility over to, but quite simply, yes. We have a lot of partners we work with around the world—the National Weather Service, DWD in Germany, Meteo France, just to name but a few; there are many—and we all collaborate with data. We all iterate. You know, the American Meteorological Society holds a conference every year, which we attend. And there have been absolutely leaping changes in forecast quality and accuracy over the years.And that's why we continually upgrade our supercomputers. Like I said, yeah, there's research and stuff, but we're pulling in all this science and Meteorology is generally very chaotic systems. We're still discovering many things around how the climate works and how the weather systems work. And we're going to use them to help improve quality of life, early warnings, actually, we can say, oh, in three days time, it's going to be sunny at the beach. Be great if you could know that seven days in advance. It would be great if you knew that 14 days in advance.I mean, we might not do that because at the moment, we might have an idea, but there's also the case of understanding, you know, it's a probability-based decision. And people say, “Oh, it's not going to rain.” But actually, it's a case of, well, we said there's a 20% probability is going to rain. That doesn't mean it's not going to, but it's saying, “Two times out of ten, at this time it's going to rain.” But of course, if you go out 14 days, that's a long lead time, and you know, you talk about chaos theory, and the butterfly moves and flaps its wings, and all of a sudden a [cake 00:22:50] changes color from green to pink or something like that, some other location in the world.These are real systems that have real impacts, so we have to balance out the science of pure numbers, but what do people do with it? And what can people do with it, as well? So, that's why we talk about having timely data as well. People say, “Well, you could run these simulations and all your products take longer to process them and generate them,” but for example, in SurfaceNet, we have five minutes to process an observation once it comes in. We could spend hours fine-tuning that observation to make it perfect, but it needs to be useful.Corey: As you take a look throughout all of the things that AWS is doing—and sure, not all of these are going to necessarily apply directly to empowering the accuracy of weather forecasts, let's be clear here—but you have expressed personal interest in for example, IoT, a bunch of the serverless nonsense we're seeing out there. What excites you the most? What has you the most enthusiastic about what the future the cloud might hold? Because unlike almost everyone else I talk to in this space, you are not selling anything. You don't have a position—that I'm aware of—that oh, yeah, I super want to see this particular thing win the industry because that means you get to buy a boat.You work for the Met Office; you know that in some cases, oh, that boat is not going to have a great time in that part of the world anyway. I don't need one. So, you're a little bit more objective than most people. I have pushing a corporate story. What excites you? Where do you see the future of this industry going in ways that are neat?Jake: Different parts of the office will tell you different things, you know. We worked with Google DeepMind on AI and machine learning. We work with many partners on AI and machine learning, we use it internally, as well. On a personal level, I like quality of life improvements and things that just make my life as both the developer fun and interesting. So, CDK was a big thing.I was a CloudFormation wizard—still hate writing YAML—but the CDK came along and it was [unintelligible 00:24:52] people wouldn't say, but that wasn't, like, know when Lambda launched back in, what, 2013? 2014? No, but it made our lives easier. It meant that actually, we didn't have to worry about, okay, how do we do templating with YAML? Do we have to run some pre-processes or something?It meant that we could invest a little bit of time upfront on CDK and migrating everything over, and then that freed us up to actually doing things that we need for what we call the business or the organization, delivering value, you know? It's great playing with tech but, you know, I need to deliver value. And I think, what was it, in the Google SRE book, they limit the things they do, toiling of manual tasks that don't really contribute anything, they're more like keeping the lights on. Let's get rid of that. Let's focus on delivering value.It's why Lambda is so great. I could patch an EC2, I can automate it, you know, you got AWS Systems Manager Patch Manager, or… whatever its name is, they can go and manage all those patches for you. Why when I can do it in a Lambda and I don't need to worry about it?Corey: So, one last question that I have for you is that you're a tech lead. It's easy for folks to fall into the trap of assuming, “Oh, you're a government. It's like an enterprise only bigger, slower, and way, way, way busier.” How many hundreds of thousands of engineers are working at the Met Office along with you?Jake: So, you can have a look at our public report and you can see the number of staff we have. I think there's about 1800 staff that work at the Met Office. And that includes our account manage, that includes our scientists, that includes HR and legal. And I'd say there's probably less than 300 people who work in technology, as we call it, which is managing our IT estate, managing our Linux estate, managing our storage area networks because, funnily enough, managing petabytes of data is not an easy thing. You know, managing a supercomputer, a mainframe.There really aren't that many people here at the office, but we do so much great stuff. So, as a technical lead, I'm not just a leader of services, but I lead a team of people. I'm responsible for them, for empowering them, and helping them to develop their own careers and their own training. So, it's me and a team of four that look after SurfaceNet. And it's not just SurfaceNet; we've got other systems we look after that SurfaceNet produces data for. Sending messages around the world on the World Meteorological Organization's global telecommunications system. What a mouthful. But you know, these messages go all around the world. And some people might say, “Well, I got a huge team for that.” Well, [unintelligible 00:27:27]. We have other teams that help us—I say, help us—in their own right, they transmit that data. But we're really—I personally wouldn't say we were huge, but boy, do we pack a punch.Corey: Can I just say on a personal note, it's so great to talk to someone who's focusing on building out these environments and solving these problems for a higher purpose slash calling than—and I will get letters for this—than showing ads to people on the internet. I really want to thank you for taking time out of your day to speak with me. If people want to learn more about what you're up to, how you do it, potentially consider maybe joining you if they are eligible to work at the Met Office, where can they find you?Jake: Yeah, so you do have to be a resident in the UK, but www.metoffice.gov.uk is our home on the internet. You can find me on Twitter at @jakehendy, and I could absolutely chew Corey's ear off for many more hours about many of the wonderful services that the Met Office provides. But I can tell he's got something more interesting to do. So, uh [crosstalk 00:28:29]—Corey: Oh, you'd be surprised. It's loads of fun to—no, it's always fun to talk to people who are just in different areas that I don't get to work with very often. It turns out that most of my customers are not focused on telling you what the weather is going to do. And that's fine; it takes all kinds. It's just neat to have this conversation with a different area of the industry. Thank you so much for being so generous with your time. I appreciate it.Jake: Thank you very much for inviting me on. I guess if we get some good feedback, I'll have to come on and I will have to chew your ear off after all.Corey: Don't offer if you're not serious.Jake: Oh, I am.Corey: Jake Hendy, Tech Lead at the Met Office. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment yelling at one or both of us for having the temerity to rain on your parade.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Putting the “Fun” in Functional with Frank Chen

Screaming in the Cloud

Play Episode Listen Later Dec 21, 2021 35:42


About FrankFrank Chen is a maker. He develops products and leads software engineering teams with a background in behavior design, engineering leadership, systems reliability engineering, and resiliency research. At Slack, Frank focuses on making engineers' lives simpler, more pleasant, and more productive, in the Developer Productivity group. At Palantir, Frank has worked with customers in healthcare, finance, government, energy and consumer packaged goods to solve their hardest problems by transforming how they use data. At Amazon, Frank led a front-end team and infrastructure team to launch AWS WorkDocs, the first secure multi-platform service of its kind for enterprise customers. At Sandia National Labs, Frank researched resiliency and complexity analysis tooling with the Grid Resiliency group. He received a M.S. in Computer Science focused in Human-Computer Interaction from Stanford. Frank's thesis studied how the design / psychology of exergaming interventions might produce efficacious health outcomes. With the Stanford Prevention Research Center, Frank developed health interventions rooted in behavioral theory to create new behaviors through mobile phones. He prototyped early builds of Tiny Habits with BJ Fogg and worked in the Persuasive Technology Lab. He received a B.S. in Computer Science from UCLA. Frank researched networked systems and image processing with the Center for embedded Networked Systems. With the Rand Corporation, he built research systems to support group decision-making.Links: Slack: https://slack.com “Infrastructure Observability for Changing the Spend Curve”: https://slack.engineering/infrastructure-observability-for-changing-the-spend-curve/ “Right Sizing Your Instances Is Nonsense”: https://www.lastweekinaws.com/blog/right-sizing-your-instances-is-nonsense/ Personal webpage: https://frankc.net Twitter: @frankc TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com. Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Several people are undoubtedly angrily typing, and part of the reason they can do that, and the fact that I know that is because we're all using Slack. My guest today is Frank Chen, senior staff software engineer at Slack. So, I guess, sort of… [sales force 00:00:53]. Frank, thanks for joining me.Frank: Hey, Corey, I have been a longtime listener and follower, and just really delighted to be here.Corey: It's one of the weird things about doing a podcast is that for better or worse, people don't respond to it in the same way that they do writing a newsletter, for example, because you receive an email, and, “Oh, well, I know how to write an email. I can hit reply and send an email back and give that jackwagon a piece of my mind,” and people often do. But with podcasts, I feel like it's much more closely attuned to the idea of an AM radio talk show. And who calls into a radio talk show? Lunatics, and most people don't self-describe as lunatics, so they don't want to do that.But then when I catch up with people one-on-one or at events in person, I find out that a lot more people listen to this show than I thought they did. Because I don't trust podcast statistics because lies, damn lies, and analytics are sort of how I view this world. So, you've worked at a bunch of different companies. You're at Slack now, which, of course, upsets some people because, “Slack is ruining the way that people come and talk to me in the office.” Or it's making it easier for employees to collaborate internally in ways their employers wish they wouldn't. But that's neither here nor there.Before this, you were at Palantir, and before this, you're at Amazon, working on Amazon WorkDocs of all things, which is supposedly rumored to have at least one customer somewhere, but I've never seen them. Before that you were at Sandia National Labs, and you've gotten a master's in computer science from Stanford. You've done a lot of things and everything you've done, on some level, seems like the recurring theme is someone on Twitter will be unhappy at you for a career choice you've made. But what is the common thread—in seriousness—between the different places that you've been?Frank: One thing that's been a driver for where I work is finding amazing people to work with and building something that I believe is valuable and fun to keep doing. The thing that brought me to Slack is I became my own Slack admin, [laugh] when I met a girl and we moved in together into a small apartment in Brooklyn. And she had a cat that, you know, is a sweetheart, but also just doesn't know how to be social. Yes, you covered that with ‘cat.' Part of moving it together, I became my own Slack admin and discovered well, we can build a series of home automations to better train and inform our little command center for when the cat lies about being fed, or not fed, clipping his nails, and discovering and tracking bad behaviors. In a lot of ways this was like the human side of a lot of the data work that I had been doing at my previous role. And it was like a fun way to use the same frameworks that I use at work to better train and be a cat caretaker.Corey: Now, at some point, you know that some product manager at Amazon is listening to this and immediately sketching notes because their product strategy is, “Yes,” and this is going to be productized and shipping in two years as Amazon Prime Meow. But until then we'll enjoy the originality of having a Slack bot more or less control the home automation slash making your house seem haunted for anyone who didn't write the code themselves. There's an idea of solving real world problems that I definitely understand. I mean, and again, it might not even be a fair question entirely. Just because I am… for better or worse, staggering through my world, and trying—and failing most days—to tell a narrative that, “Oh, why did I start my tech career at a university, and then spend time in ad tech, and then spend time in consulting, and then FinTech, and the rest?” And the answer is, “Oh, I get fired an awful lot, and that sucked.”So, instead of going down that particular rabbit hole of a mess, I went in other directions. I started finding things that would pay me and pay me more money because I was in debt at the time. But that was the narrative thread that was the, “I have rent to pay and they have computers that aren't behaving properly.” And that's what dictated the shape of my career for a long time. It's only in retrospect that I started to identify some of the things that aligns with it. But it's easy to look at it with the shine of hindsight and not realize that no, no, that's sort of retconning what happened in the past.Frank: Yeah, I have a mentor and my former adviser had this way of describing, building out the jankiest prototype you can to prove out an idea. And this manifested in his class in building out paper prototypes, or really, really janky ideas for what helping people through technology might look like. And I feel like it a lot of ways, even when those prototypes fail, like, in a career or some half baked tech prototype I put together, it might succeed and great, we could keep building upon that, but when it fails, you actually discover, “Oh, this is one way that I didn't succeed.” And even in doing so, you discover things about yourself, your way of building, and maybe a little bit about your infrastructure, or whatever it is that you build on a day-to-day basis. And wrapping that back to the original question, it's like, well, we think we're human beings, right, we're static, but in a lot of ways we're human becomings. We think we know what the future might look like with our careers, what we're building on a day-to-day basis, and what we're building a year from now, but oftentimes, things change if we discover things about ourselves, the people we work with, and ultimately, the things that we put out into the world.Corey: Obviously, I've been aware of who Slack is, for a long time; I've been a paying customer for years because it basically is IRC with reaction gifs, and not having to teach someone how to sign into IRC when they work in accounting. So, the user experience alone solved the problem.Frank: And you've actually worked with us in the past before. [laugh]. Slack, it's the Searchable Log for all Content and Knowledge; I think that backronym, that's how it works. And I was delighted when I had mentioned your jokes and you're trolling [a folk 00:07:00] on Twitter and on your podcast to my former engineering manager, Chris Merrill, who was like, oh, you should search the Slack. Corey actually worked with us and he put together a lot of cool tooling and ideas for us to think about.Corey: Careful. If we talk too much, or what I did when I was at Slack years ago, someone's going to start looking into some of the old commits and whatnot and start demanding an apology, and we don't want that. It's, “Wow, you're right. You are a terrible engineer.” “Told you.” There's a reason I don't do that anymore.Frank: I think that's all of us. [laugh]. An early career mentor of mine, he was like, “Hey, Frank, listen. You think you're building perfect software at any point in time? No, you're building future tech debt.” And yeah, we should put much more emphasis on interfaces and ideas we're putting out because the implementation is going to change over time, and likely your current implementation is shit. And that is, okay.Corey: That's the beautiful part about this is that things grow and things evolve. And it's interesting working with companies, and as a consultant, I tend to build my projects in such a way that I start on day one and people know that I'm leaving with usually a very short window because I don't want to build a forever job for myself; I don't want to show up and start charging by the hour or by the day, if I can possibly avoid it. Because then it turns into eternal projects that never end because I'm billing and nothing's ever done. No, no, I like charging fixed fee and then getting out at a predetermined outcome, but then you get to hear about what happens with companies as they move on.This combines with the fact that I have a persistent alert for my name, usually because I'm looking for various ineffective character assassination from enterprise marketing types because you know, I dish it out, I should certainly be able to take it. But I found a blog post on the Slack engineering blog that mentioned my name, and it's, “Aw, crap. Are they coming after me for a refund?” No, it was not. It was you writing a fairly sizable post. Tell me more about that.Frank: Yeah, I'm part of an organization called Developer Productivity. And our goal is to help folk at Slack deliver services to their customers, where we build, test, and release high quality software. And a lot of our time is spent thinking about internal tooling and making infrastructure bets. As engineers, right, it's like, we have this idea for what the world looks like, we have this idea for what our infrastructure looks like, but what we discover using a set of techniques around observability of just asking questions—advanced questions, basic questions, and hell, even dumb questions—we discover hey, the things that we think our computers are doing aren't actually doing what they say they're doing. And the question is like, great. Now, what? How can we ask better questions? How can we better tune, change, and equip engineers with tooling so that they can do better work to make Slack customers have simple, pleasant, and productive experiences?Corey: And I have to say that there's a lot that Slack does that is incredibly helpful. I don't know that I'm necessarily completely bought into the idea that all work should happen in Slack. It's, well, on some level, I—like people like to debate the ‘should people work from home? Should people all work in an office?' Discussion.And, on some level, it seems if you look at people who are constantly fighting that debate online, it's, “Do you ever do work at all?” on some level. But I'm not here to besmirch others; I'm here to talk about, on some level, what you alluded to in your blog post. But I want to start with a disclaimer that Slack as far as companies go is not small, and if you take a look around, most companies are using Slack whether they know it or not. The list of side-channel Slack groups people have tend to extend massively.I look and I pare it down every once in a while, whenever I cross 40 signed-in Slacks on my desktop. It is where people talk for a wide variety of different reasons, and they all do different things. But if you're sitting here listening to this and you have a $2,000 a month AWS bill, this is not for you. You will spend orders of magnitude more money trying to optimize a small cost. Once you're at significant points of scale, and you have scaled out to the point where you begin to have some ability to predict over months or years, that's what a lot of this stuff starts to weigh in.So, talk to me a bit about how you wound up—and let me quote directly from the article, which is titled, “Infrastructure Observability for Changing the Spend Curve,” and I will, of course, throw a link to this in the [show notes 00:11:38]. But you talk in this about knocking, I believe it was orders of magnitude off of various cost areas within your bill.Frank: Yeah. The article itself describes three big-ish projects, where we are able to change the curve of the number of tests that we run, and a change in how much it costs to run any single test.Corey: When you say test, are you talking CI/CD infrastructure test or code test, to make sure it goes out, or are you talking something higher up the stack, as far as, “Huh, let's see how some users respond when, I don't know, we send four notifications on every message instead of the usual one,” to give a ridiculous example?Frank: Yeah, this is in the CI/CD pipelines. And one of these projects was around borrowing some concepts from data engineering: oversubscription and planning your capacity to have access capacity at peak, where at peak, your engineers might have a 5% degradation in performance, while still maintaining high resiliency and reliability of your tests in order to oversubscribe, either CPU or memory and keep throughput on the overall system stable and consistent and fast enough. I think, with spend in developer productivity, I think, both, like, the metrics you're trying to move and why you're optimizing for it at any given time are, like, this, like, calculus. Or it's like, more art than science in that there's no one right answer, right? It's like, oh, yeah—very naively—like, yeah, let's throw the biggest machines most expensive machines we can at any given problem. But that doesn't solve the crux of your problem. It's like, “Hey, what are the things in your system doing?” And what is the right guess to capitalize around how much to spend on your CI/CD [unintelligible 00:13:39] is oftentimes not precise, nor is this blog article meant to be prescriptive.Corey: Yeah, it depends entirely on what you're doing and how because it's, on some level, well, we can save a whole bunch of money if we slow all of our CI/CD runs down by 20 minutes. Yeah, but then you have a bunch of engineers sitting idle and I promise you, that costs a hell of a lot more than your cloud bill is going to be. The payroll is almost always a larger expense than your infrastructure costs, and if it's not, you should seriously consider firing at least part of your data science team, but you didn't hear it from me.Frank: Yeah. And part of the exploration on profiling and performance and resiliency was, like, around interrogating what the boundaries and what the constraints were for our CI/CD pipelines. Because Slack has grown in engineering and in the number of tests we were running on a month-to-month basis; for a while from 2017 to mid 2020, we were growing about 10% month-over-month in test suite execution numbers. Which means on a given year, we doubled almost two times, which is quite a bit of strain on internal resources and a lot of dependent services where—and internal systems, we oftentimes have more complexity and less understood changes in what dependencies your infrastructure might be using, what business logic your internal services are using to communicate with one another than you do your production.And so, by, like, performing a series of curiosity-driven development, we're able to both answer, at that point in time, what our customers internally were doing, and start to put together ideas for eliminating some bottlenecks, and hell, even adding bottlenecks with circuit breakers where you keep the overall throughput of your system stable, while deferring or canceling work that otherwise might have overloaded dependencies.Corey: There's a lot to be said for understanding what the optimization opportunities are, in an environment and understanding what it is you're attempting to achieve. Having those test for something like Slack makes an awful lot of sense because let's be very clear here, when you're building an application that acts as something people use to do expense reports—to cite one of my previous job examples—it turns out you can be down for a week and a majority of your customers will never know or care. With Slack, it doesn't work that way. Everyone more or less has a continuous monitor that they're typing into for a good portion of the day—angrily or otherwise—and as soon as it misses anything, people know. And if there's one thing that I love, on some level, seeing change when I know that Slack is having a blip, even if I'm not using Slack that day for anything in particular, because Twitter explodes about it. “Slack is down. I'm now going to tweet some stuff to my colleagues.” All right. You do you, I suppose.And credit where due, Slack doesn't go down nearly as often as it used to because as you tend to figure out how these things work, operational maturity increases through a bunch of tests. Fixing things like durability, reliability, uptime, et cetera, should always, to some extent, take precedence priority-wise over let's save some money. Because yeah, you could turn everything off and save all the money, but then you don't have a business anymore. It's focused on where to cut, where to optimize in the right way, and ideally as you go, find some of the areas in which, oh, I'm paying AWS a tax for just going about my business. And I could have flipped a switch at any point and saved—“How much money? Oh, my God, that's more than I'll make in my lifetime.”Frank: Yeah, and one thing I talk about a little bit is distributed tracing as one of the drivers for helping us understand what's happening inside of our systems. Where it helps you figure out and it's like this… [best word 00:17:24] to describe how you ask questions of deployed code? And there a lot of ways it's helped us understand existing bottlenecks and identify opportunities for performance or resiliency gains because your past janky Band-Aids become more and more obvious when you can interrogate and ask questions around what is it performing like it used to? Or what has changed recently?Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: It's also worth pointing out that as systems grow organically, that it is almost impossible for any one person to have it all in their head anymore. I saw one of the most overly complicated architecture flow trees that I think I've seen in recent memory, and it was on the Slack engineering blog about how something was architected, but it wasn't the Slack app itself; it was simply the [decision tree for ‘Should we send a notification?' 00:18:17] and it is more complicated than almost anything I've written, except maybe my newsletter content publication pipeline. It is massive. And I'll throw a link to that in the [show notes 00:18:31] as well, just because it is well worth people taking a look at.But there is so much complexity at scale for doing the right thing, and it's necessary because if I'm talking to you on Slack right now and getting notifications every time you reply on my phone, it's not going to take too long before I turn off notifications everywhere, and then I don't notice that Slack is there, and it just becomes useless and I use something else. Ideally, something better—which is hard to come by—moderately worse, like, email or completely worse, like, Microsoft Teams.Frank: I tell all my close collaborators about this. I typically set myself away on Slack because I like to make time for deep, focused work. And that's very hard with a constant stream of notifications. How people use Slack and how people notify others on Slack is, like, not incumbent on the software itself, but it's a reflection of the work culture that you're in. The expectation for an email-driven culture is, like, oh, yeah, you should be reading your email all the time and be able to respond within 30 minutes. Peace, I have friends that are lawyers, [laugh] and that is the expectation at all times of day.Corey: I married one of those. Oh, yeah, people get very salty. And she works with a global team spread everywhere, to the point where she wakes up and there's just a whole flurry of angry people that have tried to reach her in the middle of the night. Like, “Why were you sleeping at 2 a.m.? It's daytime here.” And yeah, time zones. Not everyone understands how they work, from my estimation.Frank: [laugh]. That's funny. My sweetheart is a former attorney. On our first international date, we spent an entire day-and-a-half hopping between WiFi spots in Prague so that she could answer a five minute question from a partner about standard deviations.Corey: So, one thing that you link to that really is what drew my notice to this—because, again, if you talk about AWS cost optimization, I'm probably going to stumble over it, but if you mention my name, that's sort of a nice accelerator—and you linked to my article called Why “Right Sizing Your Instances Is Nonsense.” And that is a little overblown, to some extent, but so many folks talk about it in the cost optimization space because you can get a bunch of metrics and do these things programmatically, and somewhat without observability into what's going on because, “Well, I can see how busy the computers are and if it's not busy, we could use smaller computers. Problem solved,” versus, the things that require a fair bit of insight into what is that thing doing exactly because it leads you into places of oh, turn off that idle fleet that's not doing anything is all labeled ‘backup,' where you're going to have three seconds of notice before it gets all the traffic.There's an idea of sometimes things are the way they are for a reason. And it's also not easy for a lot of things—think databases—to seamlessly just restart the thing and have it scale back up and run on a different instance class. That takes weeks of planning and it's hard. So, I find that people tend to reach for it where it doesn't often make sense. At your level of scale and operational maturity, of course, you should optimize what instance classes things are using and what sizes they are, especially since that stuff changes over time as far as what AWS has made available. But it's not the sort of thing that I suggest as being the first easy thing to go for. It's just what people think is easy because it requires no judgment and computers can do it. At least that's their opinion.Frank: I feel like you probably have a lot more experience than me, and talked about war stories, but I recall working with customers where they want to lift-and-shift on-prem hardware to VMs on-prem. I'm like, “It's not going to be as simple as you're making it out to be.” Whereas, like, the trend today is probably oh, yeah, we're going to shift on-prem VMs to AWS, or hell, like, let's go two levels deeper and just run everything on Kubernetes. Similar workloads, right? It's not going to be a huge challenge. Or [laugh] everything serverless.Corey: Spare me from that entire school of thought, my God.Frank: [laugh].Corey: Yeah, but it's fun, too, because this came out a month ago, and you're talking about using—an example you gave was a c5.9xlarge instance. Great. Well, the c6i is out now as well, so are people going to look at that someday and think, “Oh, wow. That's incredibly quaint.”It's, you wrote this a month ago, and it's already out of date, as far as what a lot of the modern story instances are. From my perspective, one of the best things that AWS has done in this space has been to get away from the reserved instance story and over into savings plans, where it's, “I know, I'm going to run some compute—maybe it's Fargate, maybe it's EC2; let's be serious, it's definitely going to be EC2—but I don't want to tie myself to specific instance types for the next three years.” Great, well, I'm just going to commit to spending some money on AWS for the next three years because if I decide today to move off of it, it's going to take me at least that long to get everything out. So okay, then that becomes something a lot more palatable for an awful lot of folks.Frank: One thing you brought up in the article I linked to is instance types. You think upgrading to the newest instance type will solve all your challenges, but oftentimes it's not obvious that it won't all the time, and in fact, you might even see degraded resiliency and degraded performance because different packages that your software relies upon might not be optimized for the given kernel or CPU type that you're running against. And ultimately, you go back to just asking really basic questions and performing some end-to-end benchmarking so that you can at least get a sense for what your customers are doing today, and maybe make a guess for what they're going to do tomorrow.Corey: I have to ask because I'm always interested in what it is that gives rise to blog posts like this—which, that's easy; it's someone had to do a project on these things, and while we learn things that would probably apply to other folks—like, you're solving what is effectively a global problem locally when you go down this path. It's part of the reason I have a consulting business is things I learned at one company apply almost identically to another company, even though that they're in completely separate industries and parts of the world because AWS billing is, for better or worse, a bounded problem space despite their best efforts to, you know, use quantum computers to fix that. What was it that gave rise to looking at the CI/CD system from an optimization point of view?Frank: So internally, I initially started writing a white paper about, hey, here's a simple question that we can answer, you know, without too much effort. Let's transition all of our C3 instances to C5 instances, and that could have been the one and done. But by thinking about it a little more and kind of drawing out, while we can actually borrow a model for oversubscription from another field, we could potentially decrease our spend by quite a bit. That eventually [laugh] evolved into a 70 page white paper—no joke—that my former engineering manager said, “Frank, no one's going to [BLEEP] read this.” [laugh].Corey: Always. Always, always. Like, here's a whole bunch of academically research and the rest. It's like, “Great. Which of these two buttons do I press?” is really the question people are getting at. And while it's great to have the research and the academic stuff, it's also a, “Great we're trying to achieve an outcome which, what is the choice?” But it's nice to know that people are doing actual research on the back end, instead, “Eh, my gut tells me to take the path on the left because why not? Left is better; right's tricky friend.”Frank: Yeah. And it was like, “Oh, yeah. I accidentally wrote a really long thing because there was, like, a lot of variables to test.” I think we had spun up 16-plus auto-scaling groups. And ran something like the cross-section of a couple of representative test suites against them, as well as configurations for a number of executors per instance.And about a year ago, I translated that into a ten page blog article that when I read through, I really didn't enjoy. [laugh]. And that template blog article is ultimately, like, about a page in the article you're reading today. And the actual kick in the butt to get this out the door was about four months ago. I spoke at o11ycon rescources which you're a part of.And it was a vendor conference by Honeycomb, and it was just so fun to share some of the things we've been doing with distributed tracing, and how we were able to solve internal problems using a relatively simple idea of asking questions about what was running. And the entire team there was wonderful in coaching and just helping me think through what questions people might have of this work. And that was, again, former academic. The last time I spoke at a conference was about a decade earlier, and it was just so fun to be part of this community of people trying to all solve the same set of problems, just in their own unique ways.Corey: One of the things I loved about working with Honeycomb was the fact that whenever I asked them a question, they have instrumented their own stuff, so they could tell me extremely quickly what something was doing, how it was doing it, and what the overall impact on this was. It's very rare to find a client that is anywhere near that level of awareness into what's going on in their infrastructure.Frank: Yeah, and that blog article, right, it's like, here's our current perspective, and here's, like, the current set of projects we're able to make to get to this result. And we think we know what we want to do, but if you were to ask that same question, “What are we doing for our spend a year from now?” the answer might be very different. Probably similar in some ways, but probably different.Corey: Well, there are some principles that we'll never get away from. It's, “Is no one using the thing? Turn that shit off.” That's one of those tried and true things. “Oh, it's the third copy of that multiple petabyte of data thing? Maybe delete it or stuff in a deep archive.” It's maybe move data less between various places. Maybe log things fewer times, given that you're paying 50 cents per gigabyte ingest, in some cases. Et cetera, et cetera, et cetera. There's a lot to consider as far as the general principles go, but the specifics, well, that's where it gets into the weeds. And at your scale, yeah, having people focus on this internally with the context and nuance to it is absolutely worth doing. Having a small team devoted to this at large companies will pay for itself, I promise. Now, I go in and advise in these scenarios, but past a certain point, this can't just be one person's part-time gig anymore.Frank: I'm kind of curious about that. How do you think about working with a company and then deprecating yourself, and allowing your tools and, like, the frameworks you put into place to continue, like, thrive?Corey: We're advisory only. We make no changes to production.Frank: Or I don't know if that's the right word, deprecate. I think… that's my own word. [laugh].Corey: No, no, it's fair. It's a—what we do is we go in and we are advisory. It's less of a cost engagement, more of an architecture engagement because in cloud, cost and architecture are the same thing. We look at what's going on, we look at the constraints of why we've been brought in, and we identify things that companies can do and the associated cost savings associated with that, and let them make their own decision. Because it's, if I come in and say, “Hey, you could save a bunch of money by migrating this whole subsystem to serverless.”Great, I sound like a lunatic evangelist because yeah, 18 months of work during which time the team doing that is not advancing the state of the business any further so it's never going to happen. So, why even suggest it? Just look at things that are within the bounds of possibility. Counterpoint: when a client says, “A full re-architecture is on the table,” well, okay, that changes the nature of what we're suggesting. But we're trying to get away from what a lot of tooling does, which is, “Great. Here's 700 things you can adjust and you'll do none of them.” We come back with a, “Here's three or four things you can do that'll blow 20% off the bill. Then let's see where you stand.” The other half of it, of course, is large scale enterprise contract negotiation, that's a bit of a horse of a different color. I want to thank you so much for taking the time to speak with me today. I really do appreciate it. If folks want to hear more about what you're up to, and how you think about these things. Where can they find you?Frank: You can find me at frankc.net. Or at me at @FrankC on Twitter.Corey: Oh, inviting people to yell at you at Twitter. That's never a great plan. Yeash. Good luck. Thanks again. We've absolutely got to talk more about this in-depth because I think this is one of those areas that you have the folks above a certain point of scale, talk about these things semi-constantly and live in the space, whereas folks who are in relatively small-scale environments are listening to this and thinking that they've got to do this.And no. No, you do not want to spend millions of dollars of engineering effort to optimize a bill that's 80 grand a year, I promise. It's focus on the thing that's right for your business. At a certain point of scale, this becomes that. But thank you so much for being so generous with your time. I appreciate it.Frank: Thank you so much, Corey.Corey: Frank Chen, senior staff software engineer at Slack. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that seems to completely miss the fact that Microsoft Teams is free because it sucks.Frank: [laugh].Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

AWS Morning Brief
...And Now Everything Is On Fire

AWS Morning Brief

Play Episode Listen Later Dec 16, 2021 6:56


Links: The internet is now on fire:https://www.engadget.com/log4shell-vulnerability-log4j-155543990.html Blog post:https://blog.cloudflare.com/exploitation-of-cve-2021-44228-before-public-disclosure-and-evolution-of-waf-evasion-patterns/ Expecting to be down for weeks:https://www.darkreading.com/attacks-breaches/kronos-suffers-ransomware-attack-expects-full-restoration-to-take-weeks- Update for the Apache Log4j2 Issue:https://aws.amazon.com/security/security-bulletins/AWS-2021-006/ Log4Shell Vulnerability Tester at log4shell.huntress.com:https://log4shell.huntress.com/ TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it's nobody in particular's job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key or a shared admin account isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open-source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more, visit goteleport.com. And no, that's not me telling you to go away; it is, goteleport.com.Corey: I think I owe the entire internet a massive apology. See, last week I titled the episode, “A Somehow Quiet Security Week.” This is the equivalent of climbing to the top of a mountain peak during a violent thunderstorm, then waving around a long metal rod. While cursing God.So, long story short, the internet is now on fire due to a vulnerability in the log4j open-source logging library. Effectively, if you can get an arbitrary string into the logs of a system that uses a vulnerable version of the log4j library, it will make outbound network requests. It can potentially run arbitrary code.The impact is massive and this one's going to be with us for years. WAF is a partial solution, but the only real answer is to patch to an updated version, or change a bunch of config options, or disallow affected systems from making outbound connections. Further, due to how thoroughly embedded in basically everything it is—like S3; more on that in a bit—a whole raft of software you run may very well be using this without your knowledge. This is, to be clear, freaking wild. I am deeply sorry for taunting fate last week. The rest of this issue of course talks entirely about this one enormous concern.Corey: This episode is sponsored in part by my friends at Cloud Academy. Something special for you folks: if you missed their offer on Black Friday or Cyber Monday or whatever day of the week doing sales it is, good news, they've opened up their Black Friday promotion for a very limited time. Same deal: $100 off a yearly plan, 249 bucks a year for the highest quality cloud and tech skills content. Nobody else is going to get this, and you have to act now because they have assured me this is not going to last for much longer. Go to cloudacademy.com, hit the ‘Start Free Trial' button on the homepage and use the promo code, ‘CLOUD' when checking out. That's C-L-O-U-D. Like loud—what I am—with a C in front of it. They've got a free trial, too, so you'll get seven days to try it out to make sure it really is a good fit. You've got nothing to lose except your ignorance about cloud. My thanks to Cloud Academy once again for sponsoring my ridiculous nonsense.Cloudflare has a blog post talking about the timeline of what they see as a global observer of exploitation attempts of this nonsense. They're automatically shooting it down for all of their customers and users—to be clear, if you're not paying for a service you are not its customer, you're a marketing expense—and they're doing this as part of the standard service they provide. Meanwhile AWS's WAF has added the ruleset to its AWSManagedRulesKnownBadInputsRuleSet—all one word—managed rules—wait a minute; they named it that? Oh, AWS. You sad, ridiculous service-naming cloud. But yeah, you have to enable AWS WAF, for which there is effectively no free tier, and configure this rule to get its protection, as I read AWS's original update. I'm sometimes asked why I use CloudFlare as my CDN instead of AWS's offerings. Well, now you know.Also, Kronos, an HR services firm, won the ransomware timing lottery. They're expecting to be down for weeks, but due to the log4shell—which is what they're calling this exploit: The log4shell problem—absolutely nobody is paying attention to companies that are having ransomware problems or data breaches. Good job, Kronos.Now, what did AWS have to say? Well, they have an ongoing “Update for the Apache Log4j2 Issue” and they've been updating it as they go. But at the time of this recording, AWS is a Java shop, to my understanding.That means that basically everything internet-facing at AWS—which is, you know, more or less everything they sell—has some risk exposure to this vulnerability. And AWS has moved with a speed that can only be described as astonishing, and mitigated this on their managed services in a timeline I wouldn't have previously believed possible given the scope and scale here. This is the best possible argument to make for using higher-level managed services instead of building your own things on top of EC2. I just hope they're classy enough not to use that as a marketing talking point.And for the tool of the week, the Log4Shell Vulnerability Tester at log4shell.huntress.com automatically generates a string and then lets you know when that is exploited by this vulnerability what systems are connecting to is. Don't misuse it obviously, but it's great for validating whether a certain code path in your environment is vulnerable. And that's what happened last week in AWS Security, and I just want to say again how deeply, deeply sorry I am for taunting fate and making everyone's year suck. I'll talk to you next week, if I live.Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
“Liqui”fying the Database Bottleneck with Robert Reeves

Screaming in the Cloud

Play Episode Listen Later Dec 16, 2021 50:45


About RobertR2 advocates for Liquibase customers and provides technical architecture leadership. Prior to co-founding Datical (now Liquibase), Robert was a Director at the Austin Technology Incubator. Robert co-founded Phurnace Software in 2005. He invented and created the flagship product, Phurnace Deliver, which provides middleware infrastructure management to multiple Fortune 500 companies.Links: Liquibase: https://www.liquibase.com Liquibase Community: https://www.liquibase.org Liquibase AWS Marketplace: https://aws.amazon.com/marketplace/seller-profile?id=7e70900d-dcb2-4ef6-adab-f64590f4a967 Github: https://github.com/liquibase Twitter: https://twitter.com/liquibase TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com. Corey: You know how Git works right?Announcer: Sorta, kinda, not really. Please ask someone else.Corey: That's all of us. Git is how we build things, and Netlify is one of the best ways I've found to build those things quickly for the web. Netlify's Git-based workflows mean you don't have to play slap-and-tickle with integrating arcane nonsense and web hooks, which are themselves about as well understood as Git. Give them a try and see what folks ranging from my fake Twitter for Pets startup, to global Fortune 2000 companies are raving about. If you end up talking to them—because you don't have to; they get why self-service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y dot com.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This is a promoted episode. What does that mean in practice? Well, it means the company who provides the guest has paid to turn this into a discussion that's much more aligned with the company than it is the individual.Sometimes it works, Sometimes it doesn't, but the key part of that story is I get paid. Why am I bringing this up? Because today's guest is someone I met in person at Monktoberfest, which is the RedMonk conference in Portland, Maine, one of the only reasons to go to Maine, speaking as someone who grew up there. And I spoke there, I met my guest today, and eventually it turned into this, proving that I am the envy of developer advocates everywhere because now I can directly tie me attending one conference to making a fixed sum of money, and right now they're all screaming and tearing off their headphones and closing this episode. But for those of you who are sticking around, thank you. My guest today is the CTO and co-founder of Liquibase. Please welcome Robert Reeves. Robert, thank you for joining me, and suffering the slings and arrows I'm about to hurled directly into your arse, as a warning shot.Robert: [laugh]. Man. Thanks for having me. Corey, I've been looking forward to this for a while. I love hanging out with you.Corey: One of the things I love about the Monktoberfest conference, and frankly, anything that RedMonk gets up to is, forget what's on stage, which is uniformly excellent; forget the people at RedMonk who are wonderful and I aspire to do more work with them in different ways; they're great, but the people that they attract are invariably interesting, they are invariably incredibly diverse in terms of not just demographics, but interests and proclivities. It's just a wonderful group of people, and every time I get the opportunity to spend time with those folks I do, and I've never once regretted it because I get to meet people like you. Snark and cynicism about sponsoring this nonsense aside—for which I do thank you—you've been a fascinating person to talk to you because you're better at a lot of the database-facing things than I am, so I shortcut to instead of forming my own opinions, I just skate off of yours in some cases. You're going to get letters now.Robert: Well, look, it's an occupational hazard, right? Releasing software, it's hard so you have to learn these platforms, and part of it includes the database. But I tell you, you're spot on about Monktoberfest. I left that conference so motivated. Really opened my eyes, certainly injecting empathy into what I do on a day-to-day basis, but it spurred me to action.And there's a lot of programs that we've started at Liquibase that the germination for that seed came from Monktoberfest. And certainly, you know, we were bummed out that it's been canceled two years in a row, but we can't wait to get back and sponsor it. No end of love and affection for that team. They're also really smart and right about a hundred percent of the time.Corey: That's the most amazing part is that they have opinions that generally tend to mirror my own—which, you know—Robert: [laugh].Corey: —confirmation bias is awesome, but they almost never get it wrong. And that is one of the impressive things is when I do it, I'm shooting from the hip and I already have an apology half-written and ready to go, whereas when dealing with them, they do research on this and they don't have the ‘I'm a loud, abrasive shitpostter on Twitter' defense to fall back on to defend opinions. And if they do, I've never seen them do it. They're right, and the fact that I am as aligned with them as I am, you'd think that one of us was cribbing from the other. I assure you that's not the case.But every time Steve O'Grady or Rachel Stephens, or Kelly—I forget her last name; my apologies is all Twitter, but she studied medieval history, I remember that—or James Governor writes something, I'm uniformly looking at this and I feel a sense of dismay, been, “Dammit. I should have written this. It's so well written and it makes such a salient point.” I really envy their ability to be so consistently on point.Robert: Well, they're the only analysts we pay money to. So, we vote with our dollars with that one. [laugh].Corey: Yeah. I'm only an analyst when people have analyst budget. Other than that, I'm whatever the hell you describe me. So, let's talk about that thing you're here to show. You know, that little side project thing you found and are the CTO of.I wasn't super familiar with what Liquibase does until I looked into it and then had this—I got to say, it really pissed me off because I'm looking at it, and it's how did I not know that this existed back when the exact problems that you solve are the things I was careening headlong into? I was actively annoyed. You're also an open-source project, which means that you're effectively making all of your money by giving things away and hoping for gratitude to come back on you in the fullness of time, right?Robert: Well, yeah. There's two things there. They're open-source component, but also, where was this when I was struggling with this problem? So, for the folks that don't know, what Liquibase does is automate database schema change. So, if you need to update a database—I don't care what it is—as part of your application deployment, we can help.Instead of writing a ticket or manually executing a SQL script, or generating a bunch of docs in a NoSQL database, you can have Liquibase help you out with that. And so I was at a conference years ago, at the booth, doing my booth thing, and a managing director of a very large bank came to me, like, “Hey, what do you do?” And saw what we did and got angry, started yelling at me. “Where were you three years ago when I was struggling with this problem?” Like, spitting mad. [laugh]. And I was like, “Dude, we just started”—this was a while ago—it was like, “We just started the company two years ago. We got here as soon as we could.”But I struggled with this problem when I was a release manager. And so I've been doing this for years and years and years—I don't even want to talk about how long—getting bits from dev to test to production, and the database was always, always, always the bottleneck, whether it was things didn't run the same in test as they did, eventually in production, environments weren't in sync. It's just really hard. And we've automated so much stuff, we've automated application deployment, lowercase a compiled bits; we're building things with containers, so everything's in that container. It's not a J2EE app anymore—yay—but we haven't done a damn thing for the database.And what this means is that we have a whole part of our industry, all of our database professionals, that are frankly struggling. I always say we don't sell software Liquibase. We sell piano recitals, date nights, happy hours, all the stuff you want to do but you can't because you're stuck dealing with the database. And that's what we do at Liquibase.Corey: Well, you're talking about database people. That's not how I even do it. I would never call myself that, for very good reason because you know, Route 53 remains the only database I use. But the problem I always had was that, “Great. I'm doing a deployment. Oh, I'm going to put out some changes to some web servers. Okay, what's my rollback?” “Well, we have this other commit we can use.” “Oh, we're going to be making a database schema change. What's your rollback strategy,” “Oh, I've updated my resume and made sure that any personal files I had on my work laptop been backed up somewhere else when I immediately leave the company when we can't roll back.” Because there's not really going to be a company anymore at that point.It's one of those everyone sort of holds their breath and winces when it comes to anything that resembles a schema change—or an ALTER TABLE as we used to call it—because that is the mistakes will show territory and you can hope and plan for things in pre-prod environments, but it's always scary. It's always terrifying because production is not like other things. That's why I always call my staging environment ‘theory' because things work in theory but not in production. So, it's how do you avoid the mess of winding up just creating disasters when you're dealing with the reality of your production environments? So, let's back up here. How do you do it? Because it sounds like something people would love to sell me but doesn't exist.Robert: [laugh]. Well, it's real simple. We have a file, we call it the change log. And this is a ledger. So, databases need to be evolved. You can't drop everything and recreate it from scratch, so you have to apply changes sequentially.And so what Liquibase will do is it connects to the database, and it says, “Hey, what version are you?” It looks at the change log, and we'll see, ehh, “There's ten change sets”—that's what components of a change log, we call them change sets—“There's ten change sets in there and the database is telling me that only five had been executed.” “Oh, great. Well, I'll execute these other five.” Or it asks the database, “Hey, how many have been executed?” And it says, “Ten.”And we've got a couple of meta tables that we have in the database, real simple, ANSI SQL compliant, that store the changes that happen to the database. So, if it's a net new database, say you're running a Docker container with the database in it on your local machine, it's empty, you would run Liquibase, and it says, “Oh, hey. It's got that, you know, new database smell. I can run everything.”And so the interesting thing happens when you start pointing it at an environment that you haven't updated in a while. So, dev and test typically are going to have a lot of releases. And so there's going to be little tiny incremental changes, but when it's time to go to production, Liquibase will catch it up. And so we speak SQL to the database, if it's a NoSQL database, we'll speak their API and make the changes requested. And that's it. It's very simple in how it works.The real complex stuff is when we go a couple of inches deeper, when we start doing things like, well, reverse engineering of your database. How can I get a change log of an existing database? Because nobody starts out using Liquibase for a project. You always do it later.Corey: No, no. It's one of those things where when you're doing a project to see if it works, it's one of those, “Great, I'll run a database in some local Docker container or something just to prove that it works.” And, “Todo: fix this later.” And yeah, that todo becomes load-bearing.Robert: [laugh]. That's scary. And so, you know, we can help, like, reverse engineering an entire database schema, no problem. We also have things called quality checks. So sure, you can test your Liquibase change against an empty database and it will tell you if it's syntactically correct—you'll get an error if you need to fix something—but it doesn't enforce things like corporate standards. “Tables start with T underscore.” “Do not create a foreign key unless those columns have an ID already applied.” And that's what our quality checks does. We used to call it rules, but nobody likes rules, so we call it quality checks now.Corey: How do you avoid the trap of enumerating all the bad things you've seen happen because at some point, it feels like that's what leads to process ossification at large companies where, “Oh, we had this bad thing happen once, like, a disk filled up, so now we have a check that makes sure that all the disks are at least 20, empty.” Et cetera. Great. But you keep stacking those you have thousands and thousands and thousands of those, and even a one-line code change then has to pass through so many different tests to validate that this isn't going to cause the failure mode that happened that one time in a unicorn circumstance. How do you avoid the bloat and the creep of stuff like that?Robert: Well, let's look at what we've learned from automated testing. We certainly want more and more tests. Look, DevOp's algorithm is, “All right, we had a problem here.” [laugh]. Or SRE algorithm, I should say. “We had a problem here. What happened? What are we going to change in the future to make sure this doesn't happen?” Typically, that involves a new standard.Now, ossification occurs when a person has to enforce that standard. And what we should do is seek to have automation, have the machine do it for us. Have the humans come up and identify the problem, find a creative way to look for the issue, and then let the machine enforce it. Ossification happens in large organizations when it's people that are responsible, not the machine. The machines are great at running these things over and over again, and they're never hung over, day after Super Bowl Sunday, their kid doesn't get sick, they don't get sick. But we want humans to look at the things that we need that creative energy, that brain power on. And then the rote drudgery, hand that off to the machine.Corey: Drudgery seems like sort of a job description for a lot of us who spend time doing operation stuff.Robert: [laugh].Corey: It's drudgery and it's boring, punctuated by moments of sheer terror. On some level, you're more or less taking some of the adrenaline high of this job away from people. And you know, when it comes to databases, I'm kind of okay with that as it turns out.Robert: Yeah. Oh, yeah, we want no surprises in database-land. And that is why over the past several decades—can I say several decades since 1979?Corey: Oh, you can s—it's many decades, I'm sorry to burst your bubble on that.Robert: [laugh]. Thank you, Corey. Thank you.Corey: Five, if we're being honest. Go ahead.Robert: So, it has evolved over these many decades where change is the enemy of stability. And so we don't want change, and we want to lock these things down. And our database professionals have become changed from sentinels of data into traffic cops and TSA. And as we all know, some things slip through those. Sometimes we speed, sometimes things get snuck through TSA.And so what we need to do is create a system where it's not the people that are in charge of that; that we can set these policies and have our database professionals do more valuable things, instead of that adrenaline rush of, “Oh, my God,” how about we get the rush of solving a problem and saving the company millions of dollars? How about that rush? How about the rush of taking our old, busted on-prem databases and figure out a way to scale these up in the cloud, and also provide quick dev and test environments for our developer and test friends? These are exciting things. These are more fun, I would argue.Corey: You have a list of reference customers on your website that are awesome. In fact, we share a reference customer in the form of Ticketmaster. And I don't think that they will get too upset if I mention that based upon my work with them, at no point was I left with the impression that they played fast and loose with databases. This was something that they take very seriously because for any company that, you know, sells tickets to things you kind of need an authoritative record of who's bought what, or suddenly you don't really have a ticket-selling business anymore. You also reference customers in the form of UPS, which is important; banks in a variety of different places.Yeah, this is stuff that matters. And you support—from the looks of it—every database people can name except for Route 53. You've got RDS, you've got Redshift, you've got Postgres-squeal, you've got Oracle, Snowflake, Google's Cloud Spanner—lest people think that it winds up being just something from a legacy perspective—Cassandra, et cetera, et cetera, et cetera, CockroachDB. I could go on because you have multiple pages of these things, SAP HANA—whatever the hell that's supposed to be—Yugabyte, and so on, and so forth. And it's like, some of these, like, ‘now you're just making up animals' territory.Robert: Well, that goes back to open-source, you know, you were talking about that earlier. There is no way in hell we could have brought out support for all these database platforms without us being open-source. That is where the community aligns their goals and works to a common end. So, I'll give you an example. So, case in point, recently, let me see Yugabyte, CockroachDB, AWS Redshift, and Google Cloud Spanner.So, these are four folks that reached out to us and said, either A) “Hey, we want Liquibase to support our database,” or B) “We want you to improve the support that's already there.” And so we have what we call—which is a super creative name—the Liquibase test harness, which is just genius because it's an automated way of running a whole suite of tests against an arbitrary database. And that helped us partner with these database vendors very quickly and to identify gaps. And so there's certain things that AWS Redshift—certain objects—that AWS Redshift doesn't support, for all the right reasons. Because it's data warehouse.Okay, great. And so we didn't have to run those tests. But there were other tests that we had to run, so we create a new test for them. They actually wrote some of those tests. Our friends at Yugabyte, CockroachDB, Cloud Spanner, they wrote these extensions and they came to us and partnered with us.The only way this works is with open-source, by being open, by being transparent, and aligning what we want out of life. And so what our friends—our database friends—wanted was they wanted more tooling for their platform. We wanted to support their platform. So, by teaming up, we help the most important person, [laugh] the most important person, and that's the customer. That's it. It was not about, “Oh, money,” and all this other stuff. It was, “This makes our customers' lives easier. So, let's do it. Oop, no brainer.”Corey: There's something to be said for making people's lives easier. I do want to talk about that open-source versus commercial divide. If I Google Liquibase—which, you know, I don't know how typing addresses in browsers works anymore because search engines are so fast—I just type in Liquibase. And the first thing it spits me out to is liquibase.org, which is the Community open-source version. And there's a link there to the Pro paid version and whatnot. And I was just scrolling idly through the comparison chart to see, “Oh, so ‘Community' is just code for shitty and you're holding back advanced features.” But it really doesn't look that way. What's the deal here?Robert: Oh, no. So, Liquibase open-source project started in 2006 and Liquibase the company, the commercial entity, started after that, 2012; 2014, first deal. And so, for—Nathan Voxland started this, and Nathan was struggling. He was working at a company, and he had to have his application—of course—you know, early 2000s, J2EE—support SQL Server and Oracle and he was struggling with it. And so he open-sourced it and added more and more databases.Certainly, as open-source databases grew, obviously he added those: MySQL, Postgres. But we're never going to undo that stuff. There's rollback for free in Liquibase, we're not going to be [laugh] we're not going to be jerks and either A) pull features out or, B) even worse, make Stephen O'Grady's life awful by changing the license [laugh] so he has to write about it. He loves writing about open-source license changes. We're Apache 2.0 and so you can do whatever you want with it.And we believe that the things that make sense for a paying customer, which is database-specific objects, that makes sense. But Liquibase Community, the open-source stuff, that is built so you can go to any database. So, if you have a change log that runs against Oracle, it should be able to run against SQL Server, or MySQL, or Postgres, as long as you don't use platform-specific data types and those sorts of things. And so that's what Community is about. Community is about being able to support any database with the same change log. Pro is about helping you get to that next level of DevOps Nirvana, of reaching those four metrics that Dr. Forsgren tells us are really important.Corey: Oh, yes. You can argue with Nicole Forsgren, but then you're wrong. So, why would you ever do that?Robert: Yeah. Yeah. [laugh]. It's just—it's a sucker's bet. Don't do it. There's a reason why she's got a PhD in CS.Corey: She has been a recurring guest on this show, and I only wish she would come back more often. You and I are fun to talk to, don't get me wrong. We want unbridled intellect that is couched in just a scintillating wit, and someone is great to talk to. Sorry, we're both outclassed.Robert: Yeah, you get entertained with us; you learn with her.Corey: Exactly. And you're still entertained while doing it is the best part.Robert: [laugh]. That's the difference between Community and Pro. Look, at the end of the day, if you're an individual developer just trying to solve a problem and get done and away from the computer and go spend time with your friends and family, yeah, go use Liquibase Community. If it's something that you think can improve the rest of the organization by teaming up and taking advantage of the collaboration features? Yes, sure, let us know. We're happy to help.Corey: Now, if people wanted to become an attorney, but law school was too expensive, out of reach, too much time, et cetera, but they did have a Twitter account, very often, they'll find that they can scratch that itch by arguing online about open-source licenses. So, I want to be very clear—because those people are odious when they email me—that you are licensed under the Apache License. That is a bonafide OSI approved open-source license. It is not everyone except big cloud companies, or service providers, which basically are people dancing around—they mean Amazon. So, let's be clear. One, are you worried about Amazon launching a competitive service with a dumb name? And/or have you really been validated as a product if AWS hasn't attempted and failed to launch a competitor?Robert: [laugh]. Well, I mean, we do have a very large corporation that has embedded Liquibase into one of their flagship products, and that is Oracle. They have embedded Liquibase in SQLcl. We're tickled pink because that means that, one, yes, it does validate Liquibase is the right way to do it, but it also means more people are getting help. Now, for Oracle users, if you're just an Oracle shop, great, have fun. We think it's a great solution. But there's not a lot of those.And so we believe that if you have Liquibase, whether it's open-source or the Pro version, then you're going to be able to support all the databases, and I think that's more important than being tied to a single cloud. Also—this is just my opinion and take it for what it's worth—but if Amazon wanted to do this, well, they're not the only game in town. So, somebody else is going to want to do it, too. And, you know, I would argue even with Amazon's backing that Liquibase is a little stronger brand than anything they would come out with.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense. Corey: So, I want to call out though, that on some level, they have already competed with you because one of database that you do not support is DynamoDB. Let's ignore the Route 53 stuff because, okay. But the reason behind that, having worked with it myself, is that, “Oh, how do you do a schema change in DynamoDB?” The answer is that you don't because it doesn't do schemas for one—it is schemaless, which is kind of the point of it—as well as oh, you want to change the primary, or the partition, or the sort key index? Great. You need a new table because those things are immutable.So, they've solved this Gordian Knot just like Alexander the Great did by cutting through it. Like, “Oh, how do you wind up doing this?” “You don't do this. The end.” And that is certainly an approach, but there are scenarios where those were first, NoSQL is not a acceptable answer for some workloads.I know Rick [Horahan 00:26:16] is going to yell at me for that as soon as he hears me, but okay. But there are some for which a relational database is kind of a thing, and you need that. So, Dynamo isn't fit for everything. But there are other workloads where, okay, I'm going to just switch over. I'm going to basically dump all the data and add it to a new table. I can't necessarily afford to do that with anything less than maybe, you know, 20 milliseconds of downtime between table one and table two. And they're obnoxious and difficult ways to do it, but for everything else, you do kind of need to make ALTER TABLE changes from time to time as you go through the build and release process.Robert: Yeah. Well, we certainly have plans for DynamoDB support. We are working our way through all the NoSQLs. Started with Mongo, and—Corey: Well, back that out a second then for me because there's something I'm clearly not grasping because it's my understanding, DynamoDB is schemaless. You can put whatever you want into various arbitrary fields. How would Liquibase work with something like that?Robert: Well, that's something I struggled with. I had the same question. Like, “Dude, really, we're a schema change tool. Why would we work with a schemaless database?” And so what happened was a soon-to-be friend of ours in Europe had reached out to me and said, “I built an extension for MongoDB in Liquibase. Can we open-source this, and can y'all take care of the care and feeding of this?” And I said, “Absolutely. What does it do?” [laugh].And so I looked at it and it turns out that it focuses on collections and generating data for test. So, you're right about schemaless because these are just documents and we're not going to go through every single document and change the structure, we're just going to have the application create a new doc and the new format. Maybe there's a conversion log logic built into the app, who knows. But it's the database professionals that have to apply these collections—you know, indices; that's what they call them in Mongo-land: collections. And so being able to apply these across all environments—dev, test, production—and have consistency, that's important.Now, what was really interesting is that this came from MasterCard. So, this engineer had a consulting business and worked for MasterCard. And they had a problem, and they said, “Hey, can you fix this with Liquibase?” And he said, “Sure, no problem.” And he built it.So, that's why if you go to the MongoDB—the liquibase-mongodb repository in our Liquibase org, you'll see that MasterCard has the copyright on all that code. Still Apache 2.0. But for me, that was the validation we needed to start expanding to other things: Dynamo, Couch. And same—Corey: Oh, yeah. For a lot of contributors, there's a contributor license process you can go through, assign copyright. For everything else, there's MasterCard.Robert: Yeah. Well, we don't do that. Look, you know, we certainly have a code of conduct with our community, but we don't have a signing copyright and that kind of stuff. Because that's baked into Apache 2.0. So, why would I want to take somebody's ability to get credit and magical internet points and increase the rep by taking that away? That's just rude.Corey: The problem I keep smacking myself into is just looking at how the entire database space across the board goes, it feels like it's built on lock-in, it's built on it is super finicky to work with, and it generally feels like, okay, great. You take something like Postgres-squeal or whatever it is you want to run your database on, yeah, you could theoretically move it a bunch of other places, but moving databases is really hard. Back when I was at my last, “Real job,” quote-unquote, years ago, we were late to the game; we migrated the entire site from EC2 Classic into a VPC, and the biggest pain in the ass with all of that was the RDS instance. Because we had to quiesce the database so it would stop taking writes; we would then do snapshot it, shut it down, and then restore a new database from that RDS snapshot.How long does it take, at least in those days? That is left as an experiment for the reader. So, we booked a four hour maintenance window under the fear that would not be enough. It completed in 45 minutes. So okay, there's that. Sparked the thing up and everything else was tested and good to go. And yay. Okay.It took a tremendous amount of planning, a tremendous amount of work, and that wasn't moving it very far. It is the only time I've done a late-night deploy, where not a single thing went wrong. Until I was on the way home and the Uber driver sideswiped a city vehicle. So, there we go—Robert: [laugh].Corey: —that's the one. But everything else was flawless on this because we planned these things out. But imagine moving to a different provider. Oh, forget it. Or imagine moving to a different database engine? That's good. Tell another one.Robert: Well, those are the problems that we want our database professionals to solve. We do not want them to be like janitors at an elementary school, cleaning up developer throw-up with sawdust. The issue that you're describing, that's a one time event. This is something that doesn't happen very often. You need hands on the keyboard, you want people there to look for problems.If you can take these database releases away from those folks and automate them safely—you can have safety and speed—then that frees up their time to do these other herculean tasks, these other feats of strength that they're far better at. There is no silver bullet panacea for database issues. All we're trying to do is take about 70% of DBAs time and free it up to do the fun stuff that you described. There are people that really enjoy that, and we want to free up their time so they can do that. Moving to another platform, going from the data center to the cloud, these sorts of things, this is what we want a human on; we don't want them updating a column three times in a row because dev couldn't get it right. Let's just give them the keys and make sure they stay in their lane.Corey: There's something glorious about being able to do that. I wish that there were more commonly appreciated ways of addressing those pains, rather than, “Oh, we're going to sell you something big and enterprise-y and it's going to add a bunch of process and not work out super well for you.” You integrate with existing CI/CD systems reasonably well, as best I can tell because the nice thing about CI/CD—and by nice I mean awful—is that there is no consensus. Every pipeline you see, in a release engineering process inherently becomes this beautiful bespoke unicorn.Robert: Mm-hm. Yeah. And we have to. We have to integrate with whatever CI/CD they have in place. And we do not want customers to just run Liquibase by itself. We want them to integrate it with whatever is driving that application deployment.We're Switzerland when it comes to databases, and CI/CD. And I certainly have my favorite of those, and it's primarily based on who bought me drinks at the last conference, but we cannot go into somebody's house and start rearranging the furniture. That's just rude. If they're deploying the app a certain way, what we tell that customer is, “Hey, we're just going to have that CI/CD tool call Liquibase to update the database. This should be an atomic unit of deployment.” And it should be hidden from the person that pushes that shiny button or the automation that does it.Corey: I wish that one day that you could automate all of the button pushing, but the thing that always annoyed me in release engineering was the, “Oh, and here's where we stop to have a human press the button.” And I get it. That stuff's scary for some folks, but at the same time, this is the nature of reality. So, you're not going to be able to technology your way around people. At least not successfully and not for very long.Robert: It's about trust. You have to earn that database professional's trust because if something goes wrong, blaming Liquibase doesn't go very far. In that company, they're going to want a person [laugh] who has a badge to—with a throat to choke. And so I've seen this pattern over and over again.And this happened at our first customer. Major, major, big, big, big bank, and this was on the consumer side. They were doing their first production push, and they wanted us ready. Not on the call, but ready if there was an issue they needed to escalate and get us to help them out. And so my VP of Engineering and me, we took it. Great. Got VP of engineering and CTO. Right on.And so Kevin and I, we stayed home, stayed sober [laugh], you know—a lot of places to party in Austin; we fought that temptation—and so we stayed and I'm texting with Kevin, back and forth. “Did you get a call?” “No, I didn't get a call.” It was Friday night. Saturday rolls around. Sunday. “Did you get a—what's going on?” [laugh].Monday, we're like, “Hey. Everything, okay? Did you push to the next weekend?” They're like, “Oh, no. We did. It went great. We forgot to tell you.” [laugh]. But here's what happened. The DBAs push the Liquibase ‘make it go' button, and then they said, “Uh-Oh.” And we're like, “What do you mean, uh-oh?” They said, “Well, something went wrong.” “Well, what went wrong?” “Well, it was too fast.” [laugh]. Something—no way. And so they went through the whole thing—Corey: That was my downtime when I supposed to be compiling.Robert: Yeah. So, they went through the whole thing to verify every single change set. Okay, so that was weekend one. And then they go to weekend two, they do it the same thing. All right, all right. Building trust.By week four, they called a meeting with the release team. And they said, “Hey, process change. We're no longer going to be on these calls. You are going to push the Liquibase button. Now, if you want to integrate it with your CI/CD, go right ahead, but that's not my problem.” Dev—or, the release team is tier one; dev is tier two; we—DBAs—are tier three support, but we'll call you because we'll know something went wrong. And to this day, it's all automated.And so you have to earn trust to get people to give that up. Once they have trust and you really—it's based on empathy. You have to understand how terrible [laugh] they are sometimes treated, and to actively take care of them, realize the problems they're struggling with, and when you earn that trust, then and only then will they allow automation. But it's hard, but it's something you got to do.Corey: You mentioned something a minute ago that I want to focus on a little bit more closely, specifically that you're in Austin. Seems like that's a popular choice lately. You've got companies that are relocating their headquarters there, presumably for tax purposes. Oracle's there, Tesla's there. Great. I mean, from my perspective, terrific because it gets a number of notably annoying CEOs out of my backyard. But what's going on? Why is Austin on this meteoric rise and how'd it get there?Robert: Well, a lot of folks—overnight success, 40 years in the making, I guess. But what a lot of people don't realize is that, one, we had a pretty vibrant tech hub prior to all this. It all started with MCC, Microcomputer Consortium, which in the '80s, we were afraid of the Japanese taking over and so we decided to get a bunch of companies together, and Admiral Bobby Inman who was director planted it in Austin. And that's where it started. You certainly have other folks that have a huge impact, obviously, Michael Dell, Austin Ventures, a whole host of folks that have really leaned in on tech in Austin, but it actually started before that.So, there was a time where Willie Nelson was in Nashville and was just fed up with RCA Records. They would not release his albums because he wanted to change his sound. And so he had some nice friends at Atlantic Records that said, “Willie, we got this. Go to New York, use our studio, cut an album, we'll fix it up.” And so he cut an album called Shotgun Willie, famous for having “Whiskey River” which is what he uses to open and close every show.But that album sucked as far as sales. It's a good album, I like it. But it didn't sell except for one place in America: in Austin, Texas. It sold more copies in Austin than anywhere else. And so Willie was like, “I need to go check this out.”And so he shows up in Austin and sees a bunch of rednecks and hippies hanging out together, really geeking out on music. It was a great vibe. And then he calls, you know, Kris, and Waylon, and Merle, and say, “Come on down.” And so what happened here was a bunch of people really wanted to geek out on this new type of country music, outlaw country. And it started a pattern where people just geek out on stuff they really like.So, same thing with Austin film. You got Robert Rodriguez, you got Richard Linklater, and Slackers, his first movie, that's why I moved to Austin. And I got a job at Les Amis—a coffee shop that's closed—because it had three scenes in that. There was a whole scene of people that just really wanted to make different types of films. And we see that with software, we see that with film, we see it with fashion.And it just seems that Austin is the place where if you're really into something, you're going to find somebody here that really wants to get into it with you, whether it's board gaming, D&D, noise punk, whatever. And that's really comforting. I think it's the community that's just welcoming. And I just hope that we can continue that creativity, that sense of community, and that we don't have large corporations that are coming in and just taking from the system. I hope they inject more.I think Oracle's done a really good job; their new headquarters is gorgeous, they've done some really good things with the city, doing a land swap, I think it was forty acres for nine acres. They coughed up forty for nine. And it was nine acres the city wasn't even using. Great. So, I think they're being good citizens. I think Tesla's been pretty cool with building that factory where it is. I hope more come. I hope they catch what is ever in the water and the breakfast tacos in Austin.Corey: [laugh]. I certainly look forward to this pandemic ending; I can come over and find out for myself. I'm looking forward to it. I always enjoyed my time there, I just wish I got to spend more of it.Robert: How many folks from Duckbill Group are in Austin now?Corey: One at the moment. Tim Banks. And the challenge, of course, is that if you look across the board, there really aren't that many places that have more than one employee. For example, our operations person, Megan, is here in San Francisco and so is Jesse DeRose, our manager of cloud economics. But my business partner is in Portland; we have people scattered all over the country.It's kind of fun having a fully-distributed company. We started this way, back when that was easy. And because all right, travel is easy; we'll just go and visit whenever we need to. But there's no central office, which I think is sort of the dangerous part of full remote because then you have this idea of second-class citizens hanging out in one part of the country and then they go out to lunch together and that's where the real decisions get made. And then you get caught up to speed. It definitely fosters a writing culture.Robert: Yeah. When we went to remote work, our lease was up. We just didn't renew. And now we have expanded hiring outside of Austin, we have folks in the Ukraine, Poland, Brazil, more and more coming. We even have folks that are moving out of Austin to places like Minnesota and Virginia, moving back home where their family is located.And that is wonderful. But we are getting together as a company in January. We're also going to, instead of having an office, we're calling it a ‘Liquibase Lounge.' So, there's a number of retail places that didn't survive, and so we're going to take one of those spots and just make a little hangout place so that people can come in. And we also want to open it up for the community as well.But it's very important—and we learned this from our friends at GitLab and their culture. We really studied how they do it, how they've been successful, and it is an awareness of those lunch meetings where the decisions are made. And it is saying, “Nope, this is great we've had this conversation. We need to have this conversation again. Let's bring other people in.” And that's how we're doing at Liquibase, and so far it seems to work.Corey: I'm looking forward to seeing what happens, once this whole pandemic ends, and how things continue to thrive. We're long past due for a startup center that isn't San Francisco. The whole thing is based on the idea of disruption. “Oh, we're disruptive.” “Yes, we're so disruptive, we've taken a job that can be done from literally anywhere with internet access and created a land crunch in eight square miles, located in an earthquake zone.” Genius, simply genius.Robert: It's a shame that we had to have such a tragedy to happen to fix that.Corey: Isn't that the truth?Robert: It really is. But the toothpaste is out of the tube. You ain't putting that back in. But my bet on the next Tech Hub: Kansas City. That town is cool, it has one hundred percent Google Fiber all throughout, great university. Kauffman Fellows, I believe, is based there, so VC folks are trained there. I believe so; I hope I'm not wrong with that. I know Kauffman Foundation is there. But look, there's something happening in that town. And so if you're a buy low, sell high kind of person, come check us out in Austin. I'm not trying to dissuade anybody from moving to Austin; I'm not one of those people. But if the housing prices [laugh] you don't like them, check out Kansas City, and get that two-gig fiber for peanuts. Well, $75 worth of peanuts.Corey: Robert, I want to thank you for taking the time to speak with me so extensively about Liquibase, about how awesome RedMonk is, about Austin and so many other topics. If people want to learn more, where can they find you?Robert: Well, I think the best place to find us right now is in AWS Marketplace. So—Corey: Now, hand on a second. When you say the best place for anything being the AWS Marketplace, I'm naturally a little suspicious. Tell me more.Robert: [laugh]. Well, best is, you know, it's—[laugh].Corey: It is a place that is there and people can find you through it. All right, then.Robert: I have a list. I have a list. But the first one I'm going to mention is AWS Marketplace. And so that's a really easy way, especially if you're taking advantage of the EDP, Enterprise Discount Program. That's helpful. Burn down those dollars, get a discount, et cetera, et cetera. Now, of course, you can go to liquibase.com, download a trial. Or you can find us on Github, github.com/liquibase. Of course, talking smack to us on Twitter is always appreciated.Corey: And we will, of course, include links to that in the [show notes 00:46:37]. Robert Reeves, CTO and co-founder of Liquibase. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment complaining about how Liquibase doesn't support your database engine of choice, which will quickly be rendered obsolete by the open-source community.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Find and Eject the Wizards with Danielle Baskin

Screaming in the Cloud

Play Episode Listen Later Dec 15, 2021 35:45


About DanielleDanielle Baskin is a serial entrepreneur and multimedia artist whose work has been featured in The New York Times, The Guardian, NPR, The New Yorker, WSJ, and more. She's also the CEO of Dialup, a globally acclaimed voice-chat app.Links: Dialup: https://dialup.com Twitter: https://twitter.com/djbaskin Cofounder Quest: https://cofounder.quest Personal Website: https://daniellebaskin.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com. Corey: You know how Git works right?Announcer: Sorta, kinda, not really. Please ask someone else.Corey: That's all of us. Git is how we build things, and Netlify is one of the best ways I've found to build those things quickly for the web. Netlify's Git-based workflows mean you don't have to play slap-and-tickle with integrating arcane nonsense and web hooks, which are themselves about as well understood as Git. Give them a try and see what folks ranging from my fake Twitter for Pets startup, to global Fortune 2000 companies are raving about. If you end up talking to them—because you don't have to; they get why self-service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y dot com.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. It's always fun when I get the opportunity to talk to people whose work inspires me, and makes me reflect more deeply upon how I go about doing things in various ways. Now, for folks who have been following my journey for a while, it's pretty clear that humor plays a big part in this, but that is not something that I usually talk about with respect to whose humor inspires me.Today that's going to change a little bit. My guest is Danielle Baskin, who among so many other things is the CEO of a company called Dialup, but more notably is renowned for pulling a bunch of—I don't know if we'd call them pranks. I don't know if we would call them performance art. I don't know if we would call them shitposting in real life, but they are all amazing. Danielle, thank you so much for joining. How do you describe what it is that you do?Danielle: Thanks for having me. Yeah, I've used a few different terms. I've called it situation design. I've called it serious jokes. I have called what I do business art, but all the things you said, shitposting IRL, that's part of it too.Corey: It's been an absolute pleasure to just watch what you've done since I first became aware of you, which our mutual friend, Chloe Condon first pointed me in your general direction with, “Hey, Corey, you think you're funny? You should watch what Danielle is doing.” That's not how she framed it, but that's what I took from it because I'm incredibly egotistical, which is now basically a brand slash core personality trait. There you have it.And I encountered you for the first time in person—I believe only time to date—at I believe it was Oracle OpenWorld on the expo floor. She had been talking about you a couple of days before, and I saw someone who could only be you because you were dressed as a seer to be at Oracle OpenWorld. The joke should be clear to folks but we'll explain it later for the folks who are—might need to replay that a bit. I staggered up to you with, “Hey, are you Chloe's friend?”Let me give listeners here some advice through counterexample. Don't do that. It makes you look like a sketchy person who has no clue how social graces work. No one has any context and as soon as you said, “No,” I realized, “Oh, I came across as a loon.” I am going to say, “Never mind. My mistake,” and walk away like a sensible person will after bungling an introduction like that. I'm not usually that inartful about these things. I don't know what the hell happened, but it happens often when we meet people that we consider celebrities, and sorry, for some of us that's you.Danielle: [laugh] yeah, also in fairness to you I was probably fully immersed in character being my wizard self, and so I was not there to, you know, be pulled back to reality. For some context, I was at Oracle OpenWorld because I made a thing called same exact name, oracleopenworld.org, but it's a divination conference for oracles, for fortune-tellers, for wizards, for seers, and it happened at the exact same place in time, so there was a whole crew of people dressed up with capes, and robes, and tall pointy hats doing tarot readings and practicing our divination skills.Corey: Now, I could wind up applying about two dozen different adjectives to Oracle, but playful is absolutely not one of them. I would not ever accuse Oracle, or frankly any large company of that scale of having anything even remotely resembling a sense of humor. As someone who does have to factor in the not that remote possibility of getting kicked out of events that I attend, how do you handle that and not find yourself arrested?Danielle: Oh, we were kicked out every single time.Corey: Oh, good good good.Danielle: I've done this for four years. The first year we were kicked out just because we didn't have badges. I made up our own conference lanyard; of course, there's security issues with that. We were pushed out onto the sidewalk, but I wanted to be inside the conference and closer to the building.The next year I did a two-layer conference badge, so I put the real one underneath the fake one so that if security went up to us we had the right to be there. What sort of happened—so, like, the first year we got kicked out was because we were all distributed; maybe there was like 20 of us. Sometimes we were together. Sometimes we were having our own adventures. My friend Brian decided do a séance for the Deloitte team.Corey: Well, that's Deloitte-ful. Tell me more.Danielle: [laugh]. Brian has never done a séance before, but he is a good improv actor and also a spiritual person, so this is, like, perfect for him. As the Deloitte team if they wanted to do a séance they were, like, sure because I think they didn't have anything going—I mean, people are bored at this conference.Corey: Oh, of course, they are.Danielle: Especially if your boss flew you there to stand at your booth and you've been saying the same thing over and over again; you're looking for something interesting. So, he grabs the pillows from a lounge area and little tea light candles and makes a whole circle so that the team can sit down.He's wearing a bright rainbow cape and he stands in the middle and he could have a booming voice if he wants to. So, he just starts riffing and going—he just goes into séance mode, and this was enough to trigger security noticing that something really weird was happening. And when they went—Corey: They come over and say, “What the hell is this?” The answer was “Kubernetes.”Danielle: I had said everyone can blame—if you get in trouble just blame me just say, “I'm doing this with my friend, Danielle,” and have them talk to me. I wanted more people to come and be wizards. I don't want them to worry about it, so I will take all of the issues on me. He said that he should talk to his manager, Danielle, or I don't know.He said something that made it seem we were all part of a company. Which then makes it seem like our whole project was secret guerilla marketing for something. And we didn't pay for booth. We were not selling anything. We were just trolling. Or not troll—I mean, we were having our own divination summit. We were genuinely—Corey: You were virally marketing is the right answer and from my perspective—Danielle: Yeah, no, I wasn't doing viral marketing. They think anything that's unusual and getting people's attention has the ultimate goal of selling something, which it's not a philosophy I live by.Corey: No, it feels like the weird counter-intuitive thing here is the way to get the blessing of everyone from this would've—the only step you missed was charging Deloitte for doing it at their booth because it attracts attention.Danielle: Oh, sure. Oracle should have been paying us a lot of money for entertaining people. Actually, genuinely I had some real heart-to-heart conversations with people who wanted to have a tarot reading about how should they talk to their boss about not listening to them. This is something magical that happens when you are dressed up in costume and you are acting really weird people feel they can say anything because you're acting way more unusual than them, so it sort of takes away people's barriers. So, people are very honest with me about their situation.People had questions about their family. Anyway, I was in the middle of a heart-to-heart tarot reading, and security at Oracle was alerted to find anyone with a cape. Find the wizards and kick them out because they didn't pay to be here. There's some weird marketing thing happen.Corey: “Find and eject the wizards,” is probably the most surreal thing that they have been told that year.Danielle: Oh, yeah. And they didn't know why. The message why I did not transmit to all the security, but they were just told to find us. Two guards with their walkie-talkies in their uniforms went up to me and they had to escort me off the premises. Which means we had to walk through the conference together and I asked them, “Why?” They're like, “We don't know. We were just told to find you.”Corey: Imagine them trying to find you stopping and asking people, “Excuse me, have you seen the wizard?”Danielle: Exactly.Corey: It is hard to be taken seriously when asking questions like that.Danielle: Totally, totally. So yeah, unfortunately, we had to leave and that has consistently happened because I've done it four times. The final year I went, there was a message before the event even started that you're not allowed to wear a cape.Corey: The fact that you can have actual changes made to company policy for large-scale, incredibly expensive events like that is a sign that you've made it.Danielle: It doesn't even point to any particular incident. Yeah, it's cool to have this sort of lore. When I asked in the last year I went, “I asked why can't we wear a cape?” And one of the event organizer security, I don't know what her role was. She said, “There was an incident the previous year.” Which she was talking about me and my friends.Corey: Of course, but that is the best part of it.Danielle: It's just lore than something once happened with these, like, dark spirits that tried to mess up the Oracle conference with their magic.Corey: Times change and events evolve. Years ago I attended an AWS Summit with a large protest sign that said on it AMI has three syllables, and it got a bit of an eyebrow raise from people at the door, but okay, great. Then people started protesting those events for one of the very many reasons people have to protest Amazon, and they keep piling more on that pile all the time which is neither here nor there.I realized, okay, I can't do that anymore because regardless of what the sign says I will get tackled at the door for trying to bring something like that in, and I don't try and actively disrupt keynotes. So okay, it's time to move on and not get myself viewed through certain lenses that are unhelpful, but it's always a question of moving on and try to top what I did previous years. Weren't you also at Dreamforce wearing pajamas?Danielle: I did a few things at Dreamforce. One year I literally set up a tent. They spend millions of dollars on beautiful fake trees and rocks, and also Dreamforce gets taken over every time the event occurs. I did a few things. I thought I should make it seem like this is real nature so I brought camping gear and a tent and just brought a hiking backpack in.Set it up in the middle of the conference floor laying by the waterfall, but there were people in suits networking around me that did not ask me any questions. I just stayed in the tent, but then I decided to list it on Airbnb. So, inside my tent, I was making an Airbnb listing telling people that they could stay at Dreamforce and explore the beautiful nature there, but it took an hour-and-a-half to get kicked out.Corey: The emails that you must have back and forth with places like Airbnb's customer support line and the rest have got to be legendary at this point.Danielle: [laugh] I get interesting cease-and-desists. I wish there was more dialogue. With Airbnb I just got my listing taken down and I couldn't talk to a human, and even when I got kicked out of Dreamforce they wanted me to leave immediately. I totally snuck in; I didn't have a badge or anything. So, I guess they're in the right for that. The second year at Dreamforce I wore a ghillie suit so I hid. So, I stayed a little bit after the conference ended by hiding as a bush.Corey: That is both amazing and probably terrifying for the worker that encountered you while trying to clean up.Danielle: Oh, I mean often employees—like it depends. Some people find my pranks really delightful because it shakes up their day. Security guards also find this amusing. There's some type of organizer that absolutely hates my pranks.Corey: There's something to be said for self-selecting your own audience. One question that I—sure you get; if I get it I know you get it—where it's difficult for people to sometimes draw the line between the fun whimsical things that you do as pranks and the actual things that you do. A great example of this is something you've been doing for, I think, four years now, the decruiter.Danielle: Yeah. The decruiter a service that's the opposite of a recruiter so it is—Corey: At the first re:Invent AWS had a slide that was apparently he made the night before or something and they misspelled security as decurity. From that perspective, what's a decruiter?Danielle: Yes, I love decurity as a way to talk about infiltrating a space, like, “No I'm a decurity officer.” Yeah, decruiter is basically a service where you talk to us to find out if you should quit your job. Instead of finding out if you should work at a place or figuring out what opportunities there are, we discuss the unemployed life—or the inbet—like, being self-employed, between jobs, switching careers, it's a whole spectrum but there's a few recruiters and we're all like very experienced not having an employer or working for a company. And so, we ask people about how would you spend your free time. What's your financial situation? Are you able to afford leaving? It gets pretty personal, but it's highly specific therapy, but we also don't have a high acceptance rate. I've only decruited like 15% people that I've talked to.Corey: Most of them realize that, oh, there's a lot of things I would have to do if I didn't have a job and I'm just going to stay where I am?Danielle: Yeah. Well, I think a lot of people think that as soon as they leave their job a lot of other things in their life will magically transform, or they'll finally be able to do their creative project they've always wanted to do. This is true some percentage of the time, but I always encourage people to do things outside of work and not seek in their whole fulfillment through their job.There's plenty of time where you can explore other ideas and even overlap them to make sure that like when you quit you have things lined up. A lot of people don't know how to answer, “If you suddenly left tomorrow and could just float for three months, what would you do?” If people give me a good answer—and this is similar to an actual job interview I was like, “Why are you excited about working this company?”If people give me a good answer, that's a conversation. A lot of people have no idea, but they're just stuck in a situation where there's things they could do in their outside of work life that would make them feel happier. That's why it's sort of like therapy, but there's a lot of internal company issues that I talk about. A common reason that people want to leave is that they love their role, they love the company's mission, but they do not like their manager, but their manager is really good friends with the CEO and they absolutely can't say anything. This is so common.Corey: They always say people they'll quit jobs they quit managers and there is something to be said for that.Danielle: Yes, it's scary for people to speak up or who do you write a letter to? How do you secretly talk with your team about it? Are you the only one feeling that way? Typically the people that are the most nervous about saying anything are kind of young either in their early 20s and they feel like they can't say anything.I encourage them to come up with a strategy for making change within their corporation but sometimes it's not worth it. If there's tons of other opportunities for them it's not worth them fixing their company.Corey: It's also I think not incumbent upon people to fix their entire corporate culture unless they're at a somewhat higher executive level. That's a fun thing. The derecruiter.com we'll definitely throw a link to that in the [show notes 00:15:49] and I'll start driving people to it when they ask me for advice on these things. Then you decided, okay, that's fun.You're one of those people I feel has a bit of the same alignment that I do which is, why do one thing when I could do a bunch of things? And you decided, ah, you're going to do a startup. What is the best thing that you can do that really can capitalize on emerging cultural trends? That's right. Getting millennial to make phone calls to each other. Tell me about that story.Danielle: Yeah, and it's not just millennials, though I'm millennial. So, a lot of millennials use Dialup. I mean, Dialup started as a project where basically me and a friend set up a robocall between ourselves. So, like a bot would call our phones and if we would pick up we'd both be connected, but neither of us was actually calling each other. So, it was a way to just always be catching up with each other.So, many friends asked me if they could join the robocalls. That was sort of the seat of Dialup is getting serendipitous phone calls throughout the day that connect you to a person that you might know or might want to meet. Because there's overlap of interest or overlap of someone you know. It grew from me and 20 friends to now 31,000 people who are actively using it all over the world and these conversations can be really incredible.Sometimes people stay on the phone for four hours. People have flown out to meet each other. I get notes every day of how a call has impacted someone one. So, that's what I'm up to now, but I'm trying to do more interesting things with voice technology. I just like realized, oh, the voice as a medium it just transports you to other worlds. You have space to imagine.I mean, people listening to this podcast right now they're not seeing us, but they probably are imagining us, what our rooms look like, what we look like. They're imagining the stories that we're telling them without the distraction of video. I want to do more interesting things with intimate audio—not broadcast stuff. Not Clubhouse or Spaces or anything like that, but just more interesting ways to connect people in one-on-ones.Corey: Something I've noticed is that the voice has a power that text does not. It makes it easier to remember that there's a human on the other side of things. It is far easier for me to send off an incendiary tweet at someone than it is for me to call them up and then berate them, not really my style.The more three-dimensional someone becomes in various capacities and the higher bandwidth the communication takes on, I think the easier it is to remember that most people who don't work at Facebook wake up in the morning hoping to do a good job today. Extending empathy to the rest of the world, that's an important thing.Danielle: Yeah, for sure. It's incredible that humans can detect emotional qualities in a voice call. It's hard to describe why, but people can detect pauses and little mutters. You can sort of know when someone's laughing or when someone's listening even though you're missing all of the visual cues.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: Taking a glance at dialup.com, it appears to be a completely free service. You mentioned that it has 30,000 folks involved. Are you taking the VC model of we're going to get a whole bunch of users first and then figure out how to make money later? Sometimes it works super well. Other times it basically becomes Docker retold.Danielle: I've been thinking about this a lot and I swing back and forth. Right now Dialup is its own thing, connecting strangers. It's free though I do have some paying clients because I do serendipitous one-on-ones within organizations. I've got a secret B2B page, and so that is a little bit of revenue. Right now I'm trying to sort of expand beyond Dialup and make a new thing, in which case I am leaning more towards building a sustainable and profitable company rather than do the raise-VC-money-until-you-die model.Corey: I think it's long past time to disrupt the trope of starving artist. What about well-paid artist? It seems like that would inspire and empower people to create a lot more art when they're not worrying about freezing to death. To that end or presumably to that end you are in the process of looking for a co-founder in what is arguably the most Danielle Baskin possible way. How are you doing it?Danielle: Oh, yeah. I could have done a regular LinkedIn post linking to a Google Doc, but that is not my style, and as a self-employed person I can't reach out to old coworkers and be like, “Oh, you're on my team a few years ago. What are you up to now?” So, I'm sort of under-networked and I thought I should make a game that sort of explains what I'm doing, but have people discover the game in an interesting way. So, I bought a bunch of floppy discs—I have a floppy disc dealer outside of LA.Corey: For those who are not millennials and are in fact younger than that—and of course let's not forget Gen X, the Baby Boom Generation, the Silent Generation which I can only assume is comprised entirely of people who represent big companies from a PR point of view because they never comment on anything. What is a floppy disc for someone who was born in, I don't know, 2005?Danielle: Oh, a floppy disk is how you would run software on your computer.Corey: Yeah, a USB stick with no capacity you can wreck with a magnet.Danielle: Yes, it's like a flat wide USB stick, but it only contains—Corey: 1.44 megabytes on the three-and-a-half-inch version.Danielle: I think some of them then went up to 2.88.Corey: Ohh.Danielle: You can't even fit a picture—a modern picture. You could do a super low-resolution pixel art.Corey: This picture of grandma has a whopping eight pixels in it. Oh, okay, great. I guess.Danielle: Yeah. More complex software would be eight floppy disks that you have to insert disk A, insert disk B.Corey: Anti-piracy warnings in that day of ‘don't copy that floppy.' It was a seminal thing for a long time.Danielle: I have it in my game; it says ‘don't make illegal copies of this game.' My game is not literally on the floppy disc. All floppy discs come with pretty interesting artwork on the label. There's a little space for a sticker, and because I have hundreds of floppy disks, I sort of looked at—I had a ton of design inspiration.So, I made floppy discs in the aesthetic of the other ones that say Cofounder Quest—like it's this game—and it leads you to a website. I scattered these in strategic places around the bay area, and I also mailed some to people outside of the bay area. If you stumble across this in person or on the internet, it leads you to this adventure game that's around seven minutes to play.It really explains what I want to do with Dialup, and explains me, and explains my aesthetic, and the sort of playful experiences that I'm into without telling you. So, you get to really experience it. At the end, it basically leads you to a job description and tells you to reach out to me if you're interested.Corey: I was independent for years and I finally decided to take on a business partner. As it turns out, Mike Julian, who's the CEO of The Duckbill Group and I go back ten years, he's my best friend. I kept correcting him. He introduced me as his friend. I said, “No, Mike, your best friend.” Then I got him on audio at one point saying, “Oh, Corey Quinn? He's my best friend.” I have that on my soundboard and I play it every time he gets uppity. That's the sort of nonsense it's important in a co-founder relationship. It is a marriage in some respects.Danielle: Oh, for sure.Corey: It's a business entity. Each one of you can destroy the other financially in different ways. You have to have shared values. The idea of speed-dating your way through finding some random co-founder as a job application, on some level, has always struck me as a little dissonant. I like the approach you're taking of this is who I am and how I go about things. If this aligns then we should talk, and if you don't like this you're not going to like any of the rest of this.Danielle: For sure. I'm definitely self-selecting with who would actually reach out after playing. I also understand. I'm not going to find a co-founder in a few weeks. I'm just starting conversations with people and then seeing who I should continue talking to or seeing if we could do a mini-project together.Yeah, it's weird. It's a very intense relationship. That's why people do end up becoming co-founders with someone that they already know who's a friend. It's possible I already know my co-founder and they've been in front of me this whole time. I think these sorts of moments happen, but I also think that it's cool to totally expand your network and meet someone who maybe has an overlap in spirit, but is someone that you would've never otherwise met. That there could be this great overlap or convergence there. I wanted to cast a very wide net with who this would reach, but it's still going to be a multi-month-long process or longer.Corey: It's not these one-off projects that are the most interesting part to me. It is the sheer variety and consistency of this. During the pandemic I believe you wound up having the verified checkmark badges for houses and fill out this form if you want one and for folks in San Francisco. Absolutely, of course, I filled that out. I read a fairly bad take news article on it of a bunch of people fell for this prank.No, absolutely not. If people are familiar with your work then they know exactly what they're getting into with something like this and you support the kinds of things you want to see more of in the world. I didn't fall for anything. I wanted to see where it led and that's how I feel on everything you do.Danielle: Yeah, you appreciated the joke.Corey: Yeah.Danielle: Yeah, I think people who are familiar with my work understand that I take jokes very seriously. So, it's not simply—like, usually it's not just a website that's like, huh, this was a trick. It's more of an ongoing theater piece. So, I actually did go through all of the applicants for the Blue Check Homes. Oh, for some context, I made a website where you could apply to have a blue verified badge and a plaster crest put on your house if you are a dignified authentic person that lives in the house.So, I'm interviewing—I narrowed it down to 50 people from all the applicants and I'm going through and interviewing people with a committee. I'm recording all of the interviews because I think this will make an interesting mini-documentary. I'm actually making one in installing one, but I'm documenting all of it.When I started it—for a lot of projects I don't have the ending planned yet. I like the sort of joke to unfold on the internet in real-time, and then figure out what the next thing I should do from there is and continue the project in a sort of curious exploratory mindset as opposed to just saying, “All right, the joke is done.”Corey: What is your process for coming up with this stuff? Because for me the most intimidating thing I ever see in the course of a week is not the inevitable cease and desist I get from every large cloud company for everything I do. Rather an empty page where it's all right time for me to write a humorous blog post, or start drafting the bones of a Twitter thread, or start writing my resignation and if I don't come with an idea by the end of it, I'll submit it. Where does the creative process start from with you?Danielle: Yeah. I rarely have creative brainstorming sessions. I'm a person who thinks of a million bad ideas and then there's one good one. My mind leaps to a ton of ideas. I rarely write down ideas. I don't do any sort of—you might imagine I'm in a room of whiteboards and post-it notes, workshopping things and doing creative brainstorm sessions, but I don't.I think I act upon the things that I feel just extremely excited about and feel like I must do this immediately. It's hard to explain, but with a lot of my ideas, I just feel this surge of energy. I have to do this because no one else will do it and it's funny at this moment. If I don't feel that way I kind of don't do anything and see if the idea keeps reemerging. With a lot of ideas I may be thought of it a year ago and it just kept resurfacing, but I don't really force myself to churn out creative projects if that makes sense. People have told me that my work reminds them of Mischief. It's like as a company that puts out a prank on a Tuesday every two weeks.Corey: Not familiar with them, but there have been a whole bunch of flash mob groups, and other folks who affected just wind up being professional pranksters, which I love the concept.Danielle: Yeah, yeah, for sure. I do churn out a lot of pranks and I even have my own prank calendar. I'm not strict with my own deadlines and I also think timing is important. So, you might think of a good idea, but then it's just the spirit of the zeitgeist doesn't want you to do it that week. I improvise the things that I want to launch. I mostly do things that I just feel are rich in something I could explore.Like, with Cofounder Quest I was always on the fence about it because it feels to me annoying to tell people you're trying to hire someone or to put yourself out there and be pitching your startup. So, I was kind of nervous about that, but I also thought if I leave a floppy disk in the park, and then put a picture on the internet it'll lead to something—there's something that it will lead to.It might lead to finding a co-founder. It might lead to meeting interesting people, but also I've never built an interactive game with audio and so I was interested in learning that, but yeah, I tend to land on ideas that I think are rich in terms of things I could learn. Things that I could turn into more immersive theater and things that keep resurfacing as opposed to keeping myself on a strict schedule of creative ideas if that makes sense.Corey: It makes a lot of sense. It's one of those things that it is not commonly understood for those of us who came up in the nose of the grindstone 40 hours a week, have a work ethic. Even if you're not busy look busy. Sometimes work looks a lot more like getting up and going to a coffee shop and meeting some stranger from the internet than it does sitting down churning out code.Danielle: For sure. I think that it is important to continue being in conversations with people. I think good ideas emerge while you're in the middle of talking, and you realize your own limitations and ideas when you have to explain things to other people. While something you're very clear in your head as soon as there's a person you don't know and they ask you, “What are you working on?” You realize, oh, there's so many gaps. It made perfect sense to me, but there's a lot of gaps. So yeah, I think it's important to stay in dialogue and also have to explain yourself to new people instead of just sort of making ideas in a vacuum.Corey: I want to thank you for being so generous with your time and talking to me about all the various things you have going on. If people want to follow along and learn more about what you're up to, where can they find you?Danielle: I post a lot of my projects on Twitter. So, I'm @djbaskin. If you want to play Cofounder Quest, it's cofounder.quest. That is an actual domain. I also have a website daniellebaskin.com, which has a lot of my projects, many of which we didn't discuss. I also do, similar to Oracle OpenWorld, I like to host popup events that involve lots of people trolling. So, if you want to get involved in anything you see I'm always happy to bring more wizards on board.Corey: We will, of course, put links to that in the [show notes 00:31:10]. Danielle, thank you so much for taking the time to speak with me today.Danielle: Oh yeah, thanks for having me. It was great talking with you.Corey: Danielle Baskin, CEO of Dialup, and oh so very much more. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast please leave a five-star review on your podcast platform of choice, along with a long rambling comment applying to be the co-host of this podcast, viewing it of course as a podcasting call.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
MongoDB's Purposeful Application Data Platform with Sahir Azam

Screaming in the Cloud

Play Episode Listen Later Dec 14, 2021 35:01


About SahirSahir is responsible for product strategy across the MongoDB portfolio. He joined MongoDB in 2016 as SVP, Cloud Products & GTM to lead MongoDB's cloud products and go-to-market strategy ahead of the launch of Atlas and helped grow the cloud business from zero to over $150 million annually. Sahir joined MongoDB from Sumo Logic, an SaaS machine-data analytics company, where he managed platform, pricing, packaging and technology partnerships. Before Sumo Logic, Sahir was the Director of Cloud Management Strategy & Evangelism at VMware, where he launched VMware's first organically developed SaaS management product and helped grow the management tools business to over $1B in revenue. Earlier in his career, Sahir held a variety of technical and sales-focused roles at DynamicOps, BMC Software, and BladeLogic.Links:MongoDB: https://www.mongodb.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. Set up a meeting with a Redis expert during re:Invent, and you'll not only learn how you can become a Redis hero, but also have a chance to win some fun and exciting prizes. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous nonsense.  Corey: Are you building cloud applications with a distributed team? Check out Teleport, an open source identity-aware access proxy for cloud resources. Teleport provides secure access to anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps and databases. Teleport gives engineers superpowers! Get access to everything via single sign-on with multi-factor. List and see all SSH servers, kubernetes clusters or databases available to you. Get instant access to them all using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility and ensuring compliance. And best of all, Teleport is open source and a pleasure to use.Download Teleport at https://goteleport.com. That's goteleport.com. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. For the first in-person recording in ages we have a promoted guest joining us from MongoDB. Sahir Azam is the chief product officer. Thank you so much for joining me.Sahir: Thank you for having me, Corey. It's really exciting to be able to talk and actually meet in person.Corey: I know it feels a little scandalous these days, when we're in a position of meeting people in person, like you're almost like you're doing something wrong somehow. So, MongoDB has been a staple of the internet for a long time. It's oh good; another database to keep track of. What do you do these days? What is MongoDB in this ecosystem?Sahir: That's a great question. I think we're fortunate that MongoDB has been very popular for a very long time. We're seeing, you know, massive adoption grow across the globe and the massive developer community is sort of adopting the technology. What I would bring across is today MongoDB is really one of the leading cloud database companies in the world. The majority of the company's business comes from our cloud service; we partner very heavily with AWS and other cloud providers on making sure we have global availability of that. That's our flagship product.And we've invested really heavily in the last I would say, five or six years, and really extending the capabilities of the product to not just be the, sort of, database for modern web scale applications, but also to be able to handle mission-critical use cases across every vertical, you know, enterprises to startups, and doing so in a way that really empowers a general purpose strategy for any app they want to build.Corey: You're talking about general purpose which, I guess, leads to the obvious question that AWS has been pushing for a while, the idea of purpose-built databases, which makes sense from a certain point of view, and then they, of course, take that way beyond the bounds of normalcy. I don't know what the job is for someone whose role is to disambiguate between the 20 different databases that they offer by, who knows, probably the end of this year. And I don't know what that looks like. What's your take on that whole idea of a different database for every problem slash every customer slash every employee slash API request.Sahir: What we see is customers clearly moved to the cloud because they want to be able to move faster, innovate faster, be more competitive in whatever market or business or organization they're in. And certainly, I think the days of a single vendor database to rule all use cases are gone. We're not [laugh] by any means supportive of that. However, the idea that you would have 15 different databases that need to be rationalized, integrated, scripted together, frankly, may be interesting for technical teams who want to cobble together, you know, a bespoke architecture. But when we look at it from, sort of a skills, repeatability, cost, simplicity, perspective of architecture, we're seeing these, sort of like, almost like Rube Goldbergian sort of architectures.And in a large organization that wants to adopt the cloud en mass, the idea of every development team coming up with their own architecture and spending all of that time and duplication and integration of work is a distraction from ultimately their core mission, which is driving more capability and differentiation in the application for their end customer. So, to be blunt, we actually think the idea of having 15 different databases, ‘the right tool for the job' is the wrong approach. We think that there's certain key technologies that most organizations will use for 70, 80% of use cases, and then use the niche technologies for where they really need specialized solutions for particular needs.Corey: So, if you're starting off with a general-purpose database then, what is the divergence point at which point—like in my case, eventually I have to admit that using TXT records in Route 53 as a database starts to fall down for certain use cases. Not many, mind you, but one or two here and there. At what point when you're sticking with a general-purpose database does migrating to something else—what's the tipping point there?Sahir: Yeah, I think what we see is if you have a general-purpose database that hits the majority of your needs, oftentimes, especially with a microservices kind of modern architecture, it's not necessarily replacing your general-purpose database with a completely different solution, it may be augmenting it. So, you may have a particular need for, I don't know deep graph capabilities, for example, for a particular traversal use case. Maybe you augment that with a specialized solution for that. But the idea is that there's a certain set of velocity you can enable an organization by building skill set and consolidation around a technology provider that gives much more repeatability, security, less data duplication, and ultimately focuses your organization in teams on innovation as opposed to plumbing and that's where the 15 different databases been cobbled together may be interesting, but it's not really focusing on innovation, it's focusing more on the technology problems that you solved.Corey: So, we're recording this on site in Las Vegas, as re:Invent, thankfully and finally, draws to a close. How was your conference?Sahir: It's been fantastic. And to be clear, we are huge fans and partners of AWS. This is one of our most exciting conferences we sponsor. We go big, [laugh] we throw a party, we have a huge presence, we have hundreds of customer meetings. So, although I'm a little ragged, as you can probably tell from my voice from many meetings and conversations and drinks with friends, it's actually been a really great week.Corey: It is one of those things where having taken a year off, you forget so much of it, where it's, “Oh, I can definitely walk between those two hotels,” and then you sort of curse the name of God as you wind up going down that path. It was a relief, honestly, to not see, for example, another managed database service being launched that I can recall in that flurry of announcements, did you catch any?Sahir: I didn't catch any new particular database services that at least caught my eye. Granted, I've been in meetings most of the time, however, we're really excited about a lot of the infrastructure innovation. You know, I just happened to have a meeting with the compute teams on the Amazon side and what they're doing with, you know, Wavelength, and Local Zones, and new hardware, and chips with Graviton, it's all stuff we're really excited about. So, it is always interesting to see the innovation coming out of AWS.Corey: You mentioned that you are a partner with AWS, and I get it, but AWS is also one of those companies whose product strategy is ‘yes.' And they a couple years ago launched their DocumentDB, in parentheses with MongoDB compatibility, which they say, “Oh, customers were demanding this,” but no, no, they weren't. I've been talking to customers; what they wanted was actual MongoDB. The couple of folks I'm talking to who are using it are using it for one reason and one reason only, and that is replication traffic between AZs on native AWS services is free; everyone else must pay. So, there's some sub-offering in many respects that is largely MongoDB compatible to a point. Okay, but… how do you wind up, I guess, addressing the idea of continuing to partner with a company that is also heavily advantaging its own first party services, even when those are not the thing that best serves customers.Sahir: Yeah, I've been in technology for a while, and you know, the idea of working with major platform players in the context of being, in our case, a customer, a partner, and a competitor is something we're more than comfortable with, you know, and any organization at our scale and size is navigating those same dynamics. And I think on the outside, it's very easy to pay way more attention to the competitive dynamics of oh, you run in AWS but you compete with them, but the reality is, honestly, there's a lot more collaboration, both on the engineering side but also in the field. Like, we go jointly work with customers, getting them onto our platform, way more often than I think the world sees. And that's a really positive relationship. And we value that and we're investing heavily on our side to make sure you know, we're good partners in that sense.The nuances of DocumentDB versus the real MongoDB, the reality of the situation is yes, if you want the minimal MongoDB experience for, you know, a narrow percentage of our functionality, you can get that from that technology, but that's not really what customers want. Customers choose MongoDB for the breadth of capabilities that we have, and in particular, in the last few years, it's not just the NoSQL query capability of Mongo, we've integrated rich aggregation capabilities for analytics, transactional guarantees, a globally distributed architecture that scales horizontally and across regions much further than anything a relational architecture can accomplish. And we've integrated other domains of data, so things like full text, search, analytics, mobile synchronization are all baked into our Atlas platform. So, to be honest, when customers compare the two on the merits of the technology, we're more than happy to be competitors with AWS.Corey: No, I think that everyone competes with AWS, including its own product teams amongst each other because, you know, that's how you, I guess, innovate more rapidly. What do I know? I don't run a hyperscale platform. Thankfully.If I go and pull up your website, it's mongodb.com. It is natural for me to assume that you make a database, but then I start reading; after the big text and the logo, it says that you are an application data platform. Tell me more about that.Sahir: Yeah, and this has been a relatively new area of focus for us over the last couple of years. You know, I think many people know MongoDB as a non-relational modern database. Clearly, that's our core product. I think in general, we have a lot of capabilities in the database that many customers are unaware of in terms of transactional guarantees and schema management and others, so that's kind of all within the core database. But over the last few years, we've both built and acquired technology, things like Realm, that allows for mobile synchronization; event-driven architectures; APIs to be created on your data Easily; Atlas data lake, which allows for data transformation and analytics to be done using the same API as the core Mongo database; as I mentioned a couple minutes ago, things like search, where we actually allow customers to remove the need for a separate search engine for their application and make it really seamless operationally, and from the developer experience standpoint.And you know, there's no real term in the industry for that, so we kind of describe ourselves as an application data platform because really, what we're trying to do is simplify the data architecture for applications, so you don't need ten different niche database technologies to be able to build a powerful, modern, scalable application; you can build it in a unified way with an amazing developer experience that allows your teams to focus on differentiation and competitiveness as opposed to plumbing together the data infrastructure.Corey: So, when I hear platform, I think about a number of different things that may or may not be accurate, but the first thing that I think is, “Oh. There's code running on this then, as sort of part of an ecosystem.” Effectively is their code running on the data platform that you built today that wasn't written by people at MongoDB?Sahir: Yes, but it's typically the customer's code as part of their application. So, you know, I'll give you a couple of simple examples. We provide SDKs to be able to build web and mobile applications. We handle the synchronization of data from the client and front end of an application back to the back end seamlessly through our Realm platform. So, we're certainly, in that case, operating some of the business logic, or extending beyond sort of just the back end data.Similarly, a lot of what we focus on is modern event-driven architectures with MongoDB. So, to make it easier to create reactive applications, trigger off of changes in your data, we built functions and triggers natively in the platform. Now, we're not trying to be a full-on application hosting platform; that's not our business, our business is a data platform, but we really invest in making sure that platform is open, accessible, provides APIs, and functional capabilities make it very easy to integrate into any application our customers want to build.Corey: It seems like a lot of different companies now are trying to, for lack of a better term, get some of the love that Snowflake has been getting for, “Oh, their data cloud is great.” But when you take a step back and talk to people about, “So, what do you think about Mongo?” The invariable response you're going to get every time is, “Oh, you mean the database?” Like, “No, no. The character from the Princess Bride. Yes, the database.” How do you view that?Sahir: Yeah, it's easy to look at all the data landscape through a simple lens, but the reality is, there's many sub markets within the database and data market overall. And for MongoDB we're, frankly, an operational data company. And we're not focused on data warehousing, although you can use MongoDB for various analytical capabilities. We're focused on helping organizations build amazing software, and leveraging data as an enabler for great customer experiences, for digital transformation initiatives, for solving healthcare problems, or [unintelligible 00:12:51] problems in the government, or whatever it might be. We're not really focused on selling customers'—or platforms of data from—not the customers' data, but other—allowing people to monetize their data. We're focused on their applications and developers building those experiences.Corey: Yeah. So, you're if you were selling customers' data, you just rebrand as FacebookDB and be done with it, or MetaDB now—Sahir: MetaDB?Corey: Yeah. As far as the general Zeitgeist around Mongo goes, back when I was first hearing about it, in I don't know, I want to say the first half of the 2010s, the running gag was, “Oh, Mongo. It's Snapchat for databases,” with the gag being that it lost production data was unsafe for a bunch of things. To be clear, based upon my Route 53 comments, I am not a database expert by any stretch of the imagination. Now, the most common thing in my experience that loses production data is me being allowed near it. But what was the story? What gave rise to that narrative?Sahir: Yeah, I think that—thank you for bringing that up. I mean, to be clear, you know, if a database doesn't keep your data safe, consistent, and guaranteed, the rest of the functionality doesn't matter, and we take that extremely seriously at MongoDB. Now, you know, MongoDB, has been around a long time, and for better or worse—I think there's, frankly, good things and bad things about this—the database exploded in popularity extremely fast, partially because it was so easy to use for developers and it was also very different than the traditional relational database models. And so I think in many ways, customer's expectation of where the technology was compared to where we were from a maturity standpoint, combined with running an operating it the same way as a traditional system, which was, frankly, wrong for a distributed database caused, unfortunately, some situations where customers stubbed their toes and, you know, we weren't able to get to them and help them as easily as we could. Thankfully, you know, none of those issues fundamentally are, like, foundational problems. You know, we've matured the product for many, many years, you know, we work with 30,000-plus customers worldwide on mission-critical applications. I just want to make sure that everyone understands that, like, we take any issue that has to do with data loss or data corruption, as sort of the foundational [P zero 00:14:56] problem we always have to solve.Corey: I tend to form a lot of my opinions based upon very little on what, you know, sorry to say it, execs say and a lot more about what I see. There was a whole buzz going around on Twitter that HSBC was moving a whole bunch of its databases over to Mongo. And everyone was saying, “Oh, they're going to lose all their data.” But I've done work with a fair number of financial services companies, and of all the people I talk to, they're pretty far on one end of that spectrum of, “How cool are we with losing data?” So, voting with a testimonial and a wallet like that—because let's be clear, getting financial services companies to reference anything for anyone anywhere is like pulling teeth—that says a lot more than any, I guess, PR talking points could.Sahir: Yeah, I appreciate you saying that. I mean, we're very fortunate to have a very broad customer base, everything from the world's largest gaming companies to the world's largest established banks, the world's most fastest growing fintechs, to health care organizations distributing vaccines with technologies built on Mongo. Like, you name it, there's a use case in any vertical, as mission critical as you can think, built on our technology. So, customers absolutely don't take our word for granted. [laugh]. They go, you know, get comfortable with a new database technology over a span of years, but we've really hit sort of mainstream adoption for the majority of organizations. You mentioned financial services, but it's really any vertical globally, you know, we can count on our customer list.Corey: How do you, I guess, for lack of a better term, monetize what it is you do when you're one of the open-source—and yes, if you're an open-source zealot who wants to complain about licensing, it's imperative that you do not email me—but you are available for free—for certain definitions of free; I know, I know—that I can get started with a two o'clock in the morning and start running it myself in my environment. What is the tipping point that causes people to say, “Well, that was a good run. Now, I'm going to pay you folks to run it for me.”Sahir: Yeah, so there's two different sides to that, first and foremost, the majority of our engineering investment for our business goes in our core database, and our core database is free. And the way we actually, you know, survive and make money as a business, so we can keep innovating, you know, on top of the billion dollars of investment we've put in our technology over the years is, for customers who are self-managing in their own data center, we provide a set of management tools, enterprise security integrations, and others that are commercially licensed to be able to manage MongoDB for mission-critical applications in production, that's a product called Enterprise Advanced. It's typically used for large enterprise accounts in their own data centers. The flagship product for the company these days, the fastest growing part of the business is a product we call Atlas—or platform we call Atlas. That's a cloud data service.So, you know, you can go onto our website, sign up with our free tier, swipe a credit card, all consumption-based, available in every AWS region, as well as Azure and GCP, has the ability to run databases across AWS, Azure, and GCP, which is quite unique to us. And that, like any cloud data technology, is then used in conjunction with a bunch of other application components in the cloud, and customers pay us for the consumption of that database and how much they use.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: I want to zero in a little bit on something you just said, where you can have data shared between all three of the primary hyperscalers. That sounds like a story that people like to tell a lot, but you would know far better than I: how common is that use case?Sahir: It's definitely one from a strategic standpoint, especially in large enterprises, that's really important. Now, to your point, the actual usage of cross-cloud databases is still very early, but the fact that customers know that we can go, in three minutes, spin up a database cluster that allows them to either migrate, or span data across multiple regions from multiple providers for high availability, or extend their data to another cloud for analytics purposes or whatnot, is something that it almost is like science fiction to them, but it's crucial as a capability I know they will need in the future.Now, to our surprise, we've seen more real production adoption of it probably sooner than we would have expected, and there's kind of three key use cases that come into play. One—you know, for example, I was with a challenger bank from Latin America yesterday; they need high availability in Latin America. In the countries they're in, no single infrastructure cloud provider has multiple regions. They need to span across multiple regions. They mix and match cloud providers, in their case AWS being their primary, and they have a secondary cloud provider, in their case GCP, for high availability.But it's also regulatorily-driven because the banking SEC regulations in that country state that they need to be able to show portability because they don't want concentration risk of their banking sector to be on a single cloud provider or single cloud provider's region. So, we see that in multiple countries happening right now. That's one use case.The other tends to be geographic reach. So, we work with a very large international gaming company, majority of their use cases happen to be run out of the US. They happen to have a spike of customers using their game [unintelligible 00:19:58] gamers using it in Taiwan; their cloud provider of choice didn't have a region in Taiwan, but they were able to seamlessly extend a replica into a different cloud to serve low-latency performance in that country. That's the second.And then the third, which is a little bit more emerging is kind of the analytic-style use case where you may have your operational data running in a particular cloud provider, but you want to leverage the best of every cloud provider's, newest, fanciest services on top of your data. So, isn't it great if you can just hit a couple clicks, we'll extend your data and keep it in sync in near real time, and allow you to plumb into some new service from another cloud provider.Corey: In an ideal world with all things being equal, this is a wonderful vision. There's been a lot of noise made—a fair bit of it by me, let's be fair—around the data egress pricing for—it's easy to beat up on AWS because they are the largest cloud provider and it's not particularly close, but they all do it. Does that serve as a brake on that particular pattern?Sahir: Thankfully, for a database like ours and various mechanisms we use, it's not a barrier to entry. It's certainly a cost component to enabling this capability, for sure. We absolutely would love to see the industry be more open and use less of egress fees as a way to wall people into a particular cloud providers. We certainly have that belief, and would push that notion and continually do in the industry. But it hasn't been a barrier to adoption because it's not the major cost component of operating a multi-cloud database.Corey: Well, [then you start 00:21:27] doing this whole circular replication thing, at which point, wow. It just goes round and round and round and lives on the network all the time. I'm told that's what a storage area network is because I'm about as good at storage as I am at databases. As you look at Atlas, since you are in all of the major hyperscalers, is the experience different in any way, depending upon which provider you're running in?Sahir: By and large, it's pretty consistent. However, what we are not doing is building to the lowest common denominator. If there's a service integration that our customers on AWS want, and that service doesn't integrate, it doesn't exist on another cloud provider, or vice versa, we're not going to stop ourselves from building a great customer experience and integration point. And the same thing goes for infrastructure; if there's some infrastructure innovation that delivers price, performance, great value for our customers and it's only on a single cloud, we're not going to stop ourselves from delivering that value to customers. So, there's a line there, you know, we want to provide a great experience, portability across the cloud providers, consistency where it makes sense, but we are not going to water down our experience on a particular cloud provider if customers are asking for some native capabilities.Corey: It always feels like a strange challenge historically to wind up—at least in large, regulated environments—getting a new vendor in. Originally an end run around this was using the AWS Marketplace or whatever marketplace you were using at any given cloud provider. Then procurement caught on and in some cases banned in the Marketplace outright and now, the Marketplace is sort of reformed, in some ways, to being a tool for procurement to use. Have you seen significant uptake of your offering through the various cloud marketplaces?Sahir: We do work with all the cloud marketplaces. In fact, we just made an announcement with AWS that we're going to be implementing the pay-as-you-go marketplace model for self-service as well on AWS. So, it is definitely a driver for our business. It tends to be used most heavily when we're selling with the, you know, sales teams from the cloud providers, and customers want to benefit from a single bill, benefit from, you know, drawing down on their large commitments that they might have with any given cloud providers. So, it drives really good alignment between the customer, us as a third-party on AWS or Azure GCP, and the infrastructure cloud provider. And so we're all aligned on a motion. So, in that sense, it's definitely been helpful, but it's largely been a procurement and fulfillment sort of value proposition to drive that alignment, I'd say, by and large today.Corey: I don't know if you're able to answer this without revealing anything confidential, so please feel free not to, but as you look across the total landscape—since I would say that you have a fairly reasonable snapshot of the industry as a whole—am I right when I say that AWS is the behemoth in the space, or is it a closer horse race than most people would believe, based upon your perspective?Sahir: I think in general, for sure AWS is the market share leader. It would be crazy to say anything otherwise. They innovated this model, you know, the amount of innovation happening at AWS is incredible, you know, and we're benefiting from it as a customer as well. However, we do believe it's a multi-cloud future. I mean, look at the growth of Azure. You know, we're seeing Google show up in large enterprises across the globe as well.And even beyond the three American clouds, you know, we work heavily with Alibaba and Tencent in mainland China, which is a completely different market than Western world. So, I do think the trend over time will be a more heterogeneous, more multi-cloud world—which I'm biased; that does favor MongoDB, but that's the trend we're seeing—but that doesn't mean that AWS won't continue to still be a leader and a very strong player in that market.Corey: I want to talk a little bit about Jepsen. And for those who are unaware, jepsen.io is run by Kyle Kingsbury. Kyle is wonderful, and he's also nuts. If you followed him back when he was on Twitter, you've also certainly seen them.But beyond that, he is the de facto resource I go to when it comes to consistency testing and stress testing of databases. I'm a little annoyed he hasn't taken on Route 53 yet, but hope does spring eternal. He's evaluated Mongo a number of times, and his conclusions, as always are mixed sometimes, shall we say, incendiary, but they always seem relatively fair. What is your experience been, working with him? And do you share my opinion of him as being a neutral and fair arbiter of these things?Sahir: I do. I think he's got real expertise and credibility in beating up distributed database systems and finding the edges of where they don't live up to what we all hope they do, right? Whether it's us or anyone else, just to be clear. And so anytime Kyle finds some flaw in MongoDB, we take it seriously, we add it to our test suite, [laugh] we remediate, and I think we have a pretty good history of that. And in fact, we've actually worked with Kyle to welcome him beating up our database on multiple occasions, too, so it's not an adversarial relationship at all.Corey: I have to ask, since you are a more modern generation of database, then many from the previous century, but there's always been a significant, shall we say… concern, when I wind up looking at it [it again in 00:26:33] any given database, and I look in the terms and conditions and, like, “Oh, it's a great database. We're by far the best. Whatever you do, do not publish benchmarks.” What's going on with that?Sahir: I think benchmarks can be spun in any direction you want, by any vendor. And it's not just database technology. I've been in IT for a while, and you know, that applies to any technology. So, we absolutely do not shy away from our performance or benchmark or comparisons to any technology. We just think that, you know, vendors benchmarking technologies for their—are doing so largely to only make their own technologies look good versus competition.Corey: I tend to be somewhat skeptical of the various benchmark stuff. I remember repeatedly oh, I'll wind up running whatever it is—I think it's Geek Speed—on my various devices to see oh, how snappy and performant is it going to be? But then I'm sitting there opening Microsoft Word and watching the beach ball spin, and spin, and spin, and it turns out, don't care about benchmarks in a real-world use case in many scenarios.Sahir: Yeah, it's kind of a good analogy, right? I mean, performance of an application, sure, the database at the heart of it is a crucial component, but there's many more aspects of it that have to do with the overall real world performance than just some raw benchmark results for any database, right? It's the way you model your data, the way the rest of the architecture of the application interacts and hangs together with the database, many, many layers of complexity. So, I don't always think those benchmarks are indicative of how real world performance will look, but at the same time, I'm very confident in MongoDB's performance comparatively to our peers, so it's not something we're afraid of.Corey: As you take a look at where you've been and where you are now, what's next? Where are you going? Because I have a hard time believing that, “Yep, we're deciding it's feature complete and we're just going to sell this until the end of time exactly as is, we're laying off our entire engineering team and we're going to be doing support from our yacht, parked comfortably in international waters.” That's a slightly different company. What's the plan?Sahir: So, [laugh] you're—we are not parking anything, anytime soon. We are continuing to invest heavily in the innovation of the technology, and really, it's two reasons: you know, one, we're seeing an acceleration of adoption of MongoDB, either with any customers that have used us for a long time, but for more important and more use cases, but also just broader adoption globally as more and more developers learn to code, they're choosing Mongo as the place to start, increasingly. And so that's really exciting for us, and we need to keep up with those customer demands and that roadmap of asks that they have.And at the same time, customer requirements are increasing as more and more organizations are software-first organizations, the requirements of what they demand from us continually increase, which requires continual innovation in our architecture and our functionality to keep up with those and stay ahead of those customer requirements. So, what you'll see from us is, one, making sure we can build the best modern database we can. That's the core of what we do; everything we do now especially is cloud first, so working closely with our cloud partners on that. And even though we're very fortunate to be a high-performance, high-growth company with a very pervasive open technology, we're still in a giant market that has a lot of legacy technologies powering old applications. So, [laugh] you know, we have a long, long runway to become a long-standing major player in this market.And then we're going to continue this vision of an application data platform, which is really just about simplifying the capabilities and data architecture for organizations and developers so they can focus on building their application and less on the plumbing.Corey: I want to thank you so much for taking the time to speak with me today. If people want to learn more, where can they go?Sahir: Clearly, you can go to mongodb.com. You can also reach out to us on our community sites: our own or on any of the public sites that you would typically find developers hanging out. We always have folks from our teams or our champions program of advocates worldwide helping out our customers and users. And I just want to thank you, Corey, for having me. I've followed you online for a while; it's great to finally be able to meet in person.Corey: Uh-oh. It's disturbing having realized some of the things I've said on Twitter and realizing I'm now within range to get punched in the face. But, you know, we take what we can get. Thank you so much for taking the time to speak with me. I appreciate it.Sahir: My pleasure.Corey: Sahir Azam, Chief Product Officer at MongoDB. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment telling me that is not the reason that AWS is building many new databases. Tell me which one you're building and why it solves a problem other than getting you the promotion you probably don't deserve.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

AWS Morning Brief
A Somehow Quiet Security Week

AWS Morning Brief

Play Episode Listen Later Dec 9, 2021 5:30


Links: Cyber-security insurance providers are increasing their requirements to be insurable: https://Twitter.com/SwiftOnSecurity/status/1467879429707866112 “Why the C-suite doesn't need access to all corporate data”: https://www.darkreading.com/vulnerabilities-threats/why-the-c-suite-doesn-t-need-access-to-all-corporate-data “Amazon S3 Object Ownership can now disable access control lists to simplify access management for data in S3”: https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-s3-object-ownership-simplify-access-management-data-s3/ Cloud provider security mistakes: https://github.com/SummitRoute/csp_security_mistakes TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it's nobody in particular's job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: Are you building cloud applications with a distributed team? Check out Teleport, an open-source identity-aware access proxy for cloud resources. Teleport provides secure access for anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps, and databases. Teleport gives engineers superpowers. Get access to everything via single sign-on with multi-factor. List and see all of SSH servers, Kubernetes clusters, or databases available to you in one place, and get instant access to them using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility, and ensuring compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That's goteleport.com.Corey: re:Invent has come and gone, and with it remarkably few security announcements. Shockingly, it was a slow week for the industry. I'm glad but also disappointed to be proven wrong in my, “The only thing you, as a company who isn't AWS, should be announcing during re:Invent is your data breach since nobody will be paying attention,” snark. But it's for the best. It means that maybe—maybe—we're starting to see things normalize a bit.Now, from the Community, we saw some interesting stuff. Scuttlebutt has it that cyber-security insurance providers are increasing their requirements to be insurable. This makes a lot of sense; as ransomware attacks become more numerous, nobody is going to want to cut large insurance checks to folks who didn't think to have offline backups. You might want to check the specific terms and conditions of your policy.I also liked a writeup as to “Why the C-suite doesn't need access to all corporate data.” It's true, but it's super hard to defend against. When the CTO ‘requests' access to the AWS root account, who's likely to say no? If you're going to push for proper separation of duties, either do it the right way or don't even bother.Corey: This episode is sponsored in part by my friends at Cloud Academy. Something special for you folks: if you missed their offer on Black Friday or Cyber Monday or whatever day of the week doing sales it is, good news, they've opened up their Black Friday promotion for a very limited time. Same deal: $100 off a yearly plan, 249 bucks a year for the highest quality cloud and tech skills content. Nobody else is going to get this, and you have to act now because they have assured me this is not going to last for much longer. Go to cloudacademy.com, hit the ‘Start Free Trial' button on the homepage and use the promo code, ‘CLOUD' when checking out. That's C-L-O-U-D. Like loud—what I am—with a C in front of it. They've got a free trial, too, so you'll get seven days to try it out to make sure it really is a good fit. You've got nothing to lose except your ignorance about cloud. My thanks to Cloud Academy once again for sponsoring my ridiculous nonsense.Corey: And from AWS, there was really one glaring announcement that made me happy in the security context, and that was that “Amazon S3 Object Ownership can now disable access control lists to simplify access management for data in S3,” and it's huge. S3 ACLs have been a pain in everyone's side for years. Remember that S3 was the first AWS service to general availability, and a second in beta, after SQS. Meanwhile, IAM wasn't released until 2010. “Ignore bucket ACLs so you don't have to think about them” is a huge step towards normalizing security within AWS, specifically S3.And from the community's tools—I guess it's not a tool so much as it is a tip or I don't even know how you would describe it but I love it because Scott Piper is doing the lord's work by curating a list of cloud provider security mistakes. Lord knows that none of them are going to be showcasing their own failures, or—thankfully—those of their competition because I don't want to get in the middle of that mudslinging prize. This is well worth checking out and taking a look at, particularly when one provider or another starts getting a little too full of themselves around what they're doing in security. That's what happened last week in AWS security. Thank you for listening.Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Building Distributed Cognition into Your Business with Sam Ramji

Screaming in the Cloud

Play Episode Listen Later Dec 9, 2021 39:56


About SamA 25-year veteran of the Silicon Valley and Seattle technology scenes, Sam Ramji led Kubernetes and DevOps product management for Google Cloud, founded the Cloud Foundry foundation, has helped build two multi-billion dollar markets (API Management at Apigee and Enterprise Service Bus at BEA Systems) and redefined Microsoft's open source and Linux strategy from “extinguish” to “embrace”.He is nerdy about open source, platform economics, middleware, and cloud computing with emphasis on developer experience and enterprise software. He is an advisor to multiple companies including Dell Technologies, Accenture, Observable, Fletch, Orbit, OSS Capital, and the Linux Foundation.Sam received his B.S. in Cognitive Science from UC San Diego, the home of transdisciplinary innovation, in 1994 and is still excited about artificial intelligence, neuroscience, and cognitive psychology.Links: DataStax: https://www.datastax.com Sam Ramji Twitter: https://twitter.com/sramji Open||Source||Data: https://www.datastax.com/resources/podcast/open-source-data Screaming in the Cloud Episode 243 with Craig McLuckie: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/innovating-in-the-cloud-with-craig-mcluckie/ Screaming in the Cloud Episode 261 with Jason Warner: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/what-github-can-give-to-microsoft-with-jason-warner/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. Set up a meeting with a Redis expert during re:Invent, and you'll not only learn how you can become a Redis hero, but also have a chance to win some fun and exciting prizes. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: Are you building cloud applications with a distributed team? Check out Teleport, an open source identity-aware access proxy for cloud resources. Teleport provides secure access to anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps and databases. Teleport gives engineers superpowers! Get access to everything via single sign-on with multi-factor. List and see all SSH servers, kubernetes clusters or databases available to you. Get instant access to them all using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility and ensuring compliance. And best of all, Teleport is open source and a pleasure to use.Download Teleport at https://goteleport.com. That's goteleport.com.Corey: Welcome to Screaming in the Cloud, I'm Cloud Economist Corey Quinn, and recurring effort that this show goes to is to showcase people in their best light. Today's guest has done an awful lot: he led Kubernetes and DevOps Product Management for Google Cloud; he founded the Cloud Foundry Foundation; he set open-source strategy for Microsoft in the naughts; he advises companies including Dell, Accenture, the Linux Foundation; and tying all of that together, it's hard to present a lot of that in a great light because given my own proclivities, that sounds an awful lot like a personal attack. Sam Ramji is the Chief Strategy Officer at DataStax. Sam, thank you for joining me, and it's weird when your resume starts to read like, “Oh, I hate all of these things.”Sam: [laugh]. It's weird, but it's true. And it's the only life I could have lived apparently because here I am. Corey, it's a thrill to meet you. I've been an admirer of your public speaking, and public tweeting, and your writing for a long time.Corey: Well, thank you. The hard part is getting over the voice saying don't do it because it turns out that there's no real other side of public shutting up, which is something that I was never good at anyway, so I figured I'd lean into it. And again, I mean, that the sense of where you have been historically in terms of your career not, “Look what you've done,” which is a subtext that I could be accused of throwing in sometimes.Sam: I used to hear that a lot from my parents, actually.Corey: Oh, yeah. That was my name growing up. But you've done a lot of things, and you've transitioned from notable company making significant impact on the industry, to the next one, to the next one. And you've been in high-flying roles, doing lots of really interesting stuff. What's the common thread between all those things?Sam: I'm an intensely curious person, and the thing that I'm most curious about is distributed cognition. And that might not be obvious from what you see is kind of the… Lego blocks of my career, but I studied cognitive science in college when that was not really something that was super well known. So, I graduated from UC San Diego in '94 doing neuroscience, artificial intelligence, and psychology. And because I just couldn't stop thinking about thinking; I was just fascinated with how it worked.So, then I wanted to build software systems that would help people learn. And then I wanted to build distributed software systems. And then I wanted to learn how to work with people who were thinking about building the distributed software systems. So, you end up kind of going up this curve of, like, complexity about how do we think? How do we think alone? How do we learn to think? How do we think together?And that's the directed path through my software engineering career, into management, into middleware at BEA, into open-source at Microsoft because that's an amazing demonstration of distributed cognition, how, you know, at the time in 2007, I think, Sourceforge had 100,000 open-source projects, which was, like, mind boggling. Some of them even worked together, but all of them represented these groups of people, flung around the world, collaborating on something that was just fundamentally useful, that they were curious about. Kind of did the same thing into APIs because APIs are an even better way to reuse for some cases than having the source code—at Apigee. And kept growing up through that into, how are we building larger-scale thinking systems like Cloud Foundry, which took me into Google and Kubernetes, and then some applications of that in Autodesk and now DataStax. So, I love building companies. I love helping people build companies because I think business is distributed cognition. So, those businesses that build distributed systems, for me, are the most fascinating.Corey: You were basically handed a heck of a challenge as far as, “Well, help set open-source strategy,” back at Microsoft, in the days where that was a punchline. And credit where due, I have to look at Microsoft of today, and it's not a joke, you can have your arguments about them, but again in those days, a lot of us built our entire personality on hating Microsoft. Some folks never quite evolved beyond that, but it's a new ballgame and it's very clear that the Microsoft of yesteryear and the Microsoft of today are not completely congruent. What was it like at that point understanding that as you're working with open-source communities, you're doing that from a place of employment with a company that was widely reviled in the space.Sam: It was not lost on me. The irony, of course, was that—Corey: Well, thank God because otherwise the question where you would have been, “What do you mean they didn't like us?”Sam: [laugh].Corey: Which, on some levels, like, yeah, that's about the level of awareness I would have expected in that era, but contrary to popular opinion, execs at these companies are not generally oblivious.Sam: Yeah, well, if I'd been clever as a creative humorist, I would have given you that answer instead of my serious answer, but for some reason, my role in life is always to be the straight guy. I used to have Slashdot as my homepage, right? I love when I'd see some conspiracy theory about, you know, Bill Gates dressed up as the Borg, taking over the world. My first startup, actually in '97, was crushed by Microsoft. They copied our product, copied the marketing, and bundled it into Office, so I had lots of reasons to dislike Microsoft.But in 2004, I was recruited into their venture capital team, which I couldn't believe. It was really a place that they were like, “Hey, we could do better at helping startups succeed, so we're going to evangelize their success—if they're building with Microsoft technologies—to VCs, to enterprises, we'll help you get your first big enterprise deal.” I was like, “Man, if I had this a few years ago, I might not be working.” So, let's go try to pay it forward.I ended up in open-source by accident. I started going to these conferences on Software as a Service. This is back in 2005 when people were just starting to light up, like, Silicon Valley Forum with, you know, the CEO of Demandware would talk, right? We'd hear all these different ways of building a new business, and they all kept talking about their tech stack was Linux, Apache, MySQL, and PHP. I went to one eight-hour conference, and Microsoft technologies were mentioned for about 12 seconds in two separate chunks. So, six seconds, he was like, “Oh, and also we really like Microsoft SQL Server for our data layer.”Corey: Oh, Microsoft SQL Server was fantastic. And I know that's a weird thing for people to hear me say, just because I've been renowned recently for using Route 53 as the primary data store for everything that I can. But there was nothing quite like that as far as having multiple write nodes, being able to handle sharding effectively. It was expensive, and you would take a bath on the price come audit time, but people were not rolling it out unaware of those things. This was a trade off that they were making.Oracle has a similar story with databases. It's yeah, people love to talk smack about Oracle and its business practices for a variety of excellent reasons, at least in the database space that hasn't quite made it to cloud yet—knock on wood—but people weren't deploying it because they thought Oracle was warm and cuddly as a vendor; they did it because they can tolerate the rest of it because their stuff works.Sam: That's so well said, and people don't give them the credit that's due. Like, when they built hypergrowth in their business, like… they had a great product; it really worked. They made it expensive, and they made a lot of money on it, and I think that was why you saw MySQL so successful and why, if you were looking for a spec that worked, that you could talk through through an open driver like ODBC or JDBC or whatever, you could swap to Microsoft SQL Server. But I walked out of that and came back to the VC team and said, “Microsoft has a huge problem. This is a massive market wave that's coming. We're not doing anything in it. They use a little bit of SQL Server, but there's nothing else in your tech stack that they want, or like, or can afford because they don't know if their businesses are going to succeed or not. And they're going to go out of business trying to figure out how much licensing costs they would pay to you in order to consider using your software. They can't even start there. They have to start with open-source. So, if you're going to deal with SaaS, you're going to have to have open-source, and get it right.”So, I worked with some folks in the industry, wrote a ten-page paper, sent it up to Bill Gates for Think Week. Didn't hear much back. Bought a new strategy to the head of developer platform evangelism, Sanjay Parthasarathy who suggested that the idea of discounting software to zero for startups, with the hope that they would end up doing really well with it in the future as a Software as a Service company; it was dead on arrival. Dumb idea; bring it back; that actually became BizSpark, the most popular program in Microsoft partner history.And then about three months later, I got a call from this guy, Bill Hilf. And he said, “Hey, this is Bill Hilf. I do open-source at Microsoft. I work with Bill Gates. He sent me your paper. I really like it. Would you consider coming up and having conversation with me because I want you to think about running open-source technology strategy for the company.” And at this time I'm, like, 33 or 34. And I'm like, “Who me? You've got to be joking.” And he goes, “Oh, and also, you'll be responsible for doing quarterly deep technical briefings with Bill… Gates.” I was like, “You must be kidding.” And so of course I had to check it out. One thing led to another and all of a sudden, with not a lot of history in the open-source community but coming in it with a strategist's eye and with a technologist's eye, saying, “This is a problem we got to solve. How do we get after this pragmatically?” And the rest is history, as they say.Corey: I have to say that you are the Chief Strategy Officer at DataStax, and I pull up your website quickly here and a lot of what I tell earlier stage companies is effectively more or less what you have already done. You haven't named yourself after the open-source project that underlies the bones of what you have built so you're not going to wind up in the same glorious challenges that, for example, Elastic or MongoDB have in some ways. You have a pricing page that speaks both to the reality of, “It's two in the morning. I'm trying to get something up and running and I want you the hell out of my way. Just give me something that I can work with a reasonable free tier and don't make me talk to a salesperson.” But also, your enterprise tier is, “Click here to talk to a human being,” which is speaking enterprise slash procurement slash, oh, there will be contract negotiation on these things.It's being able to serve different ends of your market depending upon who it is that encounters you without being off-putting to any of those. And it's deceptively challenging for companies to pull off or get right. So clearly, you've learned lessons by doing this. That was the big problem with Microsoft for the longest time. It's, if I want to use some Microsoft stuff, once you were able to download things from the internet, it changed slightly, but even then it was one of those, “What exactly am I committing to here as far as signing up for this? And am I giving them audit rights into my environment? Is the BSA about to come out of nowhere and hit me with a surprise audit and find out that various folks throughout the company have installed this somewhere and now I owe more than the company's worth?” That was always the haunting fear that companies had back then.These days, I like the approach that companies are taking with the SaaS offering: you pay for usage. On some level, I'd prefer it slightly differently in a pay-per-seat model because at least then you can predict the pricing, but no one is getting surprise submarined with this type of thing on an audit basis, and then they owe damages and payment in arrears and someone has them over a barrel. It's just, “Oh. The bill this month was higher than we expected.” I like that model I think the industry does, too.Sam: I think that's super well said. As I used to joke at BEA Systems, nothing says ‘I love you' to a customer like an audit, right? That's kind of a one-time use strategy. If you're going to go audit licenses to get your revenue in place, you might be inducing some churn there. It's a huge fix for the structural problem in pricing that I think package software had, right?When we looked at Microsoft software versus open-source software, and particularly Windows versus Linux, you would have a structure where sales reps were really compensated to sell as much as possible upfront so they could get the best possible commission on what might be used perpetually. But then if you think about it, like, the boxes in a curve, right, if you do that calculus approximation of a smooth curve, a perpetual software license is a huge box and there's an enormous amount of waste in there. And customers figured out so as soon as you can go to a pay-per-use or pay-as-you-go, you start to smooth that curve, and now what you get is what you deserve, right, as opposed to getting filled with way more cost than you expect. So, I think this model is really super well understood now. Kind of the long run the high point of open-source meets, cloud, meets Software as a Service, you look at what companies like MongoDB, and Confluent, and Elastic, and Databricks are doing. And they've really established a very good path through the jungle of how to succeed as a software company. So, it's still difficult to implement, but there are really world-class guides right now.Corey: Moving beyond where Microsoft was back in the naughts, you were then hired as a VP over at Google. And in that era, the fact that you were hired as a VP at Google is fascinating. They preferred to grow those internally, generally from engineering. So, first question, when you were being hired as a VP in the product org, did they make you solve algorithms on a whiteboard to get there?Sam: [laugh]. They did not. I did have somewhat of an advantage [because they 00:13:36] could see me working pretty closely as the CEO of the Cloud Foundry Foundation. I'd worked closely with Craig McLuckie who notably brought Kubernetes to the world along with Joe Beda, and with Eric Brewer, and a number of others.And he was my champion at Google. He was like, “Look, you know, we need him doing Kubernetes. Let's bring Sam in to do that.” So, that was helpful. I also wrote a [laugh] 2000-word strategy document, just to get some thoughts out of my head. And I said, “Hey, if you like this, great. If you don't throw it away.” So, the interviews were actually very much not solving problems in a whiteboard. There were super collaborative, really excellent conversations. It was slow—Corey: Let's be clear, Craig McLuckie's most notable achievement was being a guest on this podcast back in Episode 243. But I'll say that this is a close second.Sam: [laugh]. You're not wrong. And of course now with Heptio and their acquisition by VMware.Corey: Ehh, they're making money beyond the wildest dreams of avarice, that's all well and good, but an invite to this podcast, that's where it's at.Sam: Well, he should really come on again, he can double down and beat everybody. That can be his landmark achievement, a two-timer on Screaming in [the] Cloud.Corey: You were at Google; you were at Microsoft. These are the big titans of their era, in some respect—not to imply that there has beens; they're bigger than ever—but it's also a more crowded field in some ways. I guess completing the trifecta would be Amazon, but you've had the good judgment never to work there, directly of course. Now they're clearly in your market. You're at DataStax, which is among other things, built on Apache Cassandra, and they launched their own Cassandra service named Keyspaces because no one really knows why or how they name things.And of course, looking under the hood at the pricing model, it's pretty clear that it really is just DynamoDB wearing some Groucho Marx classes with a slight upcharge for API level compatibility. Great. So, I don't see it a lot in the real world and that's fine, but I'm curious as to your take on looking at all three of those companies at different eras. There was always the threat in the open-source world that they are going to come in and crush you. You said earlier that Microsoft crushed your first startup.Google is an interesting competitor in some respects; people don't really have that concern about them. And your job as a Chief Strategy Officer at Amazon is taken over by a Post-it Note that simply says ‘yes' on it because there's nothing they're not going to do, or try, and experiment with. So, from your perspective, if you look at the titans, who is it that you see as the largest competitive threat these days, if that's even a thing?Sam: If you think about Sun Tzu and the Art of War, right—a lot of strategy comes from what we've learned from military environments—fighting a symmetric war, right, using the same weapons and the same army against a symmetric opponent, but having 1/100th of the personnel and 1/100th of the money is not a good plan.Corey: “We're going to lose money, going to be outcompeted; we'll make it up in volume. Oh, by the way, we're also slower than they are.”Sam: [laugh]. So, you know, trying to come after AWS, or Microsoft, or Google as an independent software company, pound-for-pound, face-to-face, right, full-frontal assault is psychotic. What you have to do, I think, at this point is to understand that these are each companies that are much like we thought about Linux, and you know, Macintosh, and Windows as operating systems. They're now the operating systems of the planet. So, that creates some economies of scale, some efficiencies for them. And for us. Look at how cheap object storage is now, right? So, there's never been a better time in human history to create a database company because we can take the storage out of the database and hand it over to Amazon, or Google, or Microsoft to handle it with 13 nines of durability on a constantly falling cost basis.So, that's super interesting. So, you have to prosecute the structure of the world as it is, based on where the giants are and where they'll be in the future. Then you have to turn around and say, like, “What can they never sell?”So, Amazon can never sell something that is standalone, right? They're a parts factory and if you buy into the Amazon-first strategy of cloud computing—which we did at Autodesk when I was VP of cloud platform there—everything is a primitive that works inside Amazon, but they're not going to build things that don't work outside of the Amazon primitives. So, your company has to be built on the idea that there's a set of people who value something that is purpose-built for a particular use case that you can start to broaden out, it's really helpful if they would like it to be something that can help them escape a really valuable asset away from the center of gravity that is a cloud. And that's why data is super interesting. Nobody wakes up in the morning and says, “Boy, I had such a great conversation with Oracle over the last 20 years beating me up on licensing. Let me go find a cloud vendor and dump all of my data in that so they can beat me up for the next 20 years.” Nobody says that.Corey: It's the idea of data portability that drives decision-making, which makes people, of course, feel better about not actually moving in anywhere. But the fact that they're not locked in strategically, in a way that requires a full software re-architecture and data model rewrite is compelling. I'm a big believer in convincing people to make decisions that look a lot like that.Sam: Right. And so that's the key, right? So, when I was at Autodesk, we went from our 100 million dollar, you know, committed spend with 19% discount on the big three services to, like—we started realize when we're going to burn through that, we were spending $60 million or so a year on 20% annual growth as the cloud part of the business grew. Thought, “Okay, let's renegotiate. Let's go and do a $250 million deal. I'm sure they'll give us a much better discount than 19%.” Short story is they came back and said, “You know, we're going to take you from an already generous 19% to an outstanding 22%.” We thought, “Wait a minute, we already talked to Intuit. They're getting a 40% discount on a $400 million spend.”So, you know, math is hard, but, like, 40% minus 22% is 18% times $250 million is a lot of money. So, we thought, “What is going on here?” And we realized we just had no credible threat of leaving, and Intuit did because they had built a cross-cloud capable architecture. And we had not. So, now stepping back into the kind of the world that we're living in 2021, if you're an independent software company, especially if you have the unreasonable advantage of being an open-source software company, you have got to be doing your customers good by giving them cross-cloud capability. It could be simply like the Amdahl coffee cup that Amdahl reps used to put as landmines for the IBM reps, later—I can tell you that story if you want—even if it's only a way to save money for your customer by using your software, when it gets up to tens and hundreds of million dollars, that's a really big deal.But they also know that data is super important, so the option value of being able to move if they have to, that they have to be able to pull that stick, instead of saying, “Nice doggy,” we have to be on their side, right? So, there's almost a detente that we have to create now, as cloud vendors, working in a world that's invented and operated by the giants.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: When we look across the, I guess, the ecosystem as it's currently unfolding, a recurring challenge that I have to the existing incumbent cloud providers is they're great at offering the bricks that you can use to build things, but if I'm starting a company today, I'm not going to look at building it myself out of, “Ooh, I'm going to take a bunch of EC2 instances, or Lambda functions, or popsicles and string and turn it into this thing.” I'm going to want to tie together things that are way higher level. In my own case, now I wind up paying for Retool, which is, effectively, yeah, it runs on some containers somewhere, presumably, I think in Azure, but don't quote me on that. And that's great. Could I build my own thing like that?Absolutely not. I would rather pay someone to tie it together. Same story. Instead of building my own CRM by running some open-source software on an EC2 instance, I wind up paying for Salesforce or Pipedrive or something in that space. And so on, and so forth.And a lot of these companies that I'm doing business with aren't themselves running on top of AWS. But for web hosting, for example; if I look at the reference architecture for a WordPress site, AWS's diagram looks like a punchline. It is incredibly overcomplicated. And I say this as someone who ran large WordPress installations at Media Temple many years ago. Now, I have the good sense to pay WP Engine. And on a monthly basis, I give them money and they make the website work.Sure, under the hood, it's running on top of GCP or AWS somewhere. But I don't have to think about it; I don't have to build this stuff together and think about the backups and the failover strategy and the rest. The website just works. And that is increasingly the direction that business is going; things commoditize over time. And AWS in particular has done a terrible job, in my experience, of differentiating what it is they're doing in the language that their customers speak.They're great at selling things to existing infrastructure engineers, but folks who are building something from scratch aren't usually in that cohort. It's a longer story with time and, “Well, we're great at being able to sell EC2 instances by the gallon.” Great. Are you capable of going to a small doctor's office somewhere in the American Midwest and offering them an end-to-end solution for managing patient data? Of course not. You can offer them a bunch of things they can tie together to something that will suffice if they all happen to be software engineers, but that's not the opportunity.So instead, other companies are building those solutions on top of AWS, capturing the margin. And if there's one thing guaranteed to keep Amazon execs awake at night, it's the idea of someone who isn't them making money somehow somewhere, so I know that's got to rankle them, but they do not speak that language. At all. Longer-term, I only see that as a more and more significant crutch. A long enough timeframe here, we're talking about them becoming the Centurylinks of the world, the tier one backbone provider that everyone uses, but no one really thinks about because they're not a household name.Sam: That is a really thoughtful perspective. I think the diseconomies of scale that you're pointing to start to creep in, right? Because when you have to sell compute units by the gallon, right, you can't care if it's a gallon of milk, [laugh] or a gallon of oil, or you know, a gallon of poison. You just have to keep moving it through. So, the shift that I think they're going to end up having to make pragmatically, and you start to see some signs of it, like, you know, they hired but could not retain Matt [Acey 00:23:48]. He did an amazing job of bringing them to some pragmatic realization that they need to partner with open-source, but more broadly, when I think about Microsoft in the 2000s as they were starting to learn their open-source lessons, we were also being able to pull on Microsoft's deep competency and partners. So, most people didn't do the math on this. I was part of the field governance council so I understood exactly how the Microsoft business worked to the level that I was capable. When they had $65 billion in revenue, they produced $24 billion in profit through an ecosystem that generated $450 billion in revenue. So, for every dollar Microsoft made, it was $8 to partners. It was a fundamentally platform-shaped business, and that was how they're able to get into doctors offices in the Midwest, and kind of fit the curve that you're describing of all of those longtail opportunities that require so much care and that are complex to prosecute. These solved for their diseconomies of scale by having 1.2 million partner companies. So, will Amazon figure that out and will they hire, right, enough people who've done this before from Microsoft to become world-class in partnering, that's kind of an exercise left to the [laugh] reader, right? Where will that go over time? But I don't see another better mathematical model for dealing with the diseconomies of scale you have when you're one of the very largest providers on the planet.Corey: The hardest problem as I look at this is, at some point, you hit a point of scale where smaller things look a lot less interesting. I get that all the time when people say, “Oh, you fix AWS bills, aren't you missing out by not targeting Google bills and Azure bills as well?” And it's, yeah. I'm not VC-backed. It turns out that if I limit the customer base that I can effectively service to only AWS customers, yeah turns out, I'm not going to starve anytime soon. Who knew? I don't need to conquer the world and that feels increasingly antiquated, at least going by the stories everyone loves to tell.Sam: Yeah, it's interesting to see how cloud makes strange bedfellows, right? We started seeing this in, like, 2014, 2015, weird partnerships that you're like, “There's no way this would happen.” But the cloud economics which go back to utilization, rather than what it used to be, which was software lock-in, just changed who people were willing to hang out with. And now you see companies like Databricks going, you know, we do an amazing amount of business, effectively competing with Amazon, selling Spark services on top of predominantly Amazon infrastructure, and everybody seems happy with it. So, there's some hint of a new sensibility of what the future of partnering will be. We used to call it coopetition a long time ago, which is kind of a terrible word, but at least it shows that there's some nuance in you can't compete with everybody because it's just too hard.Corey: I wish there were better ways of articulating these things because it seems from the all the outside world, you have companies like Amazon and Microsoft and Google who go and build out partner networks because they need that external accessibility into various customer profiles that they can't speak to super well themselves, but they're also coming out with things that wind up competing directly or indirectly, with all of those partners at the same time. And I don't get it. I wish that there were smarter ways to do it.Sam: It is hard to even talk about it, right? One of the things that I think we've learned from philosophy is if we don't have a word for it, we can't be intelligent about it. So, there's a missing semantics here for being able to describe the complexity of where are you partnering? Where are you competing? Where are you differentiating? In an ecosystem, which is moving and changing.I tend to look at the tools of game theory for this, which is to look at things as either, you know, nonzero-sum games or zero-sum games. And if it's a nonzero-sum game, which I think are the most interesting ones, can you make it a positive sum game? And who can you play positive-sum games with? An organization as big as Amazon, or as big as Microsoft, or even as big as Google isn't ever completely coherent with itself. So, thinking about this as an independent software company, it doesn't matter if part of one of these hyperscalers has a part of their business that competes with your entire business because your business probably drives utilization of a completely different resource in their company that you can partner within them against them, effectively. Right?For example, Cassandra is an amazingly powerful but demanding workload on Kubernetes. So, there's a lot of Cassandra on EKS. You grow a lot of workload, and EKS business does super well. Does that prevent us from working with Amazon because they have Dynamo or because they have Keyspaces? Absolutely not, right?So, this is when those companies get so big that they are almost their own forest, right, of complexity, you can kind of get in, hang out, do well, and pretty much never see the competitive product, unless you're explicitly looking for it, which I think is a huge danger for us as independent software companies. And I would say this to anybody doing strategy for an organization like this, which is, don't obsess over the tiny part of their business that competes with yours, and do not pay attention to any of the marketing that they put out that looks competitive with what you have. Because if you can't figure out how to make a better product and sell it better to your customers as a single purpose corporation, you have bigger problems.Corey: I want to change gears slightly to something that's probably a fair bit more insulting, but that's okay. We're going to roll with it. That seems to be the theme of this episode. You have been, in effect, a CIO a number of times at different companies. And if we take a look at the typical CIO tenure, industry-wide, it's not long; it approaches the territory from an executive perspective of, “Be sure not to buy green bananas. You might not be here by the time they ripen.” And I'm wondering what it is that drives that and how you make a mark in a relatively short time frame when you're providing inputs and deciding on strategy, and those decisions may not bear fruit for years.Sam: CIO used to—we used say it stood for ‘Career Is Over' because the tenure is so short. I think there's a couple of reasons why it's so short. And I think there's a way I believe you can have impact in a short amount of time. I think the reason that it's been short is because people aren't sure what they want the CIO role to be.Do they want it to be a glorified finance person who's got a lot of data processing experience, but now really has got, you know, maybe even an MBA in finance, but is not focusing on value creation? Do they want it to be somebody who's all-singing, all-dancing Chief Data Officer with a CTO background who did something amazing and solved a really hard problem? The definition of success is difficult. Often CIOs now also have security under them, which is literally a job I would never ever want to have. Do security for a public corporation? Good Lord, that's a way to lose most of your life. You're the only executive other than the CEO that the board wants to hear from. Every sing—Corey: You don't sleep; you wait, in those scenarios. And oh, yeah, people joke about ablative CSOs in those scenarios. Yeah, after SolarWinds, you try and get an ablative intern instead, but those don't work as well. It's a matter of waiting for an inevitability. One of the things I think is misunderstood about management broadly, is that you are delegating work, but not the responsibility. The responsibility rests with you.So, when companies have these statements blaming some third-party contractor, it's no, no, no. I'm dealing with you. You were the one that gave my data to some sketchy randos. It is your responsibility that data has now been compromised. And people don't want to hear that, but it's true.Sam: I think that's absolutely right. So, you have this high risk, medium reward, very fungible job definition, right? If you ask all of the CIO's peers what their job is, they'll probably all tell you something different that represents their wish list. The thing that I learned at Autodesk, I was only there for 15 months, but we established a fundamental transformation of the work of how cloud platform is done at the company that's still in place a couple years later.You have to realize that you're a change agent, right? You're actually being hired to bring in the bulk of all the different biases and experiences you have to solve a problem that is not working, right? So, when I got to Autodesk, they didn't even know what their uptime was. It took three months to teach the team how to measure the uptime. Turned out the uptime was 97.7% for the cloud, for the world's largest engineering software company.That is 200 hours a year of unplanned downtime, right? That is not good. So, a complete overhaul [laugh] was needed. Understanding that as a change agent, your half-life is 12 to 18 months, you have to measure success not on tenure, but on your ability to take good care of the patient, right? It's going to be a lot of pain, you're going to work super hard, you're going to have to build trust with everyone, and then people are still going to hate you at the end. That is something you just have to kind of take on.As a friend of mine, Jason Warner joined Redpoint Ventures recently, he said this when he was the CTO of GitHub: “No one is a villain in their own story.” So, you realize, going into a big organization, people are going to make you a villain, but you still have to do incredibly thoughtful, careful work, that's going to take care of them for a long time to come. And those are the kinds of CIOs that I can relate to very well.Corey: Jason is great. You're name-dropping all the guests we've had. My God, keep going. It's a hard thing to rationalize and wrap heads around. It's one of those areas where you will not be measured during your tenure in the role, in some respects. And, of course, that leads to the cynical perspective as well, where well, someone's not going to be here long and if they say, “Yeah, we're just going to keep being stewards of the change that's already underway,” well, that doesn't look great, so quick, time to do a cloud migration, or a cloud repatriation, or time to roll something else out. A bit of a different story.Sam: One of the biggest challenges is how do you get the hearts and the minds of the people who are in the organization when they are no fools, and their expectation is like, “Hey, this company's been around for decades, and we go through cloud leaders or CIOs, like Wendy's goes through hamburgers.” They could just cloud-wash, right, or change-wash all their language. They could use the new language to describe the old thing because all they have to do is get through the performance review and outwait you. So, there's always going to be a level of defection because it's hard to change; it's hard to think about new things.So, the most important thing is how do you get into people's hearts and minds and enable them to believe that the best thing they could do for their career is to come along with the change? And I think that was what we ended up getting right in the Autodesk cloud transformation. And that requires endless optimism, and there's no room for cynicism because the cynicism is going to creep in around the edges. So, what I found on the job is, you just have to get up every morning and believe everything is possible and transmit that belief to everybody.So, if it seems naive or ingenuous, I think that doesn't matter as long as you can move people's hearts in each conversation towards, like, “Oh, this person cares about me. They care about a good outcome from me. I should listen a little bit more and maybe make a 1% change in what I'm doing.” Because 1% compounded daily for a year, you can actually get something done in the lifetime of a CIO.Corey: And I think that's probably a great place to leave it. If people want to learn more about what you're up to, how you think about these things, how you view the world, where can they find you?Sam: You can find me on Twitter, I'm @sramji, S-R-A-M-J-I, and I have a podcast that I host called Open||Source||Datawhere I invite innovators, data nerds, computational networking nerds to hang out and explain to me, a software programmer, what is the big world of open-source data all about, what's happening with machine learning, and what would it be like if you could put data in a container, just like you could put code in a container, and how might the world change? So, that's Open||Source||Data podcast.Corey: And we'll of course include links to that in the [show notes 00:35:58]. Thanks so much for your time. I appreciate it.Sam: Corey, it's been a privilege. Thank you so much for having me.Corey: Likewise. Sam Ramji, Chief Strategy Officer at DataStax. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me exactly which item in Sam's background that I made fun of is the place that you work at.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Building a User-Friendly Product with Aparna Sinha

Screaming in the Cloud

Play Episode Listen Later Dec 8, 2021 42:53


About AparnaAparna Sinha is Director of Product for Kubernetes and Anthos at Google Cloud. Her teams are focused on transforming the way we work through innovation in platforms. Before Anthos and Kubernetes, Aparna worked on the Android platform. She joined Google from NetApp where she was Director of Product for storage automation and private cloud. Prior to NetApp, Aparna was a leader in McKinsey and Company's business transformation office working with CXOs on IT strategy, pricing, and M&A. Aparna holds a PhD in Electrical Engineering from Stanford and has authored several technical publications. She serves on the Governing Board of the Cloud Native Computing Foundation (CNCF).Links: DevOps Research Report: https://www.devops-research.com/research.html Twitter: https://twitter.com/apbhatnagar TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. Set up a meeting with a Redis expert during re:Invent, and you'll not only learn how you can become a Redis hero, but also have a chance to win some fun and exciting prizes. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: You know how Git works right?Announcer: Sorta, kinda, not really. Please ask someone else.Corey: That's all of us. Git is how we build things, and Netlify is one of the best ways I've found to build those things quickly for the web. Netlify's Git-based workflows mean you don't have to play slap-and-tickle with integrating arcane nonsense and web hooks, which are themselves about as well understood as Git. Give them a try and see what folks ranging from my fake Twitter for Pets startup, to global Fortune 2000 companies are raving about. If you end up talking to them—because you don't have to; they get why self-service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y dot com.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. We have a bunch of conversations on this show covering a wide gamut of different topics, things that I find personally interesting, usually, and also things I'm noticing in the industry. Fresh on the heels of Google Next, we get to ideally have conversations about both of those things. Today, I'm speaking with the Director of Product Management at Google Cloud, Aparna Sinha. Aparna, thank you so much for joining me today. I appreciate it.Aparna: Thank you, Corey. It's a pleasure to be here.Corey: So, Director of Product Management is one of those interesting titles. We've had a repeat guest here, Director of Outbound Product Management Richard Seroter, which is great. I assume—as I told him—outbound products are the ones that are about to be discontinued. He's been there a year and somehow has failed the discontinue a single thing, so okay, I'm sure that's going to show up on his review. What do you do? The products aren't outbound; they're just products, and you're managing them, but that doesn't tell me much. Titles are always strange.Aparna: Yeah, sure. Richard is one of my favorite people, by the way. I work closely with him. I am the Director of Product for Developer Platform. That's Google Cloud's developer platform.It includes many different products—actually, 30-Plus products—but the primary pieces are usually when a developer comes to Google Cloud, the pieces that they interact with, like our command-line interface, like our Cloud Shell, and all of the SDK pieces that go behind it, and then also our DevOps tooling. So, as you're writing the application in the IDE and as you're deploying it into production, that's all part of the developer platform. And then I also run our serverless platform, which is one of the most developer-friendly capabilities from a compute perspective. It's also integrated into many different services within GCP. So, behind the title, that's really what I work on.Corey: Okay, so you're, I guess, in part responsible for well, I guess, a disappointment of mine a few years ago. I have a habit on Twitter—because I'm a terrible person—of periodically spinning up a new account on various cloud providers and kicking the tires and then live-tweeting the experience, and I was really set to dunk on Google Cloud; I turned this into a whole blog post. And I came away impressed, where the developer experience was pretty close to seamless for getting up and running. It was head and shoulders above what I've seen from other cloud providers, and on the one hand, I want to congratulate you and on the other, it doesn't seem like that's that high of a bar, to be perfectly honest with you because it seems that companies get stuck in their own ways and presuppose that everyone using the product is the same as the people building the product. Google Cloud has been and remains a shining example of great developer experience across the board.If I were starting something net new and did not have deep experience with an existing cloud provider—which let's face it, the most valuable thing about the cloud is knowing how it's going to break because everything breaks—I would be hard-pressed to not pick GCP, if not as the choice, at least a strong number two. So, how did that come to be? I take a look at a lot of Google's consumer apps and, “This is a great user experience,” isn't really something I find myself saying all that often. Google Cloud is sort of its own universe. What happened?Aparna: Well, thank you, first of all, for the praise. We are very humble about it, actually. I think that we're grateful if our developers find the experience to be seamless. It is something that we measure all the time. That may be one of the reasons why you found it to be better than other places. We are continuously trying to improve the time to value for developers, how long it takes them to perform certain actions. And so what you measure is what you improve, right? If you don't measure it, you don't improve it. That's one of our SRE principles.Corey: I wish. I've been measuring certain things for years, and they don't seem to be improving at all. It's like, “Wow, my code is still terrible, but I'm counting the bugs and the number isn't getting smaller.” Turns out there might be additional steps required.Aparna: Yes, you know, we measure it, we look at it, we take active OKRs to improve these things, especially usability. Usability is extremely important for certainly the developer platform, for my group; that's something that's extremely important. I would say, stepping back, you said it's not that common to find a good user experience in the cloud, I think in general—you know, and I've spent the majority of my career, if not all of my career, working on enterprise software. Enterprise software is not always designed in the most user-friendly way; it's not something that people always think about. Some of the enterprise software I've used has been really pretty… pretty bad. Just a list of things.Corey: Oh, yeah. And it seems like their entire philosophy—I did a bit of a dive into this, and I think it was Stripe's Patrick McKenzie who wound up pointing this out originally, though; but the internet is big and people always share and reshare ideas—the actual customer for enterprise software is very often procurement or a business unit that is very organizationally distant from the person who's using it. And I think in a world of a cloud platform, that is no longer true. Yeah, there's a strategic decision of what Cloud do we use, but let's be serious, that decision often comes into play long after there's already been a shadow IT slash groundswell uprising. The sales process starts to look an awful lot less like, “Pick our cloud,” and a lot more like, “You've already picked our cloud. How about we formalize the relationship?”And developer experience with platforms is incredibly important and I'm glad to see that this is a—well, it's bittersweet to me. I am glad to see that this is something that Google is focusing on, and I'm disappointed to admit that it's a differentiator.Aparna: It is a differentiator. It is extremely important. At Google, there are a couple of reasons why this is part of our DNA, and it is actually related to the fact that we are also a consumer products company. We have a very strong user experience team, a very strong measurements-oriented—they measure everything, and they design everything, and they run focus groups. So, we have an extraordinary usability team, and it's actually one of the groups that—just like every other group—is fungible; you can move between consumer and cloud. There's no difference in terms of your training and skill set.And so, I know you said that you're not super impressed with our consumer products, but I think that the practice behind treating the user as king, treating the user as the most important part of your development, is something that we bring over into cloud. And it's just a part of how we do development, and I think that's part of the reason why our products are usable. Again, I shy away from taking any really high credit on these things because I think I always have a very high bar. I want them to be delightful, super delightful, but we do have good usability scores on some of the pieces. I think our command line, I think, is quite good. I think—there's always improvements, by the way, Corey—but I think that there are certain things that are delightful.And a lot of thought goes into it and a lot of multi-functional—meaning across product—user experience and engineering. We have end-developer relations. We have, sort of this four-way communication about—you know, with friction logs and with lots of trials and lots of discussion and measurements, is how we improve the user experience. And I would love to see that in more enterprise software. I think that my experience in the industry is that the user is becoming more important, generally, even in enterprise software, probably because of the migration to cloud.You can't ignore the user anymore. This shouldn't be all about procurement. Anybody can procure a cloud service. It's really about how easily and how quickly can they get to what they want to do as a user, which I think also the definition of what a developer is changing and I think that's one of the most exciting things about our work is that the developer can be anybody; it can be my kids, and it can be anyone across the world. And our goal is to reach those people and to make it easy for them.Corey: If I had to bet on a company not understanding that distinction, on some level, Google's reputation lends itself to that where, oh, great. It's like, I'm a little old to go back to school and join a fraternity and be hazed there, so the second option was, oh, I'll get an interview to be an SRE at Google where, “Oh, great, you've done interesting things, but can you invert a binary tree on a whiteboard?” “No, I cannot. Let's save time and admit that.” So, the concern that I would have had—you just directly contradicted—was the idea that you see at some companies where there's the expectation that all developers are like their developers.Google, for better or worse, has a high technical bar for hiring. A number of companies do not have a similar bar along similar axes, and they're looking for different skill sets to achieve different outcomes, and that's fine. To be clear, I am not saying that, oh, the engineers at Google are all excellent and the engineers all at a bank are all crap. Far from it.That is not true in either direction, but there are differences as far as how they concern themselves with software development, how they frame a lot of these things. And I am surprised that Google is not automatically assuming that developers are the type of developers that you have at Google. Where did that mindset shift come from?Aparna: Oh, absolutely not. I think we would be in trouble if we did that. I studied electrical engineering in school. This would be like assuming that the top of the class is kind of like the kind of people that we want to reach, and it's just absolutely not. Like I said, I want to reach total beginners, I want to reach people who are non-developers with our developer platform.That's our explicit goal, and so we view developers as individuals with a range of superpowers that they've gained throughout their lives, professionally and personally, and people who are always on a path to learn new things, and we want to make it easy for them. We don't treat them as bodies in an employment relationship with some organization, or people with certain minimum bar degrees, or whatever it is. As far as interviewing goes, Corey, in product management, which is the practice that I'm part of, we actually look for, in the interview, that the candidate is not thinking about themselves; they're not imposing themselves on the user base.So, can you think outside of yourself? Can you think of the user base? And are you inquisitive? Are you curious? Do you observe? And how well do you observe differences and diversity, and how well are you able to grasp what might be needed by a particular segment? How well are you able to segment the user base?That's what we look for, certainly in product management, and I'm quite sure also in user experience. You're right, on engineering, of course, we're looking for technical skills, and so on, but that's not how we design our products, that's not how we design the usability of our products.Corey: “If you people were just a little bit smarter slash more like me, then this would work a lot better,” is a common trope. Which brings us, of course, to the current state of serverless. I tend to view serverless as largely a failed initiative so far. And to be clear, I'm viewing this from an AWS-centric lens; that is the… we'll be charitable and call it pool in which I swim. And they announced Lambda in 2015; that's great. “The only code you will ever write in the future is business logic.” Yeah, I might have heard that one before about 15 other technologies dating back to the 60s, but okay.And the expectation was that it was going to take off and set the world on fire. You just needed to learn the constraints of how this worked. And there were a bunch of them, and they were obnoxious, and it didn't have a learning curve so much as a learning cliff. And nowadays, we do see it everywhere, but it's also in small doses. It's mostly used as digital spackle to plaster over the gaps between various AWS services.What I'm not seeing across the board is a radical mindset shift in the way that developers are engaging with cloud platforms that would be heralded by widespread adoption of serverless principles. That said, we are on the heels here of Google Cloud Next, and that you had a bunch of serverless announcements, I'm going to go out on a limb and guess you might not agree with my dismal take on the serverless side of the world?Aparna: Well, I think this is a great question because despite the fact that I like not to be wishy-washy about anything, I actually both agree and disagree [laugh] with what you said. And that's funny.Corey: Well, that's why we're talking about this here instead of on Twitter where two contradictory things can't possibly both be true. Wow, imagine that; nuance, it doesn't fit 280 characters. Please, continue.Aparna: So, what I agree with is that—I agree with you that the former definition of serverless and the constrained way that we are conditioned thinking about serverless is not as expansive as originally hoped, from an adoption perspective. And I think that at Google, serverless is just no longer about only event-driven programming or microservices; it's about running complex workloads at scale while still preserving the delightful developer experience. And this is where the connection to the developer experience comes in. Because the developer experience, in my mind, it's about time to value. How quickly can I achieve the outcome that I need for my business?And what are the things that get in the way of that? Well, setting up infrastructure gets in the way of that, having to scale infrastructure gets in the way of that, having to debug pieces that aren't actually related to the outcome that you're trying to get to gets in the way of that. And the beauty of serverless, it's all in how you define serverless: what does this name actually mean? If serverless only means functions and event-driven applications, then yes, actually, it has a better developer experience, but it is not expansive, and then it is limited, and it's trapped in its skin the way that you mentioned it. [laugh].Corey: And it doesn't lend itself very well to legacy applications—legacy, of course, being condescending engineering-speak for ‘it makes money.' But yeah, that's the stuff that powers the world. We're not going to be redoing all those things as serverless-powered microservices anytime soon, in most cases.Aparna: At Google Cloud, we are redefining serverless. And so what we are taking from Serverless is the delightful user experience and the fact that you don't have to manage the infrastructure, and what we're putting in the serverless is essentially serverless containers. And this is the big revolution in serverless, is that serverless—at least a Google Cloud with serverless containers and our Cloud Run offering—is able to run much bigger varieties of applications and we are seeing large enterprises running legacy applications, like you say, on Cloud Run, which is serverless from a developer experience perspective. There's no cluster, there is no server, there's no VM, there's nothing for you to set up from a scaling perspective. And it essentially scales infinitely.And it is very developer-focused; it's meant for the developer, not for the operator or the infrastructure admin. In reality in enterprise, there is very much a segmentation of roles. And even in smaller companies, there's a segmentation of roles even within the same person. Like, they may have to do some infrastructure work and they may do some development work. And what serverless—at least in the context of Google Cloud—does, is it removes the infrastructure work and maximizes the development work so that you can focus on your application and you can get to that end result, that business value that you're trying to achieve.And with Cloud Run, what we've done is we've preserved that—and I would say, actually, arguably improved that because we've done usability studies that show that we're 22 points above every other serverless offering from a usability perspective. So, it's super important to me that anybody can use this service. Anybody. Maybe even not a developer can use this service. And that's where our focus is.And then what we've done underneath is we've removed many of the restrictions that are traditionally associated with serverless. So, it doesn't have to be event-driven, it is not only a particular set of languages or a particular set of runtimes. It is not only stateless applications, and it's not only request-based billing, it's not only short-running jobs. These are the kinds of things that we have removed and I think we've just redefined serverless.Corey: [unintelligible 00:17:05], on some level, the idea of short-lived functions with a maximum cap feels like a lazy answer to one of the hard problems in computer science, the halting problem. For those not familiar, my layman's understanding of it is, “Okay, you have a program that's running in a loop. How do you deterministically say that it is done executing?” And the functional answer to that is, “Oh, after 15 minutes, it's done. We're killing it.” Which I guess is an answer, but probably not one that's going to get anyone a PhD.It becomes very prescriptive and it leads to really weird patterns trying to work around some of those limitations. And historically, yeah, by working within the constraints of the platform, it works super well. What interests me about Cloud Run is that it doesn't seem to have many of those constraints in quite the same way. It's, “Can you shove whatever monstrosity you've got into a container? You can't? Well, okay, there are ways to get there.”Full disclosure, I was very anti-container; the industry has yet again proven to me that I cannot predict the future. Here we are. “Great, can you shove a container in and hand it to some other place to run it where”—spoiler, people will argue with me on this and they are wrong—“Google engineers are better at running infrastructure to run containers than you are.” Full stop. That is the truism of how this works; economies of scale.I love the idea of being able to take something, throw it over a wall, and not have to think about the rest of it. But everything that I'm thinking about in this context looks certain ways and it's the type of application that I'm working on or that I'm looking at most recently. What are you seeing in Cloud Run as far as interesting customer use cases? What are people doing with it that you didn't expect them to?Aparna: Yeah, I think this is a great time to ask that question because with the pandemic last year—I guess we're still in the pandemic, but with the pandemic, we had developers all over the world become much more important and much more empowered, just because there wasn't really much of an operations team, there wasn't really as much coordination even possible. And so we saw a lot of customers, a lot of developers moving to cloud, and they were looking for the easiest thing that they could use to build their applications. And as a result, serverless and Cloud Run in particular, became extremely popular; I would say hockey stick in terms of usage.And we're seeing everything under the sun. ecobee—this is a home automation company that makes smart thermostats—they're using Cloud Run to launch a new camera product with multi-factor authentication and security built-in, and they had a very tight launch timeline. They were able to very quickly meet that need. Another company—and you talk about, you know, sort of brick and mortar—IKEA, which you and I all like to shop [laugh] at, particularly doing the—Corey: Oh, I love building something from 500 spare parts, badly. It's like basically bringing my AWS architecture experience into my living room. It's great. Please continue.Aparna: Yeah, it's like, yeah—Corey: The Swedish puzzle manufacturer.Aparna: Yes. They're a great company, and I think it just in the downturn and the lockdown, it was actually a very dicey time, very tricky time, particularly for retailers. Of course, everybody was refurbishing their home or [laugh], you know, improving their home environment and their furniture. And IKEA started using serverless containers along with serverless analytics—so with BigQuery, and Cloud Run, and Cloud Functions—and one of the things they did is that they were able to cut their inventory refresh rate from more than three hours to less than three minutes. This meant that when you were going to drive up and do some curbside pickup, you know the order that you placed was actually in stock, which was fantastic for CSAT and everything.But that's the technical piece that they were able to do. When I spoke with them, the other thing that they were able to do with the Cloud Run and Cloud Functions is that they were able to improve the work-life balance of their engineers, which I thought was maybe the biggest accomplishment. Because the platform, they said, was so easy for them to use and so easy for them to accomplish what they needed to accomplish, that they had a better [laugh] better life. And I think that's very meaningful.In other companies, MediaMarktSaturn, we've talked about them before; I don't know if I've spoken to you about them, but we've certainly talked about them publicly. They're a retailer in EMEA, and because of their use of Cloud Run, and they were able to combine the speed of serverless with the flexibility of containers, and their development team was able to go eight times faster while handling 145% increase in digital channel traffic. Again, there are a lot more digital channel traffic during COVID. And perhaps my favorite example is the COVID-19 exposure notifications work that we did with Apple.Corey: An unfortunate example, but a useful one. I—Aparna: Yes.Corey: —we all—I think we all wish it wasn't necessary, but here's the world in which we live. Please, tell me more.Aparna: I have so many friends in engineering and mathematics and these technical fields, and they're always looking at ways that technology can solve these problems. And I think especially something like the pandemic which is so difficult to track, so difficult with the time that it takes for this virus to incubate and so on, so difficult to track these exposures, using the smartphone, using Bluetooth, to have a record of who has it and who they've been in contact with, I think really interesting engineering problem, really interesting human problem. So, we were able to work on that, and of course, when you need a platform that's going to be easy to use, that's going to be something that you can put into production quickly, you're going to use Cloud Run. So, they used Cloud Run, and they also used Cloud Run for Anthos, which is the more hybrid version, for the on-prem piece. And so both of those were used in conjunction to back all of the services that were used in the notifications work.So, those are some of the examples. I think net-net, it's that I think usability, especially in enterprise software is extremely important, and I think that's the direction in which software development is going.Corey: Are you building cloud applications with a distributed team? Check out Teleport, an open source identity-aware access proxy for cloud resources. Teleport provides secure access to anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps and databases. Teleport gives engineers superpowers! Get access to everything via single sign-on with multi-factor. List and see all SSH servers, kubernetes clusters or databases available to you. Get instant access to them all using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility and ensuring compliance. And best of all, Teleport is open source and a pleasure to use.Download Teleport at https://goteleport.com. That's goteleport.com.Corey: It's easy for me to watch folks—like you—in keynotes at events—like Cloud Next—talk about things and say, “This is how the world is building things, and this is what the future looks like.” And I can sit there and pick to pieces all day, every day. It basically what I do because of deep-seated personality problems with me. It's very different to say that about a customer who has then taken that thing and built it into something that is transformative and solves a very real problem that they have. I may not relate to that problem that they have, but I do not believe that customers are going to have certain problems, find solutions like this and fix them, and the wrong in how they're approaching these things.No one sees the constraints that shape things; no one shows up in the morning hoping to do a crap job today unless you know you're the VP of Integrity at Facebook or something. But there's a very real sense of companies have a bunch of different drivers, and having a tool or a service or a platform that solves it for them, you'd better be very sure before you step up and start saying, “No, you're doing it wrong.” In earlier years, I did not see a whole lot of customer involvement with Cloud Next. It was always a, “Well, a bunch of Googlers are going to tell me how this stuff works, and they'll talk about theoretical things.”That's not the case anymore. You have a whole bunch of highly respectable reference customers out there doing a whole lot of really interesting things. And more to the point, they're willing to go on record talking about this. And I'm not talking about fun startups that are, “Great, it's Twitter, only for pets.” Great. I'm talking banks, companies where mistakes are going to show and leave a mark. It's really hard to reconcile what I'm seeing with Google Cloud in 2021 than what I was seeing in, let's say, five or six years ago. What drove that change?Aparna: Yes, Corey, I think you're definitely correct about that. There's no doubt about it that we have a number of really tremendous customers, we really tremendous enterprise references and so on. I run the Google Cloud Developer Platform, and for me, the developers that I work with and the developers that this platform serves are the inspiration for what we do. And in the last six or seven years that I've worked in Google Cloud, that has always been the case. So, nothing has changed from my perspective, in that regard.If anything, what has changed is that we have far more users, we have been growing exponentially, and we have many more large enterprise customers, but in terms of my journey, I started with the Kubernetes open-source project, I was one of the very early people on that, and I was working with a lot of developers, in that case, in the open-source community, a lot of them became GKE customers, and it just grew. And now we have so many [laugh] customers and so many developers, and we have developed this platform with them. We are very much—it's been a matter of co-innovation, especially on Kubernetes. It has been very much, “Okay, you tell us,” and it's a need-based relationship, you know? Something is not working, we are there and we fix it.Going back to 2017 or whenever it was that Pokemon Go was running on GKE, that was a moment when we realized, “Oh, this platform needs to scale. Okay, let's get at it.” And that's where, Corey, it really helps to have great engineers. For all the pros and cons, I think that's where you want those super-sharp, super-driven, super-intelligent folks because they can make things like that happen, they can make it happen in less than a week, so that—they can make it happen over a Saturday so that Pokemon Go can go live in Japan and everybody can be playing that game. And that's what inspires me.And that's a game, but we have a lot of customers that are running health applications. We have a customer that's running ambulances on the platform. And so this is life-threatening stuff; we have to take that very seriously, and we have to be listening to them and working with them. But I'm inspired, and I think that our roadmap, and the products, and the features that we build are inspired by what they are building on the platform. And they're combining all kinds of different things. They're taking our machine learning capabilities, they're taking our analytics capabilities, they're taking our Maps API, and they're combining it with Cloud Run, they're combining it with GKE. Often they're using both of those.And they're running new services. We've got a customer in Indonesia that's running in a food delivery service; I've got customers that are analyzing the cornfields in the middle of the country to improve crop yield. So, that's the kind of inspiring work, and each of those core, each of those users are coming back to us and saying, “Oh, you know, I need a different type of”—it's very detailed, like, “I need a different type of file system that gives me greater speed or better performance.” We just had a gaming company that was running on GKE that we really won out over a different cloud in terms of performance improvements that we were able to provide on the container startup times. It was just a significant performance improvement. We'll probably publish it in the coming few months.That's the kind of thing that drives it, and I'm very glad that I have a strong engineering team in Google Cloud, and I'm very glad that we have these amazing customers that are trying to do these amazing things, and that they're directly engaging with us and telling us what they need from us because that's what we're here for.Corey: To that end, one more area I want to go into before we call this a show, you've had Cloud Build for a little while, and that's great. Now, at—hot off the presses, you wound up effectively taking that one step further with Cloud Deploy. And I am still mostly someone with terrible build and release practices that people would be ashamed of, struggle to understand the differentiation between what I would do with Cloud Build and what I would do with Cloud Deploy. I understand they're both serverless. I understand that they are things that large companies care about. What is the story there?Aparna: Yeah, it's a journey. As you start to use containers—and these days, like you said, Corey, containers, a lot of people are using them—then you start to have a lot of microservices, and one of the benefits of container usage is that it's really quick to release new versions. You can have different versions of your application, you can test them out, you can roll them out. And so these DevOps practices, they become much more attainable, much more reachable. And we just put out the, I think, the seventh version of the DevOps Research Report—the DORA report—that shows that customers that follow best practices, they achieve their results two times better in terms of business outcomes, and so on.And there's many metrics that show that this kind of thing is important. But I think the most important thing I learned during the pandemic, as we were coming out of the pandemic, is a lot of—and you mentioned enterprises—large banks, large companies' CIOs and CEOs who basically were not prepared for the lockdown, not prepared for the fact that people aren't going to be going into branches, they came to Google Cloud and they said that, “I wish that I had implemented DevOps practices. I wish that I had implemented the capability to roll out changes frequently because I need that now. I need to be able to experiment with a new banking application that's mobile-only. I need to be able to experiment with curbside delivery. And I'm much more dependent on the software than I used to be. And I wish that I had put those DevOps practices.”And so the beginning of 2021, all our conversations were with customers, especially those, you know you said ‘legacy,' I don't think that's the right word, but the traditional companies that have been around for hundreds of years, all of them, they said, “Software is much more important. Yes, if I'm not a software company, at least a large division of my group is now a software group, and I want to put the DevOps practices into play because I know that I need that and that's a better way of working.”By the way, there's a security aspect to that I'd like to come back to because it's really important—especially in banking, financial services, and public sector—as you move to a more agile DevOps workflow, to have security built into that. So, let me come back to that. But with regard to Cloud Build and Cloud Deploy is something I've been wanting to bring into market for a couple of years. And we've been talking about it, we've been working on it actively for more than a year on my team. And I'm very, very excited about this service because what it does is it allows you to essentially put this practice, this DevOps practice into play whereas your artifacts are built and stored in the artifact repository, they can then automatically be deployed into your runtime—which is GKE Cloud Run—in the future, you can deploy them, and you can set how you want to deploy them.Do you want to deploy them to a particular environment that you want to designate the test environment, the environment to which your developers have access in a certain way? Like, it's a test environment, so they can make a lot of changes. And then when do you want to graduate from test to staging, and when do you want to graduate to production and do that gradual rollout? Those are some of the things that Cloud Deploy does.And I think it's high time because how do you manage microservices at scale? How do you really take advantage of container-based development is through this type of tooling. And that's what Cloud Deploy does. It's just the beginning of that, but it's a delightful product. I've been playing around with it; I love it, and we've seen just tremendous reception from our users.Corey: I'm looking forward to kicking the tires on it myself. I want to circle back to talk about the security aspect of it. Increasingly, I'm spending more of my attention looking at cloud security because everyone else has, too, and some of us have jobs that don't include the word security but need to care about it. That's why I have a Thursday edition of my newsletter, now, talking specifically about that. What is the story around security these days from your perspective?And again, it's a huge overall topic, and let's be clear here, I'm not asking, “What does Google Cloud think about security?” That would fill an encyclopedia. What is your take on it? And where do you want to talk about this in the context of Cloud Deploy?Aparna: Yeah, so I think about security from the perspective of the Google Cloud Developer Platform, and specifically from the perspective of the developer. And like you said, security is not often in the title of anybody in the developer organization, so how do we make it seamless? How do we make it such that security is something that is not going to catch you as you're doing your development? That's the critical piece. And at the same time, one of the things we saw during 2020 and 2021 is just the number of cyberattacks just went through the roof. I think there was a 400 to 600% increase in the number of software supply chain attacks. These are attacks where some malicious hacker has come in and inserted some malicious code into your software. [laugh]. Your software, Corey. You know, you the unsuspecting developer is—Corey: Well, it used to be my software; now there's some debate about that.Aparna: Right. That's true because most software is using open-source dependencies; and these open-source dependencies, they have a pretty intricate web of dependencies that they are themselves using. So, it's a transitive problem where you're using a language like Python, or whatever language you're using. And there's a number of—Corey: Crappy bash by default. But yes.Aparna: Well, it was actually a bash script vulnerability, I think, in the Codecov breach that happened, I think it was, in earlier this year, where a malicious bash script was injected into the build system, in fact, of Codecov. And there are all these new attack vectors that are specifically targeting developers. And whether it's nation-states or whoever it is that's causing some of these attacks, it's a problem that is of national and international magnitude. And so I'm really excited that we have the expertise in Google Cloud and beyond Google Cloud.Google, it's a very security-conscious company. This company is a very security-conscious company. [laugh]. And we have built a lot of tooling internally to avoid those kinds of attacks, so what we've done with Cloud Build, and what we're going to do with Cloud Deploy, we're building in the capability for code to be signed, for artifacts to be signed with cryptographic keys, and for that signing, that attestation—we call it an attestation—that attestation to be checked at various points along the software supply chain. So, as you're writing code, as you're submitting the code, as you're building the containers, as you're storing the containers, and then finally as you're deploying them into whatever environment you're deploying them, we check these keys, and we make sure that the software that is going through the system is actually what you intended and that there isn't this malicious code injection that's taking place.And also, we scan the software, we scan the code, we scan the artifacts to check for vulnerabilities, known vulnerabilities as well as unknown vulnerabilities. Known vulnerabilities from a Google perspective; so Google's always a little bit ahead, I would say, in terms of knowing what the vulnerabilities are out there because we do work so much on software across operating systems and programming languages, just across the full gamut of software in the industry, we work on it, and we are constantly securing software. So, we check for those vulnerabilities, we alert you, we help to remediate those vulnerabilities.Those are the type of things that we're doing. And it's all in service of certainly keeping enterprise developers secure, but also just longtail an average, everybody, helping them to be secure so that they don't get hacked and their companies don't get hacked.Corey: It's nice to see people talking about this stuff, who is not directly a security vendor. But by which I mean, you're not using this as the fear, uncertainty, and doubt angle to sell a given service that, “We have to talk about this exploit because otherwise, no one will ever buy this.” Something like Cloud Deploy is very much aligned with a best practices approach to release engineering. It's not, strictly speaking, a security product, but being able to wrap things that are very security-centric around it is valuable.Now, sponsors are always going to do interesting things at various expo halls, and oh, yeah, saw the same product warmed over. This is very much not that, and I don't interpret anything you're saying is trying to sell something via the fear, uncertainty, and doubt model. There are a lot of different areas that I will be skeptical hearing about from different companies; I do take security words from Google extremely seriously because, let's be clear, in the past 20 however many years it has been, you have established a clear track record for caring about these things.Aparna: Yeah. And I have to go back to my initial mission statement, which is to help developers accelerate time to value. And one of the things that will certainly get in the way of accelerating time to value is security breaches, by the nature of them. If you are not running a supply chain that is secure, then it is very difficult for you to empower your developers to do those releases frequently and to update the software frequently because what if the update has an issue? What if the update has a security vulnerability?That's why it's really important to have a toolchain that prevents against that, that checks for those things, that logs those things so that there's an audit trail available, and that has the capability for your security team to set policies to avoid those kinds of things. I think that's how you get speed. You get with security built in, and that's extremely important to developers and especially cloud developers.Corey: I want to thank you for taking the time to speak to me about all the things that you've been working on and how you view this industry unfolding. If people want to learn more about what you're up to, and how you think about these things, where can they find you?Aparna: Well, Corey, I'm available on Twitter, and that may be one of the best ways to reach me. I'm also available at various customer events that we are having, most of them are online now. And so I'll provide you more details on that and I can be reached that way.Corey: Excellent. I will, of course, include links to that in the [show notes 00:38:43]. Thank you so much for being so generous with your time. I appreciate it.Aparna: Thank you so much. I greatly enjoyed speaking with you.Corey: Aparna Sinha, Director of Product Management at Google Cloud. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. And that sentence needed the word ‘cloud' about four more times in it. And if you've enjoyed this episode, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a loud angry comment telling me that I just don't understand serverless well enough.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Leveling Financial Brevity with Dan Shapiro

Screaming in the Cloud

Play Episode Listen Later Dec 7, 2021 40:18


About DanAfter earning his CPA in New York, Dan dedicated his early career to education, helping to build eight schools across two continents. Three of those schools make up the charter school network Coney Island Prep, a 160-person, $20mm+ organization where Dan served as both CFO and COO.  He has served as CFO of many fast-growth start-ups, is a recurring guest lecturer at Harvard Graduate School of Education, and is an avid adventurer and musician.  But most importantly, he's a dedicated husband and an enamored father who, at this point, knows the lyrics to each and every Raffi song ever created.Links:Duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Redis, the company behind the incredibly popular open source database that is not the bind DNS server. If you're tired of managing open source Redis on your own, or you're using one of the vanilla cloud caching services, these folks have you covered with the go to manage Redis service for global caching and primary database capabilities; Redis Enterprise. Set up a meeting with a Redis expert during re:Invent, and you'll not only learn how you can become a Redis hero, but also have a chance to win some fun and exciting prizes. To learn more and deploy not only a cache but a single operational data platform for one Redis experience, visit redis.com/hero. Thats r-e-d-i-s.com/hero. And my thanks to my friends at Redis for sponsoring my ridiculous non-sense.  Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. A common myth that has sort of permeated the entire ecosystem is, when you associate a single person with a company, they're the only person is really there. It turns out that with AWS, Jeff Barr is writing an awful lot of blog posts, but I have it on good authority that there are at least three services he didn't personally create himself. Similarly here at The Duckbill Group, there are a lot of folks who aren't necessarily in the public eye as much as I am because they don't have the overriding personality flaws that I do. One of those people is my guest today. Dan Shapiro is our CFO, and I guess the closest thing you could consider to us having adult supervision. Dan, thanks for joining me.Dan: Thanks for having me, and I appreciate you considering me an adult, even though I don't always feel that way.Corey: I always assume there's someone else out there who knows what's going on and how life works, and one day they're going to bother to explain it to me because the alternative is that none of us know and we're making it up as we go along, and I'm not sure I can wrap my head around that fear.Dan: Yeah, my job is to allow you guys to have the vision, some wild ideas, try to put data to ideas, tell you if I think it's a good idea. I think, you know, I'm the balloon string to your guys' balloon is the way I think about it.Corey: It's funny, given that we fix the AWS bill and deal with customers, who are in many cases themselves working in finance, that for the first year or so that we were doing this as The Duckbill Group, and never back when I was independent did we really have anyone in a CFO position. What we were doing made sense to me from a very simplistic naive point of view. Okay, well, how is the company doing? Well, I have an app that tells me that. No, it's not QuickBooks, it's I'm going to pull up the banking app and see how much money is currently sitting in the company accounts.And that would inform things like can we hire new people, can we afford this piece of equipment, or embark on this project? And it was sort of Fisher-Price finance, from our perspective. And when you came in, you were very good at this in that you did not make us feel like the naive hayseeds we very clearly were. You were excellent at hiding your contempt, which is great. But before we dive into the specifics, let's ask the big question that I didn't know the answer to back when we first started working with you, which is, what does a CFO do?Dan: CFO does a lot of tactical things, but ultimately, I think that they have two main 30,000-foot view functions. One is to safeguard the assets of the company, and two is to ensure that those assets are employed in the best way possible for the best outcomes that the company is after. And that doesn't always necessarily have to be financial; it could be operational successes. But whatever the goals of the company are, the CFO's role is to utilize the assets to help approximate those goals as best as you can, get the most out of your resources. So, I think that, you know, you play a little bit of defense in trying to make sure that you're protected and you play a little bit of offense in trying to make sure that when you put your chips on the table, that they're in the right place.Corey: So, please take this in the spirit of which it's intended, but if I'm starting out my career today, I can attend a boot camp, learn how the ins-and-outs of software programming work, get a job somewhere, and if I manage my career right, in two or three years I'll get a job somewhere as a senior engineer. So, from that perspective, are you more or less an accountant with title inflation? What's going on? What is the difference between finance and accounting?Dan: Yeah, so to define those two words, I think of the delineator being time, right? So, today backwards is accounting. It's a historical record of everything that has happened, well categorized. And I think of finance as today forward. It is the strategy and planning and potentially analysis of what you think is going to happen in the future.So, those two buckets are delineated by time in my mind. Am I an accountant? Yes. Do I think that the best financial people have an accounting background? I do. I think accounting is the alphabet of finance and I think it's hard to really fully understand or be effective without, you know, at least some baseline accounting information. So yeah, I think accounting is super important. It's a codebase to an engineer, except there's really only one code.Corey: I will say that once you started working with us, it enabled me to understand things in a way that I hadn't before. And on some level, I feel… a little embarrassed, I'm in my mid 30s, here—which annoys the hell out of my wife because I'm 39—but I've gotten this far in my career without understanding a lot of the basic literacy of corporate finance. Now, let's be clear, I'm very good at handling personal finance aspects to my life. I can quote chapter and verse in some cases, from ERISA requirements around 401(k)s and how much I'm allowed to put in in a given year. And that's great, but business finance looks very different than personal finance in a few key ways. And I'm curious as to what your impression is of those key differences because I have thoughts, but I don't generally invite experts in areas onto this show and then explain their field to them.Dan: [laugh]. I will go into that but I am curious what you think. You know, or what you perceive in your experience in working with me, what you've seen—maybe one or two differences that you've seen between personal finance and corporate finance. I certainly feel like there's differences, but I actually don't feel like there's a tremendous differences, I just think it's much greater scale with a lot more people involved. But I'm curious what your experience has been, and then I can kind of jump off of that.Corey: Sure. From my perspective, it is—and this is probably heresy, but I tend to view finance in many ways as being more psychological than it is mathematical. And if I were to pick a random person off the street, and give them a choice of you need to come up with a thousand bucks: you can either make another $1,000 or you can save another $1,000—and let's be clear, I'm talking about someone who is in a technical-style career track; I understand the margins, this becomes a very different thing in either direction and I want to be sensitive to that—but if I ask most people, their answer is almost universally going to be to save the money because if you look at folks who are in a salary job, if they want to make more money, they need to either come up with a side project and start moonlighting, they need to petition their boss for a raise, they need to do a few other things here and there and okay, it becomes a pain. And well, I haven't updated my resume in years, I'm not great at interviewing for jobs, I don't really want to leave and upset the applecart. Whereas saving a thousand bucks when you're making six figures a year is not necessarily that hard. Eat out less, cancel Netflix, et cetera, et cetera, and you can get there by saving.Companies philosophically are the exact opposite. Because there's a theoretical upper bound of one hundred percent of a company's AWS bill that I can cut, either by moving them to another provider—which is cheating—or flying to Seattle and taking hostages—which is not particularly something we'd like to do, and it doesn't generally work more than once. And that's fine, but they can earn a multiple of that by launching the right feature or product to the right market at the right time, faster. That's always going to be more compelling for a company because they're after growth, not protecting the baseline that they have, in most cases. When that shifts, they tend to be companies in decline.Dan: Yeah, that's a great point. And I think to add onto that, when you're running a company, almost every company is going to have something like 60 to 70% of their expenses tied up in their people. So, cutting discretionary spending at a corporate level is not always going to yield the impact that you might need it to. And if you actually need to cut expenses, you're talking about very serious decisions about reducing your workforce, which happens in big steps, right? You can't just find, you know, 5k here, 10k there; you're talking about, you know, reducing your org chart, which is a very serious decision that a lot of companies take very seriously, which is great.Versus there's a lot of avenues to increasing revenue, you know, new product streams, growing your team, investing in your team, raising prices, follow-on sales with these existing customers. You know, there's just a lot more opportunity to generate revenue because companies, by definition, have a lot more opportunities to create revenue than just, you know, like you said, somebody who has a salary job.Corey: One of the things that really woke me up to what our customers are going through is I take a look at the finances of The Duckbill Group—as pointed out and categorized and [aligned 00:09:09] by you, which first, thank you, it's extremely helpful, and two, it helps me contextualize things. You raised a flag on things being out of bounds, where they had been historically. At one point, our AWS bill was a little over $2,000 a month, and your question was, “What's up with this?” It is not the sort of thing that was going to make or break the company; our payroll is six figures a month and compared to that the AWS bill is irrelevant.But given what we do, and given that it's always a good idea to be good financial stewards of the money that has been entrusted to us, “Great, let's take a look at that.” And it was dropped down to 700, 800 bucks. I think it [crested 00:09:47] at $950 last month, as of the time of this recording because I was doing some fun experiments. And sure it's annoying in that I want to keep it as low as possible just given the nature of what we do, but from a business perspective, it does not fundamentally matter. Now, that is not the case for most of our clients, it matters; it's a large expense, but payroll is still bigger.In many cases, depending on the company, real estate is bigger. And Netflix, one of the larger AWS customers, has publicly stated that their biggest expense is content. Yeah, they've built a bunch of studios, they have to get rights to all the content that they stream, they pay people through the nose, and their AWS bill is reportedly massive—it would have to be given what they do—but it's never the number one driving focus. And, on some level, it feels like a company deals with two classes of problem. There's the side that we're on of cost control, risk mitigation, et cetera—insurance hangs out here, too—and the other is speeding up time to market or growth and expansion. Our class of problem is a good diligence thing, but it isn't usually ‘summon the board in the middle of the night because of an opportunity' territory. If I look at those as the two great problems, it is more lucrative as a business to be targeting the former.Dan: Yeah. I mean, so just to go back to that AWS example for a second, right, it's contradictory, right, because on the one hand, are you really going to win the day by shaving, you know, $500 off your AWS bill a month if you're a $3 million company? You know, is that going to move the needle? Not necessarily.But I believe in hygiene and habits, and I believe as you're a growing company, that it's a lot harder to put in good financial process and good financial hygiene, the bigger you get. And so if you start doing these things early, you know, if you have a quarterly review of your subscriptions, your SaaS products, you have some kind of budget versus actual process in place where you have—even if it's wrong, right? I mean, you just take a stab at what you think, you track your actuals, and at least now you have some data to rally around from a financial perspective, as opposed to saying, “Oh, look, we spent X on Y. That's interesting.” Or, “Does that feel right? Or”—And so you have a baseline and you have a measuring stick, and then you have a process in place for figuring out what happened, and then how to better guess in the future. And so yeah, I think that hygiene is important and I think the hard part—and I think it was exhibited by you guys early on—is when you're starting a company, this is the last thing that you want to focus on, or even think you need to focus on because it's not on fire, right? Finance is never on fire until it's absolutely on fire, and it's going to kill you. And I think a lot of founders sweep finance and accounting to the back of the closet because a client is upset, or an employee quit, or their code broke, and those things are on fire today, and if they don't get fixed, the company can't move forward.And so finance, it doesn't matter, doesn't matter, doesn't matter, doesn't matter; it kills your company. So, I think it's not that the founders don't know about it, aren't smart enough for it, don't care about it. It's just that it's not on fire, and when you're starting a company, all you can do is pay attention to the things that are.Corey: It's easy to look back and beat myself up for my lack of understanding or lack of focus on things. The similarly I talked to technical people all the time where the first DevOps hire into an environment. It's been application engineers or developers building the environment so far, and it's easy to look at it in a condescending way of, “Oh, our entire field knows not to build things this way. What's wrong with you fools?” Well, it worked well enough to get to a point where they could afford to hire you, so perhaps show some respect when you're in an environment like that.This is something that most business folk, like I don't know, a CFO intrinsically understands, but many engineers still don't seem to have fully wrap their heads around just because it's easy to over-index on the area that you're focusing in. You almost certainly view companies, start to finish, through a lens of finance. A lot of engineering folks I spend time with—and used to be one of, myself—view it through a lens of well, what's their stack look like? What's their technical debt? How are they actually architected?Which is interesting, but usually not the indicator of whether a company will succeed or fail. There are other broader focuses that are important to look at, and being able to view our own company through a lens of something other than dealing with just the technology or just the sales aspect of it, or, “All right, time for me to go shitposting again because that's our substitute for marketing.” Instead, we're focusing on what the larger picture is through a finance lens. It introduces a level of rigor that I hadn't expected, though, clearly, we do still put our own stamp on it. I suspect we are probably the only company you have ever worked with that has an explicit line item in the budget labeled ‘Spite' for example, from which we make our ridiculous parody videos and other assorted nonsense, for those who are unfamiliar with the Spite Budget.Dan: That was a new one for me. I've seen, I think, every other chart of account line item before and Duckbill Group was the first one that I had to type in ‘Spite Budget' for. I didn't even really understand what it meant, and then as I, you know, got to know you a little bit better, I knew exactly what it meant. And it is—Corey: Yeah, this is what passes humor with his set.Dan: [laugh].Corey: Got it. Okay, I'll smile, nod, and continue to keep the finances in order.Dan: Yeah. So, when you leave, I just cross it out and I write marketing. But [laugh] it is what makes you you and what makes this company successful. But in all seriousness, it's an outstanding financial investment because our business is—like every business—is driven by eyeballs. And whether you're a media company or you're any other company, you need people to know about you.You need people to want to believe that your services are valuable, and want to engage in your services. And the Spite Budget brings eyeballs. And the other thing is, I think it's always wrapped up in farce, but there's always an underlying truth to it which is why people find it compelling. I think if you were out there actually, slinging shit to us, you know, [laugh] you're parlance, I think you wouldn't get 100 Twitter followers. But I think because the undercurrent is truth, you don't give yourself credit for this but there's an immense amount of knowledge behind the truth that people find it compelling. So yeah, from a serious financial perspective, I think it's one of the best investments that we make. [laugh].Corey: It's definitely a lot of fun. And it always bothered me when I would walk around re:Invent or travel for client trips and whatnot to see billboard ads in airports because I've priced out what those things cost, and for those who are wondering, it's not a small number. And okay, so you're going to go to all of the expense of buying out ads in multiple airports doing a country wide brand saturation campaign, and with all of that space, and all of those millions of dollars you're spending on this to get your message in front of folks, you fill it with something that is so anodyne, something that is so… I guess, droll, that no one notices or cares. You wind up getting all of these eyeballs and then have nothing interesting to say. And to me, that's the cardinal sin.I can build an audience, and the way I built it is by having something interesting to say. But take a look at any brand awareness billboard for a large consultancy. I mean, Accenture had one years ago in airports that I still haven't stopped making fun of them for, even more so since they blocked me on Twitter. And it said, “The new isn't on its way. It's here now. New, applied now.” So yeah, I guess they're bringing the new and it's… okay, they spent how many millions of dollars on that campaign and then wound up effectively building the tagline and the phrasing just by, I don't know, having some analyst in some junior role bang their head off a keyboard a few times? I don't get it. I just don't get it.Dan: Yeah, without being a marketing expert, I think that's the product of risk tolerance being a directly inverse relationship to the height of an org chart. I think the higher you grow in your decision-making capability and potency, the more responsibility you have on your shoulders, the less willing you are to do anything that's interesting, fun, risky, and that's why we have a lot of the marketing that we have today. We're lucky to be a small company, we're lucky to be a small company that has creative founders. I think a lot of founders for their marketing go, they use a lot of external folks, and you know, they—sometimes it works, but a lot of times they lose the actual, like, heart and soul of the company because they just aren't in it, they don't understand it. And I think it's fun, a lot of the stuff that, you know, we get to do.But I think we're fortunate that we are in a unique spot in the sense that—you know, we talk about this a lot—we're a company that has a services arm and we're a company that has a media arm. And where many companies in services have to think about marketing and sales in a very straight up and down way, we get to play in our media space, which is a revenue-generating arm of our business, in a lot of experimental ways that other companies are spending thousands, millions of dollars on marketing expenses, we get to do it and make money doing it, via media. And it's really this incredible, bifurcated business where one serves the other and vice versa. And, you know, it's fascinating.Corey: I still don't pretend to understand how we got to the place that we did. When you look back, it's easy to see a sense of plodding inevitability, where, “Oh, yeah. You did this, that led to that, and it led to this, and here you are now.” But at the time you're making the decisions, you are throwing darts blindfolded. Professional advice for those in bars, don't do that. They do not find it nearly as amusing as it sounds.Dan: I think that's how it feels, but I don't actually believe that story that you like to tell. You and Mike are—[laugh] you enjoy the self-deprecation but you're very bright guys and I think you are always fiscally responsible. You just didn't have the language that I have to show how you're fiscally responsible. And I think you really were conservative founders in the fact that you made very short-term decisions and always made conservative short-term decisions, and that puts you in a good place. I think what I have brought to the table is an ability to look a little bit further out and think about, you know, okay, it's not just next month, right? It's not just two months from now. What are we building here? What are the long-term goals, and what are the intermediary goals that will be true if we're on the right path?And that's some planning, that's using historicals, that's using some good analysis tools to try and get there, but it's also a matter of time. When you're a founder of a company, you can't spend all of your time on finance and accounting. It's just the reality. You have 8 million things to do, and so you do exactly what you guys did, which is you look at your bank account, you try and make a decision in a vacuum. But you don't have time to really grind over the details, or the strategy of the next 6, 12, 18 months because you've got people to hire, and you've got clients to appease, and you've got work to do that will generate the revenue.And so there's a whole world of fractional CFOs, some who are good, some are not good. There's a world of bookkeepers out there that are competent. And I think, even though it's not a great use of cash, or revenue-generating use of cash, which I think a lot of startups, you know, are reticent to go down that path, I think having good hygiene and good strategy on your financial front early on is necessary for the future planning of a lot of these startups. And also probably peace of mind, right? I mean, I'm not sure—Corey: Oh, I sleep way better now than I did before.Dan: [laugh]. Yeah, I hope so. I mean, I think just the not knowing creates such an undertone of stress that, you know, may or may not be recognized by founders. And I think being able to look at a document and say, “Okay, this makes sense to me, I know what the future probably holds.” And I hope it allows you to think about other things than money.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: One challenge we had was that a lot of the advice for first-time founders—since there's a lot of those in our space—is not applicable to us. The way that Mike and I built this place, we provided the initial investment personally; we're the only investors in this place, we don't have outside funding, we don't have investors, and we've given no equity away, so it is all us. And the way that most companies in tech tend to work is I would go and debase myself in front of a bunch of investors, and some VC would absolutely bite on my Twitter for Pets idea and give me $50 million to go and build the prototype.And at that point, I'm not making money. I have $50 million in the bank that I am burning through every month as I hire, as I build the MVP, et cetera, et cetera, and there's a ticking clock–it's called a runway in our space—and if we don't wind up hitting a certain milestone of being able to raise more money by the time that we've crossed a certain point, it's time to begin an orderly shutdown of the company. And that's a ticking clock hanging over the head of many founders. In our case, I was looking at it like that originally, but we are profitable, we are actively closing deals, and one of the things you taught me, for example, was when we have accounts receivable, money that is owed to us that has not yet been paid, we can count that and use that as in many ways an asset because it is. It also helps the fact that we are in a market where businesses do not generally decline to pay us money that they owe us.This is very much an understood thing in business, but my question was always, “Well, yeah, but what if they don't pay? What does that mean for us?” And that's really been a non-issue, which at first I was really grateful for and thought, “Wow, we have great customers.” It turns out that this is expected. This is like running a store and being super ecstatic because most of the people in your store aren't shoplifting.It's, yes, that is the baseline expectation for people doing business in the modern era. That was an eye opener. But it also kept me up at night because it was, “Oh, if suddenly this money in the bank is all that we're going to get in, then we only have enough runway to go two, three, four months and then we have to shut the company down.” Yeah, but we're profitable every month, so what's the concern here? It doesn't work that way.Dan: Right. We look at two key metrics as a company every single week, right? We look at a metric called months of cash.Corey: Which is pretty self-descriptive. It's in the title.Dan: Yeah, [laugh] just, you take your bank balance and you divide it by your, you know, average monthly burn, you know, total expenses out each month. And that will give you some number. You know, we really like that number to be at least two, closer to three, but I think something like having, like, six months of cash—now this is for a company that has not raised a bunch of money, right? So, if you've raised $5 million, your months of cash is going to be well more than that because you're investing in your—Corey: And the goal should be to lower that months of cash because it's designed to be used to hire and grow, not sit there, and you should not be turning a profit on that interest; you should be spending through it at a reasonable clip.Dan: Right. And for companies who have not raised money, right, even if you have a month's of cash, you know, five, six, that's probably not right either, right? I mean, there's probably opportunities for investment where you want to get your chips on the table for growth, right? Because you don't always have to be after hypergrowth. But typically, if a company's not growing, it's declining, and that's not a great thing, right? So, some amount of growth is always important for a healthy company.So, months of cash is one metric that we look at every week. And the other is what I call an adjusted quick ratio. But essentially, it's taking current assets over your current liabilities. Now, many startups are going to be debt free, short-term debt free, and so what we do is we try and talk about cash plus accounts receivable—so invoices that are out that have not been paid—and that gives you your numerator, over any outflows that you should experience over the next 30 or so days, right, we think about that as, like, our current liabilities. Because there's no real debt.In larger corporate finance, you have big, big balance sheets, quick ratios are more formulaic corporate formula, but in this case, we look at cash plus accounts receivable, over 30 days of liabilities. And that gives us a number that we're tracking constantly. And we have our own benchmarks for internally for what that number is, but that's a really great tracking of liquidity; that's more than just what's the cash in the bank, right? You're looking at your cash, you're looking at your receivables, and you're looking at your upcoming payments, you know, whether that's payroll or big vendor payments or any other, you know, non-recurring expenses that you know are coming down the pike.Corey: Yeah, a lot of things can impact that number. And that means, oh, that number is out of kilter. It's time to dig a little bit into why, but it's not itself a diagnostic, it's just an indicator that gives a snapshot of overall financial health.Dan: That's right. There are also ratios that you can track on a weekly basis and not just wait month-over-month because you can actually start to see trend lines up and down and start to ask questions about, “Hey, what happened this week?” “Oh, we sent out a big invoice, you know, we just haven't been paid on it. That's why we saw our AQR go up.” Or, “Hey, we just took in a big invoice for redoing the website. We owe $50,000 to x vendor and that's why the denominator of that equation went up. And so our AQR went down a little bit this week, but we expect it to climb back up in the coming weeks.”And so it's a really good way to track liquidity. On a longer-term, we're talking about things like margin analysis, right? Gross margins is an important thing we talk about. You know, revenue, minus the cost of goods to deliver that revenue, want to make sure that we're delivering products with efficiency. And, you know, we talk about net margin or EBITDA, or, you know, however, you want to describe the bottom line, and that's obviously your profitability.So, those are sort of the things we rally around internally. We try and look at them at least weekly, or monthly, depending on the metric. And this just goes back to hygiene and habits. You don't have to do a