Podcasts about Hacker News

Share on
Share on Facebook
Share on Twitter
Share on Reddit
Copy link to clipboard

Social news website

  • 184PODCASTS
  • 530EPISODES
  • 31mAVG DURATION
  • 1WEEKLY EPISODE
  • Jan 13, 2022LATEST
Hacker News

POPULARITY

20122013201420152016201720182019202020212022


Best podcasts about Hacker News

Latest podcast episodes about Hacker News

Screaming in the Cloud
“Cloudash”ing onto Mac with Maciej Winnicki

Screaming in the Cloud

Play Episode Listen Later Jan 13, 2022 34:41


About MaciejMaciej Winnicki is a serverless enthusiast with over 6 years of experience in writing software with no servers whatsoever. Serverless Engineer at Stedi, Cloudash Founder, ex-Engineering Manager, and one of the early employees at Serverless Inc.Links: Cloudash: https://cloudash.dev Maciej Winnicki Twitter: https://twitter.com/mthenw Tomasz Łakomy Twitter: https://twitter.com/tlakomy Cloudash email: hello@cloudash.dev TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: This episode is sponsored in part by our friends at Rising Cloud, which I hadn't heard of before, but they're doing something vaguely interesting here. They are using AI, which is usually where my eyes glaze over and I lose attention, but they're using it to help developers be more efficient by reducing repetitive tasks. So, the idea being that you can run stateless things without having to worry about scaling, placement, et cetera, and the rest. They claim significant cost savings, and they're able to wind up taking what you're running as it is in AWS with no changes, and run it inside of their data centers that span multiple regions. I'm somewhat skeptical, but their customers seem to really like them, so that's one of those areas where I really have a hard time being too snarky about it because when you solve a customer's problem and they get out there in public and say, “We're solving a problem,” it's very hard to snark about that. Multus Medical, Construx.ai and Stax have seen significant results by using them. And it's worth exploring. So, if you're looking for a smarter, faster, cheaper alternative to EC2, Lambda, or batch, consider checking them out. Visit risingcloud.com/benefits. That's risingcloud.com/benefits, and be sure to tell them that I said you because watching people wince when you mention my name is one of the guilty pleasures of listening to this podcast.Corey: Welcome to Screaming in the Cloud. I'm Cloud Economist Corey Quinn. And my guest today is Maciej Winnicki, who is the founder of Cloudash. Now, before I dive into the intricacies of what that is, I'm going to just stake out a position that one of the biggest painful parts of working with AWS in any meaningful sense, particularly in a serverless microservices way, is figuring out what the hell's going on in the environment. There's a bunch of tools offered to do this and they're all—yeee, they aspire to mediocrity. Maciej, thank you for joining me today.Corey: Welcome to Screaming in the Cloud. I'm Cloud Economist Corey Quinn. And my guest today is Maciej Winnicki, who is the founder of Cloudash. Now, before I dive into the intricacies of what that is, I'm going to just stake out a position that one of the biggest painful parts of working with AWS in any meaningful sense, particularly in a serverless microservices way, is figuring out what the hell's going on in the environment. There's a bunch of tools offered to do this and they're all—yeee, they aspire to mediocrity. Maciej, thank you for joining me today.Maciej: Thank you for having me.Corey: So, I turned out to have accidentally blown up Cloudash, sort of before you were really ready for the attention. You, I think, tweeted about it or put it on Hacker News or something; I stumbled over it because it turns out that anything that vaguely touches cloud winds up in my filters because of awesome technology, and personality defects on my part. And I tweeted about it as I set it up and got the thing running, and apparently this led to a surge of attention on this thing that you've built. So, let me start off with an apology. Oops, I didn't realize it was supposed to be a quiet launch.Maciej: I actually thank you for that. Like, that was great. And we get a lot of attention from your tweet thread, actually because at the end, that was the most critical part. At the end of the twitter, you wrote that you're staying as a customer, so we have it on our website and this is perfect. But actually, as you said, that's correct.Our marketing strategy for releasing Cloudash was to post it on LinkedIn. I know this is not, kind of, the best strategy, but that was our plan. Like, it was like, hey, like, me and my friend, Tomasz, who's also working on Cloudash, we thought like, let's just post it on LinkedIn and we'll see how it goes. And accidentally, I'm receiving a notification from Twitter, “Hey, Corey started tweeting about it.” And I was like, “Oh, my God, I'm having a heart attack.” But then I read the, you know—Corey: Oops.Maciej: [laugh]. Yeah. I read the, kind of, conclusion, and I was super happy. And again, thank you for that because this is actually when Cloudash kind of started rolling as a product and as a, kind of, business. So yeah, that was great.Corey: To give a little backstory and context here is, I write a whole bunch of serverless nonsense. I build API's Gateway, I hook them up to Lambda's Function, and then it sort of kind of works. Ish. From there, okay, I would try and track down what was going on because in a microservices land, everything becomes a murder mystery; you're trying to figure out what's broken, and things have exploded. And I became a paying customer of IOpipe. And then New Relic bought them. Well, crap.Then I became a paying customer of Epsagon. And they got acquired by Cisco, at which point I immediately congratulated the founders, who I know on a social basis, and then closed my account because I wanted to get out before Cisco ruins it because, Cisco. Then it was, what am I going to use next? And right around that time is when I stumbled across Cloudash. And it takes a different approach than any other entity in the space that I've seen because you are a native Mac desktop app. I believe your Mac only, but you seem to be Electron, so I could be way off base on that.Maciej: So, we're Linux as well right now and soon we'll be Windows as well. But yeah, so, right now is Mac OS and Linux. Yeah, that's correct. So, our approach is a little bit different.So, let me start by saying what's Cloudash? Like, Cloudash is a desktop app for, kind of, monitoring and troubleshooting serverless architectures services, like, serverless stuff in general. And the approach that we took is a little bit different because we are not web-based, we're desktop-based. And there's a couple of advantages of that approach. The first one is that, like, you don't need to share your data with us because we're not, kind of, downloading your metrics and logs to our back end and to process them, et cetera, et cetera. We are just using the credentials, the AWS profiles that you have defined on your computer, so nothing goes out of your AWS account.And I think this is, like, considering, like, from the security perspective, this is very crucial. You don't need to create a role that you give us access to or anything like that. You just use the stuff that you have on your desktop, and everything stays on your AWS account. So, nothing—we don't download it, we don't process it, we don't do anything from that. And that's one approach—well, that's the one advantage. The other advantage is, like, kind of, onboarding, as I kind of mentioned because we're using the AWS profiles that you have defined in your computer.Corey: Well, you're doing significantly more than that because I have a lot of different accounts configured different ways, and when I go to one of them that uses SSO, it automatically fires me off to the SSO login page if I haven't logged in that day for a 12 hour session—Maciej: Yes.Corey: —for things that have credentials stored locally, it uses those; and for things that are using role-chaining to use assuming roles from the things I have credentials for, and the things that I just do role assumption in, and it works flawlessly. It just works the way that most of my command-line tools do. I've never seen a desktop app that does this.Maciej: Yeah. So, we put a lot of effort into making sure that this works great because we know that, like, no one will use Cloudash if there's—like, not no one, but like, we're targeting, like, serverless teams, maybe, in enterprise companies, or serverless teams working on some startups. And in most cases, those teams or those engineers, they use SSO, or at least MFA, right? So, we have it covered. And as you said, like, it should be the onboarding part is really easy because you just pick your AWS profile, you just pick region, and just pick, right now, a CloudFormation stack because we get the information about your service based on CloudFormation stack. So yeah, we put a lot of effort in making sure that this works without any issues.Corey: There are some challenges to it that I saw while setting it up, and that's also sort of the nature of the fact you are, in fact, integrating with CloudWatch. For example, it's region specific. Well, what if I want to have an app that's multi-region? Well, you're going to have a bad time because doing [laugh] anything multi-region in AWS means you're going to have a bad time that gets particularly obnoxious and EC2 get to when you're doing something like Lambda@Edge, where, oh, where are the logs live; that's going to be in a CloudFront distribution in whatever region it winds up being accessed from. So, it comes down to what distribution endpoint or point of presence did that particular request go through, and it becomes this giant game of whack-a-mole. It's frustrating, and it's obnoxious, and it's also in no way your fault.Maciej: Yeah, I mean, we are at the beginning. Right now, it's the most straightforward, kind of pe—how people think about stacks of serverless. They're think in terms of regions because I think for us, regions, or replicated stacks, or things like that are not really popular yet. Maybe they will become—like, this is how AWS works as a whole, so it's not surprising that we're kind of following this path. I think my point is that our main goal, the ultimate goal, is to make monitoring, as I said, the troubleshooting serverless app as simple as possible.So, once we will hear from our customers, from our users that, “Hey, we would like to get a little bit better experience around regions,” we will definitely implement that because why not, right? And I think the whole point of Cloudash—and maybe we can go more deep into that later—is that we want to bring context into your metrics and logs. If you're seeing a, for example, X-Ray trace ID in your logs, you should be able with one click just see that the trace. It's not yet implemented in Cloudash, but we are having it in the backlog. But my point is that, like, there should be some journey when you're debugging stuff, and you shouldn't be just, like, left alone having, like, 20 tabs, Cloudash tabs open and trying to figure out where I was—like, where's the Lambda? Where's the API Gateway logs? Where are the CloudFront logs? And how I can kind of connect all of that? Because that's—it's an issue right now.Corey: Even what you've done so far is incredibly helpful compared to the baseline experience that folks will often have, where I can define a service that is comprised of a number of different functions—I have one set up right now that has seven functions in it—I grab any one of those things, and I can set how far the lookback is, when I look at that function, ranging from 5 minutes to 30 days. And it shows me at the top the metrics of invocations, the duration that the function runs for, and the number of errors. And then, in the same pane down below it, it shows the CloudWatch logs. So, “Oh, okay, great. I can drag and zoom into a specific timeframe, and I see just the things inside of that.”And I know this sounds like well, what's the hard part here? Yeah, except nothing else does it in an easy-to-use, discoverable way that just sort of hangs out here. Honestly, the biggest win for me is that I don't have to log in to the browser, navigate through some ridiculous other thing to track down what I'm talking about. It hangs out on my desktop all the time, and whether it's open or not, whenever I fire it up, it just works, basically, and I don't have to think about it. It reduces the friction from, “This thing is broken,” to, “Let me see what the logs say.”Very often I can go from not having it open at all to staring at the logs and having to wait a minute because there's some latency before the event happens and it hits CloudWatch logs itself. I'm pretty impressed with it, and I've been keeping an eye on what this thing is costing me. It is effectively nothing in terms of CloudWatch retrieval charges. Because it's not sitting there sucking all this data up all the time, for everything that's running. Like, we've all seen the monitoring system that winds up costing you more than it costs more than they charge you ancillary fees. This doesn't do that.I also—while we're talking about money, I want to make very clear—because disclaiming the direction the money flows in is always important—you haven't paid me a dime, ever, to my understanding. I am a paying customer at full price for this service, and I have been since I discovered it. And that is very much an intentional choice. You did not sponsor this podcast, you are not paying me to say nice things. We're talking because I legitimately adore this thing that you've built, and I want it to exist.Maciej: That's correct. And again, thank you for that. [laugh].Corey: It's true. You can buy my attention, but not my opinion. Now, to be clear, when I did that tweet thread, I did get the sense that this was something that you had built as sort of a side project, as a labor of love. It does not have VC behind it, of which I'm aware, and that's always going to, on some level, shade how I approach a service and how critical I'm going to be on it. Just because it's, yeah, if you've raised a couple 100 million dollars and your user experience is trash, I'm going to call that out.But if this is something where you just soft launched, yeah, I'm not going to be a jerk about weird usability bugs here. I might call it out as “Ooh, this is an area for improvement,” but not, “What jackwagon thought of this?” I am trying to be a kinder, gentler Corey in the new year. But at the same time, I also want to be very clear that there's room for improvement on everything. What surprised me the most about this is how well you nailed the user experience despite not having a full team of people doing UX research.Maciej: That was definitely a priority. So, maybe a little bit of history. So, I started working on Cloudash, I think it was April… 2019. I think? Yeah. It's 2021 right now. Or we're 2022. [unintelligible 00:11:33].Corey: Yeah. 2022, now. I—Maciej: I'm sorry. [laugh].Corey: —I've been screwing that up every time I write the dates myself, I'm with you.Maciej: [laugh]. Okay, so I started working on Cloudash, in 2020, April 2020.Corey: There we go.Maciej: So, after eight months, I released some beta, like, free; you could download it from GitHub. Like, you can still download on GitHub, but at that time, there was no license, you didn't have to buy a license to run it. So, it was, like, very early, like, 0.3 version that was working, but sort of, like, [unintelligible 00:12:00] working. There were some bugs.And that was the first time that I tweeted about it on Twitter. It gets some attention, but, like, some people started using it. I get some feedback, very initial feedback. And I was like, every time I open Cloudash, I get the sense that, like, this is useful. I'm talking about my own tool, but like, [laugh] that's the thing.So, further in the history. So, I'm kind of service engineer by my own. I am a software engineer, I started focusing on serverless, in, like, 2015, 2016. I was working for Serverless Inc. as an early employee.I was then working as an engineering manager for a couple of companies. I work as an engineering manager right now at Stedi; we're also, like, fully serverless. So I, kind of, trying to fix my own issues with serverless, or trying to improve the whole experience around serverless in AWS. So, that's the main purpose why we're building Cloudash: Because we want to improve the experience. And one use case I'm often mentioning is that, let's say that you're kind of on duty. Like, so in the middle of night PagerDuty is calling you, so you need to figure out what's going on with your Lambda or API Gateway.Corey: Yes. PagerDuty, the original [Call of Duty: Nagios 00:13:04]. “It's two in the morning; who is it?” “It's PagerDuty. Wake up, jackass.” Yeah. We all had those moments.Maciej: Exactly. So, the PagerDuty is calling you and you're, kind of, in the middle of night, you're not sure what's going on. So, the kind of thing that we want to optimize is from waking up into understanding what's going on with your serverless stuff should be minimized. And that's the purpose of Cloudash as well. So, you should just run one tool, and you should immediately see what's going on. And that's the purpose.And probably with one or two clicks, you should see the logs responsible, for example, in your Lambda. Again, like that's exactly what we want to cover, that was the initial thing that we want to cover, to kind of minimize the time you spent on troubleshooting serverless apps. Because as we all know, kind of, the longer it's down, the less money you make, et cetera, et cetera, et cetera.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: One of the things that I appreciate about this is that I have something like five different microservices now that power my newsletter production pipeline every week. And periodically, I'll make a change and something breaks because testing is something that I should really get around to one of these days, but when I'm the only customer, cool. Doesn't really matter until suddenly I'm trying to write something and it doesn't work. Great. Time to go diving in, and always I'm never in my best frame of mind for that because I'm thinking about writing for humans not writing for computers. And that becomes a challenge.And okay, how do I get to the figuring out exactly what is broken this time? Regression testing: It really should be a thing more than it has been for me.Maciej: You should write those tests. [laugh].Corey: Yeah. And then I fire this up, and okay, great. Which sub-service is it? Great. Okay, what happened in the last five minutes on that service? Oh, okay, it says it failed successfully in the logs. Okay, that's on me. I can't really blame you for that. But all right.And then it's a matter of adding more [print or 00:14:54] debug statements, and understanding what the hell is going on, mostly that I'm bad at programming. And then it just sort of works from there. It's a lot easier to, I guess, to reason about this from my perspective than it is to go through the CloudWatch dashboards, where it's okay, here's a whole bunch of metrics on different graphs, most of which you don't actually care about—as opposed to unified view that you offer—and then “Oh, you want to look at logs, that's a whole separate sub-service. That's a different service team, obviously, so go open that up in another browser.” And I'm sitting here going, “I don't know who designed this, but are there any windows in their house? My God.”It's just the saddest thing I can possibly experience when I'm in the middle of trying to troubleshoot. Let's be clear, when I'm troubleshooting, I am in no mood to be charitable to anyone or anything, so that's probably unfair to those teams. But by the same token, it's intensely frustrating when I keep smacking into limitations that get in my way while I'm just trying to get the thing up and running again.Maciej: As you mentioned about UX that, like, we've spent a lot of time thinking about the UX, trying different approaches, trying to understand which metrics are the most important. And as we all know, kind of, serverless simplifies a lot of stuff, and there's, like, way less metrics that you need to look into when something is happening, but we want to make sure that the stuff that we show—which is duration errors, and p95—are probably the most important in most cases, so like, covering most of this stuff. So sorry, I didn't mention that before; it was very important from the very beginning. And also, like, literally, I spent a lot of time, like, working on the colors, which sounds funny, [laugh] but I wanted to get them right. We're not yet working on dark mode, but maybe soon.Anyways, the visual part, it's always close to my heart, so we spent a lot of time going back to what just said. So, definitely the experience around using CloudWatch right now, and CloudWatch logs, CloudWatch metrics, is not really tailored for any specific use case because they have to be generic, right? Because AWS has, like, I don't know, like, 300, or whatever number of services, probably half of them producing logs—maybe not half, maybe—Corey: We shouldn't name a number because they'll release five more between now and when this publishes in 20 minutes.Maciej: [laugh]. So, CloudWatch has to be generic. What we want to do with Cloudash is to take those generic tools—because we use, of course, CloudWatch logs, CloudWatch metrics, we fetch data from them—but make the visual part more tailored for specific use case—in our case, it's the serverless use case—and make sure that it's really, kind of—it shows only the stuff that you need to see, not everything else. So again, like that's the main purpose. And then one more thing, we—like this is also some kind of measurement of success, we want to reduce number of tabs that you need to have open in your browser when you're dealing with CloudWatch. So, we tried to put most important stuff in one view so you don't need to flip between tabs, as you usually do when try to under some kind of broader scope, or broader context of your, you know, error in Lambda.Corey: What inspired you to do this as a desktop application? Because a lot of companies are doing similar things, as SaaS, as webapps. And I have to—as someone who yourself—you're a self-described serverless engineer—it seems to me that building a webapp is sort of like the common description use case of a lot of serverless stuff. And you're sitting here saying, “Nope, it's desktop app time.” Which again, I'm super glad you did. It's exactly what I was looking for. How do you get here?Maciej: I'd been thinking about both kinds of types of apps. So like, definitely webapp was the initial idea how to build something, it was the webapp. Because as you said, like, that's the default mode. Like, we are thinking webapp; like, let's build a webapp because I'm an engineer, right? There is some inspiration coming from Dynobase, which was made by a friend [unintelligible 00:18:55] who also lives in Poland—I didn't mention that; we're based in [Poznań 00:18:58], Poland.And when I started thinking about it, there's a lot of benefits of using this approach. The biggest benefit, as I mentioned, is security; and the second benefit is just most, like, cost-effective because we don't need to run in the backend, right? We don't need to download all your metrics, all your logs. We I think, like, let's think about it, like, from the perspective. Listen, so everyone in the company to start working, they have to download all of your stuff from your AWS account. Like, that sounds insane because you don't need all of that stuff elsewhere.Corey: Store multiple copies of it. Yeah I, generally when I'm looking at this, I care about the last five to ten minutes.Maciej: Exactly.Corey: I don't—Maciej: Exactly.Corey: —really care what happened three-and-a-half years ago on this function. Almost always. But occasionally I want to look back at, “Oh, this has been breaking. How long has it been that way?” But I already have that in the AWS environment unless I've done the right thing and turned on, you know, log expiry.Maciej: Exactly. So, this is a lot of, like, I don't want to be, like, you know, mean to anyone but like, that's a lot of waste. Like, that's a lot of waste of compute power because you need to download it; of cost because you need to get this data out of AWS, which you need to pay for, you know, get metric data and stuff like this. So, you need to—Corey: And almost all of its—what is it? Write once, read never. Because it's, you don't generally look at these things.Maciej: Yeah, yeah. Exactly.Corey: And so much of this, too, for every invocation I have, even though it's low traffic stuff, it's the start with a request ID and what version is running, it tells me ‘latest.' Helpful. A single line of comment in this case says ‘200.' Why it says that, I couldn't tell you. And then it says ‘End request ID.' The end.Now, there's no way to turn that off unless you disabled the ability to write to CloudWatch logs in the function, but ingest on that cost 50 cents a gigabyte, so okay, I guess that's AWS's money-making scam of the year. Good for them. But there's so much of that, it's like looking at—like, when things are working, it's like looking at a low traffic site that's behind a load balancer, where there's a whole—you have gigabytes, in some cases, of load balancer—of web server logs on the thing that's sitting in your auto-scaling group. And those logs are just load balancer health checks. 98% of it is just that.Same type of problem here, I don't care about that, I don't want to pay to store it, I certainly don't want to pay to store it twice. I get it, that makes an awful lot of sense. It also makes your security job a hell of a lot easier because you're not sitting on a whole bunch of confidential data from other people. Because, “Well, it's just logs. What could possibly be confidential in there?” “Oh, my sweet summer child, have you seen some of the crap people put in logs?”Maciej: I've seen many things in logs. I don't want to mention them. But anyways—and also, you know, like, usually when you gave access to your AWS account, it can ruin you. You know, like, there might be a lot of—like, you need to really trust the company to give access to your AWS account. Of course, in most cases, the roles are scoped to, you know, only CloudWatch stuff, actions, et cetera, et cetera, but you know, like, there are some situations in which something may not be properly provisioned. And then you give access to everything.Corey: And you can get an awful lot of data you wouldn't necessarily want out of that stuff. Give me just the PDF printout of last month's bill for a lot of environments, and I can tell you disturbing levels of detail about what your architecture is, just because when you—you can infer an awful lot.Maciej: Yeah.Corey: Yeah, I hear you. It makes your security story super straightforward.Maciej: Yeah, exactly. So, I think just repeat my, like, the some inspiration. And then when I started thinking about Cloudash, like, definitely one of the inspiration was Dynobase, from the, kind of, GUI for, like, more powerful UI for DynamoDB. So, if you're interested in that stuff, you can also check this out.Corey: Oh, yeah, I've been a big fan of that, too. That'll be a separate discussion on a different episode, for sure.Maciej: [laugh]. Yeah.Corey: But looking at all of this, looking at the approach of, the only real concern—well, not even a concern. The only real challenge I have with it for my use case is that when I'm on the road, the only thing that I bring with me for a computer is my iPad Pro. I'm not suggesting by any means that you should build this as a new an iPad app; that strikes me as, like, 15 levels of obnoxious. But it does mean that sometimes I still have to go diving into the CloudWatch console when I'm not home. Which, you know, without this, without Cloudash, that's what I was doing originally anyway.Maciej: You're the only person that requested that. And we will put that into backlog, and we will get to that at some point. [laugh].Corey: No, no, no. Smart question is to offer me a specific enterprise tier pricing—.Maciej: Oh, okay. [laugh].Corey: —that is eye-poppingly high. It's like, “Hey, if you want a subsidize feature development, we're thrilled to empower that.” But—Maciej: [laugh]. Yeah, yeah. To be honest, I like that would be hard to write [unintelligible 00:23:33] implement as iPad app, or iPhone app, or whatever because then, like, what's the story behind? Like, how can I get the credentials, right? It's not possible.Corey: Yeah, you'd have to have some fun with that. There are a couple of ways I can think of offhand, but then that turns into a sandboxing issue, and it becomes something where you have to store credentials locally, regardless, even if they're ephemeral. And that's not great. Maybe turn it into a webapp someday or something. Who knows.What I also appreciate is that we had a conversation when you first launched, and I wound up basically going on a Zoom call with you and more or less tearing apart everything you've built—and ideally constructive way—but looking at a lot of the things you've changed in your website, you listened to an awful lot of feedback. You doubled your pricing, for example. Used to be ten bucks a month; now you're twenty. Great. I'm a big believer in charging more.You absolutely add that kind of value because it's, “Well, twenty bucks a month for a desktop app. That sounds crappy.” It's, “Yeah, jackwagon, what's your time worth?” I was spending seven bucks a month in serverless charges, and 120 or 130 a month for Epsagon, and I was thrilled to pieces to be doing it because the value I got from being able to quickly diagnose what the hell was going on far outstripped what the actual cost of doing these things. Don't fall into the trap of assuming that well, I shouldn't pay for software. I can just do it myself. Your time is never free. People think it is, but it's not.Maciej: That's true. The original price of $9.99, I think that was the price was the launch promo. After some time, we've decided—and after adding more features: API Gateway support—we've decided that this is, like, solving way more problems, so like, you should probably pay a little bit more for that. But you're kind of lucky because you subscribed to it when it was 9.99, and this will be your kind of prize for the end of, you know—Corey: Well, I'm going to argue with you after the show to raise the price on mine, just because it's true. It's the—you want to support the things that you want to exist in the world. I also like the fact that you offered an annual plan because I will go weeks without ever opening the app. And that doesn't mean it isn't adding value. It's that oh, yeah, I will need that now that I'm hitting these issues again.And if I'm paying on a monthly basis, and it shows up with a, “Oh, you got charged again.” “Well, I didn't use it this month; I should cancel.” And [unintelligible 00:25:44] to an awful lot of subscriber churn. But in the course of a year, if I don't have at least one instance in which case, wow, that ten minute span justified the entire $200 annual price tag, then, yeah, you built the wrong thing or it's not for me, but I can think of three incidents so far since I started using it in the past four months that have led to that being worth everything you will charge me a year, and then some, just because it made it so clear what was breaking.Maciej: So, in that regard, we are also thinking about the team licenses, that's definitely on the roadmap. There will be some changes to that. And we definitely working on more and more features. And if we're—like, the roadmap is mostly about supporting more and more AWS services, so right now it's Lambda, API Gateway, we're definitely thinking about SQS, SNS, to get some sense how your messages are going through, probably something, like, DynamoDB metrics. And this is all kind of serverless, but why not going wider? Like, why not going to Fargate? Like, Fargate is theoretically serverless, but you know, like, it's serverless on—Corey: It's serverless with a giant asterisk next to it.Maciej: Yeah, [laugh] exactly. So, but why not? Like, it's exactly the same thing in terms of, there is some user flow, there is some user journey, when you want to debug something. You want to go from API Gateway, maybe to the container to see, I don't know, like, DynamoDB metric or something like that, so it should be all easy. And this is definitely something.Later, why not EC2 metrics? Like, it would be a little bit harder. But I'm just saying, like, first thing here is that you are not, like, at this point, we are serverless, but once we cover serverless, why not going wider? Why not supporting more and more services and just making sure that all those use cases are correctly modeled with the UI and UX, et cetera?Corey: That's going to be an interesting challenge, just because that feels like what a lot of the SaaS monitoring and observability tooling is done. And then you fire this thing up, and it looks an awful lot like the AWS console. And it's, “Yeah, I just want to look at this one application that doesn't use any of the rest of those things.” Again, I have full faith and confidence in your ability to pull this off. You clearly have done that well based upon what we've seen so far. I just wonder how you're going to wind up tackling that challenge when you get there.Maciej: And maybe not EC2. Maybe I went too far. [laugh].Corey: Yeah, honestly, even EC2-land, it feels like that is more or less a solved problem. If you want to treat it as a bunch of EC2, you can use Nagios. It's fine.Maciej: Yeah, totally.Corey: There are tools that have solved that problem. But not much that I've seen has solved the serverless piece the way that I want it solved. You have.Maciej: So, it's definitely a long road to make sure that the serverless—and by serverless, I mean serverless how AWS understands serverless, so including Fargate, for example. So, there's a lot of stuff that we can improve. It's a lot of stuff that can make easier with Cloudash than it is with CloudWatch, just staying inside serverless, it will take us a lot of time to make sure that is all correct. And correctly modeled, correctly designed, et cetera. So yeah, I went too far with EC2 sorry.Corey: Exactly. That's okay. We all go too far with EC2, I assure you.Maciej: Sorry everyone using EC2 instances. [laugh].Corey: If people want to kick the tires on it, where can they find it?Maciej: They can find it on cloudash.dev.Corey: One D in the middle. That one throws me sometimes.Maciej: One D. Actually, after talking to you, we have a double-D domain as well, so we can also try ‘Clouddash' with double-D. [laugh].Corey: Excellent, excellent. Okay, that is fantastic. Because I keep trying to put the double-D in when I'm typing it in my search tool on my desktop, and it doesn't show up. And it's like, “What the—oh, right.” But yeah, we'll get there one of these days.Maciej: Only the domain. It's only the domain. You will be redirected to single-D.Corey: Exactly.Maciej: [laugh].Corey: We'll have to expand later; I'll finance the feature request there. It'll go well. If people want to learn more about what you have to think about these things, where else can they find you?Maciej: On Twitter, and my Twitter handle is @mthenw. M-then-W, which is M-T-H—mthenw. And my co-founder @tlakomy. You can probably add that to [show notes 00:29:35]. [laugh].Corey: Oh, I certainly will. It's fine, yeah. Here's a whole bunch of letters. I hear you. My Twitter handle used to be my amateur radio callsign. It turns out most people don't think like that. And yeah, it's become an iterative learning process. Thank you so much for taking the time to speak with me today and for building this thing. I really appreciate both of them.Maciej: Thank you for having me here. I encourage everyone to visit cloudash.dev, if you have any feature requests, any questions just send us an email at hello@cloudash.dev, or just go to GitHub repository in the issues; just create an issue, describe what you want and we can talk about it.We are always happy to help. The main purpose, the ultimate goal of Cloudash is to make the serverless engineer's life easier, on very high level. And on a little bit lower level, just to make, you know, troubleshooting and debugging serverless apps easier.Corey: Well, from my perspective, you've succeeded.Maciej: Thank you.Corey: Thank you. Maciej Winnicki, founder of Cloudash. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment telling me exactly why I'm wrong for using an iPad do these things, but not being able to send it because you didn't find a good way to store the credentials.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

My First Million
Saas Companies that Anyone Can Start with Rob Walling

My First Million

Play Episode Listen Later Jan 11, 2022 77:48


Sam Parr (@theSamParr) and Ben Wilson (@BenWilsonTweets) discuss Sam's article defending Theranos, airport houses, and conducting an end of year family review. They are then joined by Rob Walling (@RobWalling) who talks about great SaaS business that anyone could start.  _____ * Do you love MFM and want to see Sam and Shaan's smiling faces? Subscribe to our Youtube channel. * Want more insights like MFM? Check out Shaan's newsletter. _____ Show Notes: (00:20) - Airline neighborhood with houses and hotels (06:45) - Sam on the front page of HackerNews for defending Elizabeth Holmes (12:20) - The family board meeting (18:30) - Rob Walling joins the show. About Rob. (30:30) - Why you should or shouldn't sell your business (42:30) - Rob's favorite Saas businesses that he's working with (50:50) - Rob's ideas for new SaaS companies that anyone could start (01:10:10) - Founding a company as a non-technical founder ----- Past guests on My First Million include Rob Dyrdek, Hasan Minhaj, Balaji Srinivasan, Jake Paul, Dr. Andrew Huberman, Gary Vee, Lance Armstrong, Sophia Amoruso, Ariel Helwani, Ramit Sethi, Stanley Druckenmiller, Peter Diamandis, Dharmesh Shah, Brian Halligan, Marc Lore, Jason Calacanis, Andrew Wilkinson, Julian Shapiro, Kat Cole, Codie Sanchez, Nader Al-Naji, Steph Smith, Trung Phan, Nick Huber, Anthony Pompliano, Ben Askren, Ramon Van Meer, Brianne Kimmel, Andrew Gazdecki, Scott Belsky, Moiz Ali, Dan Held, Elaine Zelby, Michael Saylor, Ryan Begelman, Jack Butcher, Reed Duchscher, Tai Lopez, Harley Finkelstein, Alexa von Tobel, Noah Kagan, Nick Bare, Greg Isenberg, James Altucher, Randy Hetrick and more. ----- Additional episodes you might enjoy: • #224 Rob Dyrdek - How Tracking Every Second of His Life Took Rob Drydek from 0 to $405M in Exits • #209 Gary Vaynerchuk - Why NFTS Are the Future • #178 Balaji Srinivasan - Balaji on How to Fix the Media, Cloud Cities & Crypto #169 - How One Man Started 5, Billion Dollar Companies, Dan Gilbert's Empire, & Talking With Warren Buffett • ​​​​#218 - Why You Should Take a Think Week Like Bill Gates • Dave Portnoy vs The World, Extreme Body Monitoring, The Future of Apparel Retail, "How Much is Anthony Pompliano Worth?", and More • How Mr Beast Got 100M Views in Less Than 4 Days, The $25M Chrome Extension, and More

AWS Morning Brief
Time to Give LastPass the Heave

AWS Morning Brief

Play Episode Listen Later Jan 6, 2022 5:16


Links: “Tokyo police lose 2 floppy disks containing personal info on 38 public housing applicants”: https://mainichi.jp/english/articles/20211227/p2a/00m/0na/072000c LastPass may have suffered a breach: https://news.ycombinator.com/item?id=29705957 “Worst AWS Data Breaches of 2021”: https://securityboulevard.com/2021/12/worst-aws-data-breaches-of-2021/ D.W. Morgan: https://www.hackread.com/logistics-giant-d-w-morgan-exposed-clients-data/ SEGA Europe: https://vpnoverview.com/news/sega-europe-suffers-major-security-breach/ “Identity Guide–Preventive controls with AWS Identity–SCPs”: https://aws.amazon.com/blogs/mt/identity-guide-preventive-controls-with-aws-identity-scps/ Log4j scanner: https://github.com/google/log4jscanner TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it's nobody in particular's job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: The first security round-up of the year in Last Week in AWS: Security. This is relatively light, just because it covers the last week of the year, where people didn't really “Work” so much as “Get into fights on Twitter.” Onward.So, from the community, ever see a data breach announcement that raises oh so very many more questions than it answers? I swear this headline is from a week or so ago, not 1998: “Tokyo police lose 2 floppy disks containing personal info on 38 public housing applicants”. Yes, I said floppy disks.The terrible orange website, also known as Hacker News, reports that LastPass may have suffered a breach. At the time I write this, the official LastPass blog has a, “No, it's just people reusing passwords.” Enough people I trust have seen this behavior that I'd be astounded if that were true. If you can't trust your password manager, ditch them immediately.Security Boulevard had a roundup of the “Worst AWS Data Breaches of 2021”, and it's the usual run-of-the-mill S3 bucket problems, but my personal favorite's the Twitch breach because it's particularly embarrassing, given that it is, in fact, an Amazon subsidiary.First one goes to D.W. Morgan by leaking 100GB of client data. And they're a logistics company that serves giant enterprises, so these are companies with zero sense of humor, so I would not want to be in D.W. Morgan's position this week.And the other is a little funnier. It goes to SEGA Europe, after Sonic the Hedgehog forgets to perform due diligence on his AWS environment.Corey: Are you building cloud applications with a distributed team? Check out Teleport, an open-source identity-aware access proxy for cloud resources. Teleport provides secure access for anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps, and databases. Teleport gives engineers superpowers. Get access to everything via single sign-on with multi-factor, list and see all of SSH servers, Kubernetes clusters, or databases available to you in one place, and get instant access to them using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility, and ensuring compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That's goteleport.com.AWS had only a single thing that I found interesting: “Identity Guide–Preventive controls with AWS Identity–SCPs”. I've been waiting for a while for a good explainer on SCPs to come out for a while, and this looks like it actually is a thing that I want. I've been playing around with SCPs a lot more for the past couple of weeks. If you're unfamiliar, it's a way to override what the root user can do in an organization's member accounts. It's super handy to constrain people from doing things that are otherwise foolhardy.And lastly, an interesting tool came out from Google—which I should not have to explain what that is to you folks; they turn things off, like Reader—they also released a log4j scanner. This one scans files on disk to detect the bad versions of log4j—which is most of them—and can replace them with the good version—which is, of course, print statements. And that's what happened last week in AWS security. Hopefully next week will be… well, I don't want to say less contentful, but I do want to say it's at least not as exciting as the last month has been. Thanks for listening.Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

Dead Cat
Defogging Our 2021 Amnesia

Dead Cat

Play Episode Listen Later Dec 29, 2021 62:15


Katie Benner, Tom Dotan, and Eric Newcomer look back on 2021 in Techmeme headlines for our final episode of Dead Cat for the year. We discuss some of the biggest stories of the year: In January, Microsoft said Russian hackers accessed some of its source code and the U.S. government pinned the SolarWinds hack on Russians. In February, Elon Musk drove Clubhouse listeners (and journalists blocked by Marc Andreessen) to YouTube as they tried to listen to the live interview on the platform. It would represent a peak moment of cultural relevance for Clubhouse. In March, Stripe’s valuation climbed to $95 billion. (And we talked about Stripe’s critics on Y Combinator-owned Hacker News and the coverage of Stripe’s hiring practices in Protocol.) In May, Antonio García Martínez declared that Apple had fired him over the culture war backlash to his book Chaos Monkeys. In June, the New York Times wrote about tough working conditions at Amazon. Later this year, a tornado would rip through an Amazon facility, killing six and raising further questions about how Amazon protects its workers. Also in June, Andreessen Horowitz launched its much-discussed Future — a publication that hasn’t yet taken Silicon Valley by storm but has put every venture firm on notice that they need to think about getting in the content business. We talked about Robinhood’s IPO in July and the rise of meme stocks. And we discussed how big tech executives don’t seem to want to worry about the present. Jeff Bezos stepped down as Amazon CEO in July as he spends more time on Blue Origin; Facebook CEO Mark Zuckerberg rebranded his company to Meta in October; and Jack Dorsey left behind moderation challenges at Twitter in late November and then renamed his financial services company Square to Block, hoping to emphasize the company’s crypto ambitions. Finally, Tom, Katie, and Eric offer some predictions for what 2022 holds, though no one seems quite sure after this strange year.Techmeme!My favorite tech headline aggregator, tweet tracker, and conversation setter — Techmeme — has been generously featuring me on their home page as part of a round-up of interesting tech newsletters. So I wanted to return the favor.I check Techmeme literally every couple of hours and rely on it to do my job. And in a genuine coincidence, Techmeme served as an easy-to-navigate archive for this week’s podcast. It’s a free news aggregator for tech industry folks that’s updated constantly to show the most important tech stories of the moment and the commentary surrounding those stories. They also publish a daily newsletter with stories from the past day, which is useful if you forget to visit the site. Get full access to Newcomer at www.newcomer.co/subscribe

Open Source Security Podcast
Episode 303 - Log4j Christmas Spectacular!

Open Source Security Podcast

Play Episode Listen Later Dec 27, 2021 34:37


Josh and Kurt start the show with the reading of a security themed Christmas poem. We then discuss some of the new happenings around Log4j. The basic theme is that even if we were over-investing in Log4j, it probably wouldn't have caught this. There are still a lot of things to unpack with this event, I'm sure we'll be talking about it well into the future. Log before Christmas poem 'Twas the night before Christmas, when all through the stack Not a scanner was scanning, not even a rack, The SBOMs were uploaded to the portal with care, In hopes that next year would be boring and bare The interns were nestled all snug at their beds; While visions of dashboards danced in their heads; The CISO in their 'kerchief, and I in my cap, Had just slept our laptops for a long winter's nap, When all of a sudden the pager went ack ack I sprang to my laptop with worries of attack Away to the browser I flew like a flash, Tore open the window and cleared out the cache The red of the dashboard the glow of the screen Gave a lustre of disaster my eyes rarely seen When what to my wondering eyes did we appear, But a new advisory and eight vulnerabilities to fear, Like a little old hacker all ready to play, I knew in a moment it must be Log4j More rapid than gigabit its coursers they came, And it whistled, and shouted, and called them by name: "Now, Log4Shell! now CVE! now ASF and NVD! On, CISA! on, LunaSec! on, GossiTheDog! To the top of the HackerNews! to the top of the wall! Now hack away! hack away! hack away all!" Like the bits that before the wild CDN fly by When they meet with a firewall, they mount to the sky; So up to the cloud like bastards they flew With tweets full of vulns, and Log4j too— And then, in a twinkling, I read in the slack The wailing and screaming of each analyst called back As I drew in my head, and was turning around, Down the network Log4j came with a bound. It was dressed in a hoodie, black and zipped tight, The clothes were all swag from a conference one night A bundle of vulns it had checked in its git And it looked like a pedler just being a twit The changelog—how it twinkled! its features, how merry! Its versions were like roses, its logo like a cherry! Its droll little mouth was drawn up like an at, And the beard on its chin made it look stupid and fat The stump of a diff it held tight in its teeth, And the bits, they encircled the repo like a wreath; It had a flashy readme an annoying little fad That shook when it downloaded, like a disk drive gone bad It was chubby and plump, an annoying old package, And I laughed when I saw it, in spite of the hackage A wink of its bits and a twist of its head Soon gave me to know I had everything to dread It spoke not a word, but went straight to its work, And pwnt all the servers; then turned with a jerk, And laying its patches aside of its nose, And giving a nod, up the network it rose; It sprang to its packet, to its team gave them more, And away they all fled leaving behind a back door But I heard it exclaim, ere it drove out of sight— “Merry Christmas you nerds, Log4j won tonight!”

The Swyx Mixtape
[Weekend Drop] Cloudflare vs AWS, API Economy, Learning in Public on the Changelog

The Swyx Mixtape

Play Episode Listen Later Dec 12, 2021 68:13


Listen to the Changelog: https://changelog.com/podcast/467Essays: https://www.swyx.io/LIP https://www.swyx.io/api-economy https://www.swyx.io/cloudflare-go TranscriptJerod Santo: So swyx, we have been tracking your work for years; well, you've been Learning in Public for years, so I've been (I guess) watching you learn, but we've never had you on the show, so welcome to The Changelog.Shawn Wang: Thank you. Long-time listener, first-time guest, I guess... [laughs]Adam Stacoviak: Yeah.Jerod Santo: Happy to have you here.Adam Stacoviak: Very excited to have you here.Jerod Santo: So tell us a little bit of your story, because I think it informs the rest of our conversation. We're gonna go somewhat deep into some of your ideas, some of the dots you've been connecting as you participate and watch the tech industry... But I think for this conversation it's probably useful to get to know you, and how you got to be where you are. Not the long, detailed story, but maybe the elevator pitch of your recent history. Do you wanna hook us up?Shawn Wang: For sure. For those who want the long history, I did a 2,5-hour podcast with Quincy Larson from FreeCodeCamp, so you can go check that out if you want. The short version is I'm born and raised in Singapore, came to the States for college, and was totally focused on finance. I thought people who were in the finance industry rules the world, they were masters of the universe... And I graduated just in time for the financial crisis, so not a great place to be in. But I worked my way up and did about 6-7 years of investment banking and hedge funds, primarily trading derivatives and tech stocks. And the more I covered tech stocks, the more I realized "Oh, actually a) the technology is taking over the world, b) all the value is being created pre-IPO, so I was investing in public stocks, after they were basically done growing... And you're kind of just like picking over the public remains. That's not exactly true, but...Jerod Santo: Yeah, tell that to Shopify...Shawn Wang: I know, exactly, right?Adam Stacoviak: And GitLab.Shawn Wang: People do IPO and have significant growth after, but that's much more of a risk than at the early stage, where there's a playbook... And I realized that I'd much rather be value-creating than investing. So I changed careers at age 30, I did six months of FreeCodeCamp, and after six months of FreeCodeCamp - you know, I finished it, and that's record time for FreeCodeCamp... But I finished it and felt not ready, so I enrolled myself in a paid code camp, Full Stack Academy in New York, and came out of it working for Two Sigma as a frontend developer. I did that for a year, until Netlify came along and offered me a dev rel job. I took that, and that's kind of been my claim to fame; it's what most people know me for, which is essentially being a speaker and a writer from my Netlify days, from speaking about React quite a bit.[04:13] I joined AWS in early 2020, lasted a year... I actually was very keen on just learning the entire AWS ecosystem. You know, a frontend developer approaching AWS is a very intimidating task... But Temporal came along, and now I'm head of developer experience at Temporal.Adam Stacoviak: It's an interesting path. I love the -- we're obviously huge fans of FreeCodeCamp, and Quincy, and all the work he's done, and the rest of the team has done to make FreeCodeCamp literally free, globally... So I love to see -- it makes you super-happy inside just to know how that work impacts real people.Like, you see things happen out there, and you think "Oh, that's impacting", but then you really meet somebody, and 1) you said you're a long-time listener, and now you're on the show, so it just really -- like, having been in the trenches so long, and just see all this over-time pay off just makes me really believe in that whole "Slow and steady, keep showing up, do what needs done", and eventually things happen. I just love that.Shawn Wang: Yeah. There's an infinite game mentality to this. But I don't want to diminish the concept of free, so... It bothers me a little, because Quincy actually struggles a lot with the financial side of things. He supports millions of people on like a 300k budget. 300k. If every single one of us who graduated at FreeCodeCamp and went on to a successful tech career actually paid for our FreeCodeCamp education - which is what I did; we started the hashtag. It hasn't really taken off, but I started a hashtag called #payitbackwards. Like, just go back, once you're done -- once you can afford it, just go back and pay what you thought it was worth. For me, I've paid 20k, and I hope that everyone who graduates FreeCodeCamp does that, to keep it going.Adam Stacoviak: Well, I mean, why not...?Shawn Wang: I'd also say one thing... The important part of being free is that I can do it on nights and weekends and take my time to decide if I want to change careers. So it's not just a free replacement to bootcamps, it actually is an async, self-guided, dip-your-toe-in-the-water, try-before-you-buy type of thing for people who might potentially change their lives... And that's exactly what happened for me. I kept my day job until the point I was like "Okay, I like enough of this... I'm still not good, but I like enough of this that I think I could do this full-time."Adam Stacoviak: I like the #payitbackwards hashtag. I wish it had more steam, I suppose.Jerod Santo: We should throw some weight behind that, Adam, and see if we can...Adam Stacoviak: Yeah. Well, you know, you think about Lambda School, for example - and I don't wanna throw any shade by any means, because I think what Austin has done with Lambda... He's been on Founders Talk before, and we talked deeply about this idea of making a CS degree cost nothing, and there's been a lot of movement on that front there... But you essentially go through a TL;DR of Lambda as you go through it, and you pay it after you get a job if you hit certain criteria, and you pay it based upon your earnings. So why not, right? Why not have a program like that for FreeCodeCamp, now that you actually have to commit to it... But it's a way. I love that you paid that back and you made that an avenue, an idea of how you could pay back FreeCodeCamp, despite the commitment not being there.Jerod Santo: Right.Shawn Wang: Yeah. And Quincy is very dedicated to it being voluntary. He thinks that people have different financial situations. I don't have kids, so I can afford a bit more. People should have that sort of moral obligation rather than legal obligation.I should mention that Lambda School is currently being accused of some fairly substantial fraud against its students...Jerod Santo: Oh, really?Shawn Wang: Yeah, it actually just came out like two days ago.Adam Stacoviak: I saw that news too, on Monday.Shawn Wang: Yeah. It's not evidenced in the court of law, it's one guy digging up dirt; let's kind of put this in perspective. But still, it's very serious allegations, and it should be investigated. That said, the business of changing careers and the business of teaching people to code, and this innovation of Income Share Agreements (ISA), where it actually makes financial sense for people to grow bootcamps and fund bootcamps - this is something I strongly support... Whether or not it should be a venture-funded thing, where you try to go for 10x growth every year - probably not... [laughs]Adam Stacoviak: Yeah...Jerod Santo: So after FreeCodeCamp you didn't feel quite ready, so you did do a bootcamp... Did you feel ready after that?Shawn Wang: [08:03] Yeah. [laughs] I did a reflection, by the way, of my first year of learning to code, so people can look it up... It's called "No zero days. My path to learning to code", and I think I posted it on Hacker News. And doing everything twice actually helped me a lot. Because before I came into my paid bootcamp, I had already spun up some React apps. I had already started to mess with WebPack, and I knew enough that I wasn't understanding it very much, I was just following the instructions. But the second time you do things, you have to space, to really try to experiment, to actually read the docs, which most people don't do, and actually try to understand what the hell it is you're doing. And I felt that I had an edge over the other people in my bootcamp because I did six months of FreeCodeCamp prior.Jerod Santo: So this other thing that you do, which not everybody does, is this Learning in Public idea... And you have this post, Learn in Public. You call it "The fastest way to learn", or the fastest way to build your expertise - networking, and second brain. I'm not sure what the second brain is, so help us out with that one... But also, why is learning in public faster than learning in private.Shawn Wang: Yeah. This is a reflection that came from me understanding the difference, qualitatively, between why I'm doing so well in my tech career versus my finance career. In finance, everything is private, meaning the investment memos that I wrote, the trade ideas that I had - they're just from a company; they're intellectual property of my company. In fact, I no longer own them. Some of my best work has been in that phase, and it's locked up in an email inbox somewhere, and I'll never see it again. And that's because tech is a fundamentally open and positive-sum industry, where if you share things, you don't lose anything; you actually gain from sharing things... Whereas in finance it's a zero-sum battle against who's got the secret first and who can act on it first.And I think when you're in tech, you should exploit that. I think that we have been trained our entire lives to be zero-sum, from just like the earliest days of our school, where we learn, we keep it to ourselves to try to pass the test, try to get the best scores, try to get the best jobs, the best colleges, and all that, because everything's positional. For you to win, others have to lose. But I don't see tech in that way, primarily because tech is still growing so fast. There's multiple ways for people to succeed, and that's just the fundamental baseline. You layer on top of that a bunch of other psychological phenomenon.I've been really fascinated by this, by what it is so effective. First of all, you have your skin in the game, meaning that a lot of times when your name is on the blog posts out there, or your name is on the talk that you gave, your face is there, and people can criticize you, you're just incentivized to learn better, instead of just "Oh, I'll read this and then I'll try to remember it." No, it doesn't really stick as much. So having skin in the game really helps.When you get something wrong in public, there are two effects that happen. First is people will climb over broken glass to correct you, because that's how the internet does. There's a famous XKCD comic where like "I can't go to bed yet." "Why?" "Someone's wrong on the internet. I have to correct them."Jerod Santo: Right.Shawn Wang: So people are incentivized to fix your flaws for you - and that's fantastic - if you have a small ego.Jerod Santo: I was gonna say, that requires thick skin.Shawn Wang: Yeah, exactly. So honestly -- and that's a barrier for a lot of people. They cannot get over this embarrassment. What I always say is you can learn so much on the internet, for the low, low price of your ego. If we can get over that, we can learn so much, just because you don't care. And the way to get over it is to just realize that the version that you put out today is the version you should be embarrassed about a year from now, because that shows that you've grown. So you divorce your identity from your work, and just let people criticize your work; it's fine, because it was done by you, before you knew what you know today. And that's totally fine.And then the second part, which is that once you've gotten something wrong in public, it's just so embarrassing that you just remember it in a much clearer fashion. [laughter] This built a feedback loop, because once you started doing this, and you show people that you respond to feedback, then it builds a feedback and an expectation that you'll do the next thing, and people respond to the next thing... It becomes a conversation, rather than a solitary endeavor of you just learning the source material.So I really like that viral feedback loop. It helps you grow your reputation... Because this is not just useful for people who are behind you; a lot of people, when they blog, when they write, when they speak, they're talking down. They're like "I have five years experience in this. Here's the intro to whatever. Here's the approach to beginners." They don't actually get much out of that.[12:17] That's really good, by the way, for beginners; that's really important, that experts in the field share their knowledge. They don't see this blogging or this speaking as a way to level up in terms of speaking to their experts in their fields. But I think it's actually very helpful. You can be helpful to people behind you, you can be helpful to people around you, but you can actually be helpful to people ahead of you, because you're helping to basically broadcast or personalize their message. They can check their messaging and see - if you're getting this wrong, then they're getting something wrong on their end, docs-wise, or messaging-wise. That becomes a really good conversation. I've interacted with mentors that way. That's much more how I prefer to interact with my mentors than DM-ing and saying "Hey, can you be my mentor?", which is an unspecified, unpaid, indefinitely long job, which nobody really enjoys. I like project-based mentorship, I like occasional mentorship... I really think that that develops when you learn in public.Adam Stacoviak: I've heard it say that "Today is the tomorrow you hope for."Shawn Wang: Wow.Adam Stacoviak: Because today is always tomorrow at some point, right? Like, today is the day, and today you were hoping for tomorrow to be better...Jerod Santo: I think by definition today is not tomorrow...Adam Stacoviak: No, today is the tomorrow that you hoped for... Meaning like "Seize your moment. It's here."Jerod Santo: Carpe diem. Gotcha.Adam Stacoviak: Yeah, kind of a thing like that.Shawn Wang: I feel a little shady -- obviously, I agree, but also, I feel a little shady whenever I venture into this territory, because then it becomes very motivational speaking-wise, and I'm not about that. [laughs]Adam Stacoviak: Kind of... But I think you're in the right place; keep showing up where you need to be - that kind of thing. But I think your perspective though comes from the fact that you had this finance career, and a different perspective on the way work and the way a career progressed. And so you have a dichotomy essentially between two different worlds; one where it's private, and one where it's open. That to me is pretty interesting, how you were able to tie those two together and see things differently. Because I think too often sometimes in tech, especially staying around late at night, correcting someone on the internet, you're just so deeply in one industry, and you have almost a bubble around you. You have one lens for which you see the world. And you've been able to have multi-faceted perspectives of this world, as well as others, because of a more informed career path.Jerod Santo: Yeah. When you talk about finance as a zero-sum game, I feel like there's actually been moves now to actually open up about finance as well; I'm not sure if either of you have tracked the celebrity rise of Cathie Wood and Ark Invest, and a lot of the moves that she's doing in public. They're an investment fund, and they will actually publish their moves at the end of every day. Like, "We sold these stocks. We bought these stocks." And people laughed at that for a while, but because she's been successful with early on Bitcoin, early with Tesla, she's very much into growth stocks - because of that, people started to follow her very closely and just emulate. And when she makes moves now, it makes news on a lot of the C-SPANs and the... Is C-SPAN the Congress one? What's the one that's the finance one...?Shawn Wang: CNBC?Jerod Santo: CNBC, not C-SPAN. And so she's very much learning in public. She's making her moves public, she's learning as she goes, and to a certain degree it's paid off, it's paid dividends in her career. Now, I'm not sure if everyone's doing that... When you look at crypto investors, like - okay, pseudonymous, but a lot of that stuff, public ledgers. So there's moves that are being made in public there as well. So I wonder if eventually some of that mentality will change. What do you think about that?Shawn Wang: [15:45] It's definitely changed for -- there's always been celebrity investors, and people have been copying the Buffett portfolio for 30 years. So none of that is new. What is new is that Cathie Wood is running an ETF, and just by way of regulation and by way of innovation, she does have to report those changes. [laughs] So mutual funds, hedge fund holdings - these have all been public, and people do follow them. And you're always incentivized to talk your book after you've established your position in your book...Jerod Santo: Right, but you establish it first.Shawn Wang: ...so none of that has changed. But yeah, Cathie has been leading an open approach...Jerod Santo: Is it the rate of disclosure perhaps that's new? Because it seems like it's more real-time than it has historically...Shawn Wang: Yeah. I mean, she's running an ETF, which is new, actually... Because most people just run mutual funds or hedge funds, and those are much more private. The other two I'll probably shout out is Patrick O'Shaughnessy who's been running I guess a fund of funds, and he's been fairly open. He actually adopted the "learn in public" slogan in the finance field, independently of me. And then finally, the other one is probably Ted Seides, who is on the institutional investor side of things. So he invests for universities, and teachers pensions, and stuff like that. So all these people - yeah, they've been leading that... I'm not sure if it's spreading, or they've just been extraordinarily successful in celebrity because of it.Adam Stacoviak: This idea of "in public" is happening. You see people too, like -- CopyAI is building in public... This idea of learning in public, or building in public, or exiting in public... Whatever the public might be, it's happening more and more... And I think it's definitely similar to the way that open source moves around. It's open, so it's visible to everyone. There's no barrier to see what's happening, whether it's positive or negative, with whatever it is in public. They're leveraging this to their advantage, because it's basically free marketing. And that's how the world has evolved to use social media. Social media has inherently been public, because it's social...Jerod Santo: Sure.Adam Stacoviak: Aside from Facebook being gated, with friends and stuff like that... Twitter is probably the most primary example of that, maybe even TikTok, where if I'm a creator on TikTok, I almost can't control who sees my contact. I assume it's for the world, and theoretically, controlled by the algorithm... Because if I live in Europe, I may not see content in the U.S, and the algorithm says no, or whatever. But it's almost like everybody is just in public in those spaces, and they're leveraging it to their advantage... Which is an interesting place to be at in the world. There was never an opportunity before; you couldn't do it at that level, at that scale, ten years ago, twenty years ago. It's a now moment.Jerod Santo: Yeah. Swyx, can you give us an example of something learned in public? Do you basically mean like blog when you've learned something, or ask questions? What does learning in public actually mean when it comes to -- say, take a technology. Maybe you don't understand Redux. I could raise my hand on that one... [laughter] How could I learn that in public?Shawn Wang: There are a bunch of things that you can try. You can record a livestream of you going through the docs, and that's useful to maintainers, understanding "Hey, is this useful or not?" And that's immediately useful. It's so tangible.I actually have a list -- I have a talk about this on the blog post as well... Just a suggestion of things you can do. It's not just blogging. You can speak, you can draw comics, cheatsheets are really helpful... I think Amy Hoy did a Ruby on Rails cheatsheet that basically everyone has printed out and stapled to their wall, or something... And if you can do a nice cheatsheet, I think that's also a way for you to internalize those things that you're trying to learn anyway, and it just so happens to benefit others.So I really like this idea that whatever content you're doing, it's learning exhaust, it's a side effect of you learning, and you just happen to put it out there; you understand what formats work for you, because you have abnormal talents. Especially if you can draw, do that. People love developers who can draw. And then you just put it out there, and you win anyway just by doing it. You don't need an audience. You get one if you do this long enough, but you don't need an audience right away. And you win whether or not people participate with you. It's a single-player game that can become a multiplayer game.Specifically for Redux - you know, go through source code, or go through the docs, build a sample app, do like a simple little YouTube video on it... Depending on the maturity, you may want to try to speak at a meetup, or whatever... You don't have to make everything a big deal. I'm trying to remove the perception from people that everything has to be this big step, like it has to be top of Hacker News, or something. No. It could just be helpful for one person. I often write blog posts with one persona in mind. I mean, I don't name that person, but if you focus on that target persona, actually often it does better than when you try to make some giant thesis that shakes the world...Adam Stacoviak: [20:22] Yeah. Too often we don't move because we feel like the weight of the move is just too much. It's like "How many people have to read this for me to make this a success for me?" You mentioned it's a learning exhaust... And this exhaust that you've put out before - has it been helpful really to you? Is that exhaust process very helpful to you? Is that ingrained in the learnings that you've just gone through, just sort of like synthesize "Okay, I learned. Here's actually what I learned"?Shawn Wang: Yeah. This is actually an opportunity to tie into that second brain concept which maybe you wanted to talk a little bit about. Everything that you write down becomes your second brain. At this point I can search Google for anything I've ever written on something, and actually come up on my own notes, on whatever I had. So I'm not relying on my memory for that. Your human brain, your first brain is not very good at storage, and it's not very good at search; so why not outsource that to computers? And the only way to do that is you have to serialize your knowledge down into some machine-readable format that's part of research. I do it in a number of places; right now I do it across GitHub, and my blog, and a little bit of my Discord. Any place where you find you can store knowledge, I think that's a really good second brain.And for Jerod, I'll give you an example I actually was gonna bring up, which is when I was trying to learn React and TypeScript - like, this goes all the way back to my first developer job. I was asked to do TypeScript, even though I'd never done it before. And honestly, my team lead was just like "You know TypeScript, right? You're a professional React dev, you have to know TypeScript." And I actually said no, and I started learning on day one.And what I did was I created the React to TypeScript cheatsheet, which literally was just copy-pasteable code of everything that I found useful and I wish I knew when I was starting out. And I've just built that over time. That thing's been live for three years now, it's got like 20,000 stars. I've taught thousands of developers from Uber, from Microsoft, React and TypeScript. And they've taught me - every time they send in a question or a PR... I think it's a very fundamental way of interacting, which is learning in public, but specifically this one - it's open source knowledge; bringing up our open source not just to code, but to everything else. I think that's a fundamental feedback loop that I've really enjoyed as well.Break: [22:31]Jerod Santo: One of the things I appreciate about you, swyx, is how you are always thinking, always writing down your thoughts... You've been watching and participating in this industry now for a while, and you've had some pretty (I think) insightful writings lately. The first one I wanna talk about is this API Economy post. The Light and Dark Side of the API Economy. You say "Developers severely underestimate the importance of this to their own career." So I figure if that's the case, we should hear more about it, right?Shawn Wang: [laughs] Happy to talk about it. So what is the API economy? The API economy is developers reshaping the world in their image. Very bold statement, but kind of true, in the sense that there is now an API for everything - API for cards, API for bank accounts, API for text, API for authentication, API for shipping physical goods... There's all sorts of APIs. And what that enables you to do as a developer is you can call an API - as long as you know REST or GraphQL these days, you know how to invoke these things and make these things function according to the rest of your program. You can just fit those things right in. They're a very powerful thing to have, because now the cost of developing one of these services just goes down dramatically, because there's another company doing that as a service for you.I wrote about it mainly because at Netlify we were pitching serverless, we were pitching static hosting, and we were pitching APIs. That's the A in JAMstack. But when I google "API economy", all the search results were terrible. Just horrible SEO, bland, meaningless stuff that did not speak to developers; it was just speaking to people who like tech buzzwords. So I wrote my own version. The people who coined it at Andreessen Horowitz, by the way, still to this day do not have a blog post on the API economy. They just have one podcast recording which nobody's gonna listen. So I just wrote my version.Jerod Santo: You're saying people don't listen to podcasts, or what?Shawn Wang: [laughs] When people are looking up a term, they are like "What is this thing?", and you give them a podcast, they're not gonna sit down and listen for 46 minutes on a topic. They just want like "Give me it, in one paragraph. Give me a visual, and I'm gonna move on with my day." So yeah, whenever I see an opportunity like that, I try to write it up. And that's the light side; a lot of people talk about the light side. But because it's a personal blog, I'm empowered to also talk about the dark side, which is that as much as it enables developers, it actually is a little bit diminishing the status of human expertise and labor and talent. So we can talk a little bit about that, but I'm just gonna give you time to respond.Jerod Santo: [28:05] Hm. I'm over here thinking now that you're not at Netlify, I'm curious - this is tangential, but what's your take on JAMstack now? I know you were a professional salesman there for a while, but... It seems like JAMstack - we've covered it for years, it's a marketing term, it's something we've already been doing, but maybe taking it to the next level... There's lots of players now - Netlify, Vercel etc. And yet, I don't see much out there in the real world beyond the people doing demos, "Here's how to build a blog, here's how to do this, here's my personal website", and I'm just curious... I'm not like down on JAMstack, but I just don't see it manifesting in the ways that people have been claiming it's going to... And maybe we're just waiting for the technology to catch up. I'd just love to hear what you think about it now.Shawn Wang: Yeah. I think that you're maybe not involved in that world, so you don't see this, but real companies are moving on to JAMstack. The phrasing that I like is that -- JAMstack has gone mainstream, and it's not even worth talking about these days, because it's just granted that that's an option for you... So PayPal.me is on the JAMstack, there's large e-commerce sites... Basically, anything that decouples your backend from your frontend, and your frontend is statically-hosted - that is JAMstack.I actually am blanking on the name, but if you go check out the recent JAMstack Conf, they have a bunch of examples of people who've not only moved to JAMstack, but obviously moved to Netlify, where they're trying to promote themselves.Jerod Santo: Sure, yeah.Shawn Wang: So yes, it's true that I'm no longer a professional spokesperson, but it's not true that JAMstack is no longer being applied in the enterprise, because it is getting adoption; it's moved on that boring phase where people don't talk about it.One thing I'll say - a thesis that I've been pursuing is that JAMstack is in its endgame. And what do I mean by that? There's a spectrum between the previous paradigm that JAMstack was pushing back on, which is the all-WordPress/server-render-everything paradigm, and then JAMstack is prerender-everything. And now people are filling in--Jerod Santo: In the middle.Shawn Wang: ...I'm gonna put my hands in the Zoom screen right now. People are filling that gap between fully dynamic and fully static. So that's what you see with Next.js and Gatsby moving into serverless rendering, partial rendering or incremental rendering... And there's a full spectrum of ways in which you can optimize your rendering for the trade-offs of updating your content, versus getting your data/content delivered as quickly as possible. There's always some amount of precompilation that you need to do, and there's always some amount of dynamicism that you have to do, that cannot be precompiled. So now there is a full spectrum between those.Why I say it's the end game is because that's it, there's nothing else to explore. It's full-dynamic, full-static, choose some mix in the middle, that's it. It's boring.Jerod Santo: Hasn't that always been the case though? Hasn't there always been sites that server-side render some stuff, and pre-render other things? You know, we cache, we pre-render, some people crawl their own websites once, and... I don't know it seems like maybe just a lot of excitement around a lot of things that we've been doing for many years.Shawn Wang: [laughs] So first of all, those are being remade in the React ecosystem of things, which a lot of us lost when a lot of the web development industry moved to React... So that's an important thing to get back.I mean, I agree, that's something that we've always had, pre-rendering, and services like that, caching at the CDN layer - we've always had that. There's some differences... So if you understand Netlify and why they're trying to push distributed persistent rendering (DVR), it's because caching is a hard problem, and people always end up turning off the cache. Because the first time you run into a bug, you're gonna turn off the cache. And the cache is gonna stay off.So the way that Netlify is trying to fix it is that we put the cache in Git, essentially. Git is the source of truth, instead of some other source of truth distributed somewhere between your CDN and your database and somewhere else. No, everything's in Git. I'm not sure if I've represented that well, to be honest... [laughter]Adam Stacoviak: Well, good thing you don't work for Netlify anymore. We're not holding you to the Netlify standard.Shawn Wang: [31:58] Exactly. All I can say is that to me now it's a good thing in the sense that it's boring. It's the good kind of boring, in the sense of like "Okay, there's a spectrum. There's all these techniques. Yes, there were previous techniques, but now these are the new hotness. Pick your choice." I can get into a technical discussion of why this technique, the first one, the others... But also, is it that interesting unless you're evaluating for your site? Probably not...Jerod Santo: Well, it does play into this API economy though, right? Because when you're full JAMstack, then the A is your most important thing, and when the A is owned by a bunch of companies that aren't yours - like, there's a little bit of dark side there, right? All of a sudden, now I'm not necessarily the proprietor of my own website, to a certain degree, because I have these contracts. I may or may not get cut off... There's a lot of concerns when everybody else is a dependency to your website.Shawn Wang: Yeah. So I don't consider that a dark side at all.Jerod Santo: No, I'm saying to me that seems like a dark side.Shawn Wang: Yeah, sure. This is the risk of lock-in; you're handing over your faith and your uptime to other people. So you have to trade that off, versus "Can you build this yourself? And are you capable of doing something like this, and are you capable of maintaining it?" And that is a very high upfront cost, versus the variable cost of just hiring one of these people to do it for you as a service.So what I would say is that the API economy is a net addition, because you as a startup - the startup cost is very little, and if you get big enough where it makes sense for you to build in-house - go ahead. But this is a net new addition for you to turn fixed costs into variable costs, and start with a small amount of investment. But I can hire -- like, Algolia was started by three Ph.D's in search, and I can hire them for cents to do search on my crummy little website. I will absolutely do that every single day, until I get to a big enough point where I cannot depend on them anymore, and I have to build my own search. Fine, I'll do that. But until then, I can just rely on them. That's a new addition there.Jerod Santo: One hundred percent. So what then do you think is the darker side? You mentioned it, but put a finer point on it.Shawn Wang: Yeah. The dark side is that there are people -- like, when I call an Uber ride, Uber is an API for teleportation, essentially. I'm here, I wanna go there. I press a button, the car shows up. I get in the car, get off, I'm there. What this papers over is that the API is calling real actual humans, who are being commoditized. I don't care who drives the car, I really don't. I mean, they may have some ratings, but I kind of don't care.Jerod Santo: That was the case with taxis though, wasn't it?Shawn Wang: That was the case with taxis, for sure. But there's a lot of people living below the API, who are economically constrained, and people who live above the API, developers, who have all the upside, essentially... Because the developers are unique, the labor is commoditized. My DoorDash pickers, my Instacart deliverers - all these are subsumed under the API economy. They're commodities forever, they know it, and there's no way out for them, unless they become developers themselves. There's a class system developing below and above the API. And the moment we can replace these people under the API with robots, you better believe we'll do that, because robots are way cheaper, and they complain less, they can work 24 hours, all this stuff.Jerod Santo: Yeah.Shawn Wang: So that's the dark side, which is, yeah, as a developer now - fantastic. I can control most parts of the economy with just a single API call. As a startup founder, I can develop an API for literally anything, and people will buy it. The downside is human talent is being commoditized, and I don't know how to feel about that. I think people are not talking enough about it, and I just wanna flag it to people.Jerod Santo: Yeah.Adam Stacoviak: So dark side could mean a couple things. One, it could mean literally bad; dark as synonymous with bad. Or dark as in shady. And we're not sure, it's obscured in terms of what's happening. And so let's use an Instacarter or a Dasher - to use their terminology. I happen to be a DoorDash user, so I know they're called Dashers; that's the only reason I know that. It's not a downplay, it's just simply what the terminology is...[35:59] You could say it's below the API, but I wonder, if you've spoken with these people, or people that live in what you call below the API, because I would imagine they're not doing that because they're being forced. Like, it's an opportunity for them.Shawn Wang: Oh, yeah.Adam Stacoviak: And I remember when I was younger and I had less opportunity because I had less "above the API" (so to speak) talent... And I do agree there's a class here, but I wonder if it's truly bad; that dark is truly bad, or if it's just simply obscure in terms of how it's gonna play out.Shawn Wang: This is about upside. They will never get to that six figures income with this thing.Adam Stacoviak: Not that job.Jerod Santo: No.Shawn Wang: It's really about the class system, which is the dark side. You don't want to have society splinter into like a serving class and whatever the non-serving class is. It's also about the upside - like, I don't see a way for these people to break out unless, they really just take a hard stop and just go to a completely different career track.Jerod Santo: Right.Adam Stacoviak: Here's where I have a hard time with that... I'm not pushing back on that you're wrong, I'm just wondering more deeply...Shawn Wang: Sure.Adam Stacoviak: I imagine at one point in my life I was a DoorDasher.Shawn Wang: Yeah.Adam Stacoviak: I washed dishes, I did definitely unique jobs at a young age before I had skill. And so the path is skill, and as long as we have a path to skill, which you've show-cased through FreeCodeCamp in your path, then I think that dark side is just simply shady, and not bad.Shawn Wang: Okay.Adam Stacoviak: And I'm just trying to understand it, because I was truly a DoorDasher before DoorDash was available. I washed dishes, delivered papers, I had servant-level things; I was literally a server at a restaurant before... And I loved doing that kind of work, but my talents have allowed me to go above that specific job, and maybe even the pay that came with that job. I've served in the military before, got paid terrible dollars, but I loved the United States military; it's great. And I love everybody who's served in our military. But the point is, I think the path is skill, and as long as we have a pathway to skill, and jobs that can house that skill and leverage that skill to create new value for the world, I just wonder if it's just necessary for society to have, I suppose, above and below API things.Jerod Santo: Until we have all the robots. Then there is nobody underneath. At that point it's all robots under the API.Shawn Wang: Yes, and that is true in a lot of senses, actually. Like, farming is mostly robots these days. You do have individual farmers, but they're much less than they used to be. I don't know what to say about that, shady or dark... I think it's just -- there's no career track. You have to go break out of that system yourself. Thank God there's a way to do it. But back in the day, you used to be able to go from the mailroom to the boardroom.Adam Stacoviak: I see.Shawn Wang: I see these stories of people who used to be janitors at schools become the principal. Companies used to invest in all their people and bring them up. But now we're just hiring your time, and then if you wanna break out of that system - good luck, you're on your own. I think that that lack of upward mobility is a problem, and you're not gonna see it today. It's a slow-moving train wreck. But it's gonna happen where you have society split in two, and bad things happen because of it.Adam Stacoviak: I mean, I could agree with that part there, that there definitely is no lateral movement from Dasher to CEO of DoorDash.Shawn Wang: It's just not gonna happen.Adam Stacoviak: Or VP of engineering at DoorDash. I think because there is no path, the path would be step outside of that system, because that system doesn't have a path. I could agree with that, for sure.Jerod Santo: Yeah. I mean, the good news is that we are creating -- there are paths. This is not like a path from X to Y through that system, but there are other alternate paths that we are creating and investing in, and as well as the API gets pushed further and further down in terms of reachability - we now have more and more access to those things. It's easier now, today, than it ever has been, because of what we were talking about, to be the startup founder, right? To be the person who starts at CEO because the company has one person in it, and they're the CEO. And to succeed in that case, and become the next DoorDash.Adam Stacoviak: True.Jerod Santo: So there are opportunities to get out, it's just not a clear line... And yeah, it takes perhaps some mentorship, perhaps ingenuity... A lot of the things that it takes to succeed anyway, so...Shawn Wang: [40:05] I'll give a closing note for developers who are listening, because you're already a developer... So the analogy is if you're above the API, you tell machines what to do; if you're below the API, machines tell you what to do. So here's the developer analogy, which is there's another division in society, which is the kanban board. If you're below the kanban board, the kanban board tells you what to do. If you're above it, you tell developers what to do. [laughs]Jerod Santo: There you go.Shawn Wang: So how do you break out of that class division? I'll leave it out to you, but just keep in mind, there's always layers.Jerod Santo: I love that.Adam Stacoviak: I love the discussion around it, but I'm also thankful you approached the subject by a way of a blog post, because I do believe that this is interesting to talk about, and people should talk about it, for sure. Because it provides introspection into, I guess, potentially something you don't really think about, like "Do I live below or above the APi?" I've never thought about that in that way until this very moment, talking to you, so... I love that.Break: [40:58]Jerod Santo: So another awesome post you have written lately is about Cloudflare and AWS. Go - not the language, the game Go... I know very little about the language, and I know even less about the game... And Chess... How Cloudflare is approaching things, versus how AWS and Google and others are... Given us the TL;DR of that post, and then we'll discuss.Shawn Wang: Okay. The TL;DR of that post is that Cloudflare is trying to become the fourth major cloud after AWS, Azure and GCP. The way they're doing it is fundamentally different than the other three, and the more I've studied them - I basically observed Cloudflare for the entire time since I joined Netlify. Netlify kind of is a competitor to Cloudflare, and it's always this uncomfortable debate between "Should you put Cloudflare in front of Netlify? Netlify itself is a CDN. Why would you put a CDN in front of another CDN?" Oh, because Netlify charges for bandwidth, and Cloudflare does not. [laughter]Jerod Santo: It's as simple as that.Shawn Wang: And then there's DDOS protection, all that stuff; very complicated. Go look up the Netlify blog post on why you should not put Cloudflare in front of Netlify, and decide for yourself. But Netlify now taking on AWS S3 - S3 is like a crown jewel of AWS. This is the eighth wonder of the world. It provides eleven nines of durability. Nothing less than the sun exploding will take this thing down... [laughs]Jerod Santo: Right? You know what's funny - I don't even consider us at Changelog AWS customers; I don't even think of us that way. But of course, we use S3, because that's what you do. So yeah, we're very much AWS customers, even though I barely even think about it, because S3 is just like this thing that of course you're gonna use.Shawn Wang: There's been a recent history of people putting out S3-compatible APIs, just because it's so dominant that it becomes the de-facto standard. Backblaze did it recently. But Cloudflare putting out R2 and explicitly saying "You can slurp up the S3 data, and by the way, here's all the cost-benefit of AWS egress charges that's what Matthew Prince wrote about in his blog post is all totally true, attacks a part of AWS that it cannot compromise on and just comes at the top three clouds from a different way, that they cannot respond to.[44:17] So I always like these analogies of how people play destruction games. I'm a student of destruction, and I study Ben Thompson and Clay Christensen, and that entire world, very quickly... So I thought this was a different model of destruction, where you're essentially embracing rather than trying to compete head-on. And wrapping around it is essentially what Go does versus chess, and I like -- you know, there's all these comparisons, like "You're playing 2D chess, I'm playing 3D chess. You're playing chess, I'm playing Go." So Cloudflare is playing Go by surrounding the S3 service and saying "Here is a strict superset. You're already a consumer of S3. Put us on, and magically your costs get lower. Nothing else about it changes, including your data still lives in AWS if you ever decide to leave us." Or if you want to move to Cloudflare, you've just gotta do the final step of cutting off S3.That is a genius, brilliant move that I think people don't really appreciate, and it's something that I study a lot, because I work at companies that try to become the next big cloud. I worked at Netlify, and a lot of people are asking, "Can you build a large public company on top of another cloud? Our second-layer cloud is viable." I think Vercel and Netlify are proving that partially it is. They're both highly valued. I almost leaked some info there... When does this go out? [laughs]Jerod Santo: Next week, probably...Shawn Wang: Okay, alright... So they're both highly valued, and - like, can they be hundred-billion-dollar companies? I don't know. We don't know the end state of cloud, but I think people are trying to compete there, and every startup -- I nearly joined Render.com as well. Every startup that's trying to pitch a second-layer cloud thesis is always working under the shadows of AWS. And this is the first real thesis that I've seen, that like "Oh, okay, you not only can credibly wrap around and benefit, you can actually come into your own as a fourth major cloud." So I'm gonna stop there... There's so many thoughts I have about Cloudflare.Jerod Santo: Yeah. So do you see that R2 then -- I think it's a brilliant move, as you described it... As I read your post, I started to appreciate, I think, the move, more than I did when I first read about it and I was like "Oh, they're just undercutting." But it seems they are doing more than just that. But do you think that this R2 then is a bit of a loss leader in order to just take a whole bunch of AWS customers, or do you think there's actually an economic -- is it economically viable as a standalone service, or do you think Cloudflare is using it to gain customers? What are your thoughts in their strategy of Why?Shawn Wang: This is the top question on Twitter and on Hacker News when they launch. They are going to make money on this thing, and the reason is because of all the peering agreements that they've established over the past five years. As part of the normal business strategy of Cloudflare, they have peering agreements with all of the ISPs; bandwidth is free for them. So... For them in a lot of cases. Again, I have to caveat all this constantly, because I should note to people that I am not a cloud or networking expert. I'm just learning in public, just like the rest of you, and here's what I have so far. So please, correct me if I'm wrong, and I'll learn from it.But yeah, I mean - straight on, it's not a loss leader. They plan to make money on it. And the reason they can is because they have worked so hard to make their cost structure completely different in AWS, and they've been a friend to all the other ISPs, rather than AWS consuming everything in its own world. Now you're starting to see the benefits of that strategy play out. And by the way, this is just storage, but also they have data store, also they have service compute, all following the same model.Jerod Santo: So what do you think is a more likely path over the next two years? Cloudflare --Adam Stacoviak: Prediction time!Jerod Santo: ...Cloudflare steals just massive swathes of AWS customers, or AWS slashes prices to compete?Shawn Wang: So I try not to do the prediction business, because I got out of that from the finance days... All I'm doing is nowcasting. I observe what I'm seeing now and I try to put out the clearest vision of it, so the others can follow.I think that it makes sense for them to be replicating the primitives of every other cloud service. So in 2017 they did service compute with Cloudflare Workers. In 2018 they did eventually consistent data store. In 2019 - website hosting; that's the Netlify competitor. In 2020 they did strongly-consistent data store, with Durable Objects. In 2021 object storage. What's next on that list? Go on to your AWS console and go shopping. And instead of seven different ways to do async messaging in AWS, probably they're gonna do one way in Cloudflare. [laughs]Adam Stacoviak: [48:34] A unified API, or something like that...Jerod Santo: Yeah, they'll just look at AWS' offerings, the ones they like the best, and do it that way, right?Shawn Wang: Yeah, just pick it up.Adam Stacoviak: Maybe the way to get a prediction out of you, swyx, might be rather than directly predict, maybe describe how you win Go.Shawn Wang: How you win Go...Adam Stacoviak: Yeah, what's the point of Go? How do you win Go? Because that might predict the hidden prediction, so to speak.Shawn Wang: Okay. For listeners who don't know Go, let me draw out the analogy as well. So most people are familiar with chess; individual chess pieces have different values and different points, and they must all support each other. Whenever you play chess, you need the Knight to support the pawns, something like that... Whereas in Go, you place your pieces everywhere, and they're all indistinguishable from each other. And it's more about claiming territory; at the end of the day, that's how you win Go, you claim the most territory compared to the others... And it's never a winner-take-all situation. Most likely, it's like a 60/40. You won 60% of the territory and your competitor has 40% of the territory. That's more likely a mapping of how cloud is gonna play out than chess, where winner-takes-all when you take the King. There's no King in the cloud, but--Jerod Santo: Are you sure...?Shawn Wang: ...there's a lot likely of territory claiming, and Cloudflare is really positioned very well for that. It's just part of the final realization that I had at the end of the blog post. And partially, how you take individual pieces of territory is that you surround all the pieces of the enemy and you place the final piece and you fill up all the gaps, such that the enemy is completely cut off from everything else and is surrounded. And that's what R2 does to S3 - it surrounds S3, and it's up to you to place that final piece. They call it, Atari, by the way, which is the name of the old gaming company, Atari. They have placed AWS S3 in Atari, and it's up to the customers to say "I'm gonna place that final piece. I'm gonna pay the cost of transferring all my data out of S3 and cut S3 off", and they cut off all the remaining liberties. So how do you win in Go? You claim the most amount of territory, and you surround the pieces of the enemy.Adam Stacoviak: Which, if you thought maybe that was oxygen, the territory, you might suck the oxygen away from them, so they can't live anymore, so to speak... And maybe you don't take it by killing it. Maybe you sort of suffocate it almost, if their space becomes small enough; if you take enough territory and it begins to shrink enough, it's kind of like checkmate, but not.Shawn Wang: Yeah. There's also a concept of sente in Go, which is that you make a move that the opponent has to respond to, which is kind of like a check, or checkmate -- actually, not; just the check, in chess. And right now, AWS doesn't feel the need to respond. Cloudflare is not big enough. Like, these are names to us, but let's just put things in numbers. Cloudflare's market cap is 36 billion, AWS' market cap is 1.6 trillion; this is Amazon's total market cap. Obviously, AWS is a subset of that.Jerod Santo: Sure.Shawn Wang: So your competitor is 40 times larger than you. Obviously, Cloudflare is incentivized to make a lot of noise and make themselves seem bigger than it is. But until AWS has to respond, this is not real.Adam Stacoviak: Nice.Jerod Santo: So as a developer, as a customer of potentially one or both of these... Let's say you have a whole bunch of stuff on S3 - I'm asking you personally now, swyx - and R2 becomes available... Is that a no-brainer for you, or is there any reason not to use that?Shawn Wang: You're just adding another vendor in your dependency tree. I think for anyone running silicon bandwidth, it is a no-brainer.Jerod Santo: Yeah. So over the course of n months, where n equals when they launch plus a certain number - I mean, I think this is gonna end up eventually on Amazon's radar, to where it's gonna start affecting some bottom lines that important people are gonna notice. So I just wonder - I mean, how much territory can Cloudflare grab before there's a counter-move? It's gonna be interesting to watch.Shawn Wang: [52:12] So Ben from Vantage actually did a cost analysis... Vantage is a startup that is made up former AWS Console people; they're trying to build a better developer experience on top of AWS. They actually did a cost analysis on the R2 move, and they said that there's probably a hundred billion dollars' worth of revenue at stake for Amazon. So if they start to have a significant dent in that, let's say like 40%, AWS will probably have to respond. But until then, there's nothing to worry about. That's literally how it is in Amazon; you have to see the numbers hit before you respond.Jerod Santo: Yeah. It hasn't even been a blip on the radar at this point, the key metrics to the people who are important enough to care are watching. You said you started watching all of these CDNs. Of course, you worked at Netlify... You take an interest in backends. There's something you mentioned in the break about frontenders versus backend, and where you've kind of been directing your career, why you're watching Cloudflare so closely, what you're up to now with your work... Do you wanna go there?Shawn Wang: Let's go there. So if you track my career, I started out as a frontend developer. I was developing design systems, I was working with Storybook, and React, and all that... Then at Netlify I was doing more serverless and CLI stuff. At AWS more storage and database and AppSync and GraphQL stuff... And now at Temporal I'm working on a workflow engine, pure backend. I just went to KubeCon two weeks ago...Jerod Santo: Nice!Shawn Wang: What is a frontend developer doing at KubeCon...?Adam Stacoviak: New territory.Shawn Wang: It's a frontend developer who realizes that there's a career ceiling for frontend developers. And it's not a polite conversation, and obviously there are exceptions to frontend developers who are VPs of engineering, frontend developers who are startup founders... And actually, by the way, there's a lot of VC funding coming from frontend developers, which is fantastic for all my friends. They're all getting funded, left, right and center. I feel left out. But there is a Career ceiling, in a sense that survey a hundred VPs of engineering, how many of them have backend backgrounds, and how many of them have frontend backgrounds? And given that choice, what's more likely for you and your long-term career progression? Do you want to specialize in frontend or do you want to specialize in backend? Different people have different interests, and I think that you can be successful in whatever discipline you pick. But for me, I've been moving towards the backend for that reason.Adam Stacoviak: Describe ceiling. What exactly do you mean when you say "ceiling"?Shawn Wang: Career ceiling. What's your terminal title.Jerod Santo: Like your highest role, or whatever. Highest salary, highest role, highest title...Adam Stacoviak: Gotcha.Shawn Wang: Like, straight up, how many VPs of engineering and CTOs have backend backgrounds versus frontend.Jerod Santo: Yeah. I mean, just anecdotally, I would agree with you that it's probably 8 or 9 out of 10 CTOs have -- is that what you said, 8 or 9?Shawn Wang: Yeah, yeah. So there's obviously an economic reasoning for this; it's because there's a bias in the industry that frontend is not real development, and backend is. And that has to be combated. But also, there's an economic reasoning, and I always go back to the economics part, because of my finance background... Which is that your value to the company, your value to the industry really depends on how many machines run through you. You as an individual unit of labor, how much money do you control, and how much machine process, or compute, or storage, or whatever runs through you. And just straight-up frontend doesn't take as much. [laughs] Yes, frontend is hard, yes, design is hard, yes, UX is crucially important, especially for consumer-facing products... But at the end of the day, your compute is being run on other people's machines, and people don't value that as much as the compute that I pay for, that I need to scale, and therefore I need an experienced leader to run that, and therefore that is the leader of my entire eng.Jerod Santo: I wonder if that changes at all for very product-focused orgs, where I think a lot of frontenders, the moves are into product design and architecture, and away from - not software architecture, but product design. And it seems like maybe if you compare - not VP of engineering, but VP of product, you'd see a lot of former frontenders.Shawn Wang: [56:03] Yeah.Jerod Santo: Maybe that's their path. Do you think that's --Shawn Wang: Totally. But you're no longer a frontend dev. You suddenly have to do mocks...Jerod Santo: Yeah, but when you're VP of engineering you're not a backend dev either.Shawn Wang: Yeah.Jerod Santo: So you're kind of both ascending to that degreeShawn Wang: Backends devs will never report to you, let's put it that way.Jerod Santo: Okay. Fair.Shawn Wang: [laughter] But somehow, frontend devs have to report to backend devs, for some reason; just because they're superior, or something. I don't know, it's just like an unspoken thing... It's a very impolite conversation, but hey, it's a reality, man.Jerod Santo: So do you see this personally, or do you see this by looking around?Shawn Wang: Yeah.Jerod Santo: Yeah. You felt like you had reached a ceiling.Shawn Wang: Well, again, this is very impolite; there's a ton of ways to succeed, and there are definitely exceptions. Emily Nakashima at Honeycomb - former frontend person, now VP of engineering. I don't know, I could have done that. I have interest in backend and I'm pursuing that. So I will say that - this is a soft ceiling, it's a permeable ceiling. It's not a hard ceiling.Jerod Santo: Sure.Shawn Wang: But there's a ceiling though, because you can see the numbers.Adam Stacoviak: What is it in particular the VP of engineering does that would make a frontender less likely to have that role? What specifically? I mean, engineering is one of the things, right? Commanding the software... Which is not necessarily frontend.Jerod Santo: Well, frontend is also an engineering discipline.Adam Stacoviak: I guess it kind of depends on the company, too. Honeycomb is probably a different example.Shawn Wang: I haven't been a VP of engineering, so I only have some theories. I suggest you just ask the next VP of engineering that you talk to, or CTO.Adam Stacoviak: Yeah.Jerod Santo: Yeah. That'd be a good one to start asking people.Adam Stacoviak: What do you do here? What is it you do here?Shawn Wang: What is it you do here?Jerod Santo: Exactly.Shawn Wang: [laughs]Adam Stacoviak: Well, I just wondered if there was a specific skillset that happens at that VP of engineering level that leads more towards a backender being more likely than a frontender to get hired into the role.Shawn Wang: I think there's some traditional baggage. Power structures persist for very long times... And for a long time UX and frontend was just not valued. And we're like maybe five years into the shift into that. It's just gonna take a long time.Jerod Santo: I agree with that. So tell us what you're up to now. You said you're doing workflows... I saw a quick lightning talk; you were talking about "React for the backend." So you're very much taking your frontend stuff into the backend here, with React for the backend. Tell us about that.Shawn Wang: Let's go for it. So at Netlify and at AWS I was essentially a developer advocate for serverless. So this is very cool - it does pay-as-you-go compute, and you can do a lot of cool stuff with it. But something that was always at the back of my mind bothering me, that serverless does not do well, is long-running jobs. It just does not do well. You have to chain together a bunch of stuff, and it's very brittle; you cannot test it... It's way more expensive than you would do in a normal environment.Jerod Santo: Yeah.Shawn Wang: And it made me realize that in this move to take apart everything and make everything as a service, we have gained scalability, but we've lost basically everything else. And what I was trying to do was "How do we reconstruct the experience of the monolith? What are the jobs to be done?" When you break it down, what does a computer do for you, and what is not adequately addressed by the ecosystem?I went through the exercise... I wrote a blog post called "Reconstructing the monolith, and I actually listed it out." So what are the jobs of cloud for a computer? You want static file serving, you want functions, you want gateway, you want socket management, job runners, queue, scheduler, cold storage, hot storage. There's meta jobs like error logging, usage logging, dashboarding, and then edge computing is like a unique to cloud thing. But everything else, you can kind of break it up and you can locate it on one machine, or you can locate it on multiple machines, some of them owned by you, some of them not owned by you.The thing that serverless -- that had a whole in the ecosystem was job running. Not good. Basically, as an AWS developer right now, the answer is you set a CloudWatch schedule function, and you pull an endpoint, and that should read some states from a database, and check through where you are, and compute until the 15-minute timeout for Lambda, and then save it back in, and then wait for the next pull, and start back up again. Super-brittle, and just a terrible experience; you would never want to go this way.[01:00:08.13] The AWS current response to that is AWS Step Functions, which is a JSON graph of what happens after the other, and this central orchestrator controls all of that. I think we could do better, and that's eventually what got me to temporal. So essentially, this blog post that I wrote - people found me through that, and hired both our head of product and myself from this single blog post. So it's probably the highest ROI blog post I've ever written.Jerod Santo: Wow. That's spectacular.Shawn Wang: It's just the VC that invested in Temporal. So what Temporal does is it helps you write long-running workflows in a doable fashion; every single state transition is persisted to a database, in idiomatic code. So idiomatic Java, idiomatic Go, idiomatic JavaScript, and PHP. This is different from other systems, because other systems force you to learn their language. For Amazon, you have to learn Amazon States Language. For Google Workflows - Google Workflows has a very long, very verbose JSON and YAML language as well.And these are all weird perversions of -- like, you wanna start simple; JSON is very simple, for doing boxes and arrows, and stuff like that... But you start ending up having to handwrite the AST of a general-purpose programming language, because you want variables, you want loops, you want branching, you want all that god stuff. And the best way to model asynchronous and dynamic business logic is with a general-purpose programming language, and that's our strong opinion there.So Temporal was created at Uber; it runs over 300 use cases at Uber, including driver onboarding, and marketing, and some of the trips stuff as well. It was open source, and adopted at Airbnb, and Stripe, and Netflix, and we have all those case studies on -- DoorDash as well, by the way, runs on the Uber version of Temporal.Jerod Santo: There you go, Adam.Shawn Wang: And yeah, they spun out to a company two years ago, and we're now trying to make it as an independent cloud company. And again, the

Screaming in the Cloud
Striking a Balance on the Cloud with Rachel Stephens

Screaming in the Cloud

Play Episode Listen Later Nov 29, 2021 39:11


About RachelRachel Stephens is a Senior Analyst with RedMonk, a developer-focused industry analyst firm. RedMonk focuses on how practitioners drive technological adoption. Her research covers a broad range of developer and infrastructure products, with a particular focus on emerging growth technologies and markets. (But not crypto. Please don't talk to her about NFTs.)Before joining RedMonk, Rachel worked as a database administrator and financial analyst. Rachel holds an MBA from Colorado State University and a BA in Finance from the University of Colorado.Links: RedMonk: https://redmonk.com/ Great analysis: https://redmonk.com/rstephens/2021/09/30/a-new-strategy-r2/ “Convergent Evolution of CDNs and Clouds”: https://redmonk.com/sogrady/2020/06/10/convergent-evolution-cdns-cloud/ “Everything is Securities Fraud?”: https://cafe.com/stay-tuned/everything-is-securities-fraud-with-matt-levine/ Twitter: https://twitter.com/rstephensme TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by my friends at ThinkstCanary. Most companies find out way too late that they've been breached. ThinkstCanary changes this and I love how they do it. Deploy canaries and canary tokens in minutes and then forget about them. What's great is the attackers tip their hand by touching them, giving you one alert, when it matters. I use it myself and I only remember this when I get the weekly update with a “we're still here, so you're aware” from them. It's glorious! There is zero admin overhead  to this, there are effectively no false positives unless I do something foolish. Canaries are deployed and loved on all seven continents. You can check out what people are saying at canary.love. And, their Kub config canary token is new and completely free as well. You can do an awful lot without paying them a dime, which is one of the things I love about them. It is useful stuff and not an, “ohh, I wish I had money.” It is speculator! Take a look; that's canary.love because it's genuinely rare to find a security product that people talk about in terms of love. It really is a unique thing to see. Canary.love. Thank you to ThinkstCanary for their support of my ridiculous, ridiculous non-sense.   Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. The last time I spoke to Rachel Stephens over at RedMonkwas in December of 2019. Well, on this podcast anyway; we might have exchanged conversational tidbits here and there at some point since then. But really, if we look around the world there's nothing that's materially different than it was today from December in 2019, except, oh, that's right everything. Rachel Stephens, you're still a senior analyst at RedMonk, which hey, in this day and age, longevity at a company is something that is almost enough to occasion comment on its own. Thanks for coming back for another round, I appreciate it.Rachel: Oh, I'm so happy to be here, and it's exciting to talk about the state of the world a few years later than the last time we talked. But yeah, it's been a hell of a couple years.Corey: Really has, but rather than rehashing pandemic stuff because I feel like unless people have been living in a cave for the last couple of years—because we've all been living in caves for the last couple of years—they know what's up with that. What's new in your world? What has changed for you aside from all of this in the past couple of years working in one of the most thankless of all jobs, an analyst in the cloud computing industry?Rachel: Well, the job stuff is all excellent and I've had wonderful time working at RedMonk. So, RedMonk overall is an analyst firm that is focused on helping people understand technology trends, particularly from the view of the developer or the practitioner. So, helping to understand how the people who are using technologies are actually driving their overall adoption. And so there has been all kinds of interesting things that have happened in that score in the last couple of years. We've seen a lot of interesting trends, lots of fun things to look at in the space and it's been a lot.On a personal side, like a week into lockdown I found out that I was pregnant, so I went through all of locking down and the heart of the pandemic pregnant. I had my maternity leave earlier this year and came back and so excited to be back in. But it's also just been a lot to catch up on in the space as you come back from leave which I'm sure you are well familiar with.Corey: Yes, I did the same thing, slightly differently timed. My second daughter Josephine was born at the end of September. When did your kiddo arrive on the scene to a world of masked strangers?Rachel: So, I have an older daughter who just turned four, and then my youngest is coming up on his first birthday. He was born in December.Corey: Excellent. It sounds like our kids are basically the same age, in both directions. And from my perspective, at least looking back, what advice would I give someone for having a baby in a pandemic? It distills down to ‘don't,' just because it changes so much, it's no longer a trivial thing to have a grandparent come out and spend time with the kid. It's the constant… drumbeat of is this over? Is this not over?And that manifested a bunch of different ways. And I'm glad that I got the opportunity to take some time off to spend time with my family during that timeframe, but at the same time, it would've great if there were options such as not being stuck at home with every rambunctious—at the time—three-year-old as I went through that entire joy of having the kid.Rachel: Yeah. No, for the longest time, my thing was like, okay, like, there's no amount of money you could pay me to go back to middle school. I would never do it. And my new high bar there is no amount of money that you could pay me to go back to April 2020. That was the hardest month of my entire life was getting through that, like, first trimester, both parents at home, toddler at home, nowhere to go, no one to help. That was a [BLEEP] hard month. [laugh] that was bad.Corey: Oh, my God, yes, and we don't talk about this because we're basically communicating with people on social media, and everyone feels bad looking at social media because they're comparing their blooper reel with everyone else's highlights. And it feels odd on some level to complain about things like that. And let's be very clear as a man, I wind up in society getting lauded for even deigning to mention that I have children, whereas when mothers wind up talking about anything even slightly negative it's, “Oh, you sound like a bad mom.” And it is just one of the most abhorrent things out there in the world, I suppose. It's a strange inverted thing but one of the things that surprised me the most when I was expecting my first kid was looking at the different parenting forums, and the difference in tone was palpable where on the dad forums, everyone is super supportive and you got this dude it's great. You're fine. You're doing your best.Sure, these the occasional, “I gave my toddler beer and now people yell at me,” and it's, “What is wrong with you asshole?” But everyone else is mostly sane and doing their best. Whereas a lot of the ‘mommy' forums seem to bias more toward being relatively dismissive other people's parenting choices. And I understand I'm stereotyping wildly, not all forums, not all people, et cetera, et cetera, but it really was an interesting window into an area that as a stereotypical white man world, I don't see a lot of places I hang out with that are traditionally male that are overwhelmingly supportive in quite the same way. It was really an eye-opening experience for me.Rachel: I think you hit on some really important trends. One of the things that I have struggled with is—so I came into RedMonk—I've had a Twitter account forever, and it was always just, like, my personal Twitter account until I started working at RedMonk five years ago. And then all of a sudden I'm tweeting in technical and work capacities as well. And finding that balance initially was always a challenge.But then finding that balance again after having kids was very different because I would always—it was, kind of, mix of my life and also what I'm seeing in the industry and what I'm working on and this mix of things. And once you started tweeting about kids, it very much changes the potential perception that people have of who you are, what you're doing. I know this is just a mommy blogger kind of thing.You have to be really cognizant of that balance and making sure that you continue to put yourself in a place where you can still be your authentic self, but you really as a mom in the workspace and especially in tech have to be cognizant of not leaning too far into that. Because it can really damage your credibility with some audiences which is a super unfortunate thing, but also something I've learned just, like, I have to be really careful about how much mom stuff I share on Twitter.Corey: It's bizarre to me that we have to shade aspects of ourselves like this. And I don't know what the answer is. It's a weird thing that I never thought about before until suddenly I find that, oh, I'm a parent. I guess I should actually pay attention to this thing now. And it's one of those once you see it you can't unsee it things and it becomes strange and interesting and also more than a little sad in some respects.Rachel: I think there are some signs that we are getting to a better place, but it's a hard road for parents I think, and moms, in particular, working moms, all kinds of challenges out there. But anyways, it's one of those ones that is nice because love having my kids as a break, but sometimes Mondays come and it's such a relief to come back to work after a weekend with kids. Kids are a lot of work. And so it has brought elements of joy to my personal life, but it has also brought renewed elements of joy to my work life as really being able to lean into that side of myself. So, it's been a good year.Corey: Now that I have a second kid, I'm keenly aware of why parents are always very reluctant to wind up—the good parents at least to say, “Oh yeah, I have X number of kids, but that one's my favorite.” And I understand now why my mom always said with my brother and I that, “I can't stand either one of you.” And I get that now. Looking at the children of cloud services it's like, which one is my favorite?Well, I can't stand any of them, but the one that I hate the most is the Managed NAT Gateway because of its horrible pricing. In fact, anything involving bandwidth pricing in this industry tends to be horrifying, annoying, ill-behaved, and very hard to discipline. Which is why I think it's probably time we talk a little bit about egress charges in cloud providers.You had a great analysis of Cloudflare's R2, which is named after a robot in Star Wars and is apparently also the name of their S3 competitor, once it launches. Again, this is a pre-announcement, yeah, I could write blog posts that claim anything; the proof is really going to be in the pudding. Tell me more about, I guess, what you noticed from that announcement and what drove you to, “Ah, I have thoughts on this?”Rachel: So, I think it's an interesting announcement for several reasons. I think one of them is that it makes their existing offering really compelling when you start to add in that object store to something like the CDN, or to their edge functions which is called their Workers platform. And so, if you start to combine some of those functionalities together with a better object storage story, it can make their existing offering a lot more compelling which I think is an interesting aspect of this.I think one of the aspects that is probably gotten a lot more of the traction though is their lack of egress pricing. So, I think that's really what took everyone's imaginations by storm is what does the world look like when we are not charging egress pricing on object storage?Corey: What I find interesting is that when this came out first, a lot of AWS fans got very defensive over it, which I found very odd because their egress charges are indefensible from my point of view. And their response was, “Well, if you look at how a lot of the data access patterns work this isn't as big of a deal as it looks like,” and you're right. If I have a whole bunch of objects living in an object store, and a whole bunch of people each grab one of those objects this won't help me in any meaningful sense.But if I have one object that a bunch of people grab, well, suddenly we're having a different conversation. And on some level, it turns into an interesting question of what differentiates this with their existing CDN-style approach. From my perspective, this is where the object actually lives rather than just a cache that is going to expire. And that is transformative in a bunch of different ways, but my, I guess, admittedly overstated analysis for some use cases was okay, I store a petabyte in AWS and use it with and without this thing. Great, the answer came out to something like 51 or $52,000 in egress charges versus zero on Cloudflare. That's an interesting perspective to take. And the orders of magnitude in difference are eye-popping assuming that it works as advertised, which is always the caveat.Rachel: Yeah. I remember there was a RedMonk conversation with one of the cloud vendors set us up with a client conversation that want to, kind of, showcase their products kind of thing. And it was a movie studio and they walked us through what they architecturally have to do when they drop a trailer. If you think about that thing from this use case where all of a sudden you have videos that are all going out globally at the same time, and everybody wants to watch it and you're serving it over and over, that's a super interesting and compelling use case and very different from a cost perspective.Corey: You'll notice the video streaming services all do business with something that is not AWS for what they stream to end-users from. Netflix has its own Open Connect project that effectively acts as their own homebuilt CDN that they partner with providers to put in their various environments. There are a bunch of providers that focus specifically on this. But if you do the math for the Netflix story at retail pricing—let's be clear at large scale, no one pays retail pricing for anything, but okay—even assuming that you're within hailing distance of the same universe as retail pricing; you don't have to watch too many hours of Netflix before the data egress charges cost more than you're paying a month than subscription. And I have it on good authority—read as from their annual reports—that a much larger expense for Netflix than their cloud and technology and R&D expenses is their content expenses.They're making a lot of original content. They're licensing an awful lot of content, and that's way more expensive than providing it to folks. They have to have a better economic model. They need to be able to make a profit of some kind on streaming things to people. And with the way that all the major cloud providers wind up pricing this stuff, it's not tenable. There has to be a better answer.Rachel: So, Netflix calls to mind an interesting antidote that has gone around the industry which is who can become each other faster? Can HBO become Netflix faster, or can Netflix become HBO faster? So, can you build out that technology infrastructure side, or can you build out that content side? And I think what you're talking to with their content costs speaks to that story in terms of where people are investing and trying to actually make dents in their strategic outlays.I think a similar concept is actually at play when we talk about cloud and CDN. We do have this interesting piece from my coworker, Steve O'Grady, and he called it “Convergent Evolution of CDNs and Clouds.” And they originally evolved along separate paths where CDNs were designed to do this edge-caching scenario, and they had the core compute and all of the things that go around it happening in the cloud.And I think we've seen in recent years both of them starting to grow towards each other where CDNs are starting to look a lot more cloud-like, and we're seeing clouds trying to look more CDN-like. And I think this announcement in particular is very interesting when you think about what's happening in the CDN space and what it actually means for where CDNs are headed.Corey: It's an interesting model in that if we take a look at all of the existing cloud providers they had some other business that funded the incredible expense outlay that it took to build them. For example, Amazon was a company that started off selling books and soon expanded to selling everything else, and then expanded to putting ads in all of their search portals, including in AWS and eroding customer trust.Google wound up basically making all of their money by showing people ads and also killing Google Reader. And of course, Microsoft has been a software company for a lot longer than they've been a cloud provider, and given their security lapses in Azure recently is the question of whether they'll continue to be taken seriously as a cloud provider.But what makes Cloudflare interesting from this approach is they start it from the outside in of building out the edge before building regions or anything like that. And for a lot of use cases that works super well, in theory. In practice, well, we've never seen it before. I'm curious to see how it goes. Obviously, they're telling great stories about how they envision this working out in the future. I don't know how accurate it's going to be—show, don't tell—but I can at least acknowledge that the possibility is definitely there.Rachel: I think there's a lot of unanswered questions at this point, like, will you be able to have zero egress fees, and edge-like latencies, and global distribution, and have that all make sense and actually perform the way that the customer expects? I think that's still to be seen. I think one of the things that we have watched with interest is this rise of—I think for lack of a better word techno-nationalism where we are starting to see enclaves of where people want technology to be residing, where they want things to be sourced from, all of these interesting things.And so having this global network of storage flies in the face of some of those trends where people are building more and more enclaves of we're going to go big and global. I think that's interesting and I think data residency in this global world will be an interesting question.Corey: It also gets into the idea of what is the data that's going to live there. Because the idea of data residency, yes, that is important, but where that generally tends to matter the most is things like databases or customer information. Not the thing that we're putting out on the internet for anyone who wants to, to be able to download, which has historically been where CDNs are aiming things.Yes, of course, they can restrict it to people with logins and the rest, but that type of object storage in my experience is not usually subject to heavy regulation around data residency. We'll see because I get the sense that this is the direction Cloudflare is attempting to go in, and it's really interesting to see how it works. I'm curious to know what their stories are around, okay, you have a global network. That's great. Can I stipulate which areas my data can live within or not?At some point, it's going to need to happen if they want to look at regulated entities, but not everyone has to start with that either. So, it really just depends on what their game plan is on this. I like the fact that they're willing to do this. I like the fact they're willing to be as transparent as they are about their contempt for AWS's egress fees. And yes, of course, they're a competitor.They're going to wind up smacking competition like this, but I find it refreshing because there is no defense for what they're saying, their math is right. Their approach to what customers experience from AWS in terms of egress fees is correct. And all of the defensiveness at, well, you know, no one pays retail price for this, yeah, but they see it on the website when they're doing back-of-the-envelope math, and they're not going to engage with you under the expectation that you're going to give them a 98% discount.So, figure out what the story is. And it's like beating my head against the wall. I also want to be fair. These networks are very hard to build, and there's a tremendous amount of investment. The AWS network is clearly magic in some respects just because having worked in data centers myself, the things that I see that I'm able to do between various EC2 instances at full network line rate would not have been possible in the data centers that I worked within.So, there's something going on that is magic and that's great. And I understand that it's expensive, but they've done a terrible job of messaging that. It just feels like, oh, bandwidth in is free because, you know, that's how it works. Sending it out, ooh, that's going to cost you X and their entire positioning and philosophy around it just feels unnecessary.Rachel: That's super interesting. And I think that also speaks to one of the questions that is still an open concern for what happens to Cloudflare if this is wildly successful. Which, based off of people's excitement levels at this point, it's seems like it's very potentially going to be successful. And what does this mean for the level of investment that they're going to have to make in their own infrastructure and network and order to actually be able to serve all of this?Corey: The thing that I find curious is that in a couple of comment threads on Hacker News and on Twitter, Cloudflare's CEO, Matthew Prince—who's always been extremely accessible as far as executives of giant cloud companies go—has said that at their scale and by which they he's referring to Cloudflare, and he says, “I assume that Amazon can probably get at least as decent economics on bandwidth pricing as we can,” which is a gross understatement because Amazon will spend years fighting over 50 cents.Great, but what's interesting is that he refers to bandwidth at that scale as being much closer to a fixed cost than something that's a marginal cost for everything that a customer uses. The way that companies buy and sell bandwidth back and forth is complex, but he's right. It is effectively a fixed monthly fee for a link and you can use as much or as little of that link as you want. 95th percentile billing aficionados, please don't email me.But by and large, that's the way to think about it. You pay for the size of the pipe, not how much water flows through it. And as long as you can keep the links going without saturating them to the point where more data can't fit through at a reasonable amount of time, your cost don't change. So, yeah, if there's a bunch of excitement they'll have to expand the links, but that's generally a fixed cost as opposed to a marginal cost per gigabyte.That's not how they think about it. There's a whole translation layer that's an economic model. And according to their public filings, they have something like a 77% gross margin which tells me that, okay, they are not in fact losing money on bandwidth even now where they generally don't charge on a metered basis until you're on the Cloudflare Enterprise Plan.Rachel: Yeah. I think it's going to just be really interesting to watch. I'm definitely interested to see what happens as they open this up, and like, 11 9s of availability feels like a lot of availabilities. It's just the engineering of this, the economics of this it feels like there's a lot of open questions that I'm excited to watch.Corey: You're onto my favorite part of this. So, the idea of 11 9s because it sounds ludicrous. That is well within the boundaries of probability of things such as, yeah, it is likely that gravity is going to stop working than it is that's going to lose data. How can you guarantee that? Generally speaking, although S3 has always been extremely tight-lipped about how it works under the hood, other systems have not been.And it looks an awful lot like the idea of Reed-Solomon erasure coding, where for those of us who spent time downloading large files of questionable legality due to copyright law and whatnot off of Usenet, they had the idea of parity files where they'd take these giant media files up—they're Linux ISOs; of course they are—and you'd slice them into a bunch of pieces and then generate parity files as well.So, you would wind up downloading the let's say 80 RAR files and, oh, three of them were corrupt, each parity file could wind up swapping in so as long as you had enough that added up to 80, any of those could wind up restoring the data that had been corrupted. That is almost certainly what is happening at the large object storage scale. Which is great, we're going to break this thing into a whole bunch of chunks. Let's say here is a file you've uploaded or an object.We're going to break this into a hundred chunks—let's say arbitrarily—and any 80 of those chunks can be used to reconstitute the entire file. And then you start looking at where you place them and okay, what are the odds of simultaneous drive failure in these however many locations? And that's how you get that astronomical number. It doesn't mean what people think of does. The S3 offers 11 9s of durability on their storage classes, including the One Zone storage class.Which is a single availability zone instead of something that's an entire region, which means that they're not calculating disaster recovery failure scenarios into that durability number. Which is fascinating because it's far, like, you're going to have all the buildings within the same office park burn down than it is all of the buildings within a hundred square miles burn down, but those numbers remain the same.There's a lot of assumptions baked into that and it makes for an impressive talking point. I just hear it as, oh yeah, you're a real object-store. That's how I see it. There's a lot that's yet to be explained or understood. And I think that I'm going to be going up one side and down the other as soon as this exists in the real world and I'm looking forward to seeing it. I'm just a little skeptical because it has been preannounced.The important part for me is even the idea that they can announce something like this and not be sued for securities fraud tells me that it is at least theoretically economically possible that they could be telling the truth on this. And that alone speaks volumes to just how out-of-bounds it tends to be in the context of giant cloud customers.Rachel: I mean, if you read Matt Levine, “Everything is Securities Fraud?“ so, I don't know how much we want to get excited about that.Corey: Absolutely. A huge fan of his work. Corey: You know its important to me that people learn how to use the cloud effectively. Thats why I'm so glad that Cloud Academy is sponsoring my ridiculous non-sense. They're a great way to build in demand tech skills the way that, well personally, I learn best which I learn by doing not by reading. They have live cloud labs that you can run in real environments that aren't going to blow up your own bill—I can't stress how important that is. Visit cloudacademy.com/corey. Thats C-O-R-E-Y, don't drop the “E.” Use Corey as a promo-code as well. You're going to get a bunch of discounts on it with a lifetime deal—the price will not go up. It is limited time, they assured me this is not one of those things that is going to wind up being a rug pull scenario, oh no no. Talk to them, tell me what you think. Visit: cloudacademy.com/corey,  C-O-R-E-Y and tell them that I sent you!Corey: So, we've talked a fair bit about what data egress looks like. What else have you been focusing on? What have you found that is fun, and exciting, and catches your eye in this incredibly broad industry lately?Rachel: Oh, there's all kinds of exciting things. One of the pieces of research that's been on my back-burner, usually I do it early summer, and it is—due to a variety of factors—still in my pipeline, but I always do a piece of research about base infrastructure pricing. And it's an annual piece of looking about what are all of the cloud providers doing in regards to their pricing on that core aspect of compute, and storage, and memory.And what does that look like over time, and what does that look like across providers? And it is absolutely impossible to get an apples-to-apples comparison over time and across providers. It just can't actually be done. But we do our best [laugh] and then caveat the hell out of it from there. But that's the piece of research that's most on my backlog right now and one that I'm working on.Corey: I think that there's a lot of question around the idea of what is the cost of a compute unit—or something like that—between providers? The idea of if I have this configuration will cost me more on cloud provider a or cloud provider B, my pet working theory is that whenever people ask for analyses like that—or a number of others, to be perfectly frank with you—what they're really looking for is confirmation bias to go in the direction that they wanted to go in already. I have yet to see a single scenario where people are trying to decide between cloud providers and they say, “That one because it's going to be 10% less.” I haven't seen it. That said I am, of course, at a very particular area of the industry. Have you seen it?Rachel: I have not seen it. I think users find it interesting because it's always interesting to look at trends over time. And in particular, with this analysis, it's interesting to watch the number of providers narrow and then widen back out because we've been doing this since 2012. So, we used to have [unintelligible 00:26:24] and HPE used to be in there. So, like, we used to—CenturyLink. We used to have this broader list of cloud providers that we considered that would narrow down to this doesn't really count anymore.And now why do you need to back out? It's like, okay, Oracle Cloud you're in, Alibaba, Tencent, like, let's look at you. And so, like, it's interesting to just watch the providers in the mix shift over time which I think is interesting. And I think one of just the broad trends that is interesting is early years of this, there was steep competition on price, and that leveled off for solid three, four years.We've seen some degree of competitiveness reemerge with competitors like Oracle in particular. So, those broad-brush trends are interesting. The specifics of the pricing if you're doing 10% difference kind of things I think you're missing the point of the analysis largely, but it's interesting to look at what's happening in the industry overall.Corey: If you were to ask me to set up a simple web app, if there is such a thing, and tell you in advance what it was going to cost to host, and I can get it accurate within 20%, I am on fire in terms of both analysis and often dumb luck just because it is so difficult to answer the question. Getting back to our earlier conversational topic, let's say I put CloudFront, Amazon's sorry excuse for a CDN, in front of it which is probably the closest competitor they have to Cloudflare as a CDN, what'll it cost me per gigabyte? Well, that's a fascinating question. The answer comes down to where are you visiting it from? Depending where on the planet, people who are viewing my website, or using my web app are sitting, the cost per gigabyte will vary between eight-and-a-half cents—retail pricing—and fourteen cents. That's a fairly wide margin and there's no way to predict that in advance for most use cases. It's the big open-ended question.And people build out their environments and they want to know they're making a rational decision and that their provider is not charging three times more than their competitor is for the exact same thing, but as long as it's within a certain level of confidence interval, that makes sense.Rachel: Yeah, and I think the other thing that's interesting about this analysis and one of the reasons that it's a frustrating analysis for me, in particular, is that I feel like that base compute is actually not where most of the cloud providers are actually competing anymore. So, like, it was definitely the interesting story early in cloud.I think very clearly not the focus area for most of us now. It has moved up an abstraction layer. It's moved to manage services. It has moved to other areas of their product portfolio. So, it's still useful. It's good to know. But I think that the broader portfolio of the cloud providers is definitely more the story than this individual price point.Corey: That is an interesting story because I believe it, and it speaks to the aspirational version of where a lot of companies see themselves going. And then in practice, I see companies talking like this constantly, and then I look in their environment and say, “Okay, you're basically spending 70% of your entire cloud bill on EC2 instances, running—it's a bunch of VMs that sit there.”And as much as they love to talk about the future and how other things are being considered and how their—use of machine learning in the rest, and Kubernetes, of course, a lot of this stuff all distills down to, yeah, it runs in software. It sits on top of EC2 instances and that's what you get billed for. At re:Invent it's always interesting and sad at the same time that they don't give EC2 nearly enough attention or stage time because it's not interesting, despite it being a majority of AWS bill.Rachel: I think that's a fantastic point, well made.Corey: I'll take it even one step further—and this is one where I think is almost a messaging failure on some level—Google Cloud offers sustained use discounts which apparently they don't know how to talk about appropriately, but it's genius. The way this works is if you run a VM for more than in a certain number of hours in a month, the entire month is now charged for that VM at a less than retail rate because you've been using it in a sustained way.All you have to do to capture that is don't turn it off. You know, what everyone's doing already. And sure if you commit to usage on it you get a deeper discount, but what I like about this is if you buy some reserved instances is or you buy some committed use discounting, great, you'll save more money, but okay, here's a $20 million buy. You should click the button on, people are terrified to click at that button because I don't usually get to approve dollar figure spend with multiple commas in them. That's kind of scary. So, people hem and they haw and they wait six months. This is maybe not as superior mathematically, but it's definitely an easier sell psychologically, and they just don't talk about it.It's what people say they care about when people actually do are worlds apart. And the thing that continually astounds me because I didn't expect it, but it's obvious in hindsight that when it comes to cloud economics it's more about psychology than it is about math.Rachel: I think one of the things that, having come from the finance world into the analyst world, and so I definitely have a particular point of view, but one of the things that was hardest for me when I worked in finance was not the absolute dollar amount of anything but the variability of it. So, if I knew what to expect I could work with that and we could make it work. It was when things varied in unexpected ways that it was a lot more challenging.And so I think one of the things that when people talked about, like, this shift to cloud and the move to cloud, and everyone is like, “Oh, we're moving things from the balance sheet to the income statement.” And everyone talked about that like it's a big deal. For some parts of the organization that is a big deal, but for a lot of the organization, the shift that matters is the shift from a fixed cost to a variable cost because that lack of predictability makes a lot of people's jobs, a lot more difficult.Corey: The thing that I always find fun is a thought exercise is okay, let's take a look at any given cloud company's cloud bill for the last 18 to 36 months and add all of that up. Great, take that big giant number and add 20% to it. If you could magically go back in time and offer that larger number to them as here's your cloud bill and all of your usage for the next 18 to 36 months. Here you go. Buy this instead.And the cloud providers laugh at me and they say, “Who in the world would agree to that deal?” And my answer is, “Almost everyone.” Because at the company's scale it's not like the individual developer response of, “Oh, my God, I just spent how much money? I've got to eat this month.” Companies are used to absorbing those things. It's fine. It's just a, “We didn't predict this. We didn't plan for this. What does this do to our projections, our budget, et cetera?”If you can offer them certainty and find some way to do it, they will jump at that. Most of my projects are not about make the bill lower, even though that is what is believed, in some cases by people working on these projects internally at these companies. It's about making it understandable. It's about making it predictable, it's about understanding when you see a big spike one month. What project drove that?Spoiler, it's almost always the data science team because that's what they do, but that's neither here nor there. Please don't send me letters. But yeah, it's about understanding what is going on, and that understanding and being able to predict it is super hard when you're looking at usage-based pricing.Rachel: Exactly.Corey: I want to thank you for taking so much time to speak with me. If people want to hear more about your thoughts, your observations, et cetera, where can they find you?Rachel: Probably the easiest way to get in touch with me is on Twitter, which is @rstephensme that's R-S-T-E-P-H-E-N-S-M-E.Corey: And we will, of course, put links to that in the [show notes 00:34:08]. Thank you so much for your time. I appreciate it.Rachel: Thanks for having me. This was great.Corey: Rachel Stephens, senior analyst at RedMonk. I'm Cloud Economist, Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment, angrily defending your least favorite child, which is some horrifying cloud service you have launched during the pandemic.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Building a Partnership with Your Cloud Provider with Micheal Benedict

Screaming in the Cloud

Play Episode Listen Later Nov 10, 2021 54:44


About Micheal Micheal Benedict leads Engineering Productivity at Pinterest. He and his team focus on developer experience, building tools and platforms for over a thousand engineers to effectively code, build, deploy and operate workloads on the cloud. Mr. Benedict has also built Infrastructure and Cloud Governance programs at Pinterest and previously, at Twitter -- focussed on managing cloud vendor relationships, infrastructure budget management, cloud migration, capacity forecasting and planning and cloud cost attribution (chargeback). Links: Pinterest: https://www.pinterest.com Teletraan: https://github.com/pinterest/teletraan Twitter: https://twitter.com/micheal Pinterestcareers.com: https://pinterestcareers.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You know how git works right?Announcer: Sorta, kinda, not really. Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while, I like to talk to people who work at very large companies that are not in fact themselves a cloud provider. I know it sounds ridiculous. How can you possibly be a big company and not make money by selling managed NAT gateways to an unsuspecting public? But I'm told it can be done here to answer that question. And hopefully at least one other is Pinterest. It's head of engineering productivity, Micheal Benedict. Micheal, thank you for taking the time to join me today.Micheal: Hi, Corey, thank you for inviting me today. I'm really excited to talk to you.Corey: So, exciting times at Pinterest in a bunch of different ways. It was recently reported—which of course, went right to the top of my inbox as 500,000 people on Twitter all said, “Hey, this sounds like a ‘Corey would be interested in it' thing.” It was announced that you folks had signed a $3.2 billion commitment with AWS stretching until 2028. Now, if this is like any other large-scale AWS contract commitment deal that has been made public, you were probably immediately inundated with a whole bunch of people who are very good at arithmetic and not very good at business context saying, “$3.2 billion? You could build massive data centers for that. Why would anyone do this?” And it's tiresome, and that's the world in which we live. But I'm guessing you heard at least a little bit of that from the peanut gallery.Micheal: I did, and I always find it interesting when direct comparisons are made with the total amount that's been committed. And like you said, there's so many nuances that go into how to perceive that amount, and put it in context of, obviously, what Pinterest does. So, I at least want to take this opportunity to share with everyone that Pinterest has been on the cloud since day one. When Ben initially started the company, that product was launched—it was a simple Django app—it was launched on AWS from day one, and since then, it has grown to support 450-plus million MAUs over the course of the decade.And our infrastructure has grown pretty complex. We started with a bunch of EC2 machines and persisting data in S3, and since then we have explored an array of different products, in fact, sometimes working very closely with AWS, as well and helping them put together a product roadmap for some of the items they're working on as well. So, we have an amazing partnership with them, and part of the commitment and how we want to see these numbers is how does it unlock value for Pinterest as a business over time in terms of making us much more agile, without thinking about the nuances of the infrastructure itself. And that's, I think, one of the best ways to really put this into context, that it's not a single number we pay at the end [laugh] of the month, but rather, we are on track to spending a certain amount over a period of time, so this just keeps accruing or adding to that number. And we basically come out with an amazing partnership in AWS, where we have that commitment and we're able to leverage their products and full suite of items without any hiccups.Corey: The most interesting part of what you said is the word partner. And I think that's the piece that gets lost an awful lot when we talk about large-scale cloud negotiations. It's not like buying a car, where you can basically beat the crap out of the salesperson, you can act as if $400 price difference on a car is the difference between storm out of the dealership and sign the contract. Great, you don't really have to deal with that person ever again.In the context of a cloud provider, they run your production infrastructure, and if they have a bad day, I promise you're going to have a bad day, too. You want to handle those negotiations in a way that is respectful of that because they are your partner, whether you want them to be or not. Now, I'm not suggesting that any cloud provider is going to hold an awkward negotiation against the customer, but at the same time, there are going to be scenarios in which you're going to want to have strong relationships, where you're going to need to cash in political capital to some extent, and personally, I've never seen stupendous value in trying to beat the crap out of a company in order to get another tenth of a percent discount on a service you barely use, just because someone decided that well, we didn't do well in the last negotiation so we're going to get them back this time.That's great. What are you actually planning to do as a company? Where are you going? And the fact that you just alluded to, that you're not just a pile of S3 and EC2 instances speaks, in many ways, to that. By moving into the differentiated service world, suddenly you're able to do things that don't look quite as much like building a better database and start looking a lot more like servicing your users more effectively and well.Micheal: And I think, like you said, I feel like there's like a general skepticism in viewing that the cloud providers are usually out there to rip you apart. But in reality, that's not true. To your point, as part of the partnership, especially with AWS and Pinterest, we've got an amazing relationship going on, and behind the scenes, there's a dedicated team at Pinterest, called the Infrastructure Governance Team, a cross-functional team with folks from finance, legal, engineering, product, all sitting together and working with our AWS partners—even the AWS account managers at the times are part of that—to help us make both Pinterest successful, and in turn, AWS gets that amazing customer to work with in helping build some of their newer products as well. And that's one of the most important things we have learned over time is that there's two parts to it; when you want to help improve your business agility, you want to focus not just on the bottom line numbers as they are. It's okay to pay a premium because it offsets the people capital you would have to invest in getting there.And that's a very tricky way to look at math, but that's what these teams do; they sit down and work through those specifics. And for what it's worth, in our conversations, the AWS teams always come back with giving us very insightful data on how we're using their systems to help us better think about how we should be pricing or looking things ahead. And I'm not the expert on this; like I said, there's a dedicated team sitting behind this and looking through and working through these deals, but that's one of the important takeaways I hope the users—or the listeners of this podcast then take away that you want to treat your cloud provider as your partner as much as possible. They're not always there to screw you. That's not their goal. And I apologize for using that term. It is important that you set that expectations that it's in their best interest to actually make you successful because that's how they make money as well.Corey: It's a long-term play. I mean, they could gouge you this quarter, and then you're trying to evacuate as fast as possible. Well, they had a great quarter, but what's their long-term prospect? There are two competing philosophies in the world of business; you can either make a lot of money quickly, or you can make a little bit of money and build it over time in a sustained way. And it's clear the cloud providers are playing the long game on this because they basically have to.Micheal: I mean, it's inevitable at this point. I mean, look at Pinterest. It is one of those success stories. Starting as a Django app on a bunch of EC2 machines to wherever we are right now with having a three-plus billion dollar commitment over a span of couple of years, and we do spend a pretty significant chunk of that on a yearly basis. So, in this case, I'm sure it was a great successful partnership.And I'm hoping some of the newer companies who are building the cloud from the get-go are thinking about it from that perspective. And one of the things I do want to call out, Corey, is that we did initially start with using the primitive services in AWS, but it became clear over time—and I'm sure you heard of the term multi-cloud and many of that—you know, when companies start evaluating how to make the most out of the deals they're negotiating or signing, it is important to acknowledge that the cost of any of those evaluations or even thinking about migrations never tends to get factored in. And we always tend to treat that as being extremely simple or not, but those are engineering resources you want to be spending more building on the product rather than these crazy costly migrations. So, it's in your best interest probably to start using the most from your cloud provider, and also look for opportunities to use other cloud providers—if they provide more value in certain product offerings—rather than thinking about a complete lift-and-shift, and I'm going to make DR as being the primary case on why I want to be moving to multi-cloud.Corey: Yeah. There's a question, too, of the numbers on paper look radically different than the reality of this. You mentioned, Pinterest has been on AWS since the beginning, which means that even if an edict had been passed at the beginning, that, “Thou shalt never build on anything except EC2 and S3. The end. Full stop.”And let's say you went down that rabbit hole of, “Oh, we don't trust their load balancers. We're going to build our own at home. We have load balancers at home. We'll use those.” It's terrible, but even had you done that and restricted yourselves just to those baseline building blocks, and then decide to do a cloud migration, you're still looking back at over a decade of experience where the app has been built unconsciously reflecting the various failure modes that AWS has, the way that it responds to API calls, the latency in how long it takes to request something versus it being available, et cetera, et cetera.So, even moving that baseline thing to another cloud provider is not a trivial undertaking by any stretch of the imagination. But that said—because the topic does always come up, and I don't shy away from it; I think it's something people should go into with an open mind—how has the multi-cloud conversation progressed at Pinterest? Because there's always a multi-cloud conversation.Micheal: We have always approached it with some form of… openness. It's not like we don't want to be open to the ideas, but you really want to be thinking hard on the business case and the business value something provides on why you want to be doing x. In this case, when we think about multi-cloud—and again, Pinterest did start with EC2 and S3, and we did keep it that way for a long time. We built a lot of primitives around it, used it—for example, my team actually runs our bread and butter deployment system on EC2. We help facilitate deployments across a 100,000-plus machines today.And like you said, we have built that system keeping in mind how AWS works, and understanding the nuances of region and AZ failovers and all of that, and help facilitate deployments across 1000-plus microservices in the company. So, thinking about leveraging, say, a Google Cloud instance and how that works, in theory, we can always make a case for engineering to build our deployment system and expand there, but there's really no value. And one of the biggest cases, usually, when multi-cloud comes in is usually either negotiation for price or actually a DR strategy. Like, what if AWS goes down in and us-east-1? Well, let's be honest, they're powering half the internet [laugh] from that one single—Corey: Right.Micheal: Yeah. So, if you think your business is okay running when AWS goes down and half the internet is not going to be working, how do you want to be thinking about that? So, DR is probably not the best reason for you to be even exploring multi-cloud. Rather, you should be thinking about what the cloud providers are offering as a very nuanced offering which your current cloud provider is not offering, and really think about just using those specific items.Corey: So, I agree that multi-cloud for DR purposes is generally not necessarily the best approach with the idea of being able to failover seamlessly, but I like the idea for backups. I mean, Pinterest is a publicly-traded company, which means that among other things, you have to file risk disclosures and be responsive to auditors in a variety of different ways. There are some regulations to start applying to you. And the idea of, well, AWS builds things out in a super effective way, region separation, et cetera, whenever I talk to Amazonians, they are always surprised that anyone wouldn't accept that, “Oh, if you want backups use a different region. Problem solved.”Right, but it is often easier for me to have a rehydrate the business level of backup that would take weeks to redeploy living on another cloud provider than it is for me to explain to all of those auditors and regulators and financial analysts, et cetera why I didn't go ahead and do that path. So, there's always some story for okay, what if AWS decides that they hate us and want to kick us off the platform? Well, that's why legal is involved in those high-level discussions around things like risk, and indemnity, and termination for convenience and for cause clauses, et cetera, et cetera. The idea of making an all-in commitment to a cloud provider goes well beyond things that engineering thinks about. And it's easy for those of us with engineering backgrounds to be incredibly dismissive of that of, “Oh, indemnity? Like, when does AWS ever lose data?” “Yeah, but let's say one day they do. What is your story going to be when asked some very uncomfortable questions by people who wanted you to pay attention to this during the negotiation process?” It's about dotting the i's and crossing the t's, especially with that many commas in the contractual commitments.Micheal: No, it is true. And we did evaluate that as an option, but one of the interesting things about compliance, and especially auditing as well, we generally work with the best in class consultants to help us work through the controls and how we audit, how we look at these controls, how to make sure there's enough accountability going through. The interesting part was in this case, as well, we were able to work with AWS in crafting a lot of those controls and setting up the right expectations as and when we were putting proposals together as well. Now, again, I'm not an expert on this and I know we have a dedicated team from our technical program management organization focused on this, but early on we realized that, to your point, the cost of any form of backups and then being able to audit what's going in, look at all those pipelines, how quickly we can get the data in and out it was proving pretty costly for us. So, we were able to work out some of that within the constructs of what we have with our cloud provider today, and still meet our compliance goals.Corey: That's, on some level, the higher point, too, where everything is everything comes down to context; everything comes down to what the business demands, what the business requires, what the business will accept. And I'm not suggesting that in any case, they're wrong. I'm known for beating the ‘Multi-cloud is a bad default decision' drum, and then people get surprised when they'll have one-on-one conversations, and they say, “Well, we're multi-cloud. Do you think we're foolish?” “No. You're probably doing the right thing, just because you have context that is specific to your business that I, speaking in a general sense, certainly don't have.”People don't generally wake up in the morning and decide they're going to do a terrible job or no job at all at work today, unless they're Facebook's VP of Integrity. So, it's not the sort of thing that lends itself to casual tweet size, pithy analysis very often. There's a strong dive into what is the level of risk a business can accept? And my general belief is that most companies are doing this stuff right. The universal constant in all of my consulting clients that I have spoken to about the in-depth management piece of things is, they've always asked the same question of, “So, this is what we've done, but can you introduce us to the people who are doing it really right, who have absolutely nailed this and gotten it all down?” “It's, yeah, absolutely no one believes that that is them, even the folks who are, from my perspective, pretty close to having achieved it.”But I want to talk a bit more about what you do beyond just the headline-grabbing large dollar figure commitment to a cloud provider story. What does engineering productivity mean at Pinterest? Where do you start? Where do you stop?Micheal: I want to just quickly touch upon that last point about multi-cloud, and like you said, every company works within the context of what they are given and the constraints of their business. It's probably a good time to give a plug to my previous employer, Twitter, who are doing multi-cloud in a reasonably effective way. They are on the data centers, they do have presence on Google Cloud, and AWS, and I know probably things have changed since a couple of years now, but they have embraced that environment pretty effectively to cater to their acquisitions who were on the public cloud, help obviously, with their initial set of investments in the data center, and still continue to scale that out, and explore, in this case, Google Cloud for a variety of other use cases, which sounds like it's been extremely beneficial as well.So, to your point, there is probably no right way to do this. There's always that context, and what you're working with comes into play as part of making these decisions. And it's important to take a lot of these with a grain of salt because you can never understand the decisions, why they were made the way they were made. And for what it's worth, it sort of works out in the end. [laugh]. I've rarely heard a story where it's never worked out, and people are just upset with the deals they've signed. So, hopefully, that helps close that whole conversation about multi-cloud.Corey: I hope so. It's one of those areas where everyone has an opinion and a lot of them do not necessarily apply universally, but it's always fun to take—in that case, great, I'll take the lesser trod path of everyone's saying multi-cloud is great, invariably because they're trying to sell you something. Yeah, I have nothing particularly to sell, folks. My argument has always been, in the absence of a compelling reason not to, pick a provider and go all in. I don't care which provider you pick—which people are sometimes surprised to hear.It's like, “Well, what if they pick a cloud provider that you don't do consulting work for?” Yeah, it turns out, I don't actually need to win every AWS customer over to have a successful working business. Do what makes sense for you, folks. From my perspective, I want this industry to be better. I don't want to sit here and just drum up business for myself and make self-serving comments to empower that. Which apparently is a rare tactic.Micheal: No, that's totally true, Corey. One of the things you do is help people with their bills, so this has come up so many times, and I realize we're sort of going off track a bit from that engineering productivity discussion—Corey: Oh, which is fine. That's this entire show's theme, if it has one.Micheal: [laugh]. So, I want to briefly just talk about the whole billing and how cost management works because I know you spend a lot of time on that and you help a lot of these companies be effective in how they manage their bills. These questions have come up multiple times, even at Pinterest. We actually in the past, when I was leading the infrastructure governance organization, we were working with other companies of our similar size to better understand how they are looking into getting visibility into their cost, setting sort of the right controls and expectations within the engineering organization to plan, and capacity plan, and effectively meet those plans in a certain criteria, and then obviously, if there is any risk to that, actively manage risk. That was like the biggest thing those teams used to do.And we used to talk a lot trade notes, and get a better sense of how a lot of these companies are trying to do—for example, Netflix, or Lyft, or Stripe. I recall Netflix, content was their biggest spender, so cloud spending was like way down in the list of things for them. [laugh]. But regardless, they had an active team looking at this on a day-to-day basis. So, one of the things we learned early on at Pinterest is that start investing in those visibility tools early on.No one can parse the cloud bills. Let's be honest. You're probably the only person who can reverse… [laugh] engineer an architecture diagram from a cloud bill, and I think that's like—definitely you should take a patent for that or something. But in reality, no one has the time to do that. You want to make sure your business leaders, from your finance teams to engineering teams to head of the executives all have a better understanding of how to parse it.So, investing engineering resources, take that data, how do you munch it down to the cost, the utilization across the different vectors of offerings, and have a very insightful discussion. Like, what are certain action items we want to be taking? It's very easy to see, “Oh, we overspent EC2,” and we want to go from there. But in reality, that's not just that thing; you will start finding out that EC2 is being used by your Hadoop infrastructure, which runs hundreds of thousands of jobs. Okay, now who's actually responsible for that cost? You might find that one job which is accruing, sort of, a lot of instance hours over a period of time and a shared multi-tenant environment, how do you attribute that cost to that particular cost center?Corey: And then someone left the company a while back, and that job just kept running in perpetuity. No one's checked the output for four years, I guess it can't be that necessarily important. And digging into it requires context. It turns out, there's no SaaS tool to do this, which is unfortunate for those of us who set out originally to build such a thing. But we discovered pretty early on the context on this stuff is incredibly important.I love the thing you're talking about here, where you're discussing with your peer companies about these things because the advice that I would give to companies with the level of spend that you folks do is worlds apart from what I would advise someone who's building something new and spending maybe 500 bucks a month on their cloud bill. Those folks do not need to hire a dedicated team of people to solve for these problems. At your scale, yeah, you probably should have had some people in [laugh] here looking at this for a while now. And at some point, the guidance changes based upon scale. And if there's one thing that we discover from the horrible pages of Hacker News, it's that people love applying bits of wisdom that they hear in wildly inappropriate situations.How do you think about these things at that scale? Because, a simple example: right now I spend about 1000 bucks a month at The Duckbill Group, on our AWS bill. I know. We have one, too. Imagine that. And if I wind up just committing admin credentials to GitHub, for example, and someone compromises that and start spinning things up to mine all the Bitcoin, yeah, I'm going to notice that by the impact it has on the bill, which will be noticeable from orbit.At the level of spend that you folks are at, at company would be hard-pressed to spin up enough Bitcoin miners to materially move the billing needle on a month-to-month basis, just because of the sheer scope and scale. At small bill volumes, yeah, it's pretty easy to discover the thing that spiking your bill to three times normal. It's usually a managed NAT gateway. At your scale, tripling the bill begins to look suspiciously like the GDP of a small country, so what actually happened here? Invariably, at that scale, with that level of massive multiplier, it's usually the simplest solution, an error somewhere in the AWS billing system. Yes, they exist. Imagine that.Micheal: They do exist, and we've encountered that.Corey: Kind of heartstopping, isn't it?Micheal: [laugh]. I don't know if you remember when we had the big Spectre and the Meltdown, right, and those were interesting scenarios for us because we had identified a lot of those issues early on, given the scale we operate, and we were able to, sort of, obviously it did have an impact on the builds and everything, but that's it; that's why you have these dedicated teams to fix that. But I think one of the points you made, these are large bills and you're never going to have a 3x jump the next day. We're not going to be seeing that. And if that happens, you know, God save us. [laugh].But to your point, one of the things we do still want to be doing is look at trends, literally on a week-over-week basis because even a one percentage move is a pretty significant amount, if you think about it, which could be funding some other aspects of the business, which we would prefer to be investing on. So, we do want to have enough rigor and controls in place in our technical stack to identify and alert when something is off track. And it becomes challenging when you start using those higher-order services from your public cloud provider because there's no clear insights on how do you, kind of, parse that information. One of the biggest challenges we had at Pinterest was tying ownership to all these things.No, using tags is not going to cut it. It was so difficult for us to get to a point where we could put some sense of ownership in all the things and the resources people are using, and then subsequently have the right conversation with our ads infrastructure teams, or our product teams to help drive the cost improvements we want to be seeing. And I wouldn't be surprised if that's not a challenge already, even for the smaller companies who have bills in the tunes of tens and thousands, right?Corey: It is. It's predicting the spend and trying to categorize it appropriately; that's the root of all AWS bill panic on the corporate level. It's not that the bill is 20% higher, so we're going to go broke. Most companies spend far more on payroll than they do on infrastructure—as you mentioned with Netflix, content is a significantly larger [laugh] expense than any of those things; real estate, it's usually right up there too—but instead it's, when you're trying to do business forecasting of, okay, if we're going to have an additional 1000 monthly active users, what will the cost for us be to service those users and, okay, if we're seeing a sudden 20% variance, if that's the new normal, then well, that does change our cost projections for a number of years, what happens? When you're public, there starts to become the question of okay, do we have to restate earnings or what's the deal here?And of course, all this sidesteps past the unfortunate reality that, for many companies, the AWS bill is not a function of how many customers you have; it's how many engineers you hired. And that is always the way it winds up playing out for some reason. “It's why did we see a 10% increase in the bill? Yeah, we hired another data science team. Oops.” It's always seems to be the data science folks; I know I'd beat up on those folks a fair bit, and my apologies. And one day, if they analyze enough of the data, they might figure out why.Micheal: So, this is where I want to give a shout out to our data science team, especially some of the engineers working in the Infrastructure Governance Team putting these charts together, helping us derive insights. So, definitely props to them.I think there's a great segue into the point you made. As you add more engineers, what is the impact on the bottom line? And this is one of the things actually as part of engineering productivity, we think about as well on a long-term basis. Pinterest does have over 1000-plus engineers today, and to large degree, many of them actually have their own EC2 instances today. And I wouldn't say it's a significant amount of cost, but it is a large enough number, were shutting down a c5.9xl can actually fund a bunch of conference tickets or something else.And then you can imagine that sort of the scale you start working with at one point. The nuance here is though, you want to make sure there's enough flexibility for these engineers to do their local development in a sustainable way, but when moving to, say production, we really want to tighten the flexibility a bit so they don't end up doing what you just said, spin up a bunch of machines talking to the API directly which no one will be aware of.I want to share a small anecdote because when back in the day, this was probably four years ago, when we were doing some analysis on our bills, we realized that there was a huge jump every—I believe Wednesday—in our EC2 instances by almost a factor of, like, 500 to 600 instances. And we're like, “Why is this happening? What is going on?” And we found out there was an obscure job written by someone who had left the company, calling an EC2 API to spin up a search cluster of 500 machines on-demand, as part of pulling that ETL data together, and then shutting that cluster down. Which at times didn't work as expected because, you know, obviously, your Hadoop jobs are very predictable, right?So, those are the things we were dealing with back in the day, and you want to make sure—since then—this is where engineering productivity as team starts coming in that our job is to enable every engineer to be doing their best work across code building and deploying the services. And we have done this.Corey: Right. You and I can sit here and have an in-depth conversation about the intricacies of AWS billing in a bunch of different ways because in different ways we both specialize in it, in many respects. But let's say that Pinterest theoretically was foolish enough to hire me before I got into this space as an engineer, for terrifying reasons. And great. I start day one as a typical software developer if such a thing could be said to exist. How do you effectively build guardrails in so that I don't inadvertently wind up spinning up all the EC2 instances available to me within an account, which it turns out are more than one might expect sometimes, but still leave me free to do my job without effectively spending a nine-month safari figuring out how AWS bills work?Micheal: And this is why teams like ours exist, to help provide those tools to help you get started. So today, we actually don't let anyone directly use AWS APIs, or even use the UI for that matter. And I think you'll soon realize, the moment you hit, like, probably 30 or 40 people in your organization, you definitely want to lock it down. You don't want that access to be given to anyone or everyone. And then subsequently start building some higher-order tools or abstraction so people can start using that to control effectively.In this case, if you're a new engineer, Corey, which it seems like you were, at some point—Corey: I still write code like I am, don't worry.Micheal: [laugh]. So yes, you would get access to our internal tool to actually help spin up what we call is a dev app, where you get a chance to, obviously, choose the instance size, not the instance type itself, and we have actually constrained the instance types we have approved within Pinterest as well. We don't give you the entire list you get a chance to choose and deploy to. We actually have constraint to based on the workload types, what are the instance types we want to support because in the future, if we ever want to move from c3 to c5—and I've been there, trust me—it is not an easy thing to do, so you want to make sure that you're not letting people just use random instances, and constrain that by building some of these tools. As a new engineer, you would go in, you'd use the tool, and actually have a dev app provisioned for you with our Pinterest image to get you started.And then subsequently, we'll obviously shut it down if we see you not being using it over a certain amount of time, but those are sort of the guardrails we've put in over there so you never get a chance to directly ever use the EC2 APIs, or any of those AWS APIs to do certain things. The similar thing applies for S3 or any of the higher-order tools which AWS will provide, too.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: How does that interplay with AWS launches yet another way to run containers, for example, and that becomes a valuable potential avenue to get some business value for a developer, but the platform you built doesn't necessarily embrace that capability? Or they release a feature to an existing tool that you use that could potentially be a just feature capability story, much more so than a cost savings one. How do you keep track of all of that and empower people to use those things so they're not effectively trying to reimplement DynamoDB on top of EC2?Micheal: That's been a challenge, actually, in the past for us because we've always been very flexible where engineers have had an opportunity to write their own solutions many a times rather than leveraging the AWS services, and of late, that's one of the reasons why we have an infrastructure organization—an extremely lean organization for what it's worth—but then still able to achieve outsized outputs. Where we evaluate a lot of these use cases, as they come in and open up different aspects of what we want to provide say directly from AWS, or build certain abstractions on top of it. Every time we talk about containers, obviously, we always associate that with something like Kubernetes and offerings from there on; we realized that our engineers directly never ask for those capabilities. They don't come in and say, “I need a new container orchestration system. Give that to me, and I'm going to be extremely productive.”What people actually realize is that if you can provide them effective tools and that can help them get their job done, they would be happy with it. For example, like I said, our deployment system, which is actually an open-source system called Teletraan. That is the bread and butter at Pinterest at which my team runs. We operate 100,000-plus machines. We have actually looked into container orchestration where we do have a dedicated Kubernetes team looking at it and helping certain use cases moved there, but we realized that the cost of entire migrations need to be evaluated against certain use cases which can benefit from being on Kubernetes from day one. You don't want to force anyone to move there, but give them the right incentives to move there. Case in point, let's upgrade your OS. Because if you're managing machines, obviously everyone loves to upgrade their OSes.Corey: Well, it's one of the things I love savings plans versus RIs; you talk about the c3 to c5 migration and everyone has a story about one of those, but the most foolish or frustrating reason that I ever saw not to do the upgrade was what we bought a bunch of Reserved Instances on the C3s and those have a year-and-a-half left to run. And it's foolish not on the part of customers—it's economically sound—but on the part of AWS where great, you're now forcing me to take a contractual commitment to something that serves me less effectively, rather than getting out of the way and letting me do my job. That's why it's so important to me at least, that savings plans cover Fargate and Lambda, I wish they covered SageMaker instead of SageMaker having its own thing because once again, you're now architecturally constrained based upon some ridiculous economic model that they have imposed on us. But that's a separate rant for another time.Micheal: No, we actually went through that process because we do have a healthy balance of how we do Reserved Instances and how we look at on-demand. We've never been big users have spot in the past because just the spot market itself, we realized that putting that pressure on our customers to figure out how to manage that is way more. When I say customers, in this case, engineers within the organization.Corey: Oh, yes. “I want to post some pictures on Pinterest, so now I have to understand the spot market. What?” Yeah.Micheal: [laugh]. So, in this case, when we even we're moving from C3 to C5—and this is where the partnership really plays out effectively, right, because it's also in the best interest of AWS to deprecate their aging hardware to support some of these new ones where they could also be making good enough premium margins for what it's worth and give the benefit back to the user. So, in this case, we were able to work out an extremely flexible way of moving to a C5 as soon as possible, get help from them, actually, in helping us do that, too, allocating capacity and working with them on capacity management. I believe at one point, we were actually one of the largest companies with a C3 footprint and it took quite a while for us to move to C5. But rest assured, once we moved, the savings was just immense. We were able to offset any of those RI and we were able to work behind the scenes to get that out. But obviously, not a lot of that is considered in a small-scale company just because of, like you said, those constraints which have been placed in a contractual obligation.Corey: Well, this is an area in which I will give the same guidance to companies of your scale as well as small-scale companies. And by small-scale, I mean, people on the free tier account, give or take, so I do mean the smallest of the small. Whenever you wind up in a scenario where you find yourself architecturally constrained by an economic barrier like this, reach out to your account manager. I promise you have one. Every account, even the tiny free tier accounts, have an account manager.I have an account manager, who I have to say has probably one of the most surreal jobs that AWS, just based upon the conversations I throw past him. But it's reaching out to your provider rather than trying to solve a lot of this stuff yourself by constraining how you're building things internally is always the right first move because the worst case is you don't get anywhere in those conversations. Okay, but at least you explored that, as opposed to what often happens is, “Oh, yeah. I have a switch over here I can flip and solve your entire problem. Does that help anything?”Micheal: Yeah.Corey: You feel foolish finding that out only after nine months of dedicated work, it turns out.Micheal: Which makes me wonder, Corey. I mean, do you see a lot of that happening where folks don't tend to reach out to their account managers, or rather treat them as partners in this case, right? Because it sounds like there is this unhealthy tension, I would say, as to what is the best help you could be getting from your account managers in this case.Corey: Constantly. And the challenge comes from a few things, in my experience. The first is that the quality of account managers and the technical account managers—the folks who are embedded many cases with your engineering teams in different ways—does vary. AWS is scaling wildly and bursting at the seams, and people are hard to scale.So, some are fantastic, some are decidedly less so, and most folks fall somewhere in the middle of that bell curve. And it doesn't take too many poor experiences for the default to be, “Oh, those people are useless. They never do anything we want, so why bother asking them?” And that leads to an unhealthy dynamic where a lot of companies will wind up treating their AWS account manager types as a ticket triage system, or the last resort of places that they'll turn when they should be involved in earlier conversations.I mean, take Pinterest as an example of this. I'm not sure how many technical account managers you have assigned to your account, but I'm going to go out on a limb and guess that the ratio of technical account managers to engineers working on the environment is incredibly lopsided. It's got to be a high ratio just because of the nature of how these things work. So, there are a lot of people who are actively working on things that would almost certainly benefit from a more holistic conversation with your AWS account team, but it doesn't occur to them to do it just because of either perceived biases around levels of competence, or poor experiences in the past, or simply not knowing the capabilities that are there. If I could tell one story around the AWS account management story, it would be talk to folks sooner about these things.And to be clear, Pinterest has this less than other folks, but AWS does themselves no favors by having a product strategy of, “Yes,” because very often in service of those conversations with a number of companies, there is the very real concern of are they doing research so that they can launch a service that competes with us? Amazon as a whole launching a social network is admittedly one of the most hilarious ideas I [laugh] can come up with and I hope they take a whack at it just to watch them learn all these lessons themselves, but that is again, neither here nor there.Micheal: That story is very interesting, and I think you mentioned one thing; it's just that lack of trust, or even knowing what the account managers can actually do for you. There seems to be just a lack of education on that. And we also found it the hard way, right? I wouldn't say that Pinterest figured this out on day one. We evolved sort of a relationship over time. Yes, our time… engagements are, sort of, lopsided, but we were able to negotiate that as part of deals as we learned a bit more on what we can and we cannot do, and how these individuals are beneficial for Pinterest as well. And—Corey: Well, here's a question for you, without naming names—and this might illustrate part of the challenge customers have—how long has your account manager—not the technical account managers, but your account manager—been assigned to your account?Micheal: I've been at Pinterest for five years and I've been working with the same person. And he's amazing.Corey: Which is incredibly atypical. At a lot of smaller companies, it feels like, “Oh, I'm your account manager being introduced to you.” And, “Are you the third one this year? Great.” What happens is that if the account manager excels, very often they get promoted and work with a smaller number of accounts at larger spend, and whereas if they don't find that AWS is a great place for them for a variety of reasons, they go somewhere else and need to be backfilled.So, at the smaller account, it's, “Great. I've had more account managers in a year than you've had in five.” And that is often the experience when you start seeing significant levels of rotation, especially on the customer engineering side where you wind up with you have this big kickoff, and everyone's aware of all the capabilities and you look at it three years later, and not a single person who was in that kickoff is still involved with the account on either side, and it's just sort of been evolving evolutionarily from there. One thing that we've done in some of our larger accounts as part of our negotiation process is when we see that the bridges have been so thoroughly burned, we will effectively request a full account team cycle, just because it's time to get new faces in where the customer, in many cases unreasonably, is not going to say, “Yeah but a year-and-a-half ago you did this terrible thing and we're still salty about it.” Fine, whatever. I get it. People relationships are hard. Let's go ahead and swap some folks out so that there are new faces with new perspectives because that helps.Micheal: Well, first off, if you had so many switches in account manager, I think that's something speaks about [laugh] how you've been working, too. I'm just kidding. There are a bu—Corey: Entirely possible. In seriousness, yes. But if you talk to—like, this is not just me because in my case, yeah, I feel like my account manager is whoever drew the short straw that week because frankly, yeah, that does seem like a great punishment to wind up passing out to someone who is underperforming. But for a lot of folks who are in the mid-tier, like, spending $50 to $100,000 a month, this is a very common story.Micheal: Yeah. Actually, we've heard a bit about this, too. And like you said, I think maintaining context is the most thing. You really want your account manager to vouch for you, really be your champion in those meetings because AWS, like you said is so large, getting those exec time, and reviews, and there's so many things that happen, your account manager is the champion for you, or right there. And it's important and in fact in your best interest to have a great relationship with them as well, not treat them as, oh yet another vendor.And I think that's where things start to get a bit messy because when you start treating them as yet another vendor, there is no incentive for them to do the best for you, too. You know, people relationships are hard. But that said though, I think given the amount of customers like these cloud companies are accruing, I wouldn't be surprised; every account manager seems to be extremely burdened. Even in our case, although I've been having a chance to work with this one person for a long time, we've actually expanded. We have now multiple account managers helping us out as we've started scaling to use certain aspects of AWS which we've never explored before.We were a bit constrained and reserved about what service we want to use because there have been instances where we have tried using something and we have hit the wall pretty immediately. API rate limits, or it's not ready for primetime, and we're like, “Oh, my God. Now, what do we do?” So, we have a bit more cautious. But that said, over time, having an account manager who understands how you work, what scale you have, they're able to advocate with the internal engineering teams within the cloud provider to make the best of supporting you as a customer and tell that success story all the way out.So yeah, I can totally understand how this may be hard, especially for those small companies. For what it's worth, I think the best way to really think about it is not treat them as your vendor, but really go out on a limb there. Even though you signed a deal with them, you want to make sure that you have the continuing relationship with them to have—represent your voice better within the company. Which is probably hard. [laugh].Corey: That's always the hard part. Honestly, if this were the sort of thing that were easy to automate, or you could wind up building out something that winds up helping companies figure out how to solve these things programmatically, talk about interesting business problems that are only going to get larger in the fullness of time. This is not going away, even if AWS stopped signing up new customers entirely right now, they would still have years of growth ahead of them just from organic growth. And take a company with the scale of Pinterest and just think of how many years it would take to do a full-on exodus, even if it became priority number one. It's not realistic in many cases, which is why I've never been a big fan of multi-cloud as an approach for negotiation. Yeah, AWS has more data on those points than any of us do; they're not worried about it. It just makes you sound like an unsophisticated negotiator. Pick your poison and lean in.Micheal: That is the truth you just mentioned, and I probably want to give a call out to our head of infrastructure, [Coburn 00:42:13]. He's also my boss, and he had brought this perspective as well. As part of any negotiation discussions, like you just said, AWS has way more data points on this than what we think we can do in terms of talking about, “Oh, we are exploring this other cloud provider.” And it's—they would be like, “Yeah. Do tell me more [laugh] how that's going.”And it's probably in the best interest to never use that as a negotiation tactic because they clearly know the investments that's going to build on what you've done, so you might as well be talking more—again, this is where that relationship really plays together because you want both of them to be successful. And it's in their best interest to still keep you happy because the good thing about at least companies of our size is that we're probably, like, one phone call away from some of their executive team, where we could always talk about what didn't work for us. And I know not everyone has that opportunity, but I'm really hoping and I know at least with some of the interactions we've had with the AWS teams, they're actively working and building that relationship more and more, giving access to those customer advisory boards, and all of them to have those direct calls with the executives. I don't know whether you've seen that in your experience in helping some of these companies?Corey: Have a different approach to it. It turns out when you're super loud and public and noisy about AWS and spend too much time in Seattle, you start to spend time with those people on a social basis. Because, again, I'm obnoxious and annoying to a lot of AWS folks, but I'm also having an obnoxious habit of being right in most of the things I'm pointing out. And that becomes harder and harder to ignore. I mean, part of the value that I found in being able to do this as a consultant is that I begin to compare and contrast different customer environments on a consistent ongoing basis.I mean, the reason that negotiation works well from my perspective is that AWS does a bunch of these every week, and customers do these every few years with AWS. And well, we do an awful lot of them, too, and it's okay, we've seen different ways things can get structured and it doesn't take too long and too many engagements before you start to see the points of commonality in how these things flow together. So, when we wind up seeing things that a customer is planning on architecturally and looking to do in the future, and, “Well, wait a minute. Have you talked to the folks negotiating the contract about this? Because that does potentially have bearing and it provides better data than what AWS is gathering just through looking at overall spend trends. So yeah, bring that up. That is absolutely going to impact the type of offer you get.”It just comes down to understanding the motivators that drive folks and it comes down to, I think understanding the incentives. I will say that across the board, I have never yet seen a deal from AWS come through where it was, “Okay, at this point you're just trying to hoodwink the customer and get them to sign on something that doesn't help them.” I've seen mistakes that can definitely lead to that impression, and I've seen areas where they're doing data is incomplete and they're making assumptions that are not borne out in reality. But it's not one of those bad faith type—Micheal: Yeah.Corey: —of negotiations. If it were, I would be framing a lot of this very differently. It sounds weird to say, “Yeah, your vendor is not trying to screw you over in this sense,” because look at the entire IT industry. How often has that been true about almost any other vendor in the fullness of time? This is something a bit different, and I still think we're trying to grapple with the repercussions of that, from a negotiation standpoint and from a long-term business continuity standpoint, when your faith is linked—in a shared fate context—with your vendor.Micheal: It's in their best interest as well because they're trying to build a diversified portfolio. Like, if they help 100 companies, even if one of them becomes the next Pinterest, that's great, right? And that continued relationship is what they're aiming for. So, assuming any bad faith over there probably is not going to be the best outcome, like you said. And two, it's not a zero-sum game.I always get a sense that when you're doing these negotiations, it's an all-or-nothing deal. It's not. You have to think they're also running a business and it's important that you as your business, how okay are you with some of those premiums? You cannot get a discount on everything, you cannot get the deal or the numbers you probably want almost everything. And to your point, architecturally, if you're moving in a certain direction where you think in the next three years, this is what your usage is going to be or it will come down to that, obviously, you should be investing more and negotiating that out front rather than managed NAT [laugh] gateways, I guess. So, I think that's also an important mindset to take in as part of any of these negotiations. Which I'm assuming—I don't know how you folks have been working in the past, but at least that's one of the key items we have taken in as part of any of these discussions.Corey: I would agree wholeheartedly. I think that it just comes down to understanding where you're going, what's important, and again in some cases knowing around what things AWS will never bend contractually. I've seen companies spend six weeks or more trying to get to negotiate custom SLAs around services. Let me save everyone a bunch of time and money; they will not grant them to you.Micheal: Yeah.Corey: I promise. So, stop asking for them; you're not going to get them. There are other things they will negotiate on that they're going to be highly case-dependent. I'm hesitant to mention any of them just because, “Well, wait a minute, we did that once. Why are you talking about that in public?” I don't want to hear it and confidentiality matters. But yeah, not everything is negotiable, but most things are, so figuring out what levers and knobs and dials you have is important.Micheal: We also found it that way. AWS does cater to their—they are a platform and they are pretty clear in how much engagement—even if we are one of their top customers, there's been many times where I know their product managers have heavily pushed back on some of the requests we have put in. And that makes me wonder, they probably have the same engagement even with the smallest of customers, there's always an implicit assumption that the big fish is trying to get the most out of your public cloud providers. To your point, I don't think that's true. We're rarely able to negotiate anything exclusive in terms of their product offerings just for us, if that makes sense.Case in point, tell us your capacity [laugh] for x instances or type of instances, so we as a company would know how to plan out our scale-ups or scale-downs. That's not going to happen exclusively for you. But those kind of things are just, like, examples we have had a chance to work with their product managers and see if, can we get some flexibility on that? For what it's worth, though, they are willing to find a middle ground with you to make sure that you get your answers and, obviously, you're being successful in your plans to use certain technologies they offer or [unintelligible 00:48:31] how you use their services.Corey: So, I know we've gone significantly over time and we are definitely going to do another episode talking about a lot of the other things that you're involved in because I'm going to assume that your full-time job is not worrying about the AWS bill. In fact, you do a fair number of things beyond that; I just get stuck on that one, given that it is but I eat, sleep, breathe, and dream about.Micheal: Absolutely. I would love to talk more, especially about how we're enabling our engineers to be extremely productive in this new world, and how we want to cater to this whole cloud-native environment which is being created, and make sure people are doing their best work. But regardless, Corey, I mean, this has been an amazing, insightful chat, even for me. And I really appreciate you having me on the show.Corey: No, thank you for joining me. If people want to learn more about what you're up to, and how you think about things, where can they find you? Because I'm also going to go out on a limb and assume you're also probably hiring, given that everyone seems to be these days.Micheal: Well, that is true. And I wasn't planning to make a hiring pitch but I'm glad that you leaned into that one. Yes, we are hiring and you can find me on Twitter at twitter dot com slash M-I-C-H-E-A-L. I am spelled a bit differently, so make sure you can hit me up, and my DMs are open. And obviously, we have all our open roles listed on pinterestcareers.com as well.Corey: And we will, of course, put links to that in the [show notes 00:49:45]. Thank you so much for taking the time to speak with me today. I really appreciate it.Micheal: Thank you, Corey. It was really been great on your show.Corey: And I'm sure we'll do it again in the near future. Micheal Benedict, Head of Engineering Productivity at Pinterest. I am Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a long rambling comment about exactly how many data centers Pinterest could build instead.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Founders Unfiltered
Ep43: Making Blockchain Simple ft. Matic Network

Founders Unfiltered

Play Episode Listen Later Nov 7, 2021 29:27


Jaynti was born and brought up in a small town in Ahmedabad. He studied at a local college, where students' ambition was limited to getting a job at TCS. Jaynti too followed a similar path and took a job at an Indian corporate. A friend introduced him to YC's Hacker News - a website focusing on technology and budding startups. Desiring to work with great people, he tried cold mailing people at top US startups - even offering to work for free, but didn't receive any response. He worked at a couple of startups, and learnt more about the zero to one building journey. He was always working on multiple side projects, waiting for something that really clicks. Jaynti came across bitcoin and was drawn to the endless possibilities the technology offered. Diving deeper into blockchains, he discovered the limitations of existing tech. Ethereum had made it possible for anyone to make apps on blockchain, but as the network grew transacting on ETH became increasingly slow and expensive - with cost of 1 transaction rising even higher than $80. To solve the problem of scaling blockchains, Jaynti created Matic - a layer 2 solution that augments Ethereum and drives its scalability. Using Matic brings the transaction cost to less than 1 INR, while also making it 20 times faster. Today, Matic's worth more than $10B and is used by companies worldwide for applications including NFTs, DeFi For more details - https://ajuniorvc.com/podcast/

Screaming in the Cloud
Making Multi-Cloud Waves with Betty Junod

Screaming in the Cloud

Play Episode Listen Later Nov 3, 2021 35:13


About Betty Betty Junod is the Senior Director of Multi-Cloud Solutions at VMware helping organizations along their journey to cloud. This is her second time at VMware, having previously led product marketing for end user computing products.  Prior to VMware she held marketing leadership roles at Docker and solo.io in following the evolution of technology abstractions from virtualization, containers, to service mesh. She likes to hang out at the intersection of open source, distributed systems, and enterprise infrastructure software. @bettyjunod  Links: Twitter: https://twitter.com/BettyJunod Vmware.com/cloud: https://vmware.com/cloud TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You know how git works right?Announcer: Sorta, kinda, not really Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Periodically, I like to poke fun at a variety of different things, and that can range from technologies or approaches like multi-cloud, and that includes business functions like marketing, and sometimes it extends even to companies like VMware. My guest today is the Senior Director of Multi-Cloud Solutions at VMware, so I'm basically spoilt for choice. Betty Junod, thank you so much for taking the time to speak with me today and tolerate what is no doubt going to be an interesting episode, one way or the other.Betty: Hey, Corey, thanks for having me. I've been a longtime follower, and I'm so happy to be here. And good to know that I'm kind of like the ultimate cross-section of all the things [laugh] that you can get snarky about.Corey: The only thing that's going to make that even better is if you tell me, “Oh, yeah, and I moonlight on a contract gig by naming AWS services.” And then I just won't even know where to go. But I'll assume they have to generate those custom names in-house.Betty: Yes. Yes, I think they do those there. I may comment on it after the fact.Corey: So, periodically I am, let's call it miscategorized, in my position on multi-cloud, which is that it's a worst practice that when you're designing something from scratch, you should almost certainly not be embracing unless you're targeting a very specific corner case. And I stand by that, but what that has been interpreted as by the industry, in many cases because people lack nuance when you express your opinions in tweet-sized format—who knew—as me saying, “Multi-cloud bad.” Maybe, maybe not. I'm not interested in assigning value judgment to it, but the reality is that there are an awful lot of multi-cloud deployments out there. And yes, some of them started off as, “We're going to migrate from one to the other,” and then people gave up and called it multi-cloud, but it is nuanced. VMware is a company that's been around for a long time. It has reinvented itself in a few different ways at different periods of its evolution, and it's still highly relevant. What is the Multi-Cloud Solutions group over at VMware? What do you folks do exactly?Betty: Yeah. And so I will start by multi-cloud; we're really taking it from a position of meeting the customer where they are. So, we know that if anything, the only thing that's a given in our industry is that there will be something new in the next six months, next year, and the whole idea of multi-cloud, from our perspective, is giving customers the optionality, so don't make it so that it's a closed thing for them. But if they decide—it's not that they're going to start, “Hey, I'm going to go to cloud, so day one, I'm going to go all-in on every cloud out there.” That doesn't make sense, right, as—Corey: But they all gave me such generous free credit offers when I founded my startup; I feel obligated to at this point.Betty: I mean, you can definitely create your account, log in, play around, get familiar with the console, but going from zero to being fully operationalized team to run production workloads with the same kind of SLAs you had before, across all three clouds—what—within a week is not feasible for people getting trained up and actually doing that. Our position is that meeting customers where they are and knowing that they may change their mind, or something new will come up—a new service—and they really want to use a new service from let's say GCP or AWS, they want to bring that with an application they already have or build a new app somewhere, we want to help enable that choice. And whether that choice applies to taking an existing app that's been running in their data center—probably on vSphere—to a new place, or building new stuff with containers, Kubernetes, serverless, whatever. So, it's all just about helping them actually take advantage of those technologies.Corey: So, it's interesting to me about your multi-cloud group, for lack of a better term, is there a bunch of things fall under its umbrella? I believe Bitnami does—or as I insist on calling it, ‘bitten-A-M-I'—I believe that SaltStack—which I wrote a little bit of once upon a time, which tells me you folks did no due diligence whatsoever because everything I've ever written is molten garbage—Betty: Not [unintelligible 00:04:33].Corey: And—so to be clear, SaltStack is good; just the parts that I wrote are almost certainly terrible because have you met me?Betty: I'll make a note. [laugh].Corey: You have Wavefront, you have CloudHealth, you have a bunch of other things in the portfolio, and yeah, all those things do work across multiple clouds, but there's nothing that makes using any of those things a particularly bad idea even if you're all-in on one cloud provider, too. So, it's a portfolio that applies to a whole bunch have different places from your perspective, but it can be used regardless of where folks stand ideologically.Betty: Yes. So, this goes back to the whole idea that we meet the customers where they are and help them do what they want to do. So, with that, making sure these technologies that we have work on all the clouds, whether that be in the data center or the different vendors, so that if a customer wants to just use one, or two, or three, it's fine. That part's up to them.Corey: The challenge I've run into is that—and maybe this is a ‘Twitter Bubble' problem, but unfortunately, having talked to a whole bunch of folks in different contexts, I know it isn't—there's almost this idea that you have to be incredibly dogmatic about a particular technology that you're into. I joke periodically about the Rust Evangelism Strikeforce where their entire job is talking about using Rust; their primary IDE is PowerPoint because they're giving talks all the time about it rather than writing code. And great, that's a bit of an exaggeration, but there are the idea of a technology purist who is taking, “Things must be this way,” well past a point of being reasonable, and disregarding the reality that, yeah, the world is messy in a way that architectural diagrams never are.Betty: Yeah. The architectural diagrams are always 2D, right? Back to that PowerPoint slide: how can I make pretty boxes? And then I just redraw a line because something new came out. But you and I have been in this industry for a long time, there's always something new.And I think that's where the dogmatism gets problematic because if you say we're only going to do containers this way—you know, I could see Swarm and Kubernetes, or all-in on AWS and we're going to use all the things from AWS and there's only this way. Things are generational and so the idea that you want to face the reality and say that there is a little bit of everything. And then it's kind of like, how do you help them with a part of that? As a vendor, it could be like, “I'm going to help us with a part of it, or I'm going to help address certain eras of it.” That's where I think it gets really bad to be super dogmatic because it closes you off to possibly something new and amazing, new thinking, different ways to solve the same problem.Corey: That's the problem is left to our own devices, most of us who are building things, especially for random ideas, yeah, there's a whole modern paradigm of how I can build these things, but I'm going to shortcut to the thing I know best, which may very well the architectures that I was using 15 years ago, maybe tools that I was using 15 years ago. There's a reason that Vim is still as popular as it is. Would I recommend it to someone who's a new user? Absolutely not; it's user-hostile, but back in my days of being a grumpy sysadmin, you learned vi because it was on everything you could get into, and you never knew in what environment you were going to be encountering stuff. These days, you aren't logging in to remote systems to manage them, in most cases, and when it happens, it's a rarity and a bug.The world changes; different approaches change, but you have to almost reinvent your entire philosophy on how things work and what your career trajectory looks like. And you have to give up aspects of what you've considered to be part of your identity and embrace something new. It was hard for me to accept that, for example, Docker and the wave of containerization that was rolling out was effectively displacing the world that I was deep in of configuration management with Puppet and with Salt. And the world changes; I said, “Okay, now I'll work on cloud.” And if something else happens, and mainframes are coming back again, instead, well, I'm probably not going to sit here railing against the tide. It would be ridiculous to do that from my perspective. But I definitely understand the temptation to fight against it.Betty: Mm-hm. You know, we spend so much time learning parts of our craft, so it's hard to say, “I'm now not going to be an expert in my thing,” and I have to admit that something else might be better and I have to be a newbie again. That can be scary for someone who's spent a lot of time to be really well-versed in a specific technology. It's funny that you bring up the whole Docker and Puppet config management; I just had a healthy discussion over Slack with some friends. Some people that we know and comment about some of the newer areas of config management, and the whole idea is like, is it a new category or an evolution of? And I went back to the point that I made earlier is like, it's generations. We continually find new ways to solve a problem, and one thing now is it [sigh] it just all goes so much faster, now. There's a new thing every week. [laugh] it seems sometimes.Corey: It is, and this is the joy of having been in this industry for a while—toxic and broken in many ways though it is—is that you go through enough cycles of seeing today's shiny, new, amazing thing become tomorrow's legacy garbage that we're stuck supporting, which means that—at least from my perspective—I tend to be fairly conservative with adopting new technologies with respect to things that matter. That means that I'm unlikely to wind up looking at the front page of Hacker News to pick a framework to build a banking system in, and I'm unlikely to be the first kid on my block to update to a new file system or database, just because, yeah, if I break a web server, we all laugh, we make fun of the fact that it throws an error for ten minutes, and then things are back up and running. If I break the database, there's a terrific chance that we don't have a company anymore. So, it's the ‘mistakes will show' area and understanding when to be aggressive and when to hold back as far as jumping into new technologies is always a nuanced decision. And let's be clear as well, an awful lot of VMware's customers are large companies that were founded, somehow—this is possible—before 2010. Imagine that. Did people—Betty: [laugh]. I know, right?Corey: —even have businesses or lives back then? I thought we all used horse-driven carriages and whatnot. And they did not build on cloud—not because of any perception of distrust; because it functionally did not exist at the time that they were building these things. And, “Oh, come out into the cloud. It's fine now.” It… yeah, that application is generating hundreds of millions in revenue every quarter. Maybe we treat that with a little bit of respect, rather than YOLO-ing it into some Lambda-driven monster that's constructed—Betty: One hundred—Corey: —out of popsicle sticks and glue.Betty: —percent. Yes. I think people forget that. And it's not that these companies don't want to go to cloud. It's like, “I can't break this thing. That could be, like, millions of dollars lost, a second.”Corey: I write my weekly newsletters in a custom monstrosity of a system that has something like 30-some-odd Lambda functions, a bunch of API gateways that are tied together with things, and periodically there are challenges with it that break as the system continues to evolve. And that's fine. And I'm okay with using something like that as a part of my workflow because absolute worst case, I can go back to the way that my newsletter was originally written: in Google Docs, and it doesn't look anywhere near the same way, and it goes back to just a text email that starts off with, “I have messed up.” And that would be a better story than most of the stuff I put out as a common basis. Similarly, yeah, durability is important.If this were a serious life-critical app, it would not just be hanging out in a single region of a single provider; it would probably be on one provider, as I've talked about, but going multi-region and having backups to a different cloud provider. But if AWS takes a significant enough outage to us-west-2 in Oregon, to the point where my ridiculous system cannot function to write the newsletter, that too, is a different handwritten email that goes out that week because there's no announcement they've made that anyone's going to give the slightest toss about, given the fact that it's basically Cloud Armageddon. So, we'll see. It's about understanding the blast radius and understanding your use case.Betty: Yep. A hundred percent.Corey: So, you've spent a fair bit of time doing interesting things in your career. This is your second outing at VMware, and in the interim, you were at solo.io for a bit, and before that you were in a marketing leadership role at Docker. Let's dive in, if you will. Given that you are no longer working at Docker, they recently made an announcement about a pricing model change, whereas it is free to use Docker Desktop for anyone's personal projects, and for small companies.But if you're a large company, which they define is ten million in revenue a year or 250 employees—those two things don't go alike, but okay—then you have to wind up having a paid plan. And I will say it's a novel approach, but I'm curious to hear what you have to say about it.Betty: Well, I'd say that I saw that there was a lot of flutter about that news, and it's kind of a, it doesn't matter where you draw the line in the sand for the tier, there's always going to be some pushback on it. So, you have to draw a line somewhere. I haven't kept up with the details around the pricing models that they've implemented since I left Docker a few years ago, but monetization is a really important part for a startup. You do have to make money because there are people that you have to pay, and eventually, you want to get off of raising money from VCs all the time. Docker Desktop has been something that has been a real gem from a local developer experience, right, giving the—so that has been well-received by the community.I think there was an enterprise application for it, but when I saw that, I was like, yeah, okay, cool. They need to do something with that. And then it's always hard to see the blowback. I think sometimes with the years that we've had with Docker, it's kind of like no matter what they do, the Twitterverse and Hacker News is going to just give them a hard time. I mean, that is my honest opinion on that. If they didn't do it, and then, say, they didn't make the kind of revenue they needed, people would—that would become another Twitter thread and Hacker News blow up, and if they do it, you'll still have that same reaction.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: It seems to be that Docker has been trying to figure out how to monetize for a very long time because let's be clear here; I think it is difficult to overstate just how impactful and transformative Docker was to the industry. I gave a talk “Heresy in the Church of Docker” that listed a bunch of things that didn't get solved with Docker, and I expected to be torn to pieces for it, and instead I was invited to give it at ContainerCon one year. And in time, a lot of those things stopped being issues because the industry found answers to it. Now, unfortunately, some of those answers look like Kubernetes, but that's neither here nor there. But now it's, okay, so giving everything that you do that is core and central away for free is absolutely part of what drove the adoption that it saw, but goodwill from developers is not the sort of thing that generally tends to lead to interesting revenue streams.So, they had to do something. And they've tried a few different things that haven't seemed to really pan out. Then they spun off that pesky part of their business that made money selling support contracts, over to Mirantis, which was apparently looking for something now that OpenStack was no longer going to be a thing, and Kubernetes is okay, “Well, we'll take Docker enterprise stuff.” Great. What do they do, as far as turning this into a revenue model?There's a lot of the, I guess, noise that I tend to ignore when it comes to things like this because angry people on Twitter, or on Hacker News, or other terrible cesspools on the internet, are not where this is going to be decided. What I'm interested in is what the actual large companies are going to say about it. My problem with looking at it from the outside is that it feels as if there's significant ambiguity across the board. And if there's one thing that I know about large company procurement departments, it's that they do not like ambiguity. This change takes effect in three or four months, which is underwear-outside-the-pants-superhero-style speed for a lot of those companies, and suddenly, for a lot of developers, they're so far removed from the procurement side of the house that they are never going to have a hope of getting that approved on a career-wide timespan.And suddenly, for a lot of those companies, installing and running Docker Desktop just became a fireable offense because from the company's perspective, the sheer liability side of it, if they were getting subject to audit, is going to be a problem. I don't believe that Docker is going to start pulling Oracle-like audit tactics, but no procurement or risk management group in the world is going to take that on faith. So, the problem is not that it's expensive because that can be worked around; it's not that there's anything inherently wrong with their costing model. The problem is the ambiguity of people who just don't know, “Does this apply to me or doesn't this apply to me?” And that is the thing that is the difficult, painful part.And now, as a result, the [unintelligible 00:17:28] groups and their champions of Docker Desktop are having to spend a lot more time, energy, and thought on this than it would simply be for cutting a check because now it's a risk org-wide, and how do we audit to figure out who's installed this previously free open-source thing? Now what?Betty: Yeah, I'll agree with you on that because once you start making it into corporate-issued software that you have to install on the desktop, that gets a lot harder. And how do you know who's downloaded it? Like my own experience, right? I have a locked-down laptop; I can't just install whatever I want. We have a software portal, which lets me download the approved things.So, it's that same kind of model. I'd be curious because once you start looking at from a large enterprise perspective, your developers are working on IP, so you don't want that on something that they've downloaded using their personal account because now it sits—that code is sitting with their personal account that's using this tool that's super productive for them, and that transition to then go to an enterprise, large enterprise and going through a procurement cycle, getting a master services agreement, that's no small feat. That's a whole motion that is different than someone swiping a credit card or just downloading something and logging in. It's similar to what you see sometimes with the—how many people have signed up for and paid 99 bucks for Dropbox, and then now all of a sudden, it's like, “Wow, we have all of megacorp [laugh] signed up, and then now someone has to sell them a plan to actually manage it and make sure it's not just sitting on all these personal drives.”Corey: Well, that's what AWS's original sales motion looked a lot like they would come in and talk to the CTO or whatnot at giant companies. And the CTO would say, “Great, why should we pick AWS for our cloud needs?” And the answer is, “Oh, I'm sorry. You have 87 distinct accounts within your organization that we've [unintelligible 00:19:12] up for you. We're just trying to offer you some management answers and unify the billing and this, and probably give you a discount as well because there is price breaks available at certain sizing.” It was a different conversation. It's like, “I'm not here to sell you anything. We're already there. We're just trying to formalize the relationship.” And that is a challenge.Again, I'm not trying to cast aspersions on procurement groups. I mean, I do sell enterprise consulting here at The Duckbill Group; we deal with an awful lot of procurement groups who have processes and procedures that don't often align to the way that we do things as a ten-person, fully remote company. We do not have commercial vehicle insurance, for example, because we do not have a commercial vehicle and that is a prerequisite to getting the insurance, for one. We're unlikely to buy one to wind up satisfying some contractual requirements, so we have to go back and forth and get things like that removed. And that is the nature of the beast.And we can say yes, we can say no on a lot of those questionnaires, but, “It depends,” or, “I don't know,” is the sort of thing that's going to cause giant red flags and derail everything. But that is exactly what Docker is doing. Now, it's the well, we have a sort of sloppy, weird set of habits with some of our engineers around the bring your own device to work thing. So, that's the enterprise thing. Let me be very clear, here at The Duckbill Group, we have a policy of issuing people company machines, we manage them very lightly just to make sure the drives are encrypted, so they—and that the screensaver comes out with a password, so if someone loses a laptop, it's just, “Replace the hardware,” not, “We have a data breach.”Let's be clear here; we are responsible about these things. But beyond that, it's oh, you want to have some personal thing installed on your machine or do some work on that stuff? Fine. By all means. It's a situation of we have no policy against it; we understand this is how work happens, and we trust people to effectively be grownups.There are some things I would strongly suggest that any employee—ours or anyone else—not cross the streams on for obvious IP ownership rights and the rest, we have those conversations with our team for a reason. It's, understand the nuances of what you're doing, and we're always willing to throw hardware at people to solve these problems. Not every company is like that. And ten million in revenue is not necessarily a very large company. I was doing the math out for ten million in revenue or 250 employees; assuming that there's no outside investment—which with VC is always a weird thing—it's possible—barely—to have a $10 million in revenue company that has 250 employees, but if they're full time they are damn close to a $15 an hour minimum wage. So, who does it apply to? More people than you might believe.Betty: Yeah, I'm really curious to how they're going to like—like you say, if it takes place in three or four months, roll that out, and how would you actually track it and true that up for people? So.Corey: Yeah. And there are tools and processes to do this, but it's also not in anyone's roadmap because people are not sitting here on their annual planning periods—which is always aspirational—but no one's planning for, “Oh, yeah, Q3, one of our software suppliers is going to throw a real procurement wrench at us that we have to devote time, energy, resources, and budget to figure out.” And then you have a problem. And by resources, I do mean resources of basically assigning work and tooling and whatnot and energy, not people. People are humans, they are not resources; I will die on that hill.Betty: Well, you know, actually resource-wise, the thing that's interesting is when you say supplier, if it's something that people have been able to download for free so far, it's not considered a supplier. So, it's—now they're going to go from just a thing I can use and maybe you've let your developers use to now it has to be something that goes through the official internal vetting as being a supplier. So, that's just—it's a whole different ball game entirely.Corey: My last job before I started this place, was a highly regulated financial institution, and even grabbing things were available for free, “Well, hang on a minute because what license is it using and how is it going to potentially be incorporated?” And this stuff makes sense, and it's important. Now, admittedly, I have the advantage of a number of my engineering peers in that I've been married to a corporate attorney for 11 years and have insight into that side of the world, which to be clear, is all about risk mitigation which is helpful. It is a nuanced and difficult field to—as are most things once you get into them—and it's just the uncertainty that befuddles me a bit. I wish them well with it, truly I do. I think the world is better with an independent Docker in it, but I question whether this is going to find success. That said, it doesn't matter what I think; what matters is what customers say and do, and I'm really looking forward to seeing how it plays out.Betty: A hundred percent; same here. As someone who spent a good chunk of my life there, their mark on the industry is not to be ignored, like you said, with what happened with containers. But I do wish them well. There's lot of good people over there, it's some really cool tech, and I want to see a future for them.Corey: One last topic I want to get into before we wind up wrapping this episode is that you are someone who was nominated to come on the show by a couple of folks, which is always great. I'm always looking for recommendations on this. But what's odd is that you are—if we look at it and dig a little bit beneath the titles and whatnot, you even self-describe as your history is marketing leadership positions. It is uncommon for engineering-types to recommend that I talk to marketing folks.s personally I think that is a mistake; I consider myself more of a marketer than not in some respects, but it is uncommon, which means I have to ask you, what is your philosophy of marketing because it very clearly is differentiated in the public eye.Betty: I'm flattered. I will say that—and this goes to how I hire people and how I coach teams—it's you have to be super curious because there's a ton of bad marketing out there, where it's just kind of like, “Hey, we do these five things and we always do these five things: blah, blah, blah, blah, blah.” But I think it's really being curious about what is the thing that you're marketing? There are people who are just focused on the function of marketing and not the thing. Because you're doing your marketing job in the service of a thing, this new widget, this new whatever, and you got to be super curious about it.And I'll tell you that, for me, it's really hard for me to market something if I'm not excited about it. I have to personally be super excited about the tech or something happening in the industry, and it's, kind of like, an all-in thing for me. And so in that sense, I do spend a ton of time with engineers and end-users, and I really try to understand what's going on. I want to understand how the thing works, and I always ask them, “Well”—so I'll ask the engineers, like, “So… okay, this sounds really cool. You just described this new feature and you're super excited about it because you wrote it, but how is your end-user, the person you're building this for, how did they do this before? Help me understand. How did they do this before and why is this better?”Just really dig into it because for me, I want to understand it deeply before I talk about it. I think the thing is, it shows a tremendous amount of respect for the builder, and then to try to really be empathetic, to understand what they're doing and then partner with them—I mean, this sounds so business-y the way I'm talking about this—but really be a partner with them and just help them make their thing really successful. I'm like the other end; you're going to build this great thing and now I'm going to make it sound like it's the best thing that's ever happened. But to do that, I really need to deeply understand what it is, and I have to care about it, too. I have to care about it in the way that you care about it.Corey: I cannot effectively market or sell something that I don't believe in, personally. I also, to be clear because you are a marketing professional—or at least far more of one than I ever was—I do not view what I do is marketing; I view it as spectacle. And it's about telling stories to people, it's about learning what the market thinks about it, and that informs product design in many respects. It's about understanding the product itself. It's about being able to use the product.And if people are listening to this and think, “Wait a minute, that sounds more like DevRel.” I have news for you. DevRel is marketing, they're just scared to tell you that. And I know people are going to disagree with me on that. You're wrong. But that's okay; reasonable people can disagree.And that's how I see it is that, okay, I'll talk to people building the service, I'll talk to people using the service, but then I'm going to build something with the service myself because until then, it's all a game of who sounds the most convincing in the stories that they tell. But okay, you can tell an amazing story about something, but if it falls over when I tried to use it, well, I'm sorry, you're not being accurate in your descriptions of it.Betty: A hundred percent. I hate to say, like, you're storytellers, but that's a big part of it, but it's kind of like you want to tell the story, so you do something to that people believe a certain thing. But that's part of a curated experience because you want them to try this thing in a certain way. Because you've designed it for something. “I built a spoon. I want you to use that to eat your soup because you can't eat soup with a fork.”So, then you'll have this amazing soup-eating experience, but if I build you a spoon and then not give you any directions and you start throwing it at cars, you're going to be like, “This thing sucks.” So, I kind of think of it in that way. To your point of it has to actually work, it's like, but they also need to know, “What am I supposed to use it for?”Corey: The problem I've always had on some visceral level with formal marketing departments for companies is that they can say that a product that they sell is good, they can say that the product is great, or they can choose to say nothing at all about that product, but when there's a product in the market that is clearly a turd, a marketing department is never going to be able to say that, which I think erodes its authenticity in many respects. I understand the constraints behind, that truly I do, but it's the one superpower I think that I bring to the table where even when I do sponsorship stuff it's, you can buy my attention but not my opinion. Because the authenticity of me being trusted to call them like I see them, for lack of a better term, to my mind at least outweighs any short-term benefit from saying good things about a product that doesn't deserve them. Now, I've been wrong about things, sure. I have also been misinformed in both directions, thinking something is great when it's not, or terrible when it isn't or not understanding the use case, and I am thrilled to engage in those debates. “But this is really expensive when you run for this use case,” and the answer can be, “Well, it's not designed for that use case.” But the answer should not be, “No it's not.” I promise you, expensive is in the eye of the customer not the person building the thing.Betty: Yes. This goes back to I have to believe in the thing. And I do agree it's, like not [sigh]—it's not a panacea. You're not going to make Product A and it's going to solve everything. But being super clear and focused on what it is good for, and then please just try it in this way because that's what we built it for.Corey: I want to thank you for taking the time to have a what for some people is no doubt going to be perceived as a surprisingly civil conversation about things that I have loud, heated opinions about. If people want to learn more, where can they find you?Betty: Well, they can follow me on Twitter. But um, I'd say go to vmware.com/cloud for our work thing.Corey: Exactly. VM where? That's right. VM there. And we will, of course, put links to that in the [show notes 00:30:07].Betty: [laugh].Corey: Thank you so much for taking the time to speak with me. I appreciate it.Betty: Thanks, Corey.Corey: Betty Junod, Senior Director of Multi-Cloud Solutions at VMware. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a loud, ranting comment at the end. Then, if you work for a company that is larger than 250 people or $10 million in revenue, please also Venmo me $5.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Software Social
A Conversation with Kevin Sahin, Co-Founder of ScrapingBee

Software Social

Play Episode Listen Later Nov 2, 2021 56:25


Follow Kevin! https://twitter.com/SahinKevinCheck out ScrapingBee: https://www.scrapingbee.com/AUTOMATED TRANSCRIPTColleen Schnettler  0:00  This episode of Software Social is sponsored by Hey Check It. Does your website performance keep you up at night? The creators behind Hey Check It started it for this very reason—peace of mind about their sites and the sites they manage. Hey Check It is a website performance monitoring and suggestion tool focused on SEO, accessibility, uptime, site speed and content. It includes AI-generated SEO, data, spelling and grammar checking, custom sitemaps, and a number of other tools. If you're managing multiple websites, check their agency plans with public facing dashboards to meet your clients' needs. Start a free trial today at HeyCheckIt.comMichele Hansen  0:39  Hey, welcome back to Software Social. We're doing another interview this week. I am so excited to have Kevin Sahin with me. He is co-founder of ScrapingBee. Kevin, welcome to software social.Kevin Sahin  0:57  Well, thank you, Michele, I'm excited to be here.Michele Hansen  1:01  So this kind of came about because I was on Twitter, as I often am. And I noticed, I think it was actually someone tweeted about MicroConf Europe, which I had been really wanting to go to, but conflicted with a friend's wedding. So we couldn't go. So I was just sort of following and watching everything unfold on Twitter and tweeted about how peer your co founder was, was giving a talk. And he mentioned how scraping DEA offered free API credits to customers who are willing to jump on a 15 minute call with them. And you guys ask them questions like, what else have you tried, and my interest immediately perked up. And really wanted to talk to you about those calls you had and what you learned from them, and what that added for the business. But before we jump into that, perhaps you should say for a moment, just what scraping be. Is and, and whatnot. And?Kevin Sahin  2:09  Sure. So um, so basically scraping the is an API for web scraping. When you are extracting data from the web, you often have the two same problems, which are, there are more and more websites that are using JavaScript frameworks like Vue js, react, etc. And so you have to render the page inside a web browser. And this is kind of, it's a pain to manage, especially at scale. Because you have to, you know, there are lots of DevOps skills that you need. You need big servers, you need lots of things. And it's really handy to have, you know, a headless browser accessible with a simple API call. The other thing that you have to do when you scrape the web at scale, is to manage proxies. So you can you probably need proxies for many different reasons. For example, let's say that you are extracting data from ecommerce website. Well, most ecommerce websites are internationalized, meaning that if you access the website from an IP address in Europe, you will have the prices in euro if you access the IP address or the website from an IP address in the US you will have prices in dollars. So you need some kind of proxy management system. The other thing is IP rate limit. Some websites are limiting the number of pages you can access per day from a single IP address if you need to access more pages, you need more IP addresses etc, etc. And so we bundled this inside a single API which is scrapingMichele Hansen  4:04  so I love how you're solving that because we have felt that pain personally. So I've kind of talked a little bit in the past about how my husband dies first project that was what so the one well, not at first, but the one right before geocoder that basically funded Juco was this mobile app called what's open nearby where you could open it up and see grocery stores convenience stores and coffee shops that were open near you. And how we ran that in the back end was we had a ton of scrapers running of like grocery store, you know Starbucks, whatever like their websites, scraping the hours off of them and we like just all the time there's issues you know, the parsers breaking or you get blocked or actually the the sort of recent side project we did Keren, which allowed people to get an alert when a grocery pick slot opened up on a on a grocery stores website because of COVID and everything that was also powered by scrapers basically and the back end. And so I have I have personally felt the pain of, you know, the impacts when when when, you know, scraping goes wrong or you know it can get frustrating at times.Kevin Sahin  5:29  Yeah, that's I mean, there are the, the story behind scraping is that we, we personally experienced some of those frustrations, because p&i like before launching scraping beam, we started our career in two different startups that were heavily relying on web scraping. In the business, I was working on a startup in France, which is kind of a mix between mint.com in the US and plaid.com. So for those who don't know, it's a bank account aggregation software's sublet, that comm is an API that allows third party to access your bank account. And means that comm is a bank account aggregation, personal finance management app. And so at this startup, I was really exposed to all of these issues. And Param, he was working for a real estate startup, a real estate data startup in France. And so there will relying on scraping lots of real estate portals. So we both, you know, experienced lots of these issues regarding how to handle headless browsers, how to handle proxies, how to, you know, handle blocks, etc, etc. So that was something we, we knew a little about,Michele Hansen  7:16  I love how you started with a pain that you had. But also as, as you've run the business, you're also actively reaching out to your customers to understand what they were trying to do, what problems they were having, and how they were solving those problems. So I wonder if you can kind of take us back to when you like, how did those emails come about where you were reaching out to people like, like, what what kind of prompted that?Kevin Sahin  7:47  Yeah. So that we quickly realized that we really knew when I say that we knew a little about it, it's not an a few million. Because we really knew a little about the different web scraping use cases each time. I mean, from the beginning, when we launched the API we like from day one, I'd say, we realized that some users, we're scraping, have had some use cases that we never imagined. So we quickly realized that we had to get them on the phone and knew more about about it, understand their businesses, what kind of data they they needed, what frequency for what we use case, etc, etc. But the problem that we had is that at the beginning, so we had we had the banner on the dashboard, covering that, if they had any question, they could schedule a call with me. But nobody was scheduling any call. So maybe, maybe the banner was wasn't, I mean, the copy wasn't great, maybe. The CTA wasn't clear, I don't know. But the fact is, nobody was getting any call with me. And we also had an email sequence where we, we had a few links to my county. But it wasn't working. I mean, sometimes we had a trial scheduling a call, but it was not very, not a lot. And and then we we had this idea of offering more 10x more free API calls. Then the trial offered. And then instantly, we started to get a lot of calls. So many that I had to, you know, delete some availability in my week, because I was just doing calls every day all day. And, and it was great because we will learn so much we, I mean, we will learn so many different use cases that we never thought about. For example, I don't know, we, we, we had so many diverse people. So for example, university researchers that were scraping the web for all kinds of research projects. We had government agencies that were scraping the web, to automate automatically detect security frauds. That's all those kinds of use cases we could never invented them, we like, I don't see any other way we could have learned all of this, then, you know, calling our customers and, and developing a relationship with them. And by the way, this, I mean, there are many benefits to these calls. It's not just about, you know, discovering their needs, but it's also building relationships, especially when you are one month old startup. Because, you know, it's really hard to sell your product, especially with enterprise customers, you know, government agencies, universities, etc, etc. When you say, yeah, we were launched a month ago, there's a bit of a trust issue. And developing the relationship, a relationship with them, really helped. Like, in the seven months, after our launch, we signed a big enterprise customer. And I think that we never could have done this without, you know,having them on the call. It also helped in many other ways. For example, I mentioned the the, the university researchers, we granted them free credits to the API for their research project. And like a few weeks or months later, they mentioned us in the University website, which is great for many reason for SEO for authority, etc, etc. So, I mean, there was like, it took me a lot of time to take these calls, but the, the benefits is a like, it's really worth it. And I'm glad we did.Michele Hansen  13:44  It's so interesting how you say that you You not only learn so much about why people need something like scraping be in the first place. But it also built this trust with your customers when you're very you're very new company, and they really didn't have a lot of reason to to trust you. And even though the purpose of them maybe was not, you know, making these sales, it really led to them down the road. All because you took 15 minutes to understand what they were trying to do and what they had been using before.Kevin Sahin  14:23  Yeah. Most of the time, it was more than 15 minutes, by the way. Like, especially when the conversation was getting technical. Because even those scraping visas, simple REST API, there's a whole you know that they often needed. Advice advices about how to implement it on their side. Meaning how to you know Do the scraping pipeline, the scheduling, the data storage, the error monitoring, the maintenance of the scrapers how to what kind of libraries they could use, etc, etc. So I, we we spent a lot of time with this. Sometimes this was a bit too much like, for example, when you spend one hour advising the technical team of your prospect and that that at the end, they don't end up being a customer. It's a bit frustrating. But at the same time, it was really I mean, it was a as a two months old startup, it's a really competitive advantage, I'd say that to be able to take the time to really advise and guide the prospect in the implementation. So and it really helped us to sign the first customers.Michele Hansen  16:24  I'm curious, do you remember the exact questions that you asked people?Kevin Sahin  16:30  Yes, I remember. It's not. I didn't ask a lot. But I was asking them about their, what their company is doing. What? Why they want to scrape data? I mean, is it part? Is it something that is part of their core product core business? Or is it some side thing? The, the the kind of website that they needed to extract data from the frequency? And why like, what did they tried so far? Why did it didn't it worked? Why are your other looking for another solution? Etc, etc. So that, like these five questions, or the most important one, I thinkMichele Hansen  17:35  it sounds like those questions came out of your own genuine curiosity, because you had some awareness of the some some things people might do with scraping from your own experiences. But you were aware that that was not the whole universe of things that people might possibly do. And so you genuinely did not know what the other things people why people might be doing it and what else they might be doing.Kevin Sahin  18:03  Yeah, exactly. And, and we were pretty lucky to realize this early. Because, you know, you're always tempted to just see things through your own experience. But we, as I said, early on, we, we realized all those kind of use cases we had no idea about. And so we got pretty curious about it pretty early.Michele Hansen  18:43  And in so many ways, that reminds me of how I got interested in customer research in the beginning, too, because when we launched geocode, do you know it? I mean, so it came out of our own needs, actually, because that that app I mentioned finding grocery store hours, it would show people a map, and we needed coordinates in order to show that map. And, and so it came out of our own need there. But we're not, you know, neither of us has a background in geography or geographic data analysis, GIS, any of that stuff. And when we launched and people were, you know, reaching out to us, and they're asking for us to do things, we would ask them why because we genuinely did not know because we were not do geographic information systems, people. We weren't steeped in this world. So it was as much about how do we expand our product? As you know, but what why do you want to do it in the first place? Because I just I just don't know. And following that curiosity, yeah.Kevin Sahin  19:48  And so um, the geocode IO, you launched this how many years ago,Michele Hansen  19:58  we launched in January of 2014. So we are Coming up on eight years this January. Wow, congrats. almost a decade of, you know, a couple more years. But yeah, it's kind of wild. snuck up on me.Kevin Sahin  20:17  That's a that's cool. And so how did you when you launched in 2014? What, how did you get your first customers.Michele Hansen  20:34  So we were our first customer for that app, because the app was making about like three or $400 a month in ad revenue. And basically, the idea of do codea was that, you know, we could basically if we released it as an API and threw a wall in front of it, maybe other people would pay to keep the server's going for it. And then we would, we could still keep our app going, and then not basically not be paying for for this geocoding API, rather than paying you know, a major provider, you know, 10s of 1000s of dollars a year, which we didn't. So we had, you know, two little digitalocean droplets that it was running on for 20 bucks a month. And that was our goal was to make 20 bucks a month. So we then, you know, put it on, you know, we talked talked to some other friends who are developers and had them test it out, and then put it on Hacker News. And that was how we got that initial wave of feedback, we had 1000s of signups. Most I mean, that traffic doesn't stick around, like, you look at analytics graph, and it's just like, you just we basically have to filter out our launch, because it's just, it totally breaks the graph. And but we made, we ended up making $31 that month, that that first month,Kevin Sahin  21:55  sorry, trade paid for the Digital Ocean droplet,Michele Hansen  21:59  we were over the moon, because we had made more money than we spent on it. And to us, that was a wild success.Kevin Sahin  22:10  And so how did you like, after this initial hack on your success? How did you continue to, you know, acquire customers and develop the company.Michele Hansen  22:25  So I think in the early days, it was a lot of, you know, when people expressed that they had problems that we solved, trying to be there, so I spent hours, you know, replying to stuff on Stack Overflow. And, you know, whenever something came up on Hacker News, someone asking about geocoding, whatever, we would always like pop in there, or on Twitter, or just kind of trying to be in the places where people were already looking for something like this. Of course, we had we had a website, but I don't, it wasn't super built out, you know, with, you know, case studies and example customers and testimonials and, you know, stuff like that, basically, it's for like documentation for for a long time. But um, yeah, I basically spent a lot of time on StackOverflow trying to sort of, you know, neutrally, like reply to questions and kind of, yeah, keep people coming to us,Kevin Sahin  23:33  and how, like, how did he did evolve? Like, right now, where, where does your customer are coming from?Michele Hansen  23:43  That's a really good question. Because I don't always know. We don't do a ton with analytics. But pretty much we're very SEO based. So it's still that idea that someone is already frustrated. They're already trying to find something for geocoding. Or for you know, they need you need mentioned academic researchers. So we have a lot of customers who are academic researchers, because in the US, in order to connect to any government datasets, you need this thing called a FIPS code. And you can only get that FIPS code if you have the coordinates for the address. And then the government data will be at that FIPS code level, which is basically sort of like the block. So for example, if a researcher is they know they need FIPS codes to connect to some data, there'll be googling it and so is to have tons and tons of landing pages showing people how you need to convert addresses to FIPS codes. Here's how you can do with our API. Here's how you can upload a spreadsheet. You know, if you need congressional districts, here's how you can do it. If you need time zones, here's how you can do it. And it's very content driven. On the SEO side, we we still do a little bit of replying to stuff on StackOverflow I don't think I've done that for months if not, you know like not really Really anymore? Um, pretty much it's it's about, you know, being there when someone is already looking for something.Kevin Sahin  25:08  No, we that's something that we, we also did in the beginning of scraping be. We answered Korra questions, not a lot, not a lot of Stack Overflow but a little bit, and then on forums on Twitter and indie hackers, etc, etc. And just like you like now most of our customers are coming from SEO, I'd say 90%. And we've been really focusing on that, since the beginning, we launched the blog, and even before the product was launched, so I think that our first blog was in May 2019. And we launched in August 2019. So you really treated SEO as a, like our main acquisition channel,Michele Hansen  26:17  and seems like you guys are, I don't know if you're quite like freemium. But you I noticed on your site that it says you can get started with 1000, free API calls, no credit card required. You know, in many ways, I feel like, you know, I think I think it's, you know, freemium is not a pricing model. It's a marketing tactic. And I very much feel like, you know, that combination of SEO and freemium is a huge part of why we have been able to attract customers, because people can try it out without, you know, without having to talk to us first, they can see if this is the product they need, and then they're like, okay, like, we're ready to ready to sign up, and you don't feel like you don't have to sell as hard when you have that combination of SEO and freemium, because people can just figure out for themselves if it's what they need.Kevin Sahin  27:22  Yeah, exactly. And there is only one thing that is very specific to API's. It's that in many companies, and so I learned this with the customer interviews, the developers do not necessarily have access to the company credit cards. And having a free trial without credit card is really something that can boost the activation. Because if the developer has to ask is n plus one or n plus two for a credit card? And maybe he's like, it's going to bother the developer, he's not even going to try the service, or it's going to slow things down because he needs the approval, etc. So having the free credits on the trial is really something that helped us. And I don't I don't see any, I mean, I see many drawbacks of not having it. I don't see many benefits of having, you know, a credit card. They will follow the trail when you're doing when you have an API business.Michele Hansen  28:45  Yeah, exactly. And then you know, the developers they can they're trying to get their work done. They can try it out for themselves, see if it works. And then if it is something that's going to work for them, then like they're the one selling your product within the company. You don't have to be emailing all the CTOs and directors and everything being like, Hello, we're scraping me and this is what we do. Like, it's already there. Developers within the company who are like, hey, like, we've got this project. We've got this deadline, I need to use this thing. I already tried it. It works like can you like, like, yeah, give me the card. Let's go. Let's get this over with. Exactly. Yeah. And I'm curious when you did those calls, you said you gave them free API credits? How many did you give them for those calls?Kevin Sahin  29:28  How many API credits Yeah, I mean, it was at least 10,000 acre grades, sometimes even more, depending on there. So the thing you have to keep in mind is that one API creates isn't equal to one API call. Because the the cost of the API call is depending on the parameters that you use with your API call, and it can cost up to 25 API credits per call, so it goes up quickly.Michele Hansen  29:59  Yeah. So but so basically, I'm just wondering what the, the cost to that, uh, you know, there's the cost of those interviews, but also basically like, you know, because sometimes, you know, often recommend if you're doing call somebody know, give them a 10 or $25. Amazon gift card, and I'm just kind of curious like what thatKevin Sahin  30:20  wasn't? It was not much, I'd say, but I don't have a precise figure to give you I don't know, but probably less than $1 per per 10,000. I mean, they don't even they don't like most of them didn't use the whole 10,000 free credit. So I don't think but not much. So theseMichele Hansen  30:48  customer interviews cost you maybe less than $1. Yeah, each, which actually wasn't a cash outlay, because you're just giving them credits. Half an hour, maybe an hour of your time, depending on how technical their questions were. But down the line could lead to these enterprise sales. And the customers really trusting you in a way that they maybe would not have had you not spent this time and given them those credits.Kevin Sahin  31:19  Yeah, I can't even give you a precise numbers. The first month in August 2019, we signed our first enterprise customer for seven or $800, a month after one of those calls.Michele Hansen  31:37  Wow. Do you know how many of these calls you did? You mean, you mentioned you to them over 18 months? But I'm curious if you have aKevin Sahin  31:44  I did a lot in the beginning, I'd say probably 200, something like that.Michele Hansen  31:55  And I'm curious, you know, you said you you did this for? Like, are you still doing these calls? Or?Kevin Sahin  32:01  I am but so right now, we don't offer free credits anymore. We just have some links in our email sequences. And on the website. If for the trial, period, when customers have questions that cannot be answered, with our knowledge base or recommendation. And now I would say that maybe I have four or five calls per week. Maximum.Michele Hansen  32:38  Yeah, that's, that's awesome. Yeah, I'm still sort of, you know, the the calls came about because you were just you were curious about why does anyone need this thing we made this very similar to us. And I'm curious of, you know, as as, as you were, maybe thinking about doing that, like, like, the questions you asked, you know, are very much, you know, sort of quintessential jobs to be done questions. And I'm curious, what kind of understanding you had of customer research. Before you started doing this?Kevin Sahin  33:25  I would say zero.Michele Hansen  33:30  facet came out organically.Kevin Sahin  33:33  Yeah. I mean, no, I, like, I probably read a few blog posts about how to do customer interviews. It's just not like it was a, you know, a bit of both customer interviews and sales call. So but I mean, I'm not I'm not a salesperson. I don't, I was just, you know, trying to see if, what the customer problems were and if scraping me was a good fit to solve these problems. And if it was, then I would honestly, tell them told them that I thought scripting was the best solution for them. And if it wasn't, then I just told them to. I mean, actually, I told them what if scripting wasn't the solution, I often told them what the solution was. So if I had to refer them to a specific software or consultant or whatever, I did it. And yeah, dog came, I'd say, semi organically. I had some notions about the customer. interviews and sales gold that no experience at all.Michele Hansen  35:05  Fascinating you just kind of dove like head, you know, sort of headfirst into it. And I mean, it seems like it's really helped your your business and help you understand like, like why people need scraping and how you can help them and lead to these enterprise customers and you guys are in tiny seed likeKevin Sahin  35:30  yeah, definitely it really helped.Michele Hansen  35:33  That's awesome. Cool. So I'm curious, you had mentioned that you also had some questions about geocoding. And I wanted to make sure we got time to get Yeah, soKevin Sahin  35:45  So I'm curious about the letter. So first of all, where are you based?Michele Hansen  35:50  So we are in Denmark now. But when we launched geocode, do we live? Actually, we lived in Washington, DC. We lived in Arlington, Virginia, which is just outside DC until July of 2020. So so now we're in Denmark.Kevin Sahin  36:07  Alright, that's cool. And yeah, so the question I had is, you know, the usual what, what led you to? to geocode? So you've answered this a little bit, but what what were you doing before? How did you find the date? You know, did you did some consulting on the side? Was it a side project, etc, etc. Found the stories, always fascinating.Michele Hansen  36:37  Yeah, so um, so I kind of mentioned a little bit. So we had this mobile app, which is making a couple 100 bucks a month in ad revenue. This is like 2012 2013. And we need a geocoding for it. And we ran into a point where we basically couldn't use Google anymore, because they didn't have pay as you go at the time, it was either 2500 for free per day, or enterprise contract, and we just needed 5000. So we had to, basically sort of rolled our own geo coder that was very rudimentary. And we kind of talked about this problem that we had, you know, not being able to store the data and whatnot. And, you know, developer friends had the same problem, made an API, put it on Hacker News, $31, the first month kind of vary, and got tons of feedback from people ask them, you know, why they wanted to do what they needed to do. So started, you know, adding those features as people needed them, like a big thing for us early on was was the ability to upload a spreadsheet. And I think we made our first sort of, you know, higher end sale, May of 2014. So a couple months after, and that was, I mean, that wasn't really adding that that we called the unlimited plan, which at the time was 750 a month was huge part of our growth. But so from that, the beginning as a side project, and it stayed a side project until I went full time, which is October of 2017. So currently celebrating my four year full time anniversary. I was I was a product manager before Okay, yeah, yeah, I was I was specifically like in Well, I was a first I was an operations manager that I was a technical project manager do work managing like WordPress website, builds that agency. And then I really wanted to to like dig my teeth into things. So I transitioned into being a product manager, which led into then doing product development, which is sort of where my heart is, which is how I got into customer research to is doing product development and launching a lot of stuff that didn't work out just like learning that you really need to talk to prospects and if you want something to succeed, learn that the hard way. me so I went full time 2017 and then my husband he and we're like, oh, you know, if I go full time, like it's gonna you know, maybe take some of the load off and make things a little easier. Except you know, I was full time so then our response to our customer response times got better, you know, and we actually grew more and so we're like, Okay, well now husband needs to go full time. And this is February of 2018. And he went to his boss and was like, you know, it's time for me to go full time on this thing. And his boss was like, No, and we're like, this is an interesting negotiating position to be in so he ended up going part time part salary but keeping health insurance which in the US is huge. And, but he eventually went full time by September of 2018, because I mean, basically the more we worked on it, the more you know, the better the product. Got. Yeah. And?Kevin Sahin  40:02  And yeah, did you? Do you have any employees?Michele Hansen  40:08  No, I have a VA, but we don't have any employees.Kevin Sahin  40:12  Okay, so you are very lean? Yeah, yeah, weMichele Hansen  40:15  we focus a lot on, you know, automating as many things as we can. And I think that's one reason, you know, we talking earlier about, you know, SEO and free tier and not having to, you know, sort of, you know, do cold outreach and reach out to companies. You know, partly it's because, you know, that's kind of the sort of workflow I like, when I'm starting up with a product, I like to be able to test it out, see if it works, not have to talk to anybody, like I hate when I have to have a demo to figure out if something is what I needed to do. But also, because we just don't have the time to be, you know, reaching out to people and pitching them, because it's just the two of us, but and that's also, like, a conscious decision on our part, like, we could hire another rep, or we could hire, you know, a salesperson or whatever. But we also just, we, we kind of like how calm it is with just the two of us. So SoKevin Sahin  41:06  you said, Yeah, so basically, you plan to stay just the two of you and not hire in the future.Michele Hansen  41:15  Yeah, that's the plan.Kevin Sahin  41:17  Okay. That's, I mean, there are many founders that, like, this situation that don't really like to manage employees, etc, etc. So that's great, that's working for you.Michele Hansen  41:37  I'm a very, I'm just very product driven. Like, that's what I really love doing is, is product work. And I also I do enjoy, like, sales work, too. So like my time, you know, my sort of favorite things to work on are both product and, you know, customer research and whatnot. And then also doing, like sales and negotiations. And, and yeah, if we had employees, you know, I would be spending time managing employees. And I just, I don't know, I just that that's just not really where my, my heart is. It's not in being a manager, it definitely is for some people. ButKevin Sahin  42:20  yeah, I can relate toMichele Hansen  42:21  that. Yeah.Kevin Sahin  42:25  Yeah, that's, I mean, that's, I don't have much experience managing employees. But for our blog, I worked with a lot of freelancers, you know, different kind of freelancers, constants, writers, editors, some Freelancer to help me with the SEO link builders, etc, etc. And I mean, it's really hard to hire, to manage to keep employees motivated. I mean, it's, it's pretty hard.Michele Hansen  43:10  Yeah, it's a lot of time. And, you know, I think from my own experiences, and you know, those of you know, people I know, like, having a manager who doesn't love being a manager, who, you know, doesn't love, like developing people, and helping them grow, and all that kind of stuff, like, there are people who genuinely love that those people should be managers, those of us who, you know, are a little bit more reluctant on it and enjoy other things. I think it's okay, if we allow ourselves to, to not be managers. And, you know, I sometimes think that there's this, this assumption that, that, that you have to grow and that you have to hire in order to grow. Is this sort of this baked in assumption, and I think there's a little bit of like, judgment sometimes around companies that don't hire because people like, oh, like, you're not a real company, if you don't have any employees or whatnot. I reject that. Like, I think if you can find a way to run a company, and it's successful and gives you the life you want, and for some people that involves employees, and some people it doesn't, and that's Yeah, exactly. And some people you know, it involves, like, I think, I guess, you know, my, my VA is is is you know, a contractor, like a lot of people have a lot of contractors working with them. But you know, having that responsibility also of covering someone's paycheck can, you know, can lend a lot of stress to running a business and some people like that stress and some people don't and I don't understand that like that. Yeah, I think that that sort of leadership component of it is is challenging and I sort of, you know, I asked myself, like whether I feel like at some point I could want to be a leader like that with employees. But quite frankly, I don't feel ready. You know, maybe in another season of life, I will be but at this point, you know, yeah.Kevin Sahin  45:25  Yeah. I mean, I, as I say, I totally relate to this, because it's, I mean, for me personally, I don't I don't think I totally agree with you with the fact that there is this assumption of growth and hiring and, and even sometimes raising funds, like, you have to you have to grow, you have to raise fund you have to hire, it's kind of, you know, a vanity metric in the startup ecosystem, how many employees do you have? To try etc. And, I mean, many companies that I mean, either don't hire at all or hire just, you know, a really small team, and that are doing totally fine, where the founders are happy, the employees are happy, everyone's happy. And, yeah, it's. And on the other side, there are many companies raising funds, hiring, and growing like crazy, whether founders are not happy at all, and stressed andMichele Hansen  46:43  yeah, I think, you know, that's something we, as founders, we have the decision to run our businesses in a way that, you know, to design the business. Right. And, and, you know, and for me, part of, you know, designing that business is it's, you know, setting it up in a way that, that we're running it in a way that we enjoy, and we enjoy working together. And it sounds like you and I really like working together, too.Kevin Sahin  47:12  Yeah. I mean, we've been, we've been, so we know each other since high school. So we, we've been working on many project, back in high school, and then side projects in college and the beginning of our career together. So yeah, it's been. And that's was the, it was great, because when we founded the company, we had this whole history of working together, of knowing how to talk to each other to, you know, divide the work based on, you know, what we are good at what we'd like to do, etc, etc. So it was pretty, I'd say, you know, a fluid, the work relationship.Michele Hansen  48:09  Sounds like you learned a lot from that that first side project you did together with him about how you can work together. I'm curious what that project was.Kevin Sahin  48:19  There were many projects, I'd say the most. The biggest one with a Chrome extension that we launched. I don't remember the year 2016, I'd say or 17. It was called shop tourist, it was a Chrome extension that could where users could save products on ecommerce websites that they were interested to buy. And our we had some scrapers in the backend that would refresh the price every day. And if the price dropped it send an email with a note with the with an alert that said, Hey, this product dropped 25% this night. You can buy it here. And then there was some affiliated links on the email. And like, we, we had some pretty good success marketing it on Reddit. Like we launched the we posted a Reddit post one day and it got 1000s of upvotes. And we like to overnight we got a few 1000 users on the app. And yeah, and the funny thing is that we realized Is that some customers? No, it was not customers, some users sorry. were added adding hundreds of products on their list. And we, we told ourselves, it's kind of strange, because why would I mean, unless it's, you know, the person is on the buying spree or is a has a buying problem. It's kind of weird to save, you know, hundreds of products with different variations of the same. I don't know, a T shirt or whatever. And so we realized that it was ecommerce owners that were monitoring their competitors, with our app, and they were doing it because our app was free. There were some b2b SaaS that were doing it, but it was very expensive. And so we saw an opportunity there. And we launched our first real company, pricing, but and it was a price monitoring app for ecommerce owner. And we did this in 2018. And it was a failure, we managed to get it from zero to 500, or 1000, in monthly recurring revenue. But we failed to grow it from there. And we knew nothing about marketing to ecommerce owners, or to ecommerce in general, except the previous experience we had with this little side project. And so we, we managed to sell it to one of the biggest player in this field, which which is priced to spy.com. And it's funded, what would become scraping be later. And the great thing about this failure is that with pricing, but we we had to scrape a lot of websites. So no, we had these those problems about JavaScript rendering, headless browsers, proxies, etc. So we like, we knew exactly that one, like this one kind of use case for scraping me.Michele Hansen  52:48  So interesting. And I feel like I hear so many similarities in our stories, but something that stands out to me not only how you were, you were able, you know that so that pricing bot, you know, ostensibly failed. But you were able to carry through that expertise you built in building scrapers, and understanding how difficult that can be and the problems with that. But what also carried through is I'm struck by how it seems you have this curiosity about user behavior. And you know, people were doing something and you're and you're like, Oh, that's interesting, why are they adding hundreds of products all of a sudden, and you allowed yourself to follow that, and I think that's such, like, such a great quality, and a founder to not only notice when something is strange, you know, but but follow it, you know, you could have shut your brain off that like, Oh, these people probably just have a spending problem and basically judge, right? And you could have just sort of left it at that. But instead of stopping at judgment, you instead be like, I wonder why they're doing it and follow that thread, you know, follow this sort of cookie crumbs and figure it out. Oh, it's because they're doing this ecommerce thing. Okay, well, maybe we can like pivot into doing that and then it didn't really work out but you got acquired and then you're able to use that funds to start scraping be but you had that understanding of your own use cases for scraping. And again, you were like, Why do people need this? Let me go figure it out. And you just allow yourself to follow that curiosity. And I I just love that.Kevin Sahin  54:33  Yeah, I mean, that was um, it was really a great experience. I mean, the the like, even though it was hard, you know, to fail, and both p&i we didn't. Like we had to fund the business ourselves. So it was a very hard Financially but the experience the learnings were really worth it.Michele Hansen  55:06  Yeah. It sounds like it. I feel like I could talk to you all day about this. This has been so much fun. Um, thank you so much for for coming on. I I know from this conversation that this is not going to be the last time I talked to you. So So this has been really enjoyable.Kevin Sahin  55:33  Thank you. Yeah, same for me. Thank you a lot. And maybe see you next time. I still have many questions around the geo coder, yo. And I'd like to, I'd love to talk more about it.Michele Hansen  55:52  Yeah. Hey, I'm always always happy to talk about your cardio. Cool. So if people want to know more about you keep up with what you're doing on Skype and BMI and whatnot. Where should they go?Kevin Sahin  56:03  They can go to my Twitter. It's @SahinKevin. And yeah.Michele Hansen  56:12  Awesome. Well, if you enjoyed listening to this episode, please like Kevin, and I know. And you can find us on Twitter at @softwaresocpod. Thanks. Thanks, Michele.

CHAOSScast
Episode 46: Social Science Theories with Erin Staples

CHAOSScast

Play Episode Listen Later Oct 22, 2021 36:18


Hello and welcome to CHAOSScast Community podcast, where we share use cases and experiences with measuring open source community health. Elevating conversations about metrics, analytics, and software from the Community Health Analytics Open Source Software, or short CHAOSS Project, to wherever you like to listen. Today, our guest is Erin Staples, who works as a Community Advocate at Orbit. She is with us to talk about social science theories and what we can learn from other communities. Erin tells us the importance of making sure your contributors feel valued, creating a very inclusive, mindful environment online, and she explains how we can learn a lot from how Fandom communities measure health. She goes in depth about behaviors at gatherings such as conferences and she shares advice in creating online spaces. Download this episode now to find out much more, and don't forget to subscribe for free to this podcast on your favorite podcast app and share this podcast with your friends and colleagues. [00:02:00] Erin fills us in a little on her background and about what they do at Orbit with building a healthy community in the online space. [00:03:38] How did Erin get so interested in this topic? [00:05:33] For the social science conversation and Fandom, Erin talks about how she started to explore this huge topic. She tells us about a journal article she loves from Rachel Winter, Anastasia Salter, and Mel Stanfill who wrote about the “Communities of making: Exploring parallels between Fandom and open source.” [00:09:02] Erin explains more about the behaviors and how they happen at gatherings and in the Fandom world. [00:13:30] Georg brings up how open source is changing and has changed over the years with more organizations getting involved in creation of software and paying employees to be in these communities and Erin shares her thoughts about how this may be changing the dynamic. The Founder of Linux, Linus Torvalds, comes up in conversation as well. [00:19:47] Venia tells us about a website called Budget Light Forum and Erin talks about “the medium is the message,” which is a quote from Marshall McLuhan and how this relates to the way we think about online spaces and how we transmit information. [00:24:44] Georg brings up a great point if you want to understand the community you actually have to talk to the community members and ask them how that makes them feel, if they feel welcome and included, etc., and Erin and Venia share their thoughts on this. [00:28:11] As more people are working online, maintainer burnout in open source is discussed, which existed before COVID, with pressure to maintain the quality of code and for being responsive and they're not feeling appreciated. [00:30:41] Erin talks about some action steps to creating online spaces and shares an example of the Dunning-Kruger effect. [00:32:04] Find out where you can follow Erin online. Adds (Picks) of the week: * [00:32:46] Georg's pick is re-reading the Eragon series in English. * [00:33:38] Venia's pick is a book called Systematic Methods for Analyzing Culture: A Practical Guide. * [00:34:22] Erin's pick is a book called A City is Not a Computer: Other Urban Intelligences. Panelists: * Georg Link * Venia Logan Guest: * Erin Staples Sponsor: * SustainOSS (https://sustainoss.org/) Links: * CHAOSS (https://chaoss.community/) * CHAOSS Project Twitter (https://twitter.com/chaossproj?lang=en) * CHAOSScast Podcast (https://podcast.chaoss.community/) * podcast@chaoss.community (mailto:podcast@chaoss.community) * Erin Staples Twitter (https://twitter.com/erinmikail) * Erin Staples Website (https://blog.erinmikailstaples.com/home/) * Erin Staples Linkedin (https://www.linkedin.com/in/erinmikail) * Orbit (https://orbit.love/) * Fandom (https://www.fandom.com/) * Communities of making: Exploring parallels between fandom and open source by Rachel Winter, Anastasia Salter, and Mel Stanfill (https://journals.uic.edu/ojs/index.php/fm/article/view/10870/10056) * Fans, at their core, are producers. What does this tell us about the ethics of fan labor?- Fandom Communties 002 (https://blog.erinmikailstaples.com/fans-at-their-core-are-producers-what-does-this-tells-us-about-the-ethics-of-fan-labor/) * Budget Light Forum (https://budgetlightforum.com/) * Become a Tea Duellist By Austin Sirkin (Steampunk R&D) (https://steampunk.wonderhowto.com/how-to/become-tea-duellist-0140892/) * Herbert Marshall McLuhan (Wikipedia) (https://en.wikipedia.org/wiki/Marshall_McLuhan) * Margaret Mead (Wikipedia) (https://en.wikipedia.org/wiki/Margaret_Mead) * Dunning-Kruger effect (Psychology Today) (https://www.psychologytoday.com/us/basics/dunning-kruger-effect) * Ted 2016: Linux founder not a ‘people person' By Jane Wakefield (BBC News) (https://www.bbc.com/news/technology-35599774) * Linus Torvalds apologizes for his behavior, takes time off (Hacker News) (https://news.ycombinator.com/item?id=18000698) * [The Inheritance Cycle Series 4 Book Collection Eragon, Eldestk, Brisngr Box set by Christoper Paolini](https://www.amazon.com/Inheritance-4-Book-Paperback-Eragon-Brisingr/dp/0449813223/ref=sr13?crid=5TIVV6TC74OQ&dchild=1&keywords=eragon+book+series&qid=1634154783&sprefix=eragon%2Caps%2C181&sr=8-3) * [Systematic Methods for Analyzing Culture: A Practical Guide by H.J. François ](https://www.amazon.com/Systematic-Methods-Analyzing-Culture-Practical/dp/0367551519/ref=sr11?dchild=1&keywords=systematic+methods+for+analyzing+culture&qid=1634155245&sr=8-1) * [Dengah II, Jeffrey Snodgrass, Evan R. Polzer, William Cody Nixon](https://www.amazon.com/Systematic-Methods-Analyzing-Culture-Practical/dp/0367551519/ref=sr11?dchild=1&keywords=systematic+methods+for+analyzing+culture&qid=1634155245&sr=8-1) * [A City is Not a Computer: Other Urban Intelligences by Shannon Mattern](https://www.amazon.com/City-Not-Computer-Intelligences-Places/dp/0691208050/ref=sr11?crid=8AC8JRJ020NJ&dchild=1&keywords=a+city+is+not+a+computer&qid=1634155914&sprefix=A+city+is+not+a+computer%2Caps%2C173&sr=8-1) * The Sims: A Retrospective, A Participatory Culture 14 Years On by Ludovica Price (Intensive: Cult Media Review) (https://intensitiescultmedia.files.wordpress.com/2014/08/5-price-the-sims2.pdf) Special Guest: Erin Staples.

Sustain
Episode 95: Marko Saric of Plausible Analytics, the most popular Open Source analytics platform

Sustain

Play Episode Listen Later Oct 18, 2021 43:26


Guest Marko Saric Panelists Eric Berry | Justin Dorfman | Richard Littauer Show Notes Hello and welcome to Sustain! The podcast where we talk about sustaining open source for the long haul. We hope you are as excited as we are to have as our guest today Marko Saric, who is the Co-Founder of Plausible Analytics, which is an open source and privacy friendly alternative to Google Analytics. If you've never heard about Plausible Analytics, then this is your episode to learn all about it. With over 4,000 subscribers in the past year, Marko tells us what they've done to get people to convert. He also gives us his perspective on how he sees the business surviving in the next ten years, what his future game plan is, and why it's so important that Plausible Analytics is open source. Download this episode now to learn so much more from Marko! [00:01:33] Marko tells us what he does as one of the Co-Founders, how long Plausible Analytics has been around, and how many subscribers they have. [00:03:57] Justin asks Marko how he handles the bots and how much of a threat are they in terms of making sure that they don't mess up someone's expectations in terms of traffic. [00:06:15] We find out how Justin found Marko which was from a blog post he wrote and Justin wonders how this issue has converted people that are so Google dependent in terms of Google Analytics to turn over to a paid service like this, and how the shift has been since he was brought on board. [00:10:25] Eric wonders what's to prevent developers from adding blockers to this system and is there a reason why they would or would not. [00:17:59] Marko tells us how he sees his business surviving in the next ten years, and if he sees any big plans that he is trying to push to make it so there is that harmony between advertisers and the consumers. [00:24:12] Richard wonders what Marko's game plan in twenty-five years, where he wants to go in the future, and how to build a more sustainable web for everyone. [00:27:46] Does Marko see Plausible Analytics staying independent or possibly joining a company? [00:30:40] Justin shares a conspiracy theory about what he thinks Brave is doing to Plausible Analytics and Marko shares his thoughts. [00:32:59] Richard asks Marko why it's important that Plausible is open source. [00:35:29] Marko tells us if he's worried about people taking the code and just running another “Pausable” Analytics as a fork. Quotes [00:13:14] “My thinking is let's try to make the devs better by getting website owners to use better tools for people that use ad blockers - the fact is still that most people don't use ad blockers.” [00:15:01] “There's a huge disconnect between people, like all of us here in the chat and the more kind of normal dev user.” [00:22:04] “If you actually give your vote and say no, or no to this and yes to that, you're actually voting to make a change.” [00:22:14] “That's one of the main Key Performance Indicators these days in companies is how many people are saying yes or no to that little banner we have on our sites.” [00:22:23] “I'm going to take my three seconds to click on options and then scroll down and click on reject because I know that it makes a difference.” [00:24:35] “Yeah, I mean GDPR was a great first step and I think if there can be something similar, but actually just going off to the personal data.” [00:24:48] “Many websites that I visit, the newspapers and so on, they will live from the ads.” [00:25:00] “I understand that there is a need for ads while that is the main monetization method of the web.” [00:26:15] “A few weeks ago, Ethical Ads installed Plausible and they wrote a blog post about it and I was like, “Perfect!” [00:27:19] “You can find people doing studies on their own website, and like personal ads versus contextual ads, they're seeing no difference in terms of effectiveness or in the kind of income they get or the conversion rate or whatever.” [00:27:34] “You can actually do good business, both as a publisher but also as an advertiser, just by talking to people contextually or whatever other way they can find out that's not really necessary as part of surveillance capitalism.” [00:28:45] “We just do our own thing and try to kind of do our own little sustainable business.” [00:33:19] “If you're not open source and you're talking about privacy first you will probably be excluded from the conversation. People will not take you serious.” [00:33:58] “And if you're proprietary, a lot of people with technical knowledge and people really deep into this would not trust us because we're just saying things. We don't know who you are. Why would we trust you?” [00:35:08] “I gotta trust that by being open source and having so many eyeballs on it at least if there some kind of sketchy going on or whatever, somebody will kind of flag it.” [00:35:40] “I was completely new to all this licensed system. I had no idea I was using WordPress and stuff.” [00:36:29] “And I was like, again, I was new to the open since I had no idea that this is how it can work, that they will just upfront come to us and tell us, we don't want to do anything to help you, but can you please do something so it helps us so we can kind of complete video and we have tens of thousands more of audience?” [00:37:47] “And we ended up with AGPL and we felt this was a great kind of license for our own situation.” [00:38:41] “Honestly from our perspective, like if we want to make this a thing that could become sustainable in the future, pay our own bills so we can focus on it full-time and then hopefully make a difference.” [00:39:32] “I know that my Co-Founder says that if you're doing like a database and things for developers, you probably want to be MIT because then other companies can use other projects. But I would say if you're coming from my perspective, as somebody who has to communicate the message and kind of differentiate ourselves and try to compete with what else is on the market, I was like, if you're going to sell to consumers and other businesses, like it's going to be really difficult to survive it in IT.” [00:39:57] “Again, as a beginner there are MIT licenses that have worked very well and they're sustainable, but I just don't know how I would compete with a bigger company.” Spotlight [00:40:31] Eric's spotlight is a newsletter he signed up for called Console.dev. [00:40:54] Justin's spotlight is a great read, “Developer, You May Need a Co-Founder in Marketing,” by Rauno Metsa, Microfounder of MicroFounder. [00:41:30] Richard's spotlight is Andre Greig, a Scottish poet and his book called, Getting Higher: The Complete Mountain Poems. [00:41:44] Marko's spotlights are Linux, Firefox, and WordPress. Links SustainOSS (https://sustainoss.org/) SustainOSS Twitter (https://twitter.com/SustainOSS?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor) SustainOSS Discourse (https://discourse.sustainoss.org/) Marko Saric Website (https://markosaric.com/) Marko Saric Twitter (https://twitter.com/markosaric) Plausible (https://plausible.io/) The Plausible Blog (https://plausible.io/blog) Ethical Ads Newsletter July 2021 (https://www.ethicalads.io/blog/2021/08/ethicalads-newsletter-july-2021/) “58% of Hacker News, Reddit and tech-savvy audiences block Google Analytics” by Marko Saric (https://plausible.io/blog/google-analytics-adblockers-missing-data) Console (https://console.dev/) “Developer, You May Need a Co-Founder in Marketing” by Rauno Metsa (https://microfounder.com/blog/cofounder-in-marketing) Getting Higher: The Complete Mountain Poems by Andrew Greig (https://www.amazon.com/Getting-Higher-Complete-Mountain-Poems/dp/1846971926) Linux (https://www.kernel.org/category/about.html) Mozilla Firefox (https://www.mozilla.org/en-US/firefox/new/) WordPress (https://wordpress.org/) Credits Produced by Richard Littauer (https://www.burntfen.com/) Edited by Paul M. Bahr at Peachtree Sound (https://www.peachtreesound.com/) Show notes by DeAnn Bahr at Peachtree Sound (https://www.peachtreesound.com/) Special Guest: Marko Saric.

Screaming in the Cloud
What GitHub Can Give to Microsoft with Jason Warner

Screaming in the Cloud

Play Episode Listen Later Oct 7, 2021 37:47


About JasonJason is now the Managing Director at Redpoint Ventures.Links: GitHub: https://github.com/ @jasoncwarner: https://twitter.com/jasoncwarner GitHub: https://github.com/jasoncwarner Jasoncwarner/ama: https://github.com/jasoncwarner/ama TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Jason Warner, the Chief Technology Officer at GifHub, although he pronounces it differently. Jason, welcome to the show.Jason: Thanks, Corey. Good to be here.Corey: So, GitHub—as you insist on pronouncing it—is one of those companies that's been around for a long time. In fact, I went to a training conducted by one of your early folks, Scott Chacon, who taught how Git works over the course of a couple of days, and honestly, I left more confused than I did when I entered. It's like, “Oh, this is super awful. Good thing I'll never need to know this because I'm not really a developer.” And I'm still not really a developer and I still don't really know how Git works, but here we are.And it's now over a decade later; you folks have been acquired by Microsoft, and you are sort of the one-stop-shop, from the de facto perspective of, “I'm going to go share some code with people on the internet. I'll use GitHub to do it.” Because, you know, copying and pasting and emailing Microsoft Word documents around isn't ideal.Jason: That is right. And I think that a bunch of things that you mentioned there, played into, you know, GitHub's early and sustained success. But my God, do you remember the old days when people had to email tar files around or drop them in weird spots?Corey: What the hell do you mean, by, “Old days?” It still blows my mind that the Linux kernel is managed by—they use Git, obviously. Linus Torvalds did write Git once upon a time—and it has the user interface you would expect for that. And the way that they collaborate is not through GitHub or anything like that. No, they use Git to generate patches, which they then email to the mailing list. Which sounds like I'm making it up, like, “Oh, well, yeah, tell another one, but maybe involve a fax machine this time.” But no, that is actually what they do.Jason: It blew my mind when I saw that, too, by the way. And you realize, too, that workflows are workflows, and people will build interesting workflows to solve their use case. Now, obviously, anyone that you would be talking to in 2021, if you walked in and said, “Yeah, install Git. Let's set up an email server and start mailing patches to each other and we're going to do it this way.” They would just kind of politely—or maybe impolitely—show you out of the room, and rightfully [laugh] so. But it works for one of the most important software projects in history: Linux.Corey: Yeah, and it works almost in spite of itself to some extent. You've come a long way as a company because initially, it was, “Oh, there's this amazing, decentralized version control system. How do we make it better? I know, we're going to take off the decentralized part of it and give it a central point that everything can go through.” And collaboratively, it works well, but I think that viewing GitHub as a system that is used to sell free Git repositories to people is rather dramatically missing the point. It feels like it's grown significantly beyond just code repository hosting. Tell me more about that.Jason: Absolutely. I remember talking to a bunch of folks right around when I was joining GitHub, and you know, there was still talk about GitHub as, you know, GitHub for lawyers, or GitHub for doctors, or what could you do in a different way? And you know, social coding as an aspect, and maybe turning into a social network with a resume. And all those things are true to a percentage standpoint. But what GitHub should be in the world is the world's most important software development platform, end-to-end software development platform.We obviously have grown a bunch since me joining in that way which we launched dependency management packages, Actions with built-in CI, we've got some deployment mechanisms, we got advanced security underneath it, we've Codespaces in beta and alpha on top of it now. But if you think about GitHub as, join, share, and see other people's code, that's evolution one. If you see it as world's largest, maybe most developed software development platform, that's evolution two, and in my mind, its natural place where it should be, given what it has done already in the world, is become the world's most important software company. I don't mean the most profitable. I just mean the most important.Corey: I would agree. I had a blog post that went up somewhat recently about the future of cloud being Microsoft's to lose. And it's not because Azure is the best cloud platform out there, with respect, and I don't need you to argue the point. It is very clearly not. It is not like other clouds, but I can see a path to where it could become far better than it is.But if I'm out there and I'm just learning how to write code—because I make terrible life choices—and I go to a boot camp or I follow a tutorial online or I take a course somewhere, I'm going to be writing code probably using VS Code, the open-source editor that you folks launched after the acquisition. And it was pretty clear that Atom wasn't quite where the world was going. Great. Then I'm going to host it on GitHub, which is a natural evolution. Then you take a look at things like GitHub Actions that build in CI/CD pipelines natively.All that's missing is a ‘Deploy to Azure' button that is the next logical step, and you're mostly there for an awful lot of use cases. But you can't add that button until Azure itself gets better. Done right, this has the potential to leave, effectively, every other cloud provider in the dust because no one can touch this.Jason: One hundred percent. I mean, the obvious thing that any other cloud should be looking at with us—or should have been before the acquisition, looking at us was, “Oh, no, they could jump over us. They could stop our funnel.” And I used internal metrics when I was talking to them about partnership that led to the sale, which was I showed them more about their running business than they knew about themselves. I can tell them where they were stacked-ranked against each other, based on the ingress and egress of all the data on GitHub, you know, and various reactions to that in those meetings was pretty astounding.And just with that data alone, it should tell you what GitHub would be capable of and what Azure would be capable of in the combination of those two things. I mean, you did mention the ‘Deploy to Azure' button; this has been a topic, obviously, pre and post-acquisition, which is, “When is that coming?” And it was the one hard rule I set during the acquisition was, there will be no ‘Deploy to Azure' button. Azure has to earn the right to get things deployed to, in my opinion. And I think that goes to what you're saying is, if we put a ‘Deploy to Azure' button on top of this and Azure is not ready for that, or is going to fail, ultimately, that looks bad for all of us. But if it earned the right and it gets better, and it becomes one of those, then, you know, people will choose it, and that is, to me, what we're after.Corey: You have to choose the moment because if you do it too soon, you'll set the entire initiative back five years. Do it too late, and you get leapfrogged. There's a golden window somewhere and finding it is going to be hard. And I think it's pretty clear that the other hyperscalers in this space are learning, or have learned, that the next 10 years of cloud or 15 years of cloud or whatever they want to call it, and the new customers that are going to come are not the same as the customers that have built the first half of the business. And they're trying to wrap their heads around that because a lot of where the growth is going to come from is established blue chips that are used to thinking in very enterprise terms.And people think I'm making fun of them when I say this, but Microsoft has 40 years' experience apologizing to enterprises for computer failures. And that is fundamentally what cloud is. It's about talking computers to business executives because as much as we talk about builders, that is not the person at an established company with an existing IT estate, who gets to determine where $50 million a year in cloud-spend is going to go.Jason: It's [laugh] very, [laugh] very true. I mean, we've entered a different spot with cloud computing in the bell curve of adoption, and if you think that they will choose the best technology every time, well, history of computing is littered with better technologies that have failed because the distribution was better on one side. As you mentioned, Microsoft has 40 years, and I wager that Microsoft has the best sales organizations and the best enterprise accounts and, you know, all that sort of stuff, blah, blah, blah, on that side of the world than anyone in the industry. They can sell to enterprises better than almost anyone in the industry. And the other hyperscalers—there's a reason why [TK 00:08:34] is running Google Cloud right now. And Amazon, classically, has been very, very bad assigned to the enterprises. They just happened to be the first mover.Corey: In the early days, it was easy. You'd have an Amazon salesperson roll up to a company, and the exec would say, “Great, why should we consider running things on AWS?” And the answer was, “Oh, I'm sorry, wrong conversation. Right now you have 80 different accounts scattered throughout your org. I'm just here to help you unify them, get some visibility into it, and possibly give you a discount along the way.” And it was a different conversation. Shadow IT was the sole driver of cloud adoption for a long time. That is no longer true. It has to go in the front door, and that is a fundamental shift in how you go to market.Jason: One hundred percent true, and it's why I think that Microsoft has been so successful with Azure, in the last, let's call it five years in that, is that the early adopters in the second wave are doing that; they're all enterprise IT, enterprise dev shops who are buying from the top down. Now, there is still the bottoms-up adoption that going to be happening, and obviously, bottom-up adoption will happen still going forward, but we've entered the phase where that's not the primary or sole mechanism I should say. The sole mechanism of buying in. We have tops-down selling still—or now.Corey: When Microsoft announced it was acquiring GitHub, there was a universal reaction of, “Oh, shit.” Because it's Microsoft; of course they're going to ruin GitHub. Is there a second option? No, unless they find a way to ruin it twice. And none of it came to pass.It is uniformly excellent, and there's a strong argument that could be made by folks who are unaware of what happened—I'm one of them, so maybe I'm right, maybe I'm wrong—that GitHub had a positive effect on Microsoft more than Microsoft had an effect on GitHub. I don't know if that's true or not, but I could believe it based upon what I've seen.Jason: Obviously, the skepticism was well deserved at the time of acquisition, let's just be honest with it, particularly given what Microsoft's history had been for about 15—well, 20 years before, previous to Satya joining. And I was one of those people in the late '90s who would write ‘M$' in various forums. I was 18 or 19 years old, and just got into—Corey: Oh, hating Microsoft was my entire personality.Jason: [laugh]. And it was, honestly, well-deserved, right? Like, they had anti-competitive practices and they did some nefarious things. And you know, I talked about Bill Gates as an example. Bill Gates is, I mean, I don't actually know how old he is, but I'm going to guess he's late '50s, early '60s, but he's basically in the redemption phase of his life for his early years.And Microsoft is making up for Ballmer years, and later Gates years, and things of that nature. So, it was well-deserved skepticism, and particularly for a mid-career to older-career crowd who have really grown to hate Microsoft over that time. But what I would say is, obviously, it's different under Satya, and Scott, and Amy Hood, and people like that. And all we really telling people is give us a chance on this one. And I mean, all of us. The people who were running GitHub at the time, including myself and, you know, let Scott and Satya prove that they are who they say they are.Corey: It's one of those things where there's nothing you could have said that would have changed the opinion of the world. It was, just wait and see. And I think we have. It's now, I daresay, gotten to a point where Microsoft announces that they're acquiring some other beloved company, then people, I think, would extend a lot more credit than they did back then.Jason: I have to give Microsoft a ton of credit, too, on this one for the way in which they handled acquisitions, like us and others. And the reason why I think it's been so successful is also the reason why I think so many others die post-acquisition, which is that Microsoft has basically—I'll say this, and I know I won't get fired because it feels like it's true. Microsoft is essentially a PE holding company at this point. It is acquired a whole bunch of companies and lets them run independent. You know, we got LinkedIn, you got Minecraft, Xbox is its own division, but it's effectively its own company inside of it.Azure is run that way. GitHub's got a CEO still. I call it the archipelago model. Microsoft's the landmass underneath the water that binds them all, and finance, and HR, and a couple of other things, but for the most part, we manifest our own product roadmap still. We're not told what to go do. And I think that's why it's successful. If we're going to functionally integrate GitHub into Microsoft, it would have died very quickly.Corey: You clearly don't mix the streams. I mean, your gaming division writes a lot of interesting games and a lot of interesting gaming platforms. And, like, one of the most popularly played puzzle games in the world is a Microsoft property, and that is, of course, logging into a Microsoft account correctly. And I keep waiting for that to bleed into GitHub, but it doesn't. GitHub is a terrific SAML provider, it is stupidly easy to log in, it's great.And at some level, I wish that would bleed into other aspects, but you can't have everything. Tell me what it's like to go through an acquisition from a C-level position. Because having been through an acquisition before, the process looks a lot like a surprise all-hands meeting one day after the markets close and, “Listen up, idiots.” And [laugh] there we go. I have to imagine with someone in your position, it's a slightly different experience.Jason: It's definitely very different for all C-levels. And then myself in particular, as the primary driver of the acquisition, obviously, I had very privy inside knowledge. And so, from my position, I knew what was happening the entire time as the primary driver from the inside. But even so, it's still disconcerting to a degree because, in many ways, you don't think you're going to be able to pull it off. Like, you know, I remember the months, and the nights, and the weekends, and the weekend nights, and all the weeks I spent on the road trying to get all the puzzle pieces lined up for the Googles, or the Microsofts, or the eventually AWSs, the VMwares, the IBMs of the world to take seriously, just from a product perspective, which I knew would lead to, obviously, acquisition conversations.And then, once you get the call from the board that says, “It's done. We signed the letter of intent,” you basically are like, “Oh. Oh, crap. Okay, hang on a second. I actually didn't—I don't actually believe in my heart of hearts that I was going to actually be able to pull that off.” And so now, you probably didn't plan out—or at least I didn't. I was like, “Shit if we actually pulled this off what comes next?” And I didn't have that what comes next, which is odd for me. I usually have some sort of a loose plan in place. I just didn't. I wasn't really ready for that.Corey: It's got to be a weird discussion, too, when you start looking at shopping a company around to be sold, especially one at the scale of GitHub because you're at such a high level of visibility in the entire environment, where—it's the idea of would anyone even want to buy us? And then, duh, of course they would. And you look the hyperscalers, for example. You have, well, you could sell it to Amazon and they could pull another Cloud9, where they shove it behind the IAM login process, fail to update the thing meaningfully over a period of years, to a point where even now, a significant portion of the audience listening to this is going to wonder if it's a service I just made up; it sounds like something they might have done, but Cloud9 sounds way too inspired for an AWS service name, so maybe not. And—which it is real. You could go sell to Google, which is going to be awesome until some executive changes roles, and then it's going to be deprecated in short order.Or then there's Microsoft, which is the wild card. It's, well, it's Microsoft. I mean, people aren't really excited about it, but okay. And I don't think that's true anymore at all. And maybe I'm not being fair to all the hyperscalers there. I mean, I'm basically insulting everyone, which is kind of my shtick, but it really does seem that Microsoft was far and away the best acquirer possible because it has been transformative. My question—if you can answer it—is, how the hell did you see that beforehand? It's only obvious—even knowing what I know now—in hindsight.Jason: So, Microsoft was a target for me going into it, and the reason why was I thought that they were in the best overall position. There was enough humility on one side, enough hubris on another, enough market awareness, probably, organizational awareness to, kind of, pull it off. There's too much hubris on one side of the fence with some of the other acquirers, and they would try to hug us too deeply, or integrate us too quickly, or things of that nature. And I think it just takes a deep understanding of who the players are and who the egos involved are. And I think egos has actually played more into acquisitions than people will ever admit.What I saw was, based on the initial partnership conversations, we were developing something that we never launched before GitHub Actions called GitHub Launch. The primary reason we were building that was GitHub launches a five, six-year journey, and it's got many, many different phases, which will keep launching over the next couple of years. The first one we never brought to market was a partnership between all of the clouds. And it served a specific purpose. One, it allowed me to get into the room with the highest level executive at every one of those companies.Two allow me to have a deep economic conversation with them at a partnership level. And three, it allowed me to show those executives that we knew what GitHub's value was in the world, and really flip the tables around and say, “We know what we're worth. We know what our value is in the world. We know where we sit from a product influence perspective. If you want to be part of this, we'll allow it.” Not, “Please come work with us.” It was more of a, “We'll allow you to be part of this conversation.”And I wanted to see how people reacted to that. You know how Amazon reacted that told me a lot about how they view the world, and how Google reacted to that showed me exactly where they viewed it. And I remember walking out of the Google conversation, feeling a very specific way based upon the reaction. And you know, when I talked to Microsoft, got a very different feel and it, kind of, confirmed a couple of things. And then when I had my very first conversation with Nat, who have known for a while before that, I realized, like, yep, okay, this is the one. Drive hard at this.Corey: If you could do it all again, would you change anything meaningful about how you approached it?Jason: You know, I think I got very lucky doing a couple of things. I was very intentional aspects of—you know, I tried to serendipitously show up, where Diane Greene was at one point, or a serendipitously show up where Satya or Scott Guthrie was, and obviously, that was all intentional. But I never sold a company like this before. The partnership and the product that we were building was obviously very intentional. I think if I were to go through the sale, again, I would probably have tried to orchestrate at least one more year independent.And it's not—for no other reason alone than what we were building was very special. And the world sees it now, but I wish that the people who built it inside GitHub got full credit for it. And I think that part of that credit gets diffused to saying, “Microsoft fixed GitHub,” and I want the people inside GitHub to have gotten a lot more of that credit. Microsoft obviously made us much better, but that was not specific to Microsoft because we're run independent; it was bringing Nat in and helping us that got a lot of that stuff done. Nat did a great job at those things. But a lot of that was already in play with some incredible engineers, product people, and in particular our sales team and finance team inside of GitHub already.Corey: When you take a look across the landscape of the fact that GitHub has become for a certain subset of relatively sad types of which I'm definitely one a household name, what do you think the biggest misconception about the company is?Jason: I still think the biggest misconception of us is that we're a code host. Every time I talk to the RedMonk folks, they get what we're building and what we're trying to be in the world, but people still think of us as SourceForge-plus-plus in many ways. And obviously, that may have been our past, but that's definitely not where we are now and, for certain, obviously, not our future. So, I think that's one. I do think that people still, to this day, think of GitLab as one of our main competitors, and I never have ever saw GitLab as a competitor.I think it just has an unfortunate naming convention, as well as, you know, PRs, and MRs, and Git and all that sort of stuff. But we take very different views of the world in how we're approaching things. And then maybe the last thing would be that what we're doing at the scale that we're doing it as is kind of easy. When I think that—you know, when you're serving almost every developer in the world at this point at the scale at which we're doing it, we've got some scale issues that people just probably will never thankfully encounter for themselves.Corey: Well, everyone on Hacker News believes that they will, as soon as they put up their hello world blog, so Kubernetes is the only way to do anything now. So, I'm told.Jason: It's quite interesting because I think that everything breaks at scale, as we all know about from the [hyperclouds 00:20:54]. As we've learned, things are breaking every day. And I think that when you get advice, either operational, technical, or managerial advice from people who are running 10 person, 50 person companies, or X-size sophisticated systems, it doesn't apply. But for whatever reason, I don't know why, but people feel inclined to give that feedback to engineers at GitHub directly, saying, “If you just…” and in many [laugh] ways, you're just like, “Well, I think that we'll have that conversation at some point, you know, but we got a 100-plus-million repos and 65 million developers using us on a daily basis.” It's a very different world.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: One of the things that I really appreciate personally because, you know, when you see something that company does, it's nice to just thank people from time to time, so I'm inviting the entire company on the podcast one by one, at some point, to wind up thanking them all individually for it, but Codespaces is one of those things that I think is transformative for me. Back in the before times, and ideally the after times, whenever I travel the only computer I brought with me for a few years now has been an iPad or an iPad Pro. And trying to get an editor on that thing that works reasonably well has been like pulling teeth, my default answer has just been to remote into an EC2 instance and use vim like I have for the last 20 years. But Code is really winning me over. Having to play with code-server and other things like that for a while was obnoxious, fraught, and difficult.And finally, we got to a point where Codespaces was launched, and oh, it works on an iPad. This is actually really slick. I like this. And it was the thing that I was looking for but was trying to have to monkey patch together myself from components. And that's transformative.It feels like we're going back in many ways—at least in my model—to the days of thin clients where all the heavy lifting was done centrally on big computers, and the things that sat on people's desks were mostly just, effectively, relatively simple keyboard, mouse, screen. Things go back and forth and I'm sure we'll have super powerful things in our pockets again soon, but I like the interaction model; it solves for an awful lot of problems and that's one of the things that, at least from my perspective, that the world may not have fully wrapped it head around yet.Jason: Great observation. Before the acquisition, we were experimenting with a couple of different editors, that we wanted to do online editors. And same thing; we were experimenting with some Action CI stuff, and it just didn't make sense for us to build it; it would have been too hard, there have been too many moving parts, and then post-acquisition, we really love what the VS Code team was building over there, and you could see it; it was just going to work. And we had this one person, well, not one person. There was a bunch of people inside of GitHub that do this, but this one person at the highest level who's just obsessed with make this work on my iPad.He's the head of product design, his name's Max, he's an ex-Heroku person as well, and he was just obsessed with it. And he said, “If it works on my iPad, it's got a chance to succeed. If it doesn't work on my iPad, I'm never going to use this thing.” And the first time we booted up Codespaces—or he booted it up on the weekend, working on it. Came back and just, “Yep. This is going to be the one. Now, we got to work on those, the sanding the stones and those fine edges and stuff.”But it really does unlock a lot for us because, you know, again, we want to become the software developer platform for everyone in the world, you got to go end-to-end, and you got to have an opinion on certain things, and you got to enable certain functionality. You mentioned Cloud9 before with Amazon. It was one of the most confounding acquisitions I've ever seen. When they bought it I was at Heroku and I thought, I thought at that moment that Amazon was going to own the next 50 years of development because I thought they saw the same thing a lot of us at Heroku saw, and with the Cloud9 acquisition, what they were going to do was just going to stomp on all of us in the space. And then when it didn't happen, we just thought maybe, you know, okay, maybe something else changed. Maybe we were wrong about that assumption, too. But I think that we're on to it still. I think that it just has to do with the way you approach it and, you know, how you design it.Corey: Sorry, you just said something that took me aback for a second. Wait, you mean software can be designed? It's not this emergent property of people building thing on top of thing? There's actually a grand plan behind all these things? I've only half kidding, on some level, where if you take a look at any modern software product that is deployed into the world, it seems impossible for even small aspects of it to have been part of the initial founding design. But as a counterargument, it would almost have to be for a lot of these things. How do you square that circle?Jason: I think you have to, just like anything on spectrums and timelines, you have to flex at various times for various things. So, if you think about it from a very, very simple construct of time, you just have to think of time horizons. So, I have an opinion about what GitHub should look like in 10 years—vaguely—in five years much more firmly, and then very, very concretely, for the next year, as an example. So, a lot of the features you might see might be more emergent, but a lot of long-term work togetherness has to be loosely tied together with some string. Now, that string will be tightened over time, but it loosely has to see its way through.And the way I describe this to folks is that you don't wake up one day and say, “I'm going on vacation,” and literally just throw a finger on the map. You have to have some sort of vague idea, like, “Hey, I want to have a beach vacation,” or, “I want to have an adventure vacation.” And then you can kind of pick a destination and say, “I'm going to Hawaii,” or, “I'm going to San Diego.” And if you're standing on the East Coast knowing you're going to San Diego, you basically know that you have to just start marching west, or driving west, or whatever. And now, you don't have to have the route mapped out just yet, but you know that hey, if I'm going due southeast, I'm off course, so how do I reorient to make sure I'm still going in the right direction?That's basically what I think about as high-level, as scale design. And it's not unfair to say that a lot of the stuff is not designed today. Amazon is very famous for not designing anything; they design a singular service. But there's no cohesiveness to what Amazon—or AWS specifically, I should say, in this case—has put out there. And maybe that's not what their strategy is. I don't know the internal workings of them, but it's very clear.Corey: Well, oh, yeah. When I first started working in the AWS space and looking through the console, it like, “What is this? It feels like every service's interface was designed by a different team, but that would—oh…” and then the light bulb went on. Yeah. You ship your culture.Jason: It's exactly it. It works for them, but I think if you're going to try to do something very, very, very different, you know, it's going to look a certain way. So, intentional design, I think, is part of what makes GitHub and other products like it special. And if you think about it, you have to have an end-to-end view, and then you can build verticals up and down inside of that. But it has to work on the horizontal, still.And then if you hire really smart people to build the verticals, you get those done. So, a good example of this is that I have a very strong opinion about the horizontal workflow nature of GitHub should look like in five years. I have a very loose opinion about what the matrix build system of Actions looks like. Because we have very, very smart people who are working on that specific problem, so long as that maps back and snaps into the horizontal workflows. And that's how it can work together.Corey: So, when you look at someone who is, I don't know, the CTO of a wildly renowned company that is basically still catering primarily to developers slash engineers, but let's be honest, geeks, it's natural to think that, oh, they must be the alpha geek. That doesn't really apply to you from everything I've been able to uncover. Am I just not digging deeply enough, or are you in fact, a terrible nerd?Jason: [laugh]. I am. I'm a terrible nerd. I am a very terrible nerd. I feel very lucky, obviously, to be in the position I'm in right now, in many ways, and people call me up and exactly that.It's like, “Hey, you must be king of the geeks.” And I'm like, “[laugh], ah, funny story here.” But um, you know, I joke that I'm not actually supposed to be in tech in first place, the way I grew up, and where I did, and how, I wasn't supposed to be here. And so, it's serendipitous that I am in tech. And then turns out I had an aptitude for distributed systems, and complex, you know, human systems as well. But when people dig in and they start talking about topics, I'm confounded. I never liked Star Wars, I never like Star Trek. Never got an anime, board games, I don't play video games—Corey: You are going to get letters.Jason: [laugh]. When I was at Canonical, oh, my goodness, the stuff I tried to hide about myself, and, like, learn, like, so who's this Boba Fett dude. And, you know, at some point, obviously, you don't have to pretend anymore, but you know, people still assume a bunch stuff because, quote, “Nerd” quote, “Geek” culture type of stuff. But you know, some interesting facts that people end up being surprised by with me is that, you know, I was very short in high school and I grew in college, so I decided that I wanted to take advantage of my newfound height and athleticism as you grow into your body. So, I started playing basketball, but I obsessed over it.I love getting good at something. So, I'd wake up at four o'clock in the morning, and go shoot baskets, and do drills for hours. Well, I got really good at it one point, and I end up playing in a Pro-Am basketball game with ex-NBA Harlem Globetrotter legends. And that's just not something you hear about in most engineering circles. You might expect that out of a salesperson or a marketing person who played pro ball—or amateur ball somewhere, or college ball or something like that. But not someone who ends up running the most important software company—from a technical perspective—in the world.Corey: It's weird. People counterintuitively think that, on some level, that code is the answer to all things. And that, oh, all this human interaction stuff, all the discussions, all the systems thinking, you have to fit a certain profile to do that, and anyone outside of that is, eh, they're not as valuable. They can get ignored. And we see that manifesting itself in different ways.And even if we take a look at people whose profess otherwise, we take a look at folks who are fresh out of a boot camp and don't understand much about the business world yet; they have transformed their lives—maybe they're fresh out of college, maybe didn't even go to college—and 18 weeks later, they are signing up for six-figure jobs. Meanwhile, you take a look at virtually any other business function, in order to have a relatively comparable degree of earning potential, it takes years of experience and being very focused on a whole bunch of other things. There's a massive distortion around technical roles, and that's a strange and difficult thing to wrap my head around. But as you're talking about it, it goes both ways, too. It's the idea of, “Oh, I'll become technical than branch into other things.” It sounded like you started off instead with a non-technical direction and then sort of adopted that from other sides. Is that right, or am I misremembering exactly how the story unfolds?Jason: No, that's about right. People say, “Hey, when did I start programming?” And it's very in vogue, I think, for a lot of people to say, “I started programming at three years old,” or five years old, or whatever, and got my first computer. I literally didn't get my first computer until I was 18-years-old. And I started programming when I got to a high school co-op with IBM at 17.It was Lotus Notes programming at the time. Had no exposure to it before. What I did, though, in college was IBM told me at the time, they said, “If you get a computer science degree will guarantee you a job.” Which for a kid who grew up the way I grew up, that is manna from heaven type of deal. Like, “You'll guarantee me a job inside where don't have to dig ditches all day or lay asphalt? Oh, my goodness. What's computer science? I'll go figure it out.”And when I got to school, what I realized was I was really far behind. Everyone was that ubergeek type of thing. So, what I did is I tried to hack the system, and what I said was, “What is a topic that nobody else has an advantage on from me?” And so I basically picked the internet because the internet was so new in the mid-'90s that most people were still not fully up to speed on it. And then the underpinnings in the internet, which basically become distributed systems, that's where I started to focus.And because no one had a real advantage, I just, you know, could catch up pretty quickly. But once I got into computers, it turned out that I was probably a very average developer, maybe even below average, but it was the system's thinking that I stood out on. And you know, large-scale distributed systems or architectures were very good for me. And then, you know, that applies not, like, directly, but it applies decently well to human systems. It's just, you know, different types of inputs and outputs. But if you think about organizations at scale, they're barely just really, really, really complex and kind of irrational distributed systems.Corey: Jason, thank you so much for taking the time to speak with me today. If people want to learn more about who you are, what you're up to, how you think about the world, where can they find you?Jason: Twitter's probably the best place at this point. Just @jasoncwarner on Twitter. I'm very unimaginative. My name is my GitHub handle. It's my Twitter username. And that's the best place that I, kind of, interact with folks these days. I do an AMA on GitHub. So, if you ever want to ask me anything, just kind of go to jasoncwarner/ama on GitHub and drop a question in one of the issues and I'll get to answering that. Yeah, those are the best spots.Corey: And we will, of course, include links to those things in the [show notes 00:33:52]. Thank you so much for taking the time to speak with me today. I really appreciate it.Jason: Thanks, Corey. It's been fun.Corey: Jason Warner, Chief Technology Officer at GitHub. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review in your podcast platform of choice anyway, along with a comment that includes a patch.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Non-Incidentally Keeping Tabs on the Internet with Courtney Nash

Screaming in the Cloud

Play Episode Listen Later Oct 5, 2021 33:40


About CourtneyCourtney Nash is a researcher focused on system safety and failures in complex sociotechnical systems. An erstwhile cognitive neuroscientist, she has always been fascinated by how people learn, and the ways memory influences how they solve problems. Over the past two decades, she's held a variety of editorial, program management, research, and management roles at Holloway, Fastly, O'Reilly Media, Microsoft, and Amazon. She lives in the mountains where she skis, rides bikes, and herds dogs and kids.Links: Verica: https://www.verica.io Twitter: https://twitter.com/courtneynash Email: courtney@verica.io TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Jellyfish. So, you're sitting in front of your office chair, bleary eyed, parked in front of a powerpoint and—oh my sweet feathery Jesus its the night before the board meeting, because of course it is! As you slot that crappy screenshot of traffic light colored excel tables into your deck, or sift through endless spreadsheets looking for just the right data set, have you ever wondered, why is it that sales and marketing get all this shiny, awesome analytics and inside tools? Whereas, engineering basically gets left with the dregs. Well, the founders of Jellyfish certainly did. That's why they created the Jellyfish Engineering Management Platform, but don't you dare call it JEMP! Designed to make it simple to analyze your engineering organization, Jellyfish ingests signals from your tech stack. Including JIRA, Git, and collaborative tools. Yes, depressing to think of those things as your tech stack but this is 2021. They use that to create a model that accurately reflects just how the breakdown of engineering work aligns with your wider business objectives. In other words, it translates from code into spreadsheet. When you have to explain what you're doing from an engineering perspective to people whose primary IDE is Microsoft Powerpoint, consider Jellyfish. Thats Jellyfish.co and tell them Corey sent you! Watch for the wince, thats my favorite part.Corey: This episode is sponsored in part by our friends at VMware. Let's be honest—the past year has been far from easy. Due to, well, everything. It caused us to rush cloud migrations and digital transformation, which of course means long hours refactoring your apps, surprises on your cloud bill, misconfigurations and headache for everyone trying manage disparate and fractured cloud environments. VMware has an answer for this. With VMware multi-cloud solutions, organizations have the choice, speed, and control to migrate and optimizeapplications seamlessly without recoding, take the fastest path to modern infrastructure, and operate consistently across the data center, the edge, and any cloud. I urge to take a look at vmware.com/go/multicloud. You know my opinions on multi cloud by now, but there's a lot of stuff in here that works on any cloud. But don't take it from me thats: VMware.com/go/multicloud and my thanks to them again for sponsoring my ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Periodically, websites like to fall into the sea and explode. And it's sort of a thing that we've accepted happens. Well, most of us have. My guest today is Courtney Nash, Internet Incident Librarian at Verica. Courtney, thank you for joining me.Courtney: Hi, Corey. Thanks so much for having me.Corey: So, I'm going to assume that my intro is somewhat accurate, that we've sort of accepted that sites will crash into the sea, the internet will break, and then everyone tears their hair out and complains on Twitter, assuming that's not the thing that fell over this time—Courtney: [laugh].Corey: —but what does an Internet Incident Librarian do?Courtney: Yeah, I'll come back to the first part about how—some people have accepted it and some people haven't, I think is the interesting part. So technically, I think my official real title is, like, research analyst or something really boring, but I have a background in the cognitive sciences and also in technology, and I'm really—have always been fascinated by how these socio-technical systems work. And so as an Internet Incident Librarian, I am doing a number of things to try to better understand—both for myself and, obviously, the company I work for, but for the industry as a whole—what do we really know about how incidents happen, why they happen, when they happen, and what do we do when they happen? And how do we learn from that? So, one of the first things that I'm doing along those lines is actually collecting a database of all of the public write-ups of incidents that happened at companies that are software-related.So, there's already bodies of work of people who collect airline incidents and other kinds of things. And we don't have that [laugh] as an industry, which I think is—I want to solve that problem because I think other industries that have spent some time introspecting about why things fall down, or when things fall down and how they fall down. Take the airline industry for example; planes don't really fall out of the sky very often.Corey: No. When it does, it makes news and everyone's scared about flying, but at the same time, it's yeah, do you have any idea how many people die in car crashes in a given hour?Courtney: Yeah, yeah. And we'll come back to how the media covers things in a minute because that is definitely something I have opinions about. But, I'm not trying to say I want to create the NTSB of the internet; I don't think that's quite the same thing, and I really want something in the spirit of software, and the internet, and open-source that's more collaborative and it's very open to all of us. So, the first step is to just get them in one place. There is no single place where you could go and say, “Oh, where all of the X incident reports? Where all the ones that Microsoft's written, and also Amazon, or Google, or, you know, whoever.”Corey: They have them, but they hide them so thoroughly. It turns out that they don't really put that in big letters on their corporate blog with links to it. And when you look at one incident report, they don't say, “Here, look at our previous incident reports.” They really—Courtney: Yeah.Corey: —should but no one does.Courtney: And I think that's fascinating because there's a precedent. So, there's two precedents, and I just gave you basically one side of the two, which is, the airline industry has done this and it's not like people don't fly, right? So, a lot of internet companies, a lot of software-based companies, seem to be afraid of what their customers, or what the stock market, or what folks will think. Mind you, these are publicly traded [laugh] airline companies. People aren't going to stop using Amazon just because you give more of this information out.And so I think that piece is—I would love to see that stop being the case. Because the flip side of the coin is that this is a rising tide lifts all boats kind of thing, which granted, not all companies agree on, especially really big ones because their boats already mowing all the little ones out of the ocean. But that's another story.Corey: Sure, but also, it's easy to hide an outage. “Our site is down for you can say three days. Great, if a customer didn't try to access the site at all during those three days, was the site really down in the first place?”Courtney: Oh, the tree in the forest of internet outages. Yes, it's true, although I think that companies are—they know that people go complain on social media, right? I think there's more and more of that happening now. It's not like you can hide it as easily as you could have before Twitter or Instagram or—Corey: Right. Whereas a plane falls out of the sky, generally it's one of those things that people notice.Courtney: Yeah. Even if you weren't interested in that flight at all.Corey: Right. When it lands in your garden, you sort of have a comment on this.Courtney: [laugh]. Yeah. Pieces fall out of the sky. That has happened. But I think the other flip side of the coin I already mentioned is the safety of airline industry has increased so significantly over the past, you know, whatever, 30, 40 years because of this concerted effort.And the other piece of it, then, as an industry, as technologists, as people who use software to run their businesses, some of those things are now safety-critical. And this comes back to the whole software is running the world now. Planes now actually could fall out of the sky because of software, not just because of hardware failures. And nuclear power plants are [laugh] run by software, and your electronic grid, and your health care systems, heart rate monitors, insulin pumps. There are a lot of really critical things, and now our phone services and our internet stuff is so entwined in our lives, that people can't be on their Zoom calls, people can't run their businesses. So, this stuff has a massive impact on people's lives. It's no longer just pictures of cats on the internet, which admittedly, we've really honed the machine for that.Corey: No, but now when software goes down, the biggest arguments people make, the stories people tell is, “Oh, well, it meant that the company lost this much money during that timeframe.” And great, maybe. We can argue about is that really true or is it not? It depends entirely on the company's business model, but I don't like to tend to accept those things at face value. But yeah, that's the small-scale thing, especially when you start getting to these massive platform providers. There are a lot of second and third-order effects that are a lot more interesting slash important to people's lives, than, well, we couldn't show ads to people for an hour and a half.Courtney: Right. Yes. Absolutely. So, T-Mobile had this outage, what is it, how is time—time is still not working very well, for me. I'm trying to remember if it was earlier this year, or if it was in—it was last year. I think it was 2020. And you're like, T-Mobile, oh okay, whatever. You know, like, cell phones, yadda, yadda. 911 stopped working. [laugh].And it was a fascinating outage because these are now actually regulated industries that are heavily software-backed. There was a government investigation into that the same way we have NTSB investigations into airline accidents, and they looked at all of those, kind of, second or third-order effects of people who—you know, a grandma who was stranded on the road, people who couldn't call 911, those kinds of things that are really significant impacts on people's lives. And the second-order effect is, oh, yeah, AWS goes down—like you said—and Amazon or people like to say, Jeff Bezos—I guess, now, are they going to complain about how much money Andy loses? I guess so—but [laugh] what lives on AWS, that's crazy to think about, right?Corey: Yeah, the more I learn the answer to that question, the more disturbed I become.Courtney: Well, you'd probably know a better answer to that question [laugh] than a lot of people.Corey: They have the big companies they can talk about. What's really interesting is the companies that they don't and can't. An easy example: financial services is an industry that is notorious for never granting logo rights. Like, at some point, they'll begrudgingly admit, “Yes, our multinational bank does use computers.” But it's always like pulling teeth, and I get it on some level; the entire philosophy of a lot of these companies is risk-mitigation, rather than growth and advancing the current awareness of knowledge. But it does become a problem.Courtney: Yeah. It's interesting, I need more data, which we'll get to—help me, people—but I am able to start seeing some of those interesting graphs of, kind of these cascading effects of these kinds of outages. And so I strongly believe that we need to talk about them more, that more companies need to write them up, and publish them, and be a lot more transparent about it. And I think there's a number of companies that are showing the way there that—and it has to do with your first question which is, we've all sort of accepted this, right? But I disagree with that.I think those of us who are super close to these kinds of complex, dynamic distributed systems totally know that they're going to fail, and that's not shocking, nor the case of incompetence. We are building systems that are so big and so complex, no one person, no 10X engineer out there could possibly model or hold the whole thing in their head. Especially because it's not even just your systems… we were just talking about, right? Your stuff's on GitHub; it's on AWS; there's, like, three other upstream providers; there's this API from over there. These systems are too intricate, too complex; they're going to fail.Corey: So, we're back to why all these things failed simultaneously and it comes out it's a Northern woods, middle of nowhere backhoe incident. That's right, if we look at the natural food chain of things, fiber optic cable has a natural predator in the form of a backhoe. To the point where if I'm ever lost in the woods, I will drop a length of fiber, kick some dirt over it, wait a few minutes; a backhoe will be along to sever it. Then I can follow the backhoe back to civilization. They don't teach that one and the boy scout manual, but they really should.Courtney: Yeah. Oh, my gosh. There was a beaver outage in Canada, which is the—[laugh] God, that's the most Canadian thing ever.Corey: Can you come up with a more Canadian—Courtney: No.Corey: —story than that? I would posit you could not, but give it a shot.Courtney: No, probably not. Anyhoo. So, I think, like I was saying, those of us close to it accept that, understand it, and are trying to now think about, okay, well, how do we change our approach and our philosophy about this, knowing that things will fall down? But I think if you look at a lot of the rest of the world, people are still like, “What are those idiots doing over there? Why did their site fall down?”Corey: Oh, my God—Courtney: Right?Corey: —the general population is the worst on stuff like this. The absolute worst.Courtney: The media is the worst. [laugh].Corey: It's, “How did they wind up to going down?” “Yeah, because this stuff is complicated.” Back when I was getting started in tech, I thought the whole thing worked on magic, so I started figuring out different pieces of it worked. And now I'm convinced; it runs on magic. The most amazing thing is this all works together. Because—Courtney: Yeah.Corey: —spit and duct tape and baling wire holding this stuff together would be an upgrade from a lot of the stuff that currently exists in the real world. And it's amazing.Courtney: I know the secret, Corey. You know what holds it all together?Corey: Hit me with it. Hope? Tears?Courtney: People.Corey: Mmm.Courtney: Technology is Soylent Green, Corey. It's Soylent Green. It's made of people.Corey: And that's the thing that always bugs me on Twitter. The whole HugOps movement has it right. When you see a big provider taking an outage, all their competitors are immediately there with, “Man, hope things get back together soon. Best of luck. Let us know if we can help.” And that's super reassuring because today is their outage; tomorrow it's yours.Courtney: Yep.Corey: And once in a blue moon, you see someone who's relatively new to the industry starting trying to market their stuff based on someone else's outage, and they basically get their butts fed to them, just because it's this—it's not what you do, and it's not how we operate. And it's one of the few moments where I look at this and realize that maybe people's inherent nature isn't all terrible.Courtney: [laugh]. Oh. Oh, I would hope that would be something that comes out of all of this.Corey: Yeah.Courtney: No one goes to work at their day job doing what we do, to suck. [laugh]. Right? To do a bad job.Corey: Right. Unless you're in Facebook's ethics department, I completely agree with you.Courtney: Okay. Yes. All right. There are a few caveats to that, probably. But you know, we all want to show up and do good stuff. So, nobody's going in trying to take the site down, barring bad actor stuff that's not relevant.Corey: When Azure takes an outage, AWS is not sitting there going, “Ah, we're going to win more cloud deals because of this,” because they're smarter than that. It's, no, people are going to look at this and say, “Ah, see. Told you the cloud was dangerous.” It sets the entire industry back.Courtney: Yeah. That's why we need to talk about it more, and we need to just normalize that these things happen and that we can all level up as an industry if we get a lot smarter about how we, A) think about that, and B) how we react to them. And we will develop much more useful models of our safety boundaries, right? That's really it. You don't know—no one at any of these companies hardly knows if you're five steps from the cliff, five feet, driving a Ferrari 90 miles an hour towards the edge of it.Like, we don't know, it's amazing to me just how much in the dark we are as an industry and how much of the world we're running. So, I think this is one tiny, first little step in what could be sort of a sea change about how all of this works. So, that's a big part of why I'm doing what I'm doing.Corey: Well, let's talk about something else you're doing. So, tell me a little bit about VOID?Courtney: Yeah. So, that's the first iteration of this. So, it's the [Verica Open Incident Database 00:14:10]. I feel like I have to say this almost every time John Allspaw would like me to say that it's the Verica Open Incident Report Database, but VOID is way cooler than—Corey: VOIRD?Courtney: VOIRD.Corey: Yeah, that sounds like you're trying to make fun of someone ineffectively.Courtney: Yeah. And there's a reason why he's not in marketing. But what this is is a collection of all of the publicly available incident reports in one place, easily searchable. You can search by company, you can search by technology, you can filter things by the types of, sort of, kinds of failure modes that we're seeing. And it's, I hope, valuable to a wide swath of folks, both technologists and otherwise: researchers, media and press types, analysts, and whatnot.And my biggest desire is that people will look at it, realize how incomplete it is, and then help me fill it. [laugh]. Help me fill the VOID, people. I think I have right now, at the time we're talking, about 1700, maybe 1800 of these. And they run the gamut. And I know some people who like to quibble about language—and I am one of those people having been an editor in various flavors of my life—not all of these are what a lot of people directly related to these, sort of, incident management and whatnot would call ‘incident reports.'I wanted to collect a corpus that reflects all of the public information about software-related incidents. So, it's anything from tweets—either from a company or just from people—to a status page, to a media article, a news article, an online article, to a full-blown deep-dive retrospective or post-mortem from a company that really does go into detail. It's the whole gamut. It's all of those things. I have no opinionated take on that.I want that all to be available to people. And we've collected some metadata on all of the incidents as well. So, we're collecting the obvious things like when did it happen? What date was it, if we can figure it out, or if it's explicit—how long was it? And those kinds of things and then we collect some metadata, like I said. We add some tags: was this a complete production outage, was it a partial outage? Those kinds of things.And this is all directly just taken from the language of the report. And we're not trying—like I said—we're trying not to have any sort of really subjective takes on any of that, but a bit of metadata that helps people spelunk some of this stuff. So, if it is the kind of report—these are usually from a status page, or a company post about it—what kinds of things were involved in this outage? So, sometimes you'll get lucky and the company will tell you, “It was DNS,” because, you know, it's always DNS.Corey: On some level, it always is. That's why—Courtney: It always is.Corey: —DNS is my database. It's a database problem.Courtney: It's a database problem. And sometimes you get even more detail. And so we will put as much of that that's in the report into a set of metadata about these things. So, I think there's some fascinating, really easy things that I've already seen from some of these data, and we kind of hit on one of these, which is the way that companies themselves talk about these outages versus the way that press and media and other types of organizations talk about these things. So, I think there's a whole bunch of really fascinating analysis that's going to be available to nerdy research-minded type folks like myself.I think it's a place, though, where technologists can also go and spelunk things that they're interested in, looking for patterns, anything that's really—there's an opportunity for experts in the field to add insights to what we can discern from these public incident reports. They are, like, two orders abstracted from what happened internally, but I think there's still a lot that we can learn from those. So, the first iteration of the VOID will allow people to get a first look at some of the data and to help me, hopefully, add to it, grow that corpus over time, and we'll see where that goes.This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: I love the idea of having a centralized place where outages, post-mortems, root cause analyses—I'll let you tear into that in a minute—and other things that are all tied to where can I find a list of outages. Because companies list these on their websites, they put them in blog posts, and it's always very begrudging; they don't link them from any other place, you have to know the magic incantation to find the buried link on their site. Having something that is easily searchable for outages is really something that's kind of valuable.Courtney: Yeah. And I mean, some of them are like—I'm looking at you, Microsoft—I like you for a lot of reasons, but hey, I have to scroll your status page. I can't link directly to their write-ups, and—this is Azure—and it [laugh] please stop. Make it easier. [laugh]. You're driving me crazy; I don't even have a data model to figure out how to make this work for people, other than, like, taking screenshots of them.So yeah, so there's shades of grey and black in how much they'll share, or how easy it is to find these things. So, it'll be interesting to see if there's any less-than-positive [laugh] reactions to all of this being available in one place. I'm anticipating at least a little bit of that.There is one other type of metadata that we collect for the VOID. And that is the type of analysis that is conducted if it is clear what that type of analysis is. And there, some companies explicitly say, or call it an RCA, “We did a Root Cause Analysis.” There's a few other types; some people talk about having a Contributing Factors Analysis. Most people don't consider a formal analysis type, but I am trying to collect and categorize these because I do think there are some fascinating implications buried therein, and I would like to see if I can keep track of whether or not those change over time. And yes, you've hit on one of my favorite hot-take soapbox things, which is root cause.Corey: Please, take it away.Courtney: Yeah. Well, and anyone who's close to these systems and has watched these things fall down has the inherent sense that there is no root cause. Like—[laugh]—let's—great. One of my favorite ones: human error. We don't have enough hours for this, Corey. I'm sorry. That's one of my favorite other ones. But let's say somebody fat-fingers a config change. Which happens—Corey: That was fundamentally the S3 service disruption back in—Courtney: Yes.Corey: —2017 that took down S3 for hours on end.Courtney: And took down so many other people that relied on S3.Corey: Everything was tied to that. And that's an interesting question; when something like that hits, does that mean that everything it takes down get its own entry in VOID?Courtney: I hope so. If everybody writes them up, then yes. [laugh]. So, if S3 goes down, and you go down, and you write it up, and you put it in the VOID, then we can see those things, which would be so cool. But let's go back to the fat-fingered config file—which if you haven't ever done, you're lying, first of all—Corey: Or you haven't been allowed to touch anything large and breakable yet, which, either way, you're lying on some level. So, please—Courtney: Yeah. I mean, I took down [Halloway's 00:20:53] homepage when it was on Hacker News because of YAML. So, anywho. Even if you fat-finger a config change, that's not the root cause because you have this system wherein a fat-fingered configure change can take down S3. That is a very big, complex, and I might add, socio-technical system.There are decisions that were made long ago about why it was structured that way, or why this happens that way, or what kinds of checks and balances you have. It's just, get over it people. There is no root cause. These are complex, highly dynamic systems that when they fail, they fail in unpredictable and weird ways because we've built them that way. They're complex because you're successful at pushing the envelope and your safety boundaries.So, if we could get past the root cause thing as an industry, I mean, I could probably just retire happy, honestly. [laugh]. I'm a simple woman; could we just get one thing, people? [laugh]. First of all, then it gives non-technologists, people outside of our bubble, the media, you can't hang it on these things anymore. We all have to then grapple with the complexity, which admittedly humans, not big fans of, but—Corey: People want simple stories, simple narratives. When people say, “Oh, remember the S3 outage?” They don't want to sit there and have to recount 50,000 different details. They want to say, “Oh, yeah. It took down a few big sites like Instagram, United Airlines, and it was a real mess.” The end. They want something that fits in a tweet, not something that fits in a thesis.Courtney: Well, and if you have a single root cause, then you can fix the root cause and it will never happen again. Right?Corey: That's the theory. If we're just a little bit more careful, we're never going to have outages anymore.Courtney: Yeah, if we could just train those humans to not try to make the best possible high-quality decision they could possibly make in that situation given the information they have at the time, then we'll do better. But I mean, that's why your system stay up most of the time, if you think about it. It's shocking how well these things actually work the vast majority of the time. And that's what we could learn from this, too. We could, you know—oh if we would write near-misses up, please.I mean, if I could have one more wish, I think one of the coolest things the airline industry and the government side of that did was start writing up near-misses. It's, wow, what do we learn from when we're successful, versus trying to, like, spelunk and nitpick the failures.Corey: Most of us aren't so good at the whole introspection part. We need failures, we need painful outages to really force us to make difficult, introspective, soul-searching decisions and learn from them.Courtney: Yeah. And I don't disagree with that. I just wish one of the things we would learn is that we should study our successes, too. There's more to be mined from our successes, if we can figure out how to do that, then there is from our failures. So, I have a metadata category in the VOID called ‘near-miss.'And oh man, I really wish people would write those up more. I mean, I think there's, like, five things in there that I've found so far. Because the humans hold these systems together. We make these things work the vast majority of the time. That's why there is no root cause, and even when we're involved in these things, we're also involved in preventing them, or solving them, or remediating them. So, yeah, there's no root cause. Humans aren't the problem. Those are my big hot button ones.Corey: I really wish more places would embrace that. Even Amazon uses the ‘root cause' terminology internally, and I'm not going to sit here and tell them how to run large things at scale; that's what I pay them to figure out for me. But I can't shake the feeling that by using that somewhat reductive terminology that they're glossing over an awful lot of things the rest of us could really benefit from.Courtney: Well, so the question then—one of the other things that I look at is, personally when I read and analyze these incident reports, these public ones a lot, I always ask myself, “Who's the audience for this?” And there are different audiences for different types of incident reports and different things. The vast majority of them are for customers, partners, investors.Corey: The stock market. Yes. Yes.Courtney: They're not actually for the organization. There's usually an internal one that we don't get to see—maybe—that's for the organization. But a lot of places feel that if you have a process, and a template, and a checklist, and a list of action items at the end, then you've done the right thing. You've had your incident, you've talked about it, you've got your action items. Move on.Corey: Right, and it always seems with companies, that as you get further into the company, the more honest and transparent the actual analysis is. Like, at some point, you wind up with the, like, they're very public and very cagey, and under NDA, they open up a little bit more, and a little bit more, and finally, when you work there, their executive team, it turns out, the actual thing was, “Well, Dewey was carrying arm full of boxes in the data center, tripped, went cascading face-first into the EPO cutoff switch that cut power to the entire facility.” The cagier they get, the—I guess, not to be unkind here—but the more ridiculous whatever the actual answer is. It's one of those things where, “Really? Someone tripped and hit a button. You didn't have a plan for that?” “Well, not really. We sort of assumed that people would”—Courtney: Why would you have a plan for that, right?Corey: Right.Courtney: I mean like—[laugh].Corey: Why would you have a plan for that, the first time?Courtney: Yeah. I mean, so imagine this exercise: sitting down in a room with a bunch of people and going, “What are all the things that could go wrong?” I mean, [laugh] ain't nobody got time for that? That's not how it works. You all have other jobs to do, too, and systems to build, and pressures, and customers, and partners, and features to build, so admit and acknowledge that you just won't know all of the antecedents and how do you respond when things happen?Which is a whole other, you know—I know you told me you recorded an episode with Dr. Christina Maslach on burnout, which I'm so happy you did, and there's a whole ‘nother piece of incidents and incident response, and burning people out, and blaming people, and all that stuff that's a whole ‘nother pod—it sounds like you might—you know, probably not incidents with her. But still, these things take a toll on people. And people who, like I said, show up every day really hoping to do their best job, and go up a ladder, and get a promotion, and whatever. So, I think not just treating those things as checklists has broader implications as well, just for the wellbeing of your organization.Corey: On some level, the biggest problem that I think we've run into is that, as you said, it all comes down to people. Unfortunately, legally, we can't patch those. Yet.Courtney: No, [laugh]. No, no. Not most kinds of patches, no. And that's messy. And I know some people are like, “Everyone should learn to code.” And I'm like, “Actually, everyone should get a liberal arts degree.” Come on, help me out people. Because there's so much of these socio-technical systems where the socio part of it is more relevant than the actual technical part.Corey: I believe you're right, for better or worse; there's no way around it. Thank you so much for taking the time to speak with me. If people want to learn more about what you're up to, where can they find you? And we will, of course, throw a link to VOID in the [show notes 00:28:06].Courtney: Yeah, I also like to talk on Twitter, like you do. I'm not as good at it as you are, but I try. So yeah, I'm @courtneynash on Twitter. And at Verica, you can find me at Verica as well, courtney@verica.io. And those are the best ways to find me, I would say. And yeah, please people, write up your incidents, send them to the VOID and let's all learn and get better together, please.Corey: Thank you so much for taking the time to speak with me today. I really do appreciate it.Courtney: Thank you for having me on. I know—do people say this: I'm like, “Yeah, big fan,” but I am. I'm a [laugh] big fan [laugh] of the podcast.Corey: Oh, dear Lord, find better things to listen to. My God.Courtney: [laugh]. But it's been a treat. Thank you.Corey: Courtney Nash, Internet Incident Librarian at Verica. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment making it very clear that for whatever reason the website is down, it is most certainly not your fault.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Gitting After It with Katie Sylor-Miller

Screaming in the Cloud

Play Episode Listen Later Sep 14, 2021 44:52


About KatieKatie Sylor-Miller, Frontend Architect at Etsy, has a passion for design systems, web performance, accessibility, and frontend infrastructure. She co-authored the Design Systems Handbook to spread her love of reusable components to engineers and designers. She's spoken at conferences like Smashing Conf, PerfMatters Conf, JamStack Conf, JSConf US, and FrontendConf.ch (to name a few). Her website ohshitgit.com (and the swear-free version dangitgit.com) has helped millions of people worldwide get out of their Git messes, and has been translated into 23 different languages and counting.Links: Etsy: https://www.etsy.com/ Design Systems Handbook: https://www.designbetter.co/design-systems-handbook Book of staff engineering stories: https://www.amazon.com/dp/B08RMSHYGG staffeng.com: https://staffeng.com ohshitgit.com: https://ohshitgit.com dangitgit.com: https://dangitgit.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored in part by our friends at Jellyfish. So, you're sitting in front of your office chair, bleary eyed, parked in front of a powerpoint and—oh my sweet feathery Jesus its the night before the board meeting, because of course it is! As you slot that crappy screenshot of traffic light colored excel tables into your deck, or sift through endless spreadsheets looking for just the right data set, have you ever wondered, why is it that sales and marketing get all this shiny, awesome analytics and inside tools? Whereas, engineering basically gets left with the dregs. Well, the founders of Jellyfish certainly did. That's why they created the Jellyfish Engineering Management Platform, but don't you dare call it JEMP! Designed to make it simple to analyze your engineering organization, Jellyfish ingests signals from your tech stack. Including JIRA, Git, and collaborative tools. Yes, depressing to think of those things as your tech stack but this is 2021. They use that to create a model that accurately reflects just how the breakdown of engineering work aligns with your wider business objectives. In other words, it translates from code into spreadsheet. When you have to explain what you're doing from an engineering perspective to people whose primary IDE is Microsoft Powerpoint, consider Jellyfish. Thats Jellyfish.co and tell them Corey sent you! Watch for the wince, thats my favorite part. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Katie Sylor-Miller, who is a frontend architect at Etsy. Katie, thank you for joining me.Katie: Hi, Corey. Thanks for having me.Corey: So, I met you a long time ago—before anyone had ever heard of me and the world was happier for it—but since then you've done a lot of things. You're obviously a frontend architect at Etsy. You're a co-author of the Design Systems Handbook, and you were recently interviewed and included Will Larson's book of staff engineering stories that people are mostly familiar with at staffeng.com.Katie: Yeah.Corey: So, you've done a lot of writing; you've done some talking, but let's begin with the time that we met. To my understanding, it's the only time we've ever met in person. And this harkens back to the first half—as I recall—of 2016 at the frontend conference in Zurich.Katie: Yes, before either of us were known for anything. [laugh].Corey: Exactly. And it was, oh, great. And I wound up getting invited to speak at a frontend conference. And my response was, “Uh, okay. Zurich sounds lovely. I'm thrilled to do it. Do you understand who you're asking?”There are frontend folks—which, according to the worst people on the internet is the easiest form of programming; it isn't a real engineering job, and if that's your opinion, please stop listening to anything I do ever again—secondly, then there's the backend folks who write the API side of things and what the deep [unintelligible 00:02:03] and oh, that's the way of the future. And people look at me and they think, “Oh, you're a backend person,” if their frontend. If they're backend, they look at me and think, “Oh, you're a DevOps person.” Great. And if you're on the DevOps space, you look at me and think, “What is wrong with this person?” And that's mostly it.But I was actually invited to speak at a frontend conference. And the reason that they invited me at all—turns out wasn't a mistake—was that I was giving a talk that year called, “Terrible Ideas in Git,” which is the unifying force that ties all of those different specialties together by confusing the living hell out of us.Katie: Yes. [laugh].Corey: So, I gave a talk. I thought it was pretty decent. I've done some Twitter threads on similar themes. You did something actually useful that helps people and is more lasting—and right at that same conference, I believe, you were building slash kicking it off—ohshitgit.com.Katie: Yes. Yeah. It was—Corey: Which is amazing.Katie: Thank you. Yeah, it was shortly thereafter. I think the ideas were kind of starting to percolate at that conference. Because you know—yeah I was—Corey: Because someone gave a talk about Git. Oh, I'm absolutely stealing credit for your work.Katie: No, Corey—Corey: “Oh, yeah. You know, that was my idea.”Katie: [laugh].Corey: Five years from now, I'm going to call myself the founder of it, and you're just on the implementation details.Katie: I don't—nonononono—Corey: That's right. I'm going to D.C. Bro my way through all of this.Katie: [laugh]. No, no, no, no. See, my recollection is that my talk about being a team player and a frontend expert with a T-shape happened at exactly the same time as your talk about Git because I remember I wanted to go watch your talk because at the time, I absolutely hated Git. I was still kind of learning it. So yeah, so I don't think you really get any credit because I have never actually heard that talk that you gave. [laugh].Corey: A likely story.Katie: [laugh]. However, however, I will say—so, before I was up to give my talk, the emcee of the conference was teasing me, you know, in a very good-natured ribbing sort of way, he was teasing me about my blog being totally empty and having absolutely nothing in it. And I got on the plane home from Zurich, and I was starting to think, “Oh, okay. What are some things that I could blog about? What do I have to say that would be at all interesting or new to anyone else?”And like I think a lot of people do, I had a really hard time figuring out, okay, what can I say that's, maybe, different? And, I went back home, I went back to work, and at one point, I had this idea, I had this file that I had been keeping ever since I started learning Git and I call it, like, gitshit.txt. And hopefully, your listeners don't mind lots of swears because I'm probably going to swear quite a bit.Corey: No, no. I do want to point out, you're accessible to all folks: dangitgit.com, also works but doesn't have the internal rhyming mechanism which makes it, obviously, nowhere near where it needs to be.Katie: [laugh]. Well—Corey: It's sort of a Subversion to Git if you will.Katie: Yes, exactly.Corey: I—Subversion fans, don't yell at me.Katie: [laugh]. Anyways, so I remember I tweeted something like, “Oh, what about if I took this text file that I had,” where every time I got into a Git mess, I would go on to Stack Overflow—as you do—and I would Google and I—it was so hard. I couldn't find the words to find the answers to what I was trying to fix. Because one of the big problems with Git that we can talk about it a bit more in detail later is that Git doesn't describe workflows, Git describes internal plumbing commands and everything that it exposes in its API. So, I had a really hard time with it; I had a hard time learning it.And, you know, what I said, “Okay, well, maybe if I published on my blog about these Git tips that I had saved for myself.” And I remember I tweeted, and I got a handful of likes on the tweet, including from Eric Meyer, who is one of my big idols in the frontend world. He's one of the godfathers of modern CSS. And he liked my tweet, and I was like, “Oh, okay. Maybe this is a real thing. Maybe people will actually find this interesting.”And then I had this brilliant idea for this URL, ohshitgit.com, and it was available, and I bought it. And I swear to you, I think I spent two hours writing some HTML around my text file and publishing it up to my server. And I tweeted about it, and then I went to bed.And I kind of expected maybe half a dozen of my coworkers would get a little sensible chuckle out of it, and like, that would be the end of it. But I woke up the next morning and my Twitter had blown up; I was on the front page of Hacker News. I had coworkers pinging me being like, “Oh, my God, Katie, you're on Hacker News. This is insane.” And—Corey: Wait, wait, for a good thing, or the horrifying kind of thing because, Hacker News?Katie: Well, [laugh] as I have discovered with Hacker News, whenever my site ends up on Hacker News, the response is generally, like, a mix of, “Ha ha ha, this is great. This is funny,” and, “Oh, my God, somebody actually doesn't understand Git and needs this. Wow, people are really stupid.” Which I fundamentally disagree with and I'm sure that you fundamentally [laugh] disagree with as well.Corey: Oh, absolutely.Katie: Yeah. So—Corey: It's one of those, “Oh, Git confuses you. You know what that means? It means you're human.” It confuses everyone. The only question is, at what point does it escape your fragile mortal understanding? And if you are listening to this and you don't believe me, great. I'm easy to find, I will absolutely have that discussion with you in public because I promise, one of us is going to learn something.Katie: [laugh]. Awesome. I love—I hope that people take you up on that because—Corey: Oh, that would be an amazing live stream, wouldn't it?Katie: It would. It would because Git is one of those things that I think that people who don't understand it, look at it and think, “Gosh, you know, I must be stupid,” or, “I must not be cut out to be a developer,” or, “I must not know what I'm doing.” And I know that this is how people feel because that's exactly how I felt myself, even when I made ohshitgit.com, that became this big reference that everybody looks at to help them with Git, like, I still didn't understand it. I didn't get Git at all.And since then, I've kind of been forced because people started asking me all these questions, and, “Well, what about this? What about that?” And I was just like, “Uh… I don't know. Uh…” and I didn't like that feeling, so I did what, you know, obviously, anyone would do in my situation and I sent out a proposal to give a talk about Git at a conference. [laugh].And what that did is when my talk got accepted, I had to then go off and actually learn Git and understand how it works so that I could go and teach it to other people at this conference. But it ended up being great, I think because I found a lot of really awesome books. There's A Book Apart book called Git for Humans, which is incredibly good. There's a couple of websites like learngitbranching.com.There's a bunch more that I can't think of off the top of my head. But I went out and I sort of slowly but surely developed this mental model, internally, of how Git works. And I'm a visual thinker and I'm a visual learner, and so it's a very visual model. And for what it's worth, I think that was my biggest problem with Git was, like, I came from Microsoft .NET environment before that, and we used a program called TFS, Team Foundation Server, which is basically like a SVN or a CVS type source control system that was completely integrated into Visual Studio.So, it was completely visual; you could see everything happening in your IDE as you were doing it. And then making this switch to the command line, I just could not figure it out until I had this visual mental model. So yeah, so ever since then I've just been going around and trying to teach people about Git and teach people this visual mental model that I've developed, and the tips and the tricks that I've learned for navigating Git especially on the command line. And I give talks, I do full-day training workshops, I do training workshops at work. And it's become my thing now, which is flabbergasting [laugh] because I never intended [laugh] for—I didn't set out to go and be this Git expert or to be, quote-unquote, “Famous” for a given value of famous, for knowing stuff about Git. I'm a frontend engineer. There's still a piece of me that looks at it, and is like, “How on earth did this even happen to me?” So, yeah, I don't know. So, that's my Oh shit, Git!?! story. And now—Corey: It's a great one. It's—Katie: Thank you.Corey: Git is one of those weird things where the honest truth of were, “Terrible Ideas in Git”—my talk—came from was that I kept trying and failing to understand Git, and I realized, “How do I fix this? I know. I will give a talk about something.” That is what we know as a forcing function. If I'm not quite ready, they will not move the conference. I know because I checked.Katie: Yep. [laugh]Corey: And one in Zurich was not the first time I'd given it, but it was very clearly something that everyone had problems with. The first version of that talk would have absolutely killed it, if I'd been able to give it to the core Git maintainers. And all, you know, seven of those people would have absolutely loved it, and everyone else would have been incredibly confused. So, I took the opposite tack and said, “All right. How do I expand this to as broad an audience as possible?”And in one of the times I gave it, I said, “Look, I want to make sure it is accessible to everyone, not just people who are super deep into the weeds but also be able to explain Git to my mother.” And unlike virtually every other time where that, “Let me explain something to my mom.” And that is basically coded ageism and sexism built into one. In that case, it was because my mother was sitting in the front row and does not understand what Git is. And she got part of the talk and then did the supportive mother thing of, and as for the rest of it. “Oh, you're so well-spoken. You're so funny. And people seem to love it.” Like, “Did you enjoy my discussion of rebases?”Katie: [laugh].Corey: She says, “Just so good at talking. So, good.” And it was yeah.Katie: [laugh]. Oh, yeah. No, I, I—totally—I understand that. There's this book that I picked up when I was doing all of this research, and I'm looking over at my bookshelf, it's called Version Control with Git. It's an O'Reilly book.And if I remember correctly, it was written by somebody who actually worked at Git. And the way that they started to describe how Git works to people was, they talked about all kinds of deep internals of Unix, and correlated these pieces of the deep internals of Git to these deep Unix internals, which, at the time, makes sense because Git came out of the Unix kernel project as their source control methodology, but, like, really? Like, [laugh] this book, it says at the beginning, that it's supposed to teach people who are new to Git about how to use it. And it's like, well, the first assumption that they make is that you understand the 15 years' worth of history of the Linux kernel project and how Linux works under the hood. And it's like, you've got to be absolutely kidding me that this is how anyone could think, “Oh, this is the right way to teach people Git.”I mean, it's great now, going back in and rereading that book more recently, now that I've already got that understanding of how it works under the hood. This is giving me all of this detail, but for a new person or beginner, it's absolutely the wrong way to approach teaching Git.Corey: When I first sat down to learn Git myself it was in 2008, 2009, Scott Chacon from GitHub at the time wound up doing a multi-day training at the company I worked at the time. And it was very challenging. I'm not saying that he was a bad teacher by any stretch of the imagination, but back in those days, Git was a lot less user-friendly—[laugh] not that it's tremendously good at it now—and people didn't understand how to talk about it, how to teach it, et cetera. You go to GitHub or GitLab or any of the other sites that do this stuff, and there's a 15-step intro that you can learn in 15 minutes and someone who has never used Git before now knows the basics and is not likely to completely shatter things. They've gotten the minimum viable knowledge to get started down to a very repeatable, very robust thing. And that is no small feat. Teaching people effectively is super hard.Katie: It really is. And I totally agree with you that if you go to these providers that they've invested in improving the user experience and making things easier to learn. But I think there's still this problem of what happens when everything goes wrong? What happens if you make a mistake, or what happens if you commit a file on the wrong branch? Or what happens if you make a commit but you forgot to add one of the files you wanted to put in the commit?Or what happens if you want to undo something that you did in a previous commit? And I think these are things that are still really, for some reason, not well understood. And I think that's kind of why Oh Shit, Git!?! has fallen into this little niche corner of the Git world is because the focus is really like, “Oh, shit. I just made a mistake and I don't know what to do, and I don't know what terminology to even Google for to help me figure out how to fix this problem.” And I've come out and put these very simple, like, here: step one, step two, step three.And people might disagree or argue [laugh] with some of the commands and some of the orders, but really, the focus is, like, people have this idea in their head, I think, particularly at their jobs, that Git is this big, important thing and if you screw up, you can't fix it. When really a lot of helping people to become more familiar and comfortable with Git is about ensuring them that no, no, no, the whole point of Git is that just about everything can be undone, and just about everything is fixable, and here's how you do it. So, I still think that we have a long way to go when it comes to teaching Git.Corey: I would agree wholeheartedly. And I think that most people are not thinking about this from a position of educators, they're thinking about it from the position of engineering, and it's a weird combination of the two. You're not going to generally find someone who has no engineering experience to be able to explain things in a context that resonates with the people who will need to apply it. And on the other side, you're not going to find that engineers are great at explaining things without having specific experience in that space. There are exceptions, and they are incredibly rare and extremely valuable as a result. The ability to explain complex things simply is a gift.Katie: It really is.Corey: It's also a skill and you can get better at it, but a lot of folks just seem to never put the work in in the first place.Katie: Well, you know, it's quote-unquote, “soft skills.” So [laugh].Corey: Oh, God. They're hard as hell, so it's a terrible name.Katie: [laugh]. Yeah. Though I could not agree more, I think something that I really look at as a trait of a super senior engineer is that they are somebody who has intentionally worked on and practiced developing that skill of taking something that's a really complex technical concept, and understanding your audience, and having some empathy to put yourself in the shoes of your audience and figure out okay, how do I break this down and explain it to someone who maybe doesn't have all the context that I do? Because when you think about it, if you're working at a big company, and you're an engineer, and you want to, like, do the new hotness, cool thing, and you want to make Kubernetes the thing or whatever other buzzword term you want to use, in order to get that prioritized and on a team's backlog, you have to turn around and explain to a product person why it's important for product reasons, or what benefits is this going to bring to the organization as far as scalability, and reliability. And you have to be able to put yourself in the shoes of someone whose goals are totally different than yours.Like, product people's goals are all around timelines, they're around costs, they're around things short-term versus long-term improvements. And if you can't put yourself into the shoes of that person, and figure out how to explain your cool hot tech thing to them, then you're never going to get your project off the ground. No one's ever going to approve it, nobody's going to give budget, nobody's going to put it in a team's backlog unless you have that skill.Corey: That's the hard part is that people tend to view advancement as an individual contributor or engineer purely through a lens of technical ability. And it's not. The higher you rise, the more your job involves talking to people, and the less it involves writing code in almost every case.Katie: One hundred percent. That's absolutely been my experience as an architect is that, gosh, I almost never write code these days. My entire job is basically writing docs, talking to people, meeting with people, trying to figure out, where, what is the left hand doing and what is the right hand doing so I can somehow create a bridge between them. You know, I'm trying to influence teams, and their approach, and the way that they think about writing software. And, yes there is a foundation of technical ability that has to be there.You have to have that knowledge and that experience, but at this point, it's like, my God—you know, I write more SQL as a frontend architect that I write HTML, or CSS, or JavaScript because I'm doing data analysis and [laugh] I'm doing—I'm trying to figure out what does the numbers tell us about the right thing to choose or the right way to go, or where are we having issues? And, yeah, I think that people's perceptions and the reality don't always match up when it comes to looking at the senior IC technical track.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: At some level, you hear people talking about wanting to get promoted, and what they're really saying—and it doesn't seem that they realize this—is, “I love what I do, so I'm really trying to get promoted so I can do less of what I love and a lot more of things I hate.”Katie: [laugh]. Yes. Yeah. Yeah. [laugh]. In some ways, in some ways, I think that you've got to kind of learn to accept it. And there are some people, I think that once you get past the senior engineer, or maybe even the staff engineer, maybe they don't even want to go there because they don't want to do the kind of sales pitch, people person, data numbers pitching, trying to get people to agree with you on the right way forward is really hard, and I don't think it's for everyone. But I love it. [laugh]. I absolutely love it. It's been great for me. And I feel like it really—it plays to my strengths in a lot of ways.Corey: What I always found that worked for me, as far as getting folks on board with my vision of the world is, first, I feel like I have to grab their attention, and my way is humor. With the Git talk, I have to say giving that talk a few times made me pretty confident in it. And then I was invited to the frontend conference. And in hindsight, I really, really should have seen this coming, but I'm there, I'm speaking in the afternoon, I'm watching the morning talks, and the slides are all gorgeous.Katie: Yes. [laugh].Corey: And then looking at my own, and they are dogshit. Because this was before I had the sense to hire a designer to help with these things. It was effectively black Helvetica text on a white background. And I figured, “All right, this is a problem. I only have a few hours to go, what do I do?”And my answer was, “Well, I'm not going to suddenly become an amazing designer in the four hours I have.” So, I changed some of the text to Comic Sans because if you're doing something bad, do it worse, and then make it look intentional. It was a weird experience, and it was a successful talk in that no one knew what the hell to make of what I was doing. And it really got me thinking that this was the first time I'd spoken to an audience who was frontend, and it reminded me that the DevOps problems that I normally talked about, were usually fairly restricted to DevOps. But the things that everyone touches, like Git, for example, start to be things that resonate and break down walls and silos better than a given conference ever can. But talking instead about shared pain and shared frustrations.Katie: Yes. Yes. Everyone likes to know that they are not alone in the world, particularly folks who are maybe underrepresented minorities in tech and who are afraid to speak up and say, “Oh, I don't understand.” Or, “That doesn't make any sense to me,” because they're worried that they're already being taken not as seriously as their white, male counterparts. And I feel like something I really try to lean into as a very senior woman in a very male-dominated field is if I don't understand something, or if I have a question, or something doesn't make sense is I try to raise my hand and ask those questions and say, out loud, “Okay, I don't get this.”Because I can't even tell you, Corey, the number of times I've had somebody reach out to me after a meeting and say, “Thank you. I didn't understand it either.” Or, “I thought maybe I just didn't understand the problem space, or maybe I just wasn't smart enough to understand their explanation.” And having somebody who's very senior who folks look up to, to be able to say, “Wait a minute, this doesn't make sense.” Or, you know, I don't understand that explanation.Can you explain it a different way? It's so powerful and it unblocks people and it gives them this confidence that, hey, if that person up on stage, or leading this meeting, or writing this blog post doesn't get this either, maybe I'm not so stupid, or maybe I do deserve to be in this industry, or maybe it's not just me. And I really hope that more and more people can feel empowered to do that in their daily lives more. I think that's been something that has been a tremendous learning through all of this experience with Oh shit, Git!?!For me is the number of people that come up to me after conference talks, or tweet me, or send me a message, just saying, “Thank you. I thought I was alone. I thought I was the only one that didn't get this.” And knowing that not just am I not the only one, but that people are universally frustrated, and universally Git makes them want to swear all the time, I mean, that's the best compliments that I get is when folks come up to me and say, “Thank you, I thought I was alone.”Corey: That's one of the things that I find that is simultaneously the most encouraging and also the most galling. Every once in a while I will have some company reach out to me—over a Twitter thread or something—where I'm going through their product from a naive user perspective of, like, I'm not coming at this with 15 years of experience and instinct that feed into how I approach this, but instead the, I actually haven't used this product before. I'm not going to jump ahead and make assumptions that tend to be right. I'm going to follow the predictable user path flow. And they are very often times where, “Okay. I'm hitting something. I don't understand this. Why is it like this? This is not good.”And usually, companies are appreciative when I do stuff like that, but every once in a while, I'll get some dingus who will come in, and like, “I didn't appreciate the fact that you end up intentionally misinterpreting what we're saying.” And that's basically license for me to take the gloves off and say, “No, this was not me being intentionally dumb. Sure, I didn't apply a whole bunch of outside resources I could have to this, but it wasn't me intentionally failing to get the point. I did not understand this, and you're coming back to me now reinforces that you are too close to the problem. And, on some level, when your actual customers have problems with this, they are hearing an element of contempt from you.”Katie: Totally.Corey: “This is an opportunity to fix it and make it more approachable because spoiler, not a lot of people love paying money to something that makes them feel stupid.”Katie: [laugh]. See, Corey, I don't know. You say that you're not really a frontend person, but that is a very strong UX mindset. Like that—Corey: Oh, my frontend stuff is actually pretty awesome because as soon as I have to do something that even borders on frontend, I have the insight and I guess, willingness to do the smart thing, which is to immediately stop talking and pay someone who knows what they're doing.Katie: [laugh]. Thank you. On behalf of all frontend engineers everywhere, I applaud that, and I appreciate it.Corey: It comes down to specialty. I mean, again, it would also be sort of weird from my perspective, which is my entire corporate position is I fix the horrifying AWS bill. So, if you're struggling with the bill in various capacities, first, join basically everyone, but two, you're not alone so maybe hire someone who is an expert in this specific thing to come in and help you with it. And wouldn't it be a little hypocritical of me to go in and say, “Oh, yeah, but I'm just going to YOLO my way through this nonsense?”Katie: Mm-hm. [laugh]. Yeah, [laugh] I don't know we'll want to include this in the final recording, but I have a really hilarious story, actually, about Amazon. So—Corey: Oh, please. They listen to this and they love customer feedback.Katie: [laugh].Corey: I'm not being sarcastic. I'm very sincere here.Katie: Well, this is many, many, many years ago. I mean, probably, oh, gosh, this is probably eight years ago at this point. I was interviewing for a job at Amazon. It was a job to be a frontend engineer on the homepage team, which at the time, I was like, “Oh, my God, this is Amazon. This is such an honor. I'm so excited.”Corey: And you look at amazon.com's front page, and it's, “Oh, I can fix this. There's so much to fix here.”Katie: Yes.Corey: And then reality catches up if I might not be the first person in the world to have made that observation.Katie: [laugh].Corey: What's—Katie: Well—Corey: Going on in there?Katie: Yeah. Well, I'll tell you what's going on. So, I think I did five different phone interviews. You know, before they invite you out to Seattle, there's—and again, this was eight years ago, so this was well before everyone was working at home. And in those five hours of phone interviews, I want you to make a guess at how many minutes we spent talking about HTML, CSS, and JavaScript.Corey: I am so unfamiliar with the frontend world, I don't know what the right answer is for an interview, but it's either going to be all the time or none of it, based on the way you're framing it.Katie: Yes. [laugh]. It was basically, like, half an hour. So, when you are a frontend engineer, your job is to write HTML, CSS, and JavaScript. And in five hours, I talked about that for probably half an hour.It was one small question and one small discussion, and all the rest of the time was algorithms, and data structures, and big O notation, and oh, gosh, I think they even did the whole, like, “I typed something into my browser, tell me what happens after I type a URL into my browser.” And I think that just told [laugh] me everything that I needed to know about how Amazon approached the frontend and why their website was such a hot mess was because they weren't actually hiring anyone with real frontend skills to work on the frontend. They were hiring backend people who probably—not to say that they weren't capable or didn't care, but I don't know. That's my favorite Amazon story that I have is trying to go work there, and they basically were like, “Yes, we want a frontend engineer.” And then they didn't actually ask about any frontend engineering skill sets in the job. They didn't offer me anyth—I don't think I got invited to go to Seattle, but I probably wouldn't have anyways.Corey: No. Having done it a couple of times now, again, I like the people I meet at Amazon very, very much. I want to be very clear on that. But some of their processes on the other hand, oh, my God. It shows that being a big company is clearly not necessarily a signal that you solved all of these problems. In some cases, you're basically just crashing through the problem space by sheer power of inertia.Katie: Yeah, definitely. I think you can see that when looking at their frontend. Harkening back a little bit to what we were talking about earlier is you don't go to Amazon and learn patterns of interaction that are applicable to every single site on the web. Amazon kind of expects that users are going to learn the Amazon way of shopping and that users are going to adjust how they navigate the web in order to accommodate Amazon. You know, people learn, “Oh, this is what I do on Amazon.” And then, you know, they're—Corey: Oh, that's the biggest problem with bad user experience is people feel dumb.Katie: Mm-hm.Corey: They don't think, “This company sucks at this thing.” They think, “I must not get it.” And I know this, and I am subject to it. I run into this problem all the time myself.Katie: Oh, yes.Corey: And that is a problem.Katie: Yeah. It's why I think, like you said earlier, it's so important when you work somewhere to figure out how do you get that distance between being a power user enough so that you can understand and appreciate what it's like for a regular user who's not a power user of your site. And what do they do? And UX researchers are amazing. A good UX researcher is worth absolutely their weight in gold because, I don't know if you've ever sat in on a UX session where the researcher is walking a user through completing a specific task on a website, but oh my God, it's painful.It's because [laugh] you just want to, you want to push them in the right direction, and you want to be like, “Oh, but what about in the upper right over there, that big orange button,” and you can't do that. You can't push people. You have to be very open-ended, you have to ask them questions. And every single time I've listened in on a UX research recording, or a call, I want to scream through the computer and be like, “Oh, my gosh. This is how you do it.”But, you know, you can't do that. So, [laugh] I think it's important to try to develop that kind of skill set on your own of, “Okay, if I didn't stare at this website every day, what would it be like for me to try to navigate? If I was using a keyboard for navigation or a screen reader instead of a mouse, what would my experience be like?” Having that empathy, and that ability to get outside of yourself is just really important to be a successful engineer on the web, I think.Corey: Yeah. And you really wish, on some level, that they would be able to articulate this as an industry. And I say ‘they,' I guess I'm speaking of about three companies in particular. I have a lot more sympathy for a small startup that is having problems with UX than I am for enormous companies who can basically hurl all the money at it. And maybe that's unfair, but I feel like, at some point of market dominance, it is beholden on you to set the shining example for how these things are going to work.I don't feel that way, necessarily about architecture on the backend. Sure, it can be a dangerous, scary tire fire, but that's not something your customers or users need to think about or worry about, as long as it is up from their perspective. UX is very much the opposite of that.Katie: Totally. And I think, working at a former startup, there's a tendency to really focus a lot on those backend problems. You know, you really look at, “Okay, we're going to nitpick every single RPC request. We're going to have all kinds of logging and monitoring about, okay, this is the time that it takes for a database API request to return.” And just the slightest movement and people freak out.But it's been a process that I've been working really hard on the last couple of years, to get folks to have that same kind of care and attention to the stuff that they ship to the frontend, especially for a lot of organizations that really focus on, “Well, we're a tech company,” it's easy to get into this, oh, engineering is all of these big hard systems problems, when really your customers don't care about all of that. Yes, ultimately, it does affect them because if your database calls are really, really slow, then it has an effect on how quickly the user gets a response back and we know that slow-performing websites, folks are more likely to abandon them. Not that it doesn't matter completely, but personally, I would really love it to see more universally around the industry that frontend is seen as this is the entirety of your product and if you get that wrong, then none of the rest of your architecture, or your infrastructure, or how great your DevOps is matters because you need customers to come to your site and buy things.Corey: It turns out that the relationship between customers coming to your site and buying things and the salaries engineering likes to command is sometimes attenuated in ways that potentially shouldn't be. These are interesting times, and it does help to remember the larger context of the work we do, but honestly, at some point, you wind up thinking about that all the time, and not the thing that you're brought in specifically to fix. These are weird times.Katie: Yes.Corey: Katie, thank you so much for taking the time to speak with me about several things. Usually—it's weird. Normally, when someone says thank you for speaking to me about Git, there is no way that isn't a sarcastic—Katie: [laugh].Corey: —statement. But in this case, it is in fact genuine.Katie: Yes, I will bitch about Git until I am blue on the face, so I appreciate you having me on board to talk about it, Corey. Thank you.Corey: Of course. If people want to learn more, where can they find you?Katie: They can find me at ohshitgit.com, or as you pointed out, the dangitgit.com swear-free version. As a little plug for the site, we now have had the site translated by volunteers in the community into 28 different languages. So, if English is not your first language, there's a really good chance you'll find a version of OSG—as I like to call it—that is in your language.Corey: Terrific. And we will, of course, put links to these wonderful things in the [show notes 00:39:16]. Thank you so much for taking the time to speak with me. I really appreciate it.Katie: Thank you, Corey. It's been lovely to reconnect, and gosh, look at where we are now compared to where we were almost five years ago.Corey: I know. It's amazing how the world works.Katie: Really.Corey: Katie Sylor-Miller, frontend architect at Etsy. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment written in what is clearly your preferred user interface: raw XML.Katie: [laugh].Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.This has been a HumblePod production. Stay humble.

Ask Noah Show
Episode 249: Pine64 with Lukasz Erecinski

Ask Noah Show

Play Episode Listen Later Sep 9, 2021 80:41


-- During The Show -- 02:14 VPN Links - Jscar Hawk Road Warrior Script (https://github.com/Nyr/openvpn-install) Mullvad Pricing (https://mullvad.net/en/pricing/) privacytools.io (https://www.privacytools.io/providers/vpn/) Privacy Security and OSINT Podcast (https://inteltechniques.com/podcast.html) 04:30 Email 2 - VPN Question Response - Charlie Incognet VPS (https://incognet.io) Mullvad VPN (https://mullvad.net) Nearly Free Speech Webhosting (https://www.nearlyfreespeech.net/about/faq#Payment) OpenNIC (https://www.opennic.org) HackerPublic Radio EP 3323 (https://hackerpublicradio.org/eps.php?id=3323) 09:45 Caller Ryan UniFi AP Setup Can be done with UniFi App (https://play.google.com/store/apps/details?id=com.ubnt.easyunifi) UniFi CloudKey (https://store.ui.com/collections/unifi-accessories/products/unifi-cloudkey) Check the Docker Container Author Use the docker file/container file and build it yourself UniFi+Podman (https://github.com/jdoss/unifi) Author is known in the community AP placement (20 db of separation) WiFiAnalyzer (https://f-droid.org/en/packages/com.vrem.wifianalyzer/) WiFiScanner (https://f-droid.org/en/packages/org.bitbatzen.wlanscanner/) Steve's Ubuntu setup (ADD STEVE'S LINK HERE) 25:40 Email 4 - UPS & Servers - Veritanuda Wake-On-Lan Packet 29:05 Email 5 - Comments on Windows - Jon sudo strings /sys/firmware/acpi/tables/MSDM 31:25 Proton Shares Activist IP No logging doesn't equal unable to log Privacy is not a cloak for illegal activities ProtonMail provided ways to avoid their own monitoring ProtonMail talked with legal to try and avoid handing over information Information was handed over in response to a legal request XKCD Security (https://xkcd.com/538) HackerNews (https://thehackernews.com/2021/09/protonmail-shares-activists-ip-address.html) ProtonMail Blog Post (https://protonmail.com/blog/climate-activist-arrest/) 39:40 Pine64 Interview DevZone FemtoStar How do you decide what products to make? What Devices do you want to make but are impractical/not cost effective? Pine64 router? PinePhone Pro? PinePhone Operating Systems Target Audience? PineNote Where can people help? What's next for Pine64? -- The Extra Credit Section -- For links to the articles and material referenced in this week's episode check out this week's page from our podcast dashboard! This Episode's Podcast Dashboard (http://podcast.asknoahshow.com/249) Phone Systems for Ask Noah provided by Voxtelesys (http://www.voxtelesys.com/asknoah) Join us in our dedicated chatroom #GeekLab:linuxdelta.com on Matrix (https://element.linuxdelta.com/#/room/#geeklab:linuxdelta.com) -- Stay In Touch -- Find all the resources for this show on the Ask Noah Dashboard Ask Noah Dashboard (http://www.asknoahshow.com) Need more help than a radio show can offer? Altispeed provides commercial IT services and they're excited to offer you a great deal for listening to the Ask Noah Show. Call today and ask about the discount for listeners of the Ask Noah Show! Altispeed Technologies (http://www.altispeed.com/) Contact Noah live [at] asknoahshow.com -- Twitter -- Noah - Kernellinux (https://twitter.com/kernellinux) Ask Noah Show (https://twitter.com/asknoahshow) Altispeed Technologies (https://twitter.com/altispeed)

Reversim Podcast
419 Navigation @Waze

Reversim Podcast

Play Episode Listen Later Sep 1, 2021


[קישור לקובץ mp3]שלום וברוכים הבאים לפודקאסט מספר 419 של רברס עם פלטפורמה, התאריך היום הוא ה-24 באוגוסט 2021, עדיין בקיץ הלוהט של ישראל . . . (אורי) עדיין באוגוסט . . . (רן) עדיין באוגוסט . . . אוטוטו - והכבישים כבר מתחילים להראות סימני אוגוסט, וזה רמז וסימן לפרק של היום: היום אנחנו מתכבדים לארח את חנוך מ-Waze - היי חנוך! ברוך הבא ותודה שבאת.(אורי) מצאת את בדרך? . . . . (חנוך) כן, והגעתי לפה עם ה-ETA, ממש בול . . . (רן) מעולה - אז תיכף נדבר על איך באמת עושים את זה, את כל הסיפור של (1) למצוא את הדרך ו-(2) לשערך ETA . . . חנוך עוסק בעיקר ב-Routing ב-Waze - ובוא, חנוך - ספר קצת על עצמך, מה אתה עושה שם, קצת על Waze היום - את Waze אני מניח שכולם מכירים, אבל מה עושים ב-Waze היום? אז בבקשה . . .(חנוך) אוקיי - אז שלום לכולם, אני חנוך, 4 שנים בישראל, 11 שנים ב-Googleהגעתי ל-Google אחרי התואר השלישי ב-Columbia ואמרתי להם שאני, במקצוע ב-++C ב-Hardware - אמרו “אחלה, בוא תעשה JavaScript ארבע שנים . . . “אחרי זה עשיתי כמה תפקידים ב-Google ואחרי זה עשינו עלייה - ובאתי לפה והתחלתי ב-Wazeכרגע אני “ראש ניווט” ב-Waze - זה אומר שיש לי שני צוותים - אחד שעובד על Routing - מה המסלולוהשני שאחראי על מה ה-ETA - כמה דקות זה יקח להגיע, למסלול (אורי) אז “מחשב מסלול מחדש” - זה אתה?(חנוך) כן . . . .בגדול כן - מצטער, זה אני . . .אבל גם “הגעת ליעד!” - כל הדברים הטובים.אז אני אחראי על הגרסא הנוכחית - זאת אומרת ש-Waze, כמובן, היה קיים הרבה שנים לפני כן, ויש הרבה היסטוריה של Waze בישראלכרגע, Waze הוא חלק מ-Google, כמו שרמזתי - כבר ב-2013 קנו אותם, וזה אומר שיש לנו תמיכה של החבר'ה פה בארץ ואפשר לעשות דברים שאולי בזמן שזה היה סטארטאפ קטן אז לא היו עושים.כרגע אנחנו עושים הימור גדול על Waze Carpool - אולי שמעתם על זה, יש לנו גרסא חדשה . . .זה . . . אני לא רוצה להגיד שזה הפוקוס היחיד, אבל זה אחד מהפוקוסים הגדולים שלנו.אז זה אחד מ”העתידים” של Waze - לאן Waze הולך: חלק זה גם להשקיע עדיין ב-Waze שאתם מכירים, של ה-Routing וה-ETA, שאני אוהב - וגם חלק שהוא של ה-Waze Carpool.אז אנחנו, פחות או יותר, עושים את שניהם במקביל.(רן) אוקיי, אז כן - אז למי שלא היה כאן בעשור האחרון: Waze היא היום חלק מ-Google, ואחת השאלות שעולות היא “רגע! אבל יש את Google Maps, או את Google Mobile Maps, ויש גם את -Waze . . . .” - אז מה ההבדל ביניהן? מתי אני ארצה להשתמש ב-Waze ומתי אני ארצה להשתמש ב-Google Mobile Maps?(חנוך) Google Maps, כן . . . אז זו שאלה טובה, והיא שאלה שלפעמים לא ברורה . . . וודאי יש חפיפה בין מה שהאפליקציות עושות.ה-Waze מפוקס אך ורק על הנהג, למשל, ויש הרי use cases אחרים - יש הליכה ברגל, יש חיפוש מקומות, יש Public transportation . . . אנחנו לא מפוקסים על זה בכלל - אנחנו רק מפוקסים בלהיות הכי טובים לנהג.גם ב-GMM, ה-Google Mobile Maps, יש את ה-Mode של הנהג - אז יש לנו “תחרות חברתית” פנימית על העניין הזהאבל בדרך כלל, אם אתה משתמש ב-Waze אז אנחנו אומרים “כן, אמור להיות לך גם Google Maps על הטלפון, כדי תוכל לעשות את כל הדברים האחרים שאנחנו לא תומכים בהם”.אז אנחנו לא מנסים להיות התשובה השלמה.אז למה להשתמש בנו [Waze] ברכב, במקום להשתמש ב-Google Maps, אם כבר אומרים לך שיהיו לך את שניהם?אז אנחנו [Waze] מנסים לתת לך את החווייה הכי מפוקסת והכי טובה . . . אפשר להיכנס לפרטים על ההבדלים, אבל ההבדל הכי גדול זה הרעיון של הקהילה: ש-Waze הוא, מכמה כיוונים, דומה ל-Wikipedia . . . יש משתמשים שמעדכנים את הזמן שלהם ומעדכנים את המפה כל הזמןכש-Google Maps זה יותר Top-Down - זה אומר שיש להם צוות ענק שעושה את זה, אבל זה לא אותו הדבר, שאתה יכול בעצמך להיכנס ולתקן.יש לנו חצי מיליון אנשים שמתקנים [מעדכנים] את המפה באופן אקטיבי, יש לנו מיליוני Wazer-ים שמדווחים כל הזמן מה קורה על הכביש בזמן אמיתי - ומזה אנחנו בונים תמונה מאוד מאוד “טרייה”, מאוד מאוד “נוכחית” של מה שקורה . . . (אורי) . . . “עדכני” . . .(חנוך) - “עדכני”, תודה, זו כנראה מילה יותר טובה.ולמרות שלפעמים גם, כמו ב-Wikipedia, יש משהו שהוא חדש ועדיין צריך קצת שיפור . . . אז אנחנו יותר רצים להיות היותר מעודכנים מלהיות אלו שעברו [בדיקה] מקצועית [מדוייקת].(רן) אז אמרת (1) אחד-עשר מיליון אנשים שבאופן אקטיבי . . .(חנוך) חצי-מיליון . . .(רן) חצי-מיליון, סליחה . . . חצי-מיליון אנשים שבאופן אקטיבי הולכים ועורכים מפות, מעדכנים דרכים חדשות . . .(אורי) . . . וזה בכל העולם, חשוב להדגיש . . .(רן) . . . בכל העולם, כן - המספר מאוד מאוד משמעותי.(חנוך) הכמות של הממש-אקטיביים היא כמובן הרבה יותר קטנה מזה - יש חצי-מיליון שעשו את זה בערך, אני לא יודע, בשנה האחרונה - ועשרות-אלפים שעושים את זה כל הזמן.(אורי) אז חנוך - אם ה . . איך נקרא לזה - Driver Assistant? אם אתה צריך לתת שם לקטיגוריה של ה . . .(חנוך) אני אוהב את ההגדרה של “ניווט” . . . ואנחנו מפוקסים על זה שאנחנו צריכים לתת לך את את חוויית הניווט ה . . . זה לא רק . . .בוא נדבר על ההבדל באפליקציה בתוך הרכב לבין מה שהיה לפני כן - שהדפסת משהו מ- Google Maps [!], או MapQuest בזמנו . . . אז ההבדל העיקרי הוא שאנחנו בודקים עבורך כל הזמן, מעדכנים את המסלול, בודקים את ה-ETA, נותנים לך אופציות חדשות - וכן, אנחנו איתך כל הזמן.אני לא רוצה להגיד Driver Assistant בגלל שכמו שאמרתי, הפוקוס שלנו הוא אך ורק על הנהיגה - יש צרכים שיכול להיות . . . אנחנו נותנים לך כפתור ל-Spotify, אבל בגדול זה לא הפוקוס שלנו . . .(אורי) כן . . . השאלה שלי היא שאלת Market Share - מתוך סך-כל הנהגים שמשתמשים בתוכנת ניווט כזאת או אחרת, אתה יודע להגיד כמה ב-Waze, כמה ב-Google Maps?(חנוך) ב-Waze יש לנו 140 מיליון משתמשים - חודשי [MAU], אז זה פחות או יותר.אני לא יודע להגיד לך את המספר ב-Google Maps . . . אני יכול להגיד לך, אני חושב, שאם אתה רק מתפקס על ה-Use Case שלנו, של הנהג, ולא על ה-Use Case הכללי, שיש לנו תחרות משמעותית גם מ-Google Maps, גם מ-Apple Maps . . .יש לנו חלק משמעותי מהשוק, עם הפוקוס הזה - אם אתה אומר [שואל על] אחוז האנשים שמשתמשים בטלפון כל הזמן, אז אנחנו פחות בגלל שכמו שאמרתי, אנחנו רק עושים תפקיד אחד.(אורי) לא, אני מדבר על Drivers . . . (חנוך) אז אנחנו ב-Drivers . . . אני לא יכול להגיד מספר . . . יש לנו 140 מיליון שלנואת המספר של השוק אתה יכול אולי לחפש במקומות אחרים [יש להם חטיבה-אחות שמפתחת פתרון לא רע לחיפוש] - אבל זה אחוז לא קטן של השוק.(רן) כשאתה אומר “שימושים במפות שלא ל-Driving” - הכוונה לנסיעה באוטובוסים, חיפוש מסעדות, דברים בסגנון הזה? . . .(חנוך) כן - יש לי -Google Maps על ה-iPhone [!] שלי . . . (אורי) . . . ללכת ברגל, רחמנא ליצלן . . .(חנוך) בדיוק . . .(רן) אז אתמול נסעתי באוטו ועמדתי בפקק, כמו שקורה הרבה פעמים בכביש 2, והבת שלי ישבה לידי [אינני הנהג!] ואמרתי לה שמחר אנחנו מקליטים פרק עם בחור נחמד מ-Waze, אז היא אמרה לי “אה, רגע - אבא, יש לשאלה!” . . . אז בוא נשמע את השאלה שלה - ומיד נענה . . .(חנוך) אוקי . . . .(יעל) שלום, קוראים לי יעל, אני בת 9.5 ואני מגיעה מיוקנעם - ויש לי שאלה: איך Waze יודע איפה יש שוטר ומתי?(חנוך) אז התשובה היא שאת אמרת לנו . . . יש כפתור כתום ב-Waze, בצד ימין-תחתון, לוחצים עליו - ויש אפשרות לדווח כמה דבריםאנחנו לוקחים את הדברים הללו, ואם זה נראה אמין - זאת אומרת, מי שלחץ, כמה אנשים לחצו וכו' - אז אנחנו נותנים את העדכון הזה לכולם.(יעל) ויש לי עוד שאלה - איך Waze יודע איפה יש פקקים?(חנוך) פה התשובה היא אחרת - אנחנו מודדים את התנועה על הכביש כל הזמן, בכל כביש בעולם שאנשים עם Waze עברוולפעמים אפילו בלי Waze - יש לנו גם Inputs אחריםאז אנחנו יודעים מהי המהירות הנוכחית על הכביש - וגם יש לנו את ההיסטוריה שיצרנו מכל הימים שלפני זה, של מה שאמור להיות פה.אז כשאנחנו רואים שזה חריג, שהמהירות הנוכחית על הכביש היא הרבה פחות ממה שהיינו מצפים לראות עכשיו, אז אנחנו מסמנים את זה כפקק, כי כנראה שיש משהו.במקביל, יש אפשרות לדווח, עם אותו הכפתור הכתום, יש אפשרות לדווח על פקק - וכשאנחנו רואים גם שהמהירות חריגה ואיטית מדי וגם שיש דיווחים, אנחנו נחבר את זה אחד לשני ונגיד ש“זה איטי בגלל הפקק”.אבל אנחנו יודעים על הפקק בכל מקרה - פשוט אנחנו לא כל הזמן אומרים שיש פקק עד שמדווחים . . . (רן) זאת אומרת שאתם צריכים להגיע לאיזשהו Confidence מספיק משמעותי שבאמת יש שם פקק . . .(חנוך) להגיד שיש פקק זה לא דורש Confidence משמעותי - בשביל להתחיל לשנות את ה-Inputs של המסלול ושל ה-ETA, אנחנו עושים את זה באופן אוטומטי כל הזמן.(רן) כן . . . אוקיי, אז זאת אומרת שאם, לצורך העניין, כמה מכוניות עצרו בצד, אתם יכול להיות שאפילו בטעות תחשבו שזה פקק - אבל זה לא מספיק כדי לשנות את ה . . .(חנוך) אז אם זה כמה שעצרו בצד, אז אחרי זמן לא-רב אנחנו נבין שהם לא זזים, אז זה לא שאנחנו פשוט “או, וואו! . . .”וגם אם אנשים אחרים עוברים את ה-Segment הזה, אז אנחנו נדע שאפשר לעבור פה, ופשוט הם לא . . . זה חלק מה-Modeling של מה שקורה על הכביש, שזה מסובך . . .(רן) אז עכשיו, אנחנו בעצם התכנסנו כאן בעיקר כדי לדבר על Routing, ואני מניח שהשאלה הראשונה שעולה בראש, לפני שמדברים על איך עושים Routing, זה בכלל איך מייצגים כביש . . . זאת אומרת, איזה מודל של העולם אתם מחזיקים? איך אתם ממדלים את הכבישים, את הערים את הרמזורים את ה . . .(חנוך) אתה יכול לראות, בגלל שזה פתוח לכולם - זה אחד היתרונות ב-Waze, שאתה יכול להיכנס ל-Waze Account ולראות בעצמך בדיוק איך אנחנו עושים את זה . . .וגם לשנות פרטים . . .בגדול, יש לנו Data structures פנימיים שאנחנו עשינו, שממדלים בדיוק את כל הדברים שאמרת - ועוד מיליון פיצ'רים של הכביש . . .וכל האינפורמציה הזאת, כולה - היא מהקהילה.אז אין לנו בכלל אנשים שיושבים ומתקנים - כולו נכנס דרך אנשים שאמרו, באיזור שלהם, “אני רוצה לתקן את האיזור שלי”.דוגמא טובה לאיך שזה עובד - אני גר בתל מונד [שכן!] ואצלי בבית, בגינה, לפי Google Maps יש רחוב, לא קטן, עם שני נתיבים . . . ב-Waze הוא לא קיים - וגם פיזית הוא לא קיים . . . (אורי) עד שיגיעו ויפקיעו לך חצי מהחצר. . . (חנוך) כן, אז יש מישהו מצד ימין ומצד שמאל, אז בטוח שהם יגיעו אלי אם כבר עברו בגינה של מישהו אחר, אז אני לא דואג לזה . . .אבל אני אוהב את הדוגמא הזו - בגלל שזה שנים ככה . . . אפילו פעם אחת הגעתי עם זה לראש ההנדסה של Google Maps, שכמובן שאנחנו עובדים איתם, והוא אמר לי “בוא, תפתח באג, חנוך, ואני אתקן את זה בשבילך . . .”אמרתי “לא, אני אוהב את זה!” - זו הדוגמא הכי טוב שאני יכול להשתמש בה להבדל בשיטה, באיך שאנחנו מחזיקים מפה, בין Google Maps ל-Waze . . . (רן) כן, אבל פה אתה מדבר על איזה Ground Truth יש לך, איזו אינפורמציה אתה מקבל . . . אני מדבר . . . בוא נדבר הנדסית . . .(חנוך) בעיקר זה גרף . . . הדבר הכי חשוב זה Connectivity, זה גרף של סגמנטים (Segments) עם Nodesזה קצת אחרת ממה שהיית חושב באופן טבעי - אם היית כותב את זה על ה-Whiteboard, אז ה-Node-ים היו ה-Intersections והצלעות [קשתות] היו הרחובות, איך שזה נראהאבל זה להיפך - בגלל שאתה עובר דרך Intersection לכביש, אז ה-Intersection הוא הצלע, בגדול, והרחוב עצמו הוא ה-Node, זה קצת אחרת ממה שהיית חושב.אבל אחרי זה, אנחנו מדברים על Graph-search טהור - אז על כל צלע יש כמה נתונים, שמהם אנחנו בונים את ה-Cost של לעבור אותה.ה-Cost הוא Time-dependent- גם מתי התחלת וגם כמה עברת בתוך החיפוש עצמו.חלק מהסיבוך הגדול זה לעשות ניחוש קדימה - לא רק מה הפקקים הנוכחיים אלא גם מה יקרה בעוד 30 דקות, כשאתה תגיע לכביש 2, על כביש 6 - אז אנחנו עושים אינטרפולציה (Interpolation) כל הזמן של ההיסטוריה ושל ה-Real-time - קדימה.אבל בגדול, אנחנו מדברים על משהו מה-Text-book - זה *A על גרף, שנראה כמו גרף . . .(רן) כן, אוקיי - אז אם אני מבין נכון, למעשה “כביש” זה Node, ובין כל שני כבישים יש קשת, אם יש צומת שמחברת ביניהם באופן ישיר - זאת אומרת שלצורך עניין, צומת מסויימת יכולה להתבטא במספר קשתות, כי היא יכולה לחבר בין מספר כבישים שונים . . (חנוך) כן - אני קצת מפשט את העניין, בגלל שיש כמה סוגי Nodes וכמה סוגי סגמנטים ויש Intersections ו-Junctions שמאוד קשה לעבור [- הוסף כאן את החיבור האהוב עליך לכביש 6 -]“צומת” יכול להיות מודל בפני עצמו - לפעמים צומת הוא מספיק מסובך שאנחנו מייצרים גרף קטן רק עבורונכנסים לתוך הגרף הזה ויוצאים ממנו - אז זה גרף נפרד רק לצומתאבל בגדול, אנחנו מדברים על גרף ממשי בסיסי . . .לא בסיסי - זה משהו שקשה לתחזק, אבל גרף שאתה יכול לקרוא עליו ב-Computer Science 101.עושים את זה ככה, ואז עושים Dijkstra עליו ואחרי זה כל השיפורים וכל מה שעושים אחרי Dijkstra.(רן) אוקיי - וזה גרף כזה פר-מדינה? זאת אומרת, מערכת כבישים . . .(חנוך) יש לנו שלוש סביבות - אחת זו ישראל - התחלנו מישראל, וישראל, למי שלא לא יודע זה “אי” . . . אז יש לה גרף נפרד, שבו אנחנו לפעמים מנסים פיצ'רים חדשים שאנחנו לא מוכנים עדיין להעביר לכל העולם.יש לנו מפה ל-North Americaויש לנו מפה שלישית של כל העולם . . . אז בפועל זה שלוש מפות.(רן) של כל . . . זאת אומרת, לצורך העניין - אוסטרליה זה באותה מפה עם רוסיה או . . . .(חנוך) כן, אין לזה הרבה משמעות כי אין Connectivity ביניהן, אבל זה נמצא באותו קובץ.(רן) ואיך, “פיזית”? - משתמשים באיזשהו Graph Database, או שמידלתם לכם איזושהי סכמה (Scheme) משלכם?(חנוך) הכל בזיכרון . . . אנחנו מעלים את הכל לזיכרון ובונים את כל הגרף, כי אנחנו צריכים “לרוץ" על זה מאוד מהראנחנו מדברים על milliSeconds של כל . . . בדיקת סגמנט זה משהו שחייב להיות מאוד מהיר, ממש - לא milliSeconds אלא microSeconds של מעבר על סגמנט.אז אנחנו חייבים לעבור על עשרות-אלפי סגמנטים בכל חיפוש - אז הכל חייב להיות בזיכרון, אין אפשרות אחרת.(רן) אוקיי, הבנתי - ומפת כל העולם יכולה להיכנס לתוך הזיכרון . . .(חנוך) כן, אנחנו מדברים על עשרות . . . עד 100Gb, משנה בדיוק איזו מפה ואיזו גירסא, אבל כשמשווים את זה לענן זה לא הרבה בכלל.(רן) אוקיי, אז אמרת שעל הגרף הזה מפעילים *A . . . למי שלא זוכר את החומר, בגדול, מה ה-*A עושה? אני רוצה להגיע מ-Point A ל-Point B - מה עושים?(חנוך) אגב, אני רוצה להגיד פה שאנחנו, למיטב הבנתי, היחידים שעדיין משתמשים ב-*A . . . זה אלגוריתם מאוד ישן, לפחות 30 שנים או 40 שנים, לא יודע - מאוד ישן [יותר מ-50, לפחות לטענת ויקיפדיה].ויש סיבות לכך שאנחנו משתמשים בו - אני אסביר עליו, ואחרי זה אני אסביר למה אנחנו לא עושים משהו יותר “מודרני”.אז *A הוא גרסת Graph-Search . . . בוא נתחיל מ-Breadth First Search ,שאתה מחפש בכל כיווןיש לזה גרסא יותר מתקדמת שנקראת Dijkstra, שאומר את אותו הדבר - חוץ מזה שיכולות להיות צלעות עם מחירים שונים, ואז זה לא בדיוק Breadth First אבל עדיין אתה הולך כל הזמן ומנסה למצוא את הדרך הכי קצרה לשלב הבא.(רן) כן - מה שנקרא “Shortest Path” . . .(חנוך) בדיוק - ו-*A הוא גרסת Dijkstra שמשתמשת במידע חיצוני . . . אני מכיר משהו על הגרף - זה לא רק Connectivity, יש גם משמעות פיזית - אז אני יכול להגיד לך ש”בכיוון הזה אי אפשר לחזור . . . אי אפשר שיהיה משהו”, וזו היוריסטיקה (Heuristic)(רן) אז היוריסטיקה אומרת לך ש”אפילו שהמחיר פה נראה זול, לא כדי ללכת לשם, כי . . .”(חנוך) . . . בגלל שאנחנו יודעים שאי אפשר, בכיוון הזה, שיהיה מסלול יותר מהיר.לדוגמא - היוריסטיקה הכי ידועה זה Aerial Distance - אני אומר שנניח שאתה הולך לשם, בכיוון הזה, ונניח שבכיוון הזה יש כביש ישר ומהיר - “כביש 6” הולך בדיוק מהנקודה הבאה ועד ליעד” . . . - אפילו אם זה היה נכון, זה לא היה מספיק.אז אם כן, אין סיבה להמשיך בכיוון הזה.(רן) אז אתה אומר - במקום לפזר את החיפוש ל-360 מעלות, אם אתה הולך בניגוד לכיוון היעד, באיזשהו שלב כנראה שאין טעם להמשיך, כי אפילו אם יהיה כביש ישר [לשם] זה עדיין מרחק גדול מדי.(חנוך) כן, והיתרון ב-*A הוא שמתחילת החיפוש הוא מחשב כל הזמן מחדש - כל משתמש רואה, כל פעם שלוחצים על “Routes” . . . אין שום Pre-caching, אנחנו מדברים על “מפה טהורה” וחיפוש ממש מחדש.זה מאפשר לנו פרסונליזציה (Personalization) מאוד חזקה לחיפוש - גם האופציות שאתה יכול לראות באפליקציה עצמה, גם בדברים שאנחנו שומרים עליך . . . לדוגמא, יש לנו Opt-In Feature שנקרא Personal ETA, שאנחנו רואים איך נהגת ב-30 הימים האחרונים [בהצלחה בסגר הבא . . . ] זה Opt-In לגמרי, אבל עם זה, אנחנו יכולים להגיד: “אתה מהיר מהרגיל, או איטי מהרגיל, אז אנחנו משנים את “המחיר” של כל צלע בשבילך” . . . אז יש לנו הרבה דברים כאלה.וזה מתאפשר ב-*A בגלל שאתה [עושה] הכל ממש מחדש.המחיר של זה זה זמן . . . אם אתה לחצת על Waze ביחס ל-Google Maps, אתה יכול לראות שלוקח לנו הרבה יותר זמן לעשות Route מאשר התחרות, זה יותר איטי . . . (רן) אז אם נסתכל על ה-Worst Case Scenario, לא יודע מהו - נגיד West Coast to East Cost בארה”ב . . . (חנוך) זה יכול להיות שניות . . . 10 שניות אפילו אפשר . . . (רן) ומה גודל ה . . . כמו Nodes עוברים בדרך? זאת אומרת, מה המסלול הארוך . . .(חנוך) יכול להיות . . . המסלול עצמו אולי לא יהיה הכי גדול, פחות מ-1,000 - אבל החיפוש יכול להגיע ל . . . לא יודע, מיליון או יותר.(רן) ואתה אומר שאתם היחידים שעדיין משתמשים ב-*A - אז מה אחרים עושים?(חנוך) אז יש . . . הרי יש התקדמות אקדמאית על זה . . . במשך הרבה שנים, הדבר הכי טוב היה משהו שנקרא Bijection Hierarchies, שזה מתחיל עם אותו גרף של *A ובאיזשהו סדר ידוע עושה Short-cuts ששומרים . . . Short-cuts שלא קיימים באופן פיזי, אבל שומרים על המרחק בין שתי הנקודותיש סדר ויש הרבה חוכמה - ויש הרבה דעות על איך לעשות את זה . . . .יש הרבה שיטות, אבל בפועל - אתה יכול לבנות מפה, שעליה אתה יכול לעשות חיפוש מאוד מהיר.אחרי זה יש גם משהו חדש מ-Microsoft . . . אגב, כל ה-Routing הטוב הוא מ-Microsoft Research - הם הכי טובים בזה . . .אז יש את מה שנקרא Customizable Route Planning, שזה סגו של Tree שאתה בונה -אתה לוקח את העולם ומפרק אותו לארבעה חלקים, ואחרי זה ליותר ויותר . . . זה סוג-של Quad-Treeואתה שומר את הקפיצות על איזשהו חלק, אז כשאתה יודע . . . אתה לא חייב לעבור את החלק כדי לדעת - בכל מקרה אני חייב לעבור את החלק (Segment) הזה, מהכניסה הזאת - זה המחיר לכל היציאות, זה בגדול . . .וזה עכשיו יותר פופולרי . . . לשניהם יש את אותו . . . לא בעיה, כי אלו יתרונות וחסרונות - זה הרבה יותר מהיר ודי הרבה יותר זול, אבל אתה חייב להכניס “חלק מהאמת”, כלומר, את המחיר של כל צלע - זה חייב להיות חלק מהאימפלמנטציה (Implementation).ב-Congestion hierarchies, הסדר שבו את צריך את ה-Short-cuts נגזר מהמחירים של הצלעות - וגם ה-Shortcuts עצמם, אם אתה משנה את ה-Cost שלהם - יש מצב שמה שנשאר לא יהיה נכון.אז אתה מתחיל, ואתה חייב . . . אם אתה משנה את המחיר של איזושהי צלע, בגדול - אתה חייב לבנות את כל המפה מחדש.כש-Customizable Route Planning הוא קצת יותר טוב, יש סוגי Metrics שאתה יכול לשנות On-the-fly, אבל אתה לא יכול לעשות כל מה שבא לך . . .וב-Waze, אתה יכול . . . נגיד שמחר יש לנו פיצ'ר חדש: אתה מעדיף רחובות או כבישים שמתחילים באות ר' . . . אני מעדיף דברים שמתחילים באות ר' . . . אין בעיה - אני אעשה עוד Cost Function בתוך המערכת, כדי לתת איזשהו “Punch-up” למשהו שמתחיל ב”ר'” - זורם . . . כל דבר כזה.בגדול - זה בלתי אפשרי . . . ואנחנו החלטנו - זו החלטה מהמנכ”ל לשעבר, נעם ברדין, שישבנו ודיברנו על זה כמה פעמים - שהאופי של Waze זה הפרסונליזציה (Personalizing), וזה משהו שאנחנו מוכנים לשלם עליו יותר - כסף זה זמן, ואנחנו מוכנים לשלם יותר כדי לתת לך עוד טיפה יותר, מסלול יותר טוב.וזו אחת מהסיבות שאנחנו לא עושים את כל ה . . . זה אחד מההבדלים, הפוקוס על הנהג, לעומת לעשות דברים אחרים - זה נותן לנו את האפשרות לעשות ממש Drill-down לנהג, ועדיף לעשות את זה . . . [ואז החבר'ה של DeepMind שחררו את זה . . .](רן) אני סקרן האם אתם יכולים למדוד עד כמה לקוחות אוהבים את זה, או כמה לקוחות משתמשים בזה? לצורך העניין, אם לקחתם החלטה לא טובה במימוש של ה-Routing, האם אתם רואים את ה-Retention יורד? האם אתם רואים . . . האם אתם רואים החלטות כאלה מתבטאות בהתנהגות משתמשים?(חנוך) יש לנו Checking על זה, ולצערי אני לא יכול להכנס לזה . . . אבל ודאי שיש לנו Checking על זה.אנחנו מבינים אילו סוגי Routing ואילו סוגי תשובות אנשים שמחים איתם ואילו לאאנחנו בודקים את זה כל הזמן - גם פרואקטיבית - אם איזשהו מדד עולה אז מה קרה ואם יורד אז מה קרהוגם ראקטיבית - אנשים שולחים לנו באגים כל הזמן ואנחנו בודקים כמות לא קטנה של הבאגים ורואים אם יש משהו לא נכון.אני יכול להגיד שחלק מהדברים שאנחנו עושים זה עניין של אי-אפשר . . . לדוגמא: פה בישראל זה לא מעניין, אבל בברזיל, ששם יש לנו הרבה משתמשים, יש עניין סביב באיזה יום אתה יכול להכנס לעיר עם ה-License plate הנכון, וזה משהו שדורש שתיהיה לך מפה ייעודית עבורך . . . עם כל ה . . . זה פשוט Explosion of parameters אם אתה לא עושה את כולו, אחרי זה, גם לכל License plate.אז יש סוג של דברים שאנחנו יודעים שאנשים משתמשים [עבורו] בנו - בגלל שאנחנו ממשים פיצ'רים שאי אפשר לעשות בצורות אחרות.(רן) כן, אז זה מביא אותנו באמת לשאלות מעניינות על רגולציות . . . אז, למשל, ימים שבהם מותר להיכנס לאיזור או כמו שהזכרת License Plates שמותר להם להיכנס ביום א' וכאלה שמותר להם ביום ב', לפי האם זה זוגי או לא זוגי או כל שיטה אחרת . . . (אורי) אני חושב שבישראל הייתה בעיה של שטחים מסויימים, שלא רוצים ש . . .(רן) . . . זהו, אז למשל שטחים עירוניים, שכונות יחסית שקטות, שפתאום אולי נחיל של מכוניות עובר דרכן כי היה פקק באיזור . . . דרך אגב, אני מרגיש את זה כל מוצאי-שבת דרך העיר שלי - תמיד במוצאי-שבת מאוד עמוס . . . (חנוך) מצטער . . .(רן) לא, אני מניח שכל אפליקציית Routing אחרת גם הייתה עושה את זה . . . אז כן, למזלי זה יחסית על כבישים ראשיים אבל זה בהחלט מורגש . . .(חנוך) אז אני יכול לענות על זה שמבחינת רגולציות אנחנו . . . דווקא זו אחת החוזקות שלנו, שיש לנו Model מאוד מאוד גמיש לכל סוג רעיון חדש שמישהו יגיד, שעכשיו “רק רכב לבן בשעות זוגיות יכול להיכנס לפה” - אנחנו יכולים לתמוך בזה.ועוד הרבה דברים Hyper-localized כאלואבל מה שאמרת לגבי ישוב שקט - אנחנו בכוונה אומרים שהנהג יכול ללכת לכל מקום שמותר מבחינת החוק.אז אנחנו לא מכניסים, בשום מקום בעולם - לא בישראל, יש הרבה ביקוש לזה בקליפורניה - אנחנו לא מכניסים שום Restriction שלא מגיע מבחינה חוקיתאנחנו עונים כל הזמן לאנשים שזה מדאיג אותם, שהם יכולים לדבר עם פוליטיקאים ולדבר עם הממשלה שלכם שיעשו כאן איזשהו Restriction - ואנחנו נקבל אותו.ברגע שיש כזה חוק - נכבד אותו.(רן) כן, אז אתה אומר שזו לא בעיה לממש את זה, אבל אתם צריכים לקבל את ההנחיה מהרשות המקומית . . .(חנוך) לא רק שזו לא בעיה - אנחנו עושים את זה כל הזמןיש הרבה מקומות בעולם שיש בהם הוראות כאלו - אבל אנחנו לא יכולים לעשות את זה בשבילך . . . (רן) אתם לא מחוקקים, אתם . . . (חנוך) אנחנו לא רוצים להיות בעסק הזה . . . זה interest שלך יחד עם interest של הנהג, ואנחנו לא יכולים לענות.(אורי) יש לי שאלה . . . (חנוך) בבקשה . . .(אורי) בשביל זה אני פה . . . האם אתם מתייחסים לעובדה שמישהו מחליט שלא ללכת לפי ה-Route?(חנוך) כן . . . Compliance . . . מישהו שכל הזמן לא החליט אצלנו [כמו שהמלצנו) או סתם פעם אחת?(אורי) לא, הוא . . . נתת לו Route מסויים, והוא מחליט To Challenge - להגיד “אני חושב שיש דרך יותר קצרה” . . . השאלה היא - אתם לא יודעים, הרי, שהוא אמר “וואלה, ה-Waze הזה לא יודע, אני חושב שיש דרך יותר קצרה” - אתם פשוט רואים אותו סוטה מה-Route . . . אתה תעשה “Recalculating route”, אבל השאלה היא האם אתם מתייחסים לזה כאל סיגנל, זאת אומרת . . .(חנוך) לא ב-Real-time, אבל אנחנו כן בודקים - זאת אומרת שאנחנו בודקים . . .(אורי) לאו דווקא ב-Real time, השאלה היא האם אתם מסתכלים ומתייחסים אל זה כאל סיגנל של . . . (חנוך) . . . יש משהו שאנחנו אולי לא מכירים, כן.אנחנו עושים מחקר ongoing, אבל זה לא online בכלל - כי כשאנשים לא מקשיבים לנו, ובמקרים חריגים שהם צודקים, יש לנו מדד לזה, ואנחנו בודקים כל הזמן אחוז מסוים של זה - לראות מה קרה.אגב, ברוב הפעמים יש סיבה שאנחנו לא יכולנו לתקן - לדוגמא: יש פקק, שהוא [הנהג] לא היה צודק [קודם] - אבל עכשיו הוא צודק בגלל הפקק שנוצר.או שבמקרים רבים יש מקרה של משהו לא חוקי - עשה U-Turn לא חוקי . . . אז זה לא משהו שאפשר להגיד לו . . . (אורי) לא, אני לא מדבר על זה - אני מדבר באמת על המקרים האלה ש . . .(חנוך) אז כן - זה קורה, ואנחנו בודקים את זה.הקטע הוא שכל אחד משתמש ב-Waze כל היום - פעמיים ביום, שלוש פעמים ביום - ולא זוכר בכלל כש-Waze היה ממש על המספר הנכון, וזוכר את הפעם היחידה שהוא לקח שמאלה ו”עבד” על Waze קצת . . . זה יחסית חריג . . . אבל אנחנו כן בודקים את זה - וכל אחד זוכר את המקרה שזה קרה לו, אבל אנחנו כן עושים . . .(אורי) אתם משתמשים בזה כסיגנל כדי להשתפר? (חנוך) כן - אבל לא באופן אוטומטי.אנחנו בודקים את זה ולומדים מזה - ובמקרים רבים עושים תיקונים, כמו “שיפוצים” למערכת . . . אם הבנו שיש בזה איזשהו סוג של דבר שלא חשבנו עליו, אז לפעמים מתקנים את זהלפעמים לומדים שיש בעיה עם המפה - יש מצב שאנחנו אמרנו ככה בגלל שאנחנו חשבנו שהיציאה הזאת תיקח שתי דקות, ובפועל זה עשר שניות . . . וכשאנחנו רואים את זה, אז זה כן Online נכנס לתוך המערכת ואומר “או, וואו - אנחנו טעינו פה ואפשר לתקן את זה” - ועוד יום או יומיים זה כבר יתחיל להיכנס להיסטוריה.(רן) בוא נדבר רגע על ETA, כי המילה עלתה [ואיזה מתכנת בעולם לא אוהב שמדברים איתו על ETA? . . . ]אז אני לא יודע אם אתה זוכר, אבל אני חושב שלפני כמה שנים היה איזשהו בחור, אני חושב אמריקאי, שעשה עבודה מאוד יסודית והחליט שהוא משווה בין ה-ETAs השונים שצפים ב-Google Maps, ב-Waze ו-Apple Maps, אני חושב [ב-Reddit יש כמה Waze vs Google ETA, וכמובן ב-Hacker News] - ואני חושב, אם אני זוכר נכון, שהמסקנה שהוא הגיע אליה היא ש-Waze דרך כלל אופטימיסטית, זאת אומרת - נותן ETA קצת יותר קצר מה-ETA האמיתי, Google Maps קצת יותר פסימי ולא זוכר מה הוא אמר על Apple Maps . . . אבל בוא . . . (אורי) הוא בטח אמר שה-ETA של Apple נורא יפה . . . [1+](רן) מעוצב יפה . . . (חנוך) כן . . . (רן) אז איך מחשבים ETA? איך אתה יודע באמת כמה זמן הולכת לקחת נסיעה?(חנוך) אז יש כאן שתי שאלות, ואני אשמח לענות על שתיהן . . .אז הראשונה, לגבי התחרות הזו . . . זה מאוד שונה ממקום למקום, ואני זוכר את הבלוג-פוסט שהזכרת, והוא עשה את זה באיזשהו מסלול אחד שלו, על פני כמה ימים . . . יש מקומות שאנחנו יותר מדוייקים, יש מקומות שבהם GMM יותר מדויק ויש מקומות שבהם Apple Maps כנראה . . . לא יודע, לא מצאתי, אבל כנראה אפשר לראות.אני בטוח שבאיזור ה-Headquarters שלהם הם מאוד מדוייקים . . . [בנסיעה במעגל מסביב?]אבל את זה אני יכול להגיד בוודאות - הגרסא הנוכחית של Waze, של ה-ETA, היא לגמרי אחרת ממה שהייתה לפני שנתיים או לפני שלוש שנים, וגם של Google Maps - לחלוטין.אז אנחנו כל הזמן משפרים - כולנו, כל האפליקציות - כל הזמן משפרים את זה.עכשיו, אתה שואל מבחינת האופטימיות? אז יש לנו קצת בעית אופטימיות, אני אופטימיים בקצת יותר מדקה, בממוצע . . . זה לא משהו ענק, ואנחנו כן היינו יכולים פשוט לשנות ETA לעוד דקה - הקטע הוא שאנחנו לא יודעים איפה על המסלול לעשות את זה . . .אז אנחנו מנסים לתקן את זה, אבל יש לנו בעיית אופטימיות של כדקה . . .[שזה מעניין - כי מניסיון, כששולחים את המסלול למישהו כ-Share, לפחות בארץ, זה אכן תמיד מוסיף דקה על ה-ETA הנוכחי, ב-Total . . .](רן) עכשיו, זה נשמע די פשוט . . . זאת אומרת, מקודם דיברנו על מציאה של מסלול עם המחיר הנמוך ביותר, וכשאנחנו מדברים על מחיר אנחנו מדברים כמובן על זמן . . .(חנוך) לא, זה אשכרה לא . . . זה רק אחד מהמחיריםאנחנו רוצים לתת לך. . . אם זה היה רק זמן, אז היה ממש קל לדעת אם זה עבד לנו ומי יותר טוב וכל הדברים היו מאוד קלים.הקטע הוא ש-Route טוב הוא לא רק הכי מהיר . . . (רן) אתה יודע מה - בוא נחזור לשם עוד מעט, אבל שנייה נדבר על ה-ETA . . . בכל אופן, הגעתי למסלול, ועכשיו אני, כדי לחשב את הזמן שלו, פשוט סוכם את פרקי הזמן על המסלול . . .(חנוך) אפשר לעשות את זה, אבל יש כמה אתגרים פה - דבר ראשון זה שאם אתה עושה את זה אז זה לא הכי גרוע בעולם - הגרסאות הקודמות של Waze דווקא עשו דבר כזה, וזה עובד.הקטע הוא שיש עניין של Flow - זרימה בין הסגמנטים - ויש אינטראקציות בין זה שהייתי פה והייתי ברמזור ובעוד שני סגמנטים יש עוד רמזור, אבל אם עברתי את זה אז אני ודאי אעבור את השני בלי שזה יהיה אדום בשבילי [הנחה מאוד אופטימית על סינכרון הרמזורים בארץ . . . .](רן) “הגל הירוק”, כמו שקוראים לזה בישראל . . .(חנוך) בדיוק - אז אנחנו לא מודדים דווקא את זה, אבל יש לנו דרך למודל כללי למסלול עצמו.אז מה שקורה זה ש-*A חייב להיות מהיר - על כל צלע יש לנו, כמו שאמרתי, יש לך microSeconds בודדות כדי לבדוק אותו, ואתה חייב לבנות את המחיר של המסלול מאוד מהר.אחרי שיש לך מסלול - או כמה מסלולים, כמה אופציות או אלטרנטיבות - אתה יכול לעשות דברים יותר חזקים:אתה יכול להכניס משהו שלוקח כל מסלול ומבין בעצמו את ה-Flow שיש ממקום למקום - ויכול גם להביא עוד פיצ'רים, שלא קיימים במערכתכרגע אין לנו את זה, סתם - אנחנו אומרים שבגרסא X יהיה לנו מזג אוויר [אלון!], שאי אפשר להכניס לתוך ה-Routing עצמו, אבל אפשר אחרי זה . . .או שהיום יש שלג, אז אולי נחכה עוד כמה דקות . . . לא ידנית אלא דרך המודל.אנחנו בונים מודל שיכול לקחת את זה ולחדד את זה.בהכרח זה אומר שיש מצב שאם לא ידענו את זה על כל מסלול אפשרי, שיש אולי מסלול אחר שהיה מצליח בזה טיפה יותר - אבל מאוד נדיר למצוא את זה.בדרך כלל, עדיין - עם כל הנתונים האלה, באופן כללי, על כל המקומות - אם היה לנו את הזמן לעשות את זה לכולם, היינו נותנים פחות או יותר את אותו המסלול + ETA יותר מדויק.(רן) כן - אתה יוצא מתוך נקודת הנחה שאם שני מסלולים . . . זאת אומרת שאם אורך של מסלול אחד זה X ואורך של מסלול אחר זה Y, ו-X

Screaming in the Cloud
Creatively Giving Back to the Cloud Community with Forrest Brazeal

Screaming in the Cloud

Play Episode Listen Later Sep 1, 2021 36:36


About Forrest Forrest is a cloud educator, cartoonist, author, and Pwnie Award-winning songwriter. He currently leads the content marketing team at Google Cloud. You can buy his book, The Read Aloud Cloud, from Wiley Publishing or attend his talks at public and private events around the world.Links: The Cloud Bard Speaks: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/the-cloud-bard-speaks-with-forrest-brazeal/ The Read Aloud Cloud: https://www.amazon.com/Read-Aloud-Cloud-Innocents-Inside/dp/1119677629 The Cloud Resume Challenge Book: https://forrestbrazeal.gumroad.com/l/cloud-resume-challenge-book/launch-deal The Cloud Resume Challenge: https://cloudresumechallenge.dev Twitter: https://twitter.com/forrestbrazeal TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part my Cribl Logstream. Cirbl Logstream is an observability pipeline that lets you collect, reduce, transform, and route machine data from anywhere, to anywhere. Simple right? As a nice bonus it not only helps you improve visibility into what the hell is going on, but also helps you save money almost by accident. Kind of like not putting a whole bunch of vowels and other letters that would be easier to spell in a company name. To learn more visit: cribl.ioCorey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: Welcome to Screaming in the Cloud. I am Cloud Economist Corey Quinn, and as an industry, we stand on the precipice of change. There's an awful lot of movement lately. It feels like the real triggering event for this was when Andy Jassy ascended from being the CEO of AWS—the cloud computing division of Amazon—to being the CEO of all of Amazon, including things like not just AWS, but also the underpants store. Suddenly, we have people migrating between different cloud providers constantly.Today's guest is a change I would not have expected and didn't see coming. So, last year, on episode 127, called The Cloud Bard Speaks I had Forrest Brazeal from A Cloud Guru joining me. Forrest, welcome back.Forrest: Hey, thanks, Corey. Big fan of the show; always great to be here.Corey: At the time that we're recording this, you are unemployed, which is great because it's Screaming in the Cloud. Screaming at people on your day off is always fun. But by the time it airs, you'll have started your new job as the Head of Content for Google Cloud.Forrest: Yes. And of course, that's definitely a career change for me coming directly from A Cloud Guru, which was a wonderful place to be and it was exciting to be with them right up through their acquisition earlier this summer, but when it came time to make the next move, I ended up going to Google Cloud. I'll be starting there on Monday after this recording has been completed, and just really looking forward to helping tell the story of the cloud at a much bigger scale, something that I've been doing throughout my career with increasing levels of scale. It's exciting to do it at the level of an entire cloud provider.Corey: We'll get to the future in a minute, but I want to start by looking at the past. From my perspective, you were a consultant for a while at Trek10; we've talked about that before. You have an engineering background of building things with computers, at least presumably computers—you've been a big serverless advocate and I'm told that runs on computers somewhere, but I don't want to get into that particular debate—to the point where you were—I assume were, not are anymore—an AWS Serverless Hero?Forrest: Yes, that's right, and even going back prior to Trek10, my background is in enterprise software. I helped to migrate some of the world's largest enterprise applications from data centers to cloud when I was at Infor and continued to work on that kind of thing as a consultant later on. And in that time, I was working a lot with AWS, which was the only game in town for a lot of those years, right? You go back to 2014, 2015, I'm putting an enterprise app in the cloud, what am I going to put it on? Probably AWS if I'm serious about what I'm doing.But it's been amazing to see how the industry has grown and changed and the other options that have come along. And one of the cool things about my work in A Cloud Guru is that I really got a chance to branch out and expand, not just to AWS, but also to get a much better feel for the other cloud providers, for Azure and GCP, and even beyond to Oracle and some of the other vendors that are out there. And just to get a better understanding of how these different cloud providers thrive in different niches. So yes, it is absolutely a change for me; I obviously won't be an AWS Hero anymore, I'm having to close that chapter, sadly; I love those people and that program, but it is going to be a new and interesting change. I'm going to have to be back in learning mode, back in catch-up mode as I get busy on GCP.Corey: So, one thing that I think gets occluded with you because it definitely does with me is that you and I are both distinguishable personalities in the cloud community—historically AWS, let's be clear here—and you do your own custom songs; you write a newsletter that instead of snarky is insightful—of which I'm jealous—but it still has a personality that shines through; you wrote a children's book, The Read Aloud Cloud; you wound up having a new book that just came out last week for folks listening to this the day of release, called The Cloud Resume Challenge Book, if I'm getting the terms all in the right order?Forrest: Yeah, exactly.Corey: It's like naming cloud services only naming books instead? It's still challenging to keep all the words in the right order?Forrest: You know, I think it actually transcends industries; naming things is hard whether you're in computer science or not.Corey: Whereas making fun of things' names is a lot easier. It's something you did not do—to my understanding—as an employee of A Cloud Guru, The Cloud Resume Challenge, but it's something you did as a side project because it interested you. It's effectively, you want to get into tech, into cloud.Great. Here's a list of things I want you to do. And it ranges the gamut. And we talked about it before, but to my understanding it's, build a statically hosted website that winds up building your resume, and a blog post, and how to do all these things, CI/CD, frontend, backend, the works. It's a lot of work, but by the time you're done, you know a heck of a lot more about the cloud provider you're working with than you did when you started.Forrest: Yeah, not only do you know more than you did when you started, but quite frankly, you're going to know more than a lot of people who've even been doing this kind of thing for a couple of years. That's why we have people that take The Cloud Resume Challenge, who are not only aspiring cloud engineers but who have been doing this for a while, maybe even are hiring people, and they see this project and say, “Wow. That would look good on my resume. I've never actually sat down and plugged a frontend and a backend together on AWS,” and, “Maybe I've never had to actually sit down and think carefully about how I would build a CI/CD pipeline,” or, “I really want to get my hands dirty with Terraform,” or something like that. So, we see a whole range of people.I did a survey on this actually, and I found that about 40% of all the people who take The Cloud Resume Challenge have three years or more of professional IT experience. So, that should tell you how impressive it is, if you can figure this out as a brand new person to cloud. That's why we've seen so many of these folks change careers and go from things like plumbing, and working in a bank, and working in HR, and whatever else to starting roles, now, as cloud engineers and DevOps engineers. It's not entirely due to the challenge; not even mostly due to the challenge. These are folks who are self-motivated, quick learners, and are going to succeed no matter what, but The Cloud Resume Challenge was the thing that came on at the right time for them to build those skills and show what they had.Corey: And the fact that you put this together is incredibly uplifting for folks new to the field. And that's amazing, and it's great, and it's more content, the kind that I think that we need in this industry. You also launched a newsletter last week: the cloud jobs newsletter, which is fantastic. It's a pay-to-subscribe newsletter—which I've always debated experimenting with but never did—and lists curated jobs in the industry, sorted by level of experience required and things that you find personally interesting. You might have sponsored job listings in the future that you've already said would be clearly delineated from the others, which is the ethically right thing to do. You are seemingly everywhere in the cloud space.Forrest: Well, I mean look, I'm trying to give back. I've benefited from folks like yourself and others who have made time to help lift my career over the years, and I really want to be here to help others as well. The newsletter that you mentioned the Best Jobs in Cloud, it does have a small fee associated with it, but that's really just to help gate my [laugh] referrals so that they don't end up getting overwhelmed. You actually can get free access to the newsletter with the purchase of The Cloud Resume Challenge Book we talked about before. It's really intended to be a package deal where you prepare your resume by doing these projects, and there's a lot of other advice in that book about how to get yourself positioned for a great career in the cloud.And then you have this newsletter coming into your inbox every couple of weeks that lays out a list of jobs and they're broken down by, you know, these are jobs that are best for juniors, these are jobs where you're going to need some senior-level experience. Because what I found—and honestly, I've been kind of acting as a talent agent for a lot of engineers over the past several years as my network has grown, and I've tried to give back to others and help to connect folks who are eagerly trying to find great engineers for cool projects that are working on with folks who are eagerly looking for those opportunities. And what I've realized is whether you're a junior or whether you've been doing this for a long time, let's face it, most of us are not spending all of our time being those distinguishable personalities that you mentioned a minute ago. I like how you said distinguishable and not distinguished by the way; those are two very different words. But most of us are not spending our time doing that.You know, we're working engineers; we're working, right? We're not blogging and tweeting all the time and building these gigantic personal networks. So, it helps if you can have a trusted friend standing alongside you so that when you are thinking about maybe making a switch, or maybe you're not thinking about making a switch but you should be because of where the market is, that friend is coming alongside you and saying, “Hey, this is an awesome opportunity that I think you should consider checking out; why not just do the interview. Even if you're not really looking to move, it's always important to keep your skills fresh.” That's what this newsletter is designed to do. I hope that it'll be helpful for you, no matter where you are in your cloud career, as long as you're staying in the cloud space.Corey: And the fact that's how you view this is the answer to a question a lot of folks have asked me over drinks with theoretical conversations for years of, “Well, Corey, if you went to go work at one of these big cloud providers, it destroy everything you've built because how in the world could you be authentic while working for one of these companies?” And the answer is exactly what you're doing. It's, “Yeah, the people who pay you don't own you.” I cannot imagine that even Google could afford to buy your authenticity from you because once that's gone, you don't get it back, and you're one of those people in this space, that—I'm not entirely sure that you understand where you are in this space, so let me help enlighten you with that for a minute.Forrest: Oh, great. [laugh].Corey: Oh, yeah, like, the first thing I was starting to talk about that we have in common is that we do a lot of content, both of us and that sometimes occludes the very real fact that we have a distinct level of technical expertise, historically. You and I can both feel relatively deep technical questions about cloud services, but because our job doesn't have the word engineer in the title, it doesn't lead to the same type of recognition of that fact. But I want to be very clear: you are technically excellent at what you'll do. You also have a distinguished personality and brand in the space, and your authenticity is also unparalleled. When you say something is good, it is believed that it is because you say it, and the inverse is also true.You're also someone that is very clearly aligned with fighting for the user if you want to quote Tron. It's the, you're not here to shill for things that don't get people ahead in their careers; you're not here to prop things up just because that's where the money is blowing. Your position on this is unimpeachable. And I'm going to be clear here: I am more interested in Google Cloud now than I was before you made this announcement. That is the value of having someone like you aboard, and frankly, I'm astonished they managed to grab you. It shows a forward-looking ability that historically I have not associated with cloud marketing groups.Forrest: Yeah, well I mean, the space changes fast. And I think you've said this yourself as well, even with the services; you look away for six months and you look back and it's not the same industry you remember. And that actually is a challenge when you talk about that technical credibility because that can go away very, very quickly. So, it does require some constant effort to stay fresh on that, especially if you're not building every single day. But to your point about the forward-looking-ness of Google Cloud, I really am excited about that and that's honestly the biggest thing that attracted me to what they're doing.They clearly understand, I think, their position in the space. We know they're three out of three and trying to catch up, and because of that, they're able to [laugh] be really creative. They're able to make bold choices and try things that you might not try if you were trying to maintain a market-leading position. So, that's exciting to me. I'm a creative person, I like to do things that are outside the box and I think you can look forward to seeing some more outside-the-box things coming at Google Cloud here over the next couple of years.Corey: I'd be astounded if it were otherwise. The question I have for you is that ‘Head of Cloud' is not a junior role. That's not something entry-level that you're just going to pick some rando off of LinkedIn to fill. They're going to pick a different rando: you specifically as one of those randos. And to my understanding, you've never really touched Google Cloud in anger from a technical level before. Is that right? Am I dramatically misunderstanding, “Oh yeah, you don't remember the whole musical, and three-act stage play that you put on, and the music video, and the rock opera all about Google Cloud?” It's, “No, I must have been sick that week,” because that's the level of prolific you tend to be?Forrest: [laugh].Corey: What is your experience with it?Forrest: That's yet to come. So, check back on the Google Cloud rock opera; we'll see if that takes place. So no, I'm going to be learning about Google Cloud. This will be a chance for me to kind of start over a little bit from first principles. In another sense, I've been interacting with Google services for years.Keep in mind that Google Cloud is not just Google Cloud Platform, but it's G Suite as well, and there's a lot going on there. So, I definitely am going to be going back to being a beginner a little bit here. They do say if you can teach something to a beginner, you have to really understand it at an expert level. And I know that whether I'm doing this officially on behalf of Google or otherwise, I'm going to be continuing to try to help and educate folks wherever I can. So, it's going to be incumbent on me, if I want to keep doing that, to go deep quickly and continue to learn.I'm excited about that challenge. I've been doing a lot with AWS for a long time, I don't know everything. In fact, I know less every day with the amount that they're continuing to roll out, but this is a chance for me to expand, become a more well-rounded person to see how the other cloud lives. I'm taking that very seriously; I'm not going to be an expert overnight, but stick around, follow me. I'm going to be learning, I'm going to share what I learned, and maybe we'll all get a little better Google Cloud together.Corey: The thing I can't quite get past is that when you told me that you had resigned from A Cloud Guru, I want to be selfish here and say that there were two things that went through my mind. The first was, “Okay, it's probably AWS. I hope it's AWS,” because the alternative is you're going somewhere potentially independent, and I know you keep arguing with me on this point but you are one of the few people I could point out that could start something on the basis of cloud content with a personal brand that I would view as potentially being an audience split for what I do. And it's, “Oh, you're going to go work for a big cloud company. That's awesome. Is it AW—no, it's not.” And that one threw me for a different loop where it's, that is very odd because you have identified, clearly, publicly as the leading voice in AWS in many contexts. It just really surprised me. Did you consider looking at AWS as an alternative?Forrest: I mean first, I don't know that it's fair to say that I was a leading voice for AWS. There's many wonderful people that [crosstalk 00:14:13]—Corey: To be clear, Forrest, that was not a question. You are a leading voice in the community for AWS and understanding how it works. That is one of those things that no one knows their own reputation. This is one of those areas. Take it from me—a thought leader—that it's true. Please continue.Forrest: You have led my thoughts in that direction, so thanks for that, Corey. But to your question, Corey, regarding how did I decide what career move to make, and definitely was a challenge. And it was a struggle for me to say, well, I'm going to leave behind this warm, friendly AWS community that I know, and try something brand new. But it's not the first time I've done something like that in my career. You mentioned already that I spent a number of years as a very, very technical person and I identified strongly as an engineer.I had multiple degrees in computer science and I had worked as a frontend/backend software engineer, I'd worked as a database administrator, I'd worked as a cloud engineer, and a manager of cloud engineers, and I'd consulted for companies from startups all the way up to the Fortune 50, always on cloud and always very hands-on and writing code. I've never had a job where I didn't have an IDE open and wasn't writing code every day. And it was a tremendous shock to my system when I started moving away from that, moving a little bit more into the business side of cloud, learning more about marketing, learning how to impact the bottom line of a company in other ways. That was a real challenge, and I went through months where I kind of felt like I was having an identity crisis because if I'm not writing code if I didn't create YAML today, who am I? Can I call myself an engineer? What worth do I have? And I know a lot of folks have struggled with this, and a lot of times, I think that's what sometimes holds people back in their career, saying, “Well, I can only do what I've already done because I've identified myself so strongly with it.” So, I'm encouraging anyone who's listening, if you're at that point where you feel like, “I don't know if I can leave behind what I know because will I still be able to succeed?” I would encourage you to go ahead and take that step and commit to it if you really believe that you have an opportunity because growth is ultimately going to be a good thing for you. Getting outside your comfort zone and feeling those unpleasant cracks as you start to grow and change into a different person, that ultimately is a strength-building thing.If you're not growing, you're not struggling, you're not going to be the person that you want to be. So, tying all that back, I went through one round of that already, Corey, when I moved a little bit away from technical delivery. I'm about to go through a second round of that when I move away a little bit farther from the AWS community. I believe that's going to be a growth opportunity. But yeah, it's going to be hard.Corey: It really is. The idea of walking away from the thing that you've immersed yourself in is really an interesting thing to think about. Forgive me in advance for the next question; I have to ask it. As a part of your interview process at Google, do they make you write code in a Google Doc?Forrest: Not as a part of this interview process. I interviewed at Google years ago for a developer advocate position, actually, and made it all the way through their interview process, writing many lines of code in many Google Docs, but not this time.Corey: Yeah, I confess, I did the same with an SRE job many years ago at Google, and again, you are better at writing code than I am; I did not progress past this stage. But it was moot, honestly, because the way that the interview was conducted, the person I was talking to was so adversarial at the time and so, I got to be honest, condescending that I swore I would never put myself through that process again. But I was also under the impression that the ritualistic algorithmic hazing via whiteboarding code was sort of a requirement for every role at Google. So, things change, times change, people change. I'm gratified to know that was not a part of your interview process.Forrest: Well, I mean, I think it was more just about the role. My favorite whiteboard interview—Corey: Nonsense. Every accountant must be able to solve code on a whiteboard.Forrest: No, I don't think that's true. But my favorite whiteboard interview story and I'm sure you have a few, I remember being in an interview with someone—I won't say who it was or what company it was, but it wasn't not Google—it was some sort of problem where I was having to lay out, I don't know, a path for a robot to take through an environment or something like that. And I wrote the code, and it was fine. It was, like, iterative. It was what you would do if you had ten minutes to write something.And then the interviewer looked at the code, and he said, “Great, now write it again, but don't use any variables.” And I remember sitting there for a minute thinking, “In what professional context [laugh] would someone encourage you to do that in a pair programming situation?”Corey: Right. The response there is, “What the hell does your codebase in production look like?”Forrest: [laugh]. And of course, the answer is you're supposed to be using, like, the stack, and it's kind of like this thought exercise with the local stack. But even if you were to do that, the performance hit would be tremendous. It would not be a wise or logical way to actually write the code. So, it was a pure trivial, kind of like a just academic exercise that they were recommending. And I remember being really turned off by that. So, I guess if you're considering putting problems like that in your interview process, don't. They're not helpful.Corey: Yeah, I remember hearing at one point one of the Microsoft brain teasers which they've since done away with—credit where due—where someone was asked, “How would you go about finding out the weight of a Boeing 747?” And the person responded with the exact weight of a Boeing 747 because their previous job had been at Boeing for seven years. And that was apparently not what they were expecting to hear. But yeah, it's sort of an allegory as well for, first, this has no bearing on your ability to do the job, and two, expertise is important. There's a lot of ways I could try and Hacker News first principles my way through something like that, but the easier answer is for me to call someone at Boeing and ask them, or Google it, depending on exactly how precise I need to be and whether lives hang in the balance of the [laugh] answer to the question. That's a skill that seems lost somewhere, too.Forrest: Yeah, and this takes us all the way back to the conversation about The Cloud Resume Challenge, Corey. And why it works is it takes the burden of proof off of you in the interview, or the burden of proof off the interviewer to have to come up with some kind of trivial problem that you've done under time pressure, and instead, it lets the conversation flow naturally back to, “Well, what have you done? Tell me about a story about a problem that you have solved, a challenge you ran into, and how you got past it.” That's all work that has taken place prior to the interview that you've reflected on, that's built you as a person and as an engineer, even if you don't necessarily have professional experience. That's how I try to conduct interviews and I think it's a much healthier and more sustainable way to find people that you'll like to work with.Corey: Is this going to be your first outing at a giant multinational tech company?Forrest: No, although it will be my first time with a public company. When I worked at Infor, Infor was the largest privately owned software company in the world. I don't know if that's still technically true or not, but it'll be my first time with a publicly-traded company.Corey: Fantastic. The nice thing from my perspective is it gives me a little bit more context into what companies can and can't do, and how things are structured. It feels like your content—I mean, the music videos and things and whatnot that you do—I mean, you have something that I don't, which is commonly known as musical talent. And that's great. I can write funny lyrics, but you are not just able to write lyrics, you're able to perform, you're able to sing, the unanswered question for the entire interview right now is whether you can also dance. So, we're going to find that out at some point.Forrest: You would think that I could, Corey. I definitely seem like someone who should be able to tap dance. I regret to tell you that I can't, but I want to learn.Corey: For a lot of this, it's clearly you're doing this in front of your own piano with a microphone in front of you, doing it live, and having a—I don't know if it is a built-in webcam to a laptop that's sitting in front of you or something else, but—Forrest: I'm playing with that.Corey: Yeah, well don't take this the wrong way; it's not a high definition 4k camera, et cetera. It's the Lightning's—eh, it's your home office. You're comfortable there. It's not a studio. What I'm most excited about—from my perspective, I know what you're excited about—but you're now going to be producing content for Google and I checked the numbers in preparation for this interview.It's okay, can Google wind up affording a production house of some sort to work on your videos to upscale the production value of some of what you're doing? And I have checked; it is not the likeliest scenario—and I have no inside knowledge for those who are trying to trade on this—but yes, it turns out that Google could, in fact, shore up your content by buying you Disney.Forrest: I think that's technically true, and I do expect that to happen in the next three to six months, so that is completely inside information.Corey: Oh, exactly. Have reasonable expectations, but you could let it go as long as a year because that's when the first annual review cycle comes in and you want to give people time to let that clear through M&A and make sure that they are living up to their commitments to you, of course.Forrest: That's right, yeah. We're just about to go into the quiet period there. No, but kind of to that point, though, and you bring up the amateurish quality of a lot of these videos that I put together in terms of the lighting and the staging, and everything else. And I am doing a little bit to help with that. Like, it would be great if you could see—Corey: To be clear, that is not a criticism. I'm in the same boat as you are on this. It's—[laugh]—Forrest: So, far from a criticism, it's actually pretty deliberate. The fact of the matter is, there's something very raw, very authentic about just seeing someone sitting in their house, at their piano, playing and singing. There's no tricks, there's no edits, there's no glitz, there's no makeup team behind the scenes, there's no one who's involved with this other than just me caring a lot about something and sitting down and singing about it. And I think some of that is what helps come across to people and it helps these things travel. So yeah, I'm looking forward a lot to being able to collaborate with other fantastic people at Google, and I can't exactly promise what will come out of that, but I'm quite sure there will be more fun content to come.But I hope never to lose that, kind of, DIY sensibility. Because, again, my background is as an engineer, and the things I create, whether it's music, whether it's cartoons, whether it's books, or other things I write, I never want to lose that sense of just excitement about the technologies I'm working with and the fact that I get to use the tools that are available at my disposal to share them with you as directly and honestly and humanly as possible.Corey: Up next we've got the latest hits from Veem. Its climbing charts everywhere and soon its going to climb right into your heart. Here it is!Corey: No matter how hard you try, you're not able to hide the sheer joy you take from even talking about this sort of stuff, and I think that's a powerful lesson. For folks listening to this who want to expand into their own content story and approach things that they find interesting in a way that they enjoy, don't try and do what I do; don't try to do what Forrest does; do the thing that makes you happy. I would love to be able to sing, but I can't. I can write funny lyrics, but those don't do well in pure text form. I'm fortunate that I was able to construct a structure on my end where I can pay people who do know how to sing—like Adeem the Artist and many more—to participate in a lot of the things that I get to work on.But find the way that you want to express things and do you. You're only ever going to be second best at being Forrest or being Corey, but you're always going to be number one at being whoever you happen to be. I think that's a lesson that gets overlooked an awful lot.Forrest: Yeah, I've been playing with this thought for a while that the only real [moat 00:24:24] out there is originality, is your personality. Everything else can be cloned, but you are an individual. And I mean that to us specifically, Corey, and also the general ‘you' to anybody listening to this. So, find what makes you tick. It sounds like the most cliche device in the world, but another way, it's also the only useful advice that's out there.Corey: I want to be clear, you don't work there yet and I'm not here to effectively give undue praise to large companies, but I just want to say again how the sheer vision of hiring you is just astounding to me. That it makes perfect sense, don't get me wrong, but because I know that every large company, somewhere, at some point, internally has had a conversation of, “We really should hire Corey, except…” well, I've got to level with you, Corey without the except parts looks an awful lot like you.Forrest: Yeah, you know, you brought up earlier this idea that well, hopefully, Forrest doesn't lose his authenticity at Google. And one of the things that I appreciate about the team that I've talked to there so far, is that they really do understand the power of individuals and voices. And so that's not going to happen. You know, my authenticity is not for sale. And frankly, I'm useless without it, so it wouldn't be in anyone's best interest to buy it anyway. And that would be true for you as well, Corey. Whatever you end up doing, whether you someday ascend to the head of AWS Marketing, as is apparently your divine destiny, I know that—Corey: Well, I'm starting to worry that there's not too many people left in that org, so I'm worried people took me seriously and they think I've got this in hand or something.Forrest: You may be the last man standing for all we know. You may be able to go in and just, kind of, do this non-hostile takeover where there's just no one there to defend against you, anymore.Corey: Well, speaking about takeovers and whatnot, we talk about Google acquiring Disney so you now have a production studio on this. But let's talk about actual hard problems you're going to be solving there. Do you think you can bring back Google Reader?Forrest: That would be my dream. I have no inside knowledge of what would even be required to bring that off, but I think it's obvious that it's not just about that particular product that people like—because yes, you or I could go make a startup and create something that did what Google Reader did—but it's about what it represents. It's about the commitment that it would mean to Google's customers and to their products. So yeah, something like bring Google Reader back would be a wonderful thing for everyone that subscribes to Google but it would also be a fantastic storytelling element for Google as well. So yes, I'd be entirely in favor of something like that. I hope we can make it happen someday.Corey: Oh, as would I. YOu're in Brian Hall's org, correct?Forrest: Yes.Corey: Brian is a man who was the VP of Product Marketing over at AWS, went to Google for the same role, was sued by AWS under the auspices of a non-compete, which is just the most ridiculous thing in the world, and I want to be very clear here, you can say an awful lot about Brian Hall. I say an awful lot about Brian Hall. AWS says a lot about Brian Hall in very poorly conceived depositions and lawsuits that should never have been allowed to continue, and at least have an editor go over them, but that's a separate problem. But one thing you cannot say about Brian is that he is not incredibly intelligent. And the way that I find that manifesting is, I do not accept that he is someone with such a limited vision that he would be prepared to even entertain the idea of hiring you without giving you what amounts to effectively full creative control of the things you're going to be working on.You are not someone it would make any sense to hire and then try and shove into a box. That is my assessment of everything I've read on every conversation I've had with Googlers in the marketing org; it all speaks to something like this. Was that your impression during the interview? Specifically that you have carte blanche, not that Brian is smart. You're about to be in his org; you're obligated to say it. That's okay. We'll meet at the bar until the real Brian stories later but I'm talking about their remit here.Forrest: No, my authenticity is not for sale, but at the same time. I am a big fan of Brian's and have been since his AWS days, which was honestly one of the big reasons why I ended up joining his org. But yeah, to your question about what is that role going to look like, day to day, of course obviously, that remains to be seen, but it is my understanding that it will have a consultative element and that I will have some opportunity to help to drive some influence across some different teams. Something that I've learned as I've grown in my career a little bit and I've moved into more of management type of roles is that the people that report to you are such a small fraction of the overall influence that you should be having to be really successful in a role like that, any kind of leadership role, so much more of your leadership is going to happen indirectly and by influence, and it's going to happen slowly over time, as you build support for what you're doing and you start to show value and encourage other people to come around to your side. That's just the reality of making change in large organizations.And of course, this is by far the largest organization I've ever worked in, so I know it's going to take time. But my understanding is I do have a little bit of leeway to bring some of my ideas in, and I'm excited about that, and you can sort of judge for yourself, how successful I am, over time.Corey: My last question for you is that sort that has the potential to get you in trouble, except I think I'm going to agree with your answer to this. Do you believe that they're going to Google Reader Google Cloud?Forrest: If I believed that I wouldn't be joining? So obviously, no, I don't believe that.Corey: I have to confess that for the longest time, I was convinced that this was yet another Google misadventure, where they were going to dabble with it, sort of half-ass it, and then shut it down. Because that seems to be the fate of so many Google products out there. The first AWS service that entered beta was Simple Queuing Service. What is a queue but a messaging system, and we know how Google treats messaging products. Same problem; same story.I have to say over the last year or so, my perspective has evolved considerably. They are signing ten-year deals with very large banks; they are investing heavily in hiring, in R&D, in marketing clearly, in a bunch of different areas that are doing the right thing for the long-term. The financial analysts like to beat Google Cloud up because I think two quarters ago, they showed a $5 billion loss, either for the year or for the quarter, and, “It's not making money.” It's, “No. Given Google's position in the market, I'd be horrified if it were. The only way it shouldn't be turning a profit is if there's nowhere left to invest in the platform.”They're making the investments, they're doing the right things. And I have to say I've gone from, “I don't know if I would trust that without an exodus plan,” to, “Yeah, you should have a theoretical exodus plan the same way you should with any provider, but it's not the sort of thing that I feel the need to yank away on 30-days' notice.” I have crossed that bridge myself. In all sincerity, cheap, easy jokes aside, it's clear to me from what I've seen that Google Cloud is going to be around for the long term. Now, we are talking long-term in terms of tech companies, not 150-year-old companies based in Europe, but we can aspire to it. I expect it to outlive me, and not just because I have a big mouth and piss off large companies.Forrest: Yeah. Some of my closest friends and longest-tenured colleagues, people I've worked with for years are GCP engineers, people who are not working for GCP, but they're building on GCP services at various companies. And they always come to me and I've noticed a steady increase in this over the past, I would say 12 to 18 months where they say, “I love working on GCP. I love these services. I love the way the IAM is designed. I love the way the projects are put together. It just feels right. It feels natural to me. It scratches some sort of an itch in my engineering brain.”And then they pause and they say, “Why don't more people get this? Why don't more people understand this story?” That's a problem that I can help to solve. So, I'm really excited about helping to tell the story of Google Cloud. And yeah, that chapter is just about to be written.Corey: I can't wait to see what happens next. If people want to learn more about what you're up to, and how you're approaching these things, and sign up for your various newsletters, where's the entry point? Where can they find you?Forrest: I would say go to my Twitter. I'm on Twitter @forrestbrazeal and there'll be a link in my bio that has links to all the things we've mentioned: The Cloud Resume Challenge Book, my other extremely bizarre book about cloud which is called The Read Aloud Cloud. And there you can sign up for that Best Jobs in Cloud newsletter and all the other things we talked about. So, I'll see you there.Corey: I look forward to including those links in the [show notes 00:32:24]. That's how I wind up expressing my support for all of my guests' nonsense, but particularly yours. Forrest, thank you so much for taking the time to speak with me.Forrest: Much appreciated, Corey. Always a pleasure.Corey: Forrest Brazeal, currently unemployed, but by the time you listen to this, the Head of Content at Google Cloud. I am Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a long, obnoxious, insulting comment, and then rewrite the entire insulting comment without using vowels.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

The Ecommerce Influence Podcast
319: GREATEST HITS: The Biggest Ecommerce Mattress Brand Ever

The Ecommerce Influence Podcast

Play Episode Listen Later Aug 31, 2021 72:45


In this Greatest Hits episode we're revisiting a conversation about market disruption and big goals. JT Marino shares the story of his dream to build a company that would disrupt the mattress industry. JT Marino was the co-founder of Tuft & Needle, and this is his story of bootstrapping his way from nothing to the largest online mattress retailer in the world. JT, who is now the Chief Strategy Officer for SSB Phoenix, joins us to share his incredible story. We talk about why customer-centricity was one of Tuft & Needle's core principles from the beginning, and the role it played in their eventual merger with Serta Simmons. JT shares his thoughts on the importance, as a founder, of having an intimate understanding of how every aspect of your business works, and why they decided not to take any outside investments when they first started out. This is one of my favorite founder interviews we've done and I hope you enjoy it! Episode Highlights 4:58 Introducing JT Marino 6:47 How the friendship and partnership started between the founders of Turf & Needle, JT and Daehee 11:13 How the Tuft & Needle co-founders maintain a strong friendship while running a business together 12:51 How JT's prior experience with heavily-funded, but unsuccessful, Silicon Valley start-ups influenced Tuft & Needle's decision to not take outside investments. 17:21 The importance of identifying “the problem” when you're building a business, and how this led to the creation of Tuft & Needle 19:28 Some of the other principles the Tuft & Needle founders committed to before starting their business 22:11 How JT and Daehee used a problem-solution approach to determine whether they had a product-market fit for Tuft & Needle 27:59 The benefits of being highly involved with and having an intimate understanding of every aspect of the business, from manufacturing to supply chain to marketing 31:30 How Tuft & Needle moved through the pit of sorrow and started to scale 35:12 How a Hacker News blog post contributed to explosive growth of Tuft & Needle 37:51 The importance of customer satisfaction and word-of-mouth to Tuft & Needle's success, especially when they had a limited advertising budget 39:40 The various channels that helped grow the business over time 42:45 JT's approach to tracking attribution across a variety of channels 45:55 How Tuft & Needle developed advertising techniques and targeting that span across digital and non-digital 50:34 The unconventional reason Tuft & Needle started selling on Amazon 53:39 The Net Promoter Score target T&F set before making the decision to list on Amazon 56:10 JT's advice if you're considering selling on Amazon and the importance of looking at the market picture as you start to grow 59:35 The role customer-centricity played in Tuft & Needle's decision to merge with Serta 1:06:46 The future of the mattress industry 1:09:48 JT shares his hobbies and what's in store for him in the next year   Links And Resources Tuft & Needle How to Get Startup Ideas by Paul Graham Blog: How we bootstrapped to the #1 rated mattress on Amazon.com Meet the Warby Parker of Mattresses JT on Twitter: @johnmarino JT@tn.com The Coalition @a_brawn on Twitter Review or subscribe on iTunes

The Startup Story
Vlad Magdalin, co-founder of Webflow

The Startup Story

Play Episode Listen Later Aug 31, 2021 78:25


About this episode This week's episode features an interview with Vlad Magdalin, co-founder of Webflow. For those that may not be aware, Webflow (at its core) is a website building platform, but yet it is so much more. Webflow is a platform that has enabled thousands of designers to act as an design and development agency because Webflow allows anyone to design masterfully, and develop online engagements without any need to know how to code. In fact, just a few weeks ago we had Duncan Hamra, cofounder of Memberstack on The Startup Story. In his episode we discovered that Memberstack was built ontop of Webflow. So the reality is that Webflow is not just a website builder but an entire web and software development platform that is democratizing how web design and development is achieved. Vlad is an incredible storyteller and you're going to love his full episode. But for me, one of my favourite aspects of his entrepreneurial journey. Is the fact that both he and his brother (who is also his co-founder) are refugees from Russia and who grew up in the Shadows of Silicon Valley. Having immigrated to the US only days before the collapse of the Soviet Union, Vlad knows quite a bit about starting over from scratch. And aside from overcoming many personal struggles as he adapted his life to try and fit in within the United States. It also took him and his brother four separate tries to get their now two-billion-dollar company up and running. Vlad's story is so incredibly relatable because the startup story for many companies is not one continuous thread, sometimes it has many starts and stops and Vlad was no different. In this episode, you'll hear: Vlad shares how he was born in the USSR and his parents took the massive risk to move him and his siblings to America in 1991. He shares what it was like growing up in American from the age of 9 and how he struggled with his identity and he tried to hide that he was from Russia. When Vlad was looking to go to college his parents said he should do a computer course like his brother. He did this for one term then dropped out to go to art school to do 3D animation as he aspired to work at Pixar. While he was at college Vlad had his first entrepreneurial venture. Vlad was using Quickdot to chat with his friends but the app crashed. So picked up a book on programming and started writing a clone of Quickdot but developed it and started ChatterFox. Here he fell in love with programming. Vlad shares how he first came up with the idea for Webflow when he was an intern at a design agency. He accidentally saw how the company was charging their clients hundreds of thousands of dollars while Vlad was getting paid $7 an hour. This sparked his entrepreneurial flair and he wanted to fix the problem he saw and make it better for everyone involved. Vlad shares how he pushed back starting Webflow for 6 years and experienced many ups and downs with this. He almost gave up until one day he randomly receives a trademark certificate for Webflow, that he applied for it over 5 years ago. He took this as a sign to keep going. How he went viral on Hacker News. In less than 24 hours it was the number 1 post and went viral on Twitter. Vlad shares the posting on hacker news drove over 20,000 sign ups. When Webflow launched only 30 people out of the 20,000 paid to use the software. Vlad shares that with those first 30 customers they started a group chat, to hear direct complaints, suggestions and requests they were making. How they think Webflow hasn't even scratched the surface yet of what's to come for the website and with their Series B round completed they are on to developing Webflow. Resources from this episode Join Grindology: https://grindology.com/ ExpressVPN: Get 3 Months Free → https://www.expressvpn.com/startupstory Get Emails: https://app.getemails.com/referrals/newaccount?ref=R18HWW5 The Startup Story Inner Circle: https://www.thestartupstory.co/vip The Startup Story on LinkedIn: https://www.linkedin.com/company/thestartupstory The Startup Story is now on YouTube: https://www.youtube.com/jamesmckinney The Startup Story on Instagram: https://www.instagram.com/thestartupstory Webflow Website: https://webflow.com/ Webflow Instagram: https://www.instagram.com/webflow/ Webflow Twitter: https://twitter.com/webflow Webflow Facebook: https://www.facebook.com/webflow/ Bret Victor, Investing On Principle: https://www.youtube.com/watch?v=PUv66718DII Share the podcast The Startup Story community has been so incredible in sharing our podcast with others, and we thank you! We do have more stories to tell and more people to reach. So please keep sharing!

FounderQuest
Live From The Indie Hackers' Backstage

FounderQuest

Play Episode Listen Later Aug 27, 2021 18:37


Show notes:Links:Snohomish Centennial trailIndie Hackers AMAIntro CRMFull transcript:Starr:All right. Welcome back. Welcome back, everybody. So we took a little break. We're going to have her hot vax summer, but that-Josh:Hot vax summer.Starr:It turns out that was the mirage. It turns out that was a mirage.Josh:Well, it did reach 112 degrees in Portland. So it was hot.Starr:There you go. Yeah. The summer never existed. It was just an illusion caused by our overwhelming thirst for lots of things.Josh:Mirage.Ben:Well, there were a couple of weeks there that I thought, "Yeah. This is going to work out. And then Delta.Starr:Yeah. It was a couple of nice weeks, wouldn't it?Ben:Yeah. It was. It was.Starr:Except for the panic about, "Oh, crap. I need to learn how to deal with people again."Josh:Wouldn't it be wonderful when we can just look back on those two weeks and just remember those last good two weeks?Ben:Yeah. Went 112 in Portland. That's pretty bad. It got to 116 in my garage.Starr:Yeah.Ben:It's pretty warm.Josh:Yeah. That's like melt some things if you're not careful.Ben:I did not know this until well, at the beginning of the pandemic, that there was actually a special class of freezer called the garage freezer because at the beginning of the pandemic I wanted to have a freezer in my garage. I'm like, "Okay. I'm just going to go to Home Depot and buy a freezer." Oh, no, no, no, no. You can't just buy a freezer to put in your garage. You have to have a garage freezer to put it in your garage. So we have a garage freezer and even with 116 in the garage, the stuff stayed frozen. So I guess it actually works.Josh:Nice. Yeah. My freezer survived as well.Starr:I mean, not having a garage freezer in your garage is almost as bad as wearing white after labor day, or is it before labor day? I forget.Josh:I don't know. I never wear white.Starr:I just don't wear white.Josh:Yeah.Starr:Yeah.Starr:Stains too easily.Josh:I just always dress like I'm going to a funeral.Starr:All right. So today's going to be a little bit of a short episode. So we should probably get to the content.Ben:I thought we were already in the content.Starr:I know our reader.Josh:Yeah. It might be short. I don't know.Starr:Oh, we are?Josh:Our podcasts tend to have a mind of their own.Ben:That's true.Starr:Well, that's true. But we've got this Ask Me Anything schedule.Josh:Oh, yeah.Starr:20 minutes from now.Josh:Well, the great thing about asynchronous ask me anything is that they're asynchronous so you can post them even while you're on a podcast and answer the questions whenever you want.Starr:Yeah. Maybe you can, but my brain does not work that way.Josh:Oh, I've got it all queued up.Starr:I've got a one track mind.Josh:It's just a button press. We're locked and loaded.Starr:Oh, you're like Kramer. You've got the button.Josh:No. I'm ready to go.Starr:Sell sell sell!Josh:So yeah. At 10:30, we're recording this podcast. It's 10:08 right now. Pacific. And we're going to be doing an ask me anything AMA on the indie hackers forums.Starr:Yes. And it's a last minute affair as of 20 minutes ago. I didn't have an indie hackers invite code. We're running around scrambling.Josh:Yeah.Starr:Yeah. Ben wanted to try a new podcast recording software, and I'm just like, "No. I can't handle this amount of change in my life right now."Josh:We need to title this episode, live from the indie hackers backstage, by the way.Josh:[crosstalk]Starr:Oh, yeah. I don't know if you like a live album.Josh:Yeah.Starr:Okay.Josh:We're doing it live.Starr:Well, so Ben suggested, when you talk about one work thing and one vacation thing we did. And I guess, I'll start because I didn't actually have a vacation. I just got sick a lot, which I didn't get COVID, but there was some sort of bug that was going around and I got it and I was out for a couple of weeks. And so I guess that was my vacation. I don't know. I just played a lot of Diablo III.Josh:That's cool.Starr:Yeah.Ben:We got our worst vacations in Diablo III.Josh:Yeah. We got away for a few days. We went to this lake up north of Spokane in Washington and just five nights or something. But on the trip there, we're looking at our friends who were already up there, sent us the fire map of Washington. And we are traveling, literally our destination is in the middle of six fires.Starr:Oh no.Josh:We're like, "Should we be turning around?" I don't know. But it turned out all right. We breathe too much smoke the first couple of days, but it cleared up and-Starr:Yeah. After the first couple of days you hardly notice it.Josh:I only got a minor headache.Starr:Your nerves just die. The nerves in your lungs.Josh:Yeah.Ben:It's okay. We have good health insurance.Josh:I'm an ex smoker. So I'll just tack it on, it's just like adding a couple of days.Ben:It's like getting that upgrade package when you're buying a $30,000 car. And it's like, "What's another thousand dollars?Josh:Yeah. I've already got the risk.Ben:Yeah. I stayed closer to home. I read a bunch of books and I got out for a nice bike ride, went to the Snohomish Centennial trail. So it starts in Snohomish and it goes up through Arlington and it's rails to trail conversion. So there used to be railroad tracks there, but now it's a paved trail. And the thing that's neat though, they have a bunch of trail heads and a few of them have the recreations of the old train stations. So it's like, you can act like you're getting on board that train and actually getting on-Josh:Oh, that's nice. Really nice.Ben:Yeah.Josh:That's cool.Ben:That's a lot of fun. Let's see, a work thing that I did. It's a blur.Josh:Yeah.Ben:I probably migrated something somewhere at some point. And back-filled something-Josh:You were busy.Ben:Yeah.Josh:Yeah. You did a lot.Ben:Yeah. I can't remember what I did.Starr:Yeah. I mean, there's a lot of things, right? We're working with that sales consultancy, what is it? Intro CRM people?Ben:Yeah. Did do that.Starr:Have you done some outreach? You got some replies even?Ben:Yeah. Yeah. It's been kind of a mixed bag. So I've gotten some replies, but also the outbound stuff has not really been all that productive. So I'm questioning my life choices at this point.Starr:Have you had any overt hostility though?Ben:No overt hostility.Starr:Oh, you're not pushing hard enough then. You want your OH metric to be at least 10%. At least 10%, you want death threats.Ben:I will take that under advisement.Starr:Okay. That's how you know you're really-Josh:Really selling it.Starr:Yeah. I would say coffee's for closers, but you don't drink coffee. So there you go. Oh, cool. On my end, I don't know. We published our first batch of Honeybadger intelligence reports and I don't know. Loyal listeners might remember from last time, I mean, if you don't remember how loyal are you and how much should I even trust you, but yeah. You might remember that we were working on these things. Basically, they are quarterly reports for a certain programming language where if you kind of need to keep an eye on, I don't know. Front-end JavaScript, but you don't want to just inhale the feed of news that's constantly coming out, you can just look at this beautiful quarterly report. And we are publishing them quarterly now on our blog. And the first batch went out three weeks late, maybe a month late, I don't know. I didn't give myself enough time to get them ready for publication. And then I got sick for two weeks and just could barely crawl to the computer. So I'm sorry. I'll do better next time.Josh:If that's you're going to say, if you don't want to inhale the whatever weekly newsfeed, you can inhale it once a month or once a quarter. Just all.Starr:Well, no. We're not just collating everything together.Starr:[crosstalk].Starr:We're concatenating together.Josh:It's like a curation of curation.Starr:Yeah. We're not just a pending three months worth of Hacker News together. We're going in and applying some real intelligence to it. We have real domain experts.Josh:Editorial.Starr:Curating.Josh:Occasionally?Starr:Yes. Providing you the choicest morsels.Josh:Mm-hmm (affirmative).Ben:Hand crafted morsels of information.Starr:Yeah. Maybe I should be doing these outreach emails.Ben:Yeah. I think so.Ben:I've got the wrong person writing this stuff.Starr:Yeah. They'd be like, "Are these people even professionals?"Josh:Well, that should be obvious from our website.Starr:Yes.Josh:I'll let you decide which way that goes.Ben:Wow. I've been sitting here while you're talking, thinking, what did I do? I'm like, "This is not good. If I can't remember doing anything useful for the past three months, that's probably a sign that I'm doing the wrong things."Starr:I mean, it could just be, you did a lot, Ben. I can remember things you've done. Can we got set up in a new compliance automated thing?Ben:Oh, yeah. Then the compliance-Starr:Yeah. An automated compliance thing. So you don't have to juggle all that stuff manually.Ben:Yeah. We got our SOC 2 type two report done. So we're legit now. We're officially doing the things that we said we would do.Starr:We're enterprise.Ben:Yeah. Full on enterprise.Josh:That's amazing.Ben:Yeah. And it wasn't a particularly painful process. I mean, it wasn't pleasant, but yeah. We survived.Starr:My favorite part of that was that, so as part of this automated security, your automated SOC 2 compliance stuff, all of the employees I guess, have to do mandatory security training once a year now. And it's this automated quiz where you have to read something and then it asks you questions. So it was a really weird big business moment, where I just felt, okay. I'm watching this training video. It should have 50s music in the background of it. And I hate to admit that I got stuck on the first question for 10 minutes. For 10 minutes. Because it was an easy question, but it was one of those things where it's like, "What's the correct answer? Choose one or more." And the correct answer was all of them. But for some reason, I had selected them all with my keyboard and that wasn't good enough. I had to click on them to show I really meant it because hackers generally use keyboards. So they're not really trustworthy devices.Starr:Yeah.Josh:Starr it was like a JavaScript bug.Starr:So eventually, I literally tried every combination. Eventually, I was just like, "Okay. I'm just going to try the first one again," and it worked. So there you are. There you are.Ben:I can't believe you're giving away the answers to our security questions on the podcast. That's a breach of security.Starr:Yeah. I mean, I think our security questions have some security vulnerabilities if, you can manually brute force them. You have four binary options. That's what? Four factorial combinations? You can knock that out in an hour.Ben:Starr is hacking the mainframe.Starr:I am hacking the planet.Josh:That's how Starr passed the security test.Starr:Yeah. That's also how I got such a great score on the SAT, by the way. You just take it, I don't know. 128 factorial times and then you just brute force it.Josh:Nice. How long did that take you?Starr:I don't know. I still haven't graduated from high school.Josh:I sort of graduated from high school.Starr:Well, you can tell you've been away for a while. Because I just have all this bullshit that I've saved up for you all, and it's just all coming out now.Ben:So I was surprised to learn. I don't know why this surprised me, but it surprised me nonetheless, when we had our all hands meeting recently that we have three Honeybadger employees that have children starting kindergarten this year.Starr:Oh, my God. Yes.Ben:That's pretty wild.Starr:It's pretty terrifying. It's pretty terrifying. I'm glad that I live in Seattle. You guys don't. Josh and Kevin don't, but I mean, you all live in fairly reasonable places where governors aren't banning masks in school.Josh:Yeah.Ben:As they themselves are going to get advanced treatments for their COVID infections. Yeah.Starr:Oh, yeah. Yeah. It's okay. We love you Texas. We just don't love your governor.Ben:Speaking of Texas. So this random tidbit I saw the other day, Austin, Texas of course, you know the housing market has been crazy. As far as prices go over the past several months, people have been overbidding regularly on how to just be able to-Josh:Oh, I read that.Josh:A hundred grand?Ben:Yeah. So Austin, Texas.Josh:That's what I'm asking.Ben:A hundred grand over asking price. So you have a $400,000 list price, but you actually got to pay $500,000 to get the house. That's crazy.Starr:That is wild.Josh:Yeah.Starr:Yeah. I had to drop off my car at the mechanic to get its normal service and I was walking by, and this was this morning and there's this kind of older condo building. It's not great looking or anything. And it's two bedroom condo, 900 square feet is now selling for the same price that I bought my single-family house with big yard and everything three blocks away. And that was five or six years ago? Six years ago?Ben:Crazy stuff.Starr:It's bizarre. Totally. I don't know. It's the sort of thing like it feels kind of gross even. Just because I was able to scrape together a down payment for a house, suddenly I get, I don't know. A hundred grand a year extra just in appreciation.Josh:You just hit a jackpot.Starr:Yeah. But it's just like, okay. I literally did nothing to deserve that. And meanwhile, people who could use that or I mean, I could use it, but I'm not in dire straits. I don't know. It's just like, "Wow, this whole system is just kind of backwards and weird."Ben:Yeah. It's to the point I'm getting unsolicited offers to buy my house, right?Starr:Oh, me too.Ben:I'm getting these letters in the mail like, "Hi, I'm Bob and my wife is Alice and we'd like to buy your house." And I'm like looking at the letters, "Is this is really an automated thing or do they really write this by the hand?"Starr:I've had people call me on the phone, in person.Ben:They called you?Starr:Yeah. They called me. Three houses on my block have been demolished in the past two months, three older houses, one of them was just really messed up. But two of them were these small houses on big lots. And essentially what happened is a developer bought almost every house on the opposite side of the street from me and is now basically filling up the lots with as many units as they can. So I think they're going to end up with like 18 units out of these five or six houses, which is fine. I guess. I don't mind density and everything, but it's just so wild because it's like, "Oh, it finally caught up with us." Because for a long time we were just over the edge where things were nice, we were just one block over from the nice stuff. And it finally caught up with us. So we're going to have to move now because we're not fancy enough for the neighborhood anymore.Josh:Yeah. Just cash out.Ben:Yeah. Move to Kansas.Starr:Yeah. I mean, that's the problem though. It's like, "Okay. Great." I get all this appreciation, but if I ever want to get a new house, it's like, "Okay. I've got to pay those new prices."Ben:Mm-hmm (affirmative).Josh:Yeah. We've looked at that too, or you could sell and rent for a few years and see if anything happens. That would probably be a gamble.Starr:That would be a really bad gamble I think. I mean, I don't know.Josh:Yeah.Starr:Yeah.Josh:Considering no markets decline anymore.Starr:I mean, they, they could decline, but you're trying to time it.Josh:Time the housing the market?Starr:Yeah.Starr:Maybe it'll decline, but yeah.Ben:This got me thinking, real estate agents, they want you to trade up, right? You buy your starter house and then you buy your bigger house and then eventually you downsize again because hey, why not have another transaction that a real estate agent can take a commission on, right? And it just got me thinking, why don't we have that for businesses? Why can't you trade up your business, right?Josh:Like trade it?Ben:Yeah. It's like, "Honeybadger, that's a nice little business. Why don't you trade it on up to a bigger business?Starr:So we sell Honeybadger and then by a larger business.Ben:Right. Right. Like that. Rolled into a down payment for a bigger business, yeah.Josh:Yeah.Starr:I'm not sure if you're very good at that.Josh:I love it.Starr:I don't know.Ben:Maybe this is a new marketing thing we can try. We can figure out new business models.Josh:Because we're getting trade-in program like the private equity firms.Ben:You're slapping the top of your business. You can fit so many customers in here.Josh:Might be our best bit yet.Ben:Well, I guess, we better get ready for our ask me anything session. Got a crack the knuckles and get ready to type.Starr:Crack the old knuckles.Josh:Almost time.Starr:All right. Okay. I will sign us off. All right. So this has been FounderQuest back from hot vax summer, back from vacation or being sick or whatever we call it these days. If you want to give us a review on Apple podcasts, whatever they call it, go for it. If you want to look up this AMA we're about to do on Indie Hackers, we recommend that and yeah. Otherwise, just stay cool, stay safe, and we will see you next week.Ben:Catch you later.Josh:See you.Starr:Bye.

Screaming in the Cloud
What an “Agilist” Brings to the Engineering Table with Cliff Moon

Screaming in the Cloud

Play Episode Listen Later Aug 18, 2021 39:15


About CliffCliff is an Agile Consultant and self proclaimed “computer botherer.”Links: Agile Manifesto: https://agilemanifesto.org Twitter: https://twitter.com/moonpolysoft TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part my Cribl Logstream. Cirbl Logstream is an observability pipeline that lets you collect, reduce, transform, and route machine data from anywhere, to anywhere. Simple right? As a nice bonus it not only helps you improve visibility into what the hell is going on, but also helps you save money almost by accident. Kind of like not putting a whole bunch of vowels and other letters that would be easier to spell in a company name. To learn more visit: cribl.ioCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today has done an awful lot of things over the course of his career: startup engineering; software work; founded two startups; has been an engineering manager a bunch of places; has been the CTO at UpGuard, for example; and consulted at one point on HBO's Silicon Valley. Also of note, he is now these days, a renowned Agile consultant. Cliff Moon, welcome to the show.Cliff: [laugh]. Hi, Corey. Thanks for having me.Corey: So, you and I have had energetic conversations about Agile, and based upon that context calling you an Agile consultant for enterprises is basically a deadly insult at this point. Let's get some context on that. For those who have not heard of the term because they live wonderful, blessed lives. What is Agile? A lot of people talk about it, but always the presupposition that people listening know what it is.Cliff: Yeah, that's a great place to start. So, let's go back into, sort of, prehistory. What we call Agile today is, I guess, several generations removed from the original thoughts of Agile. So, in case folks aren't aware, to kind of lay the background, there was a group of software developers, I think it might have been in 2000, might even have been earlier than that, who came up with what they call the Agile Manifesto. And I'm not going to go through a point by point, one, because I don't remember it; two, it's not germane. But—Corey: But it's called a manifesto. I mean, if you take a look at things that have been written historically that are called manifestos, very few of them are good. Like, generally ‘manifesto' sounds like something you wind up writing in a cabin somewhere, right before you wind up doing some sort of horrible crime that winds up living in infamy for 30 years.Cliff: Yeah, yeah. Manifestos, they get a bad rap for good reason. Anyway, let's not go down that [laugh] that rabbit hole. But yeah, so Agile Manifesto, right. Basically, it was this group of people, they said, “Hey, we don't think we're developing software the right way. This is unnecessarily painful. We're doing things in kind of a silly fashion. Let's refocus it around the customer. Let's do this,” yadda, yadda, yadda.And again, like you're saying, it's a manifesto; it's not very prescriptive about what to do to solve the problem, it really just points out problems and then gives a bunch of vague statements about, “Here's the things we should value,” or whatever. And so again, like we're saying, as a manifesto it kind of mutates from there, and then everyone agrees; they say, “Yep, this is the wrong way. Let's try a better way.” And down through the line, what ended up happening was a lot of people figured out that they can make money doing this, and make money being an Agile consultant, or an Agilist, if you're, [laugh] if you like that title. So, it's people who come in, and I guess, try to teach you how to do Agile the right way.And the problem is that the right way usually ends up being Scrum. And so Scrum, if we can get into that, is this idea of having a two-week sprint, you plan out the work you're going to do for that sprint, there's a bunch of meetings you have to do that are kind of mandatory: you do a stand up every day, you do a retrospective, you do sprint planning, et cetera, et cetera. And so it's this, like, cake-in-a-box, right? So, it's like a ready-made thing. And, like, ta-da, now we have Agile, we've implemented Agile, we've done this Scrum thing.The problem, of course, is that most people when they implement this—and it's certainly not the Agile consultant in most cases because they're basically there to just bake the cake, and then keep soaking hours until they're forced to leave.Corey: The problem that I had, whenever I wound up dealing with, I guess, Agile consultants in large enterprises, they always looked a lot like Agile trainers. And I don't know what they were charging because it doesn't matter what they were charging; they wound up gathering the entire engineering department in part of the building for two or three days to talk about how tickets worked, and how planning worked, and how to iterate forward, and how to wind up planning for spikes, and all the various terminology, and how to work with different tooling and the rest. And the reason didn't matter what these people cost was because it was absolutely dwarfed by the sheer cost of every engineer in the company sitting through this nonsense for the better part of a week.Cliff: Oh, absolutely. And then it's an ongoing thing, right?Corey: Well, it's supposed to be an ongoing thing.Cliff: Yeah, it's an ongoing concern. You end up having all of these meetings that you have to do every two weeks or, God forbid, every one week, or whatever the iteration speed that they've laid out is. And the thing that gets lost in the sauce here is, why are you doing this? What is the point of all of this? And I think one of my favorite things to do if I'm at a company that they're implementing Scrum, and they've got their sprints, and then we have our retrospective, one of the things I love to do is I love to touch the third wire during retrospective because that's when you're supposed to bring up, “Hey, what can we do better? What could have gone well?”And that's what I like to say, “Hey, can we just not do this?” [laugh]. And the response that I get is usually an indication of how hard of a job I'm going to have of trying to deprogram people. Because what it ends up being is that—and especially if you have an Agile consultant, and Agile teacher, whatever, they're not going to like the sound of that at all, right? It's like, “Why are we doing any of this? What is the point?”And when you dig into it, especially when we talk about Scrum, it sort of confuses a bunch of different goals that a lot of companies don't even necessarily have anymore. One of the tenets of Scrum is that every two weeks—or whatever your iteration speed is, whether it's one week, two weeks, whatever—the whole system has to be shippable. So, that means that everything has to work together correctly, and then you can ship an entire, like, vertical, or monolith, or whatever. The problem with this is that, especially related to if people are deploying to the cloud or if they're running some sort of SaaS service, this is a meaningless statement; the way that people develop software in that arena today is things get shipped immediately. The system is always shippable because the system is always up because prod is always up.And so you ship your component, everything is backwards compatible, and then your features are behind a feature flag. So, the idea that, oh, everything has to be set in stone on a two-week cycle or whatever, it doesn't mean anything anymore, unless of course, you have a physical artifact that you actually send to a customer, like a CD, or an image to download, or something. But if you're doing cloud-based software—Corey: Or a giant rocket.Cliff: Yeah. Or giant rocket, yes. Oh, God. [laugh].Corey: For some things, you always using waterfall. It's like, giant rockets going to space or—more realistically for most of us or more prosaic at least—is billing systems; people don't generally tend to iterate forward on things that charge customer credit cards. It's a lot of planning, a lot of testing, and they roll it out, and everyone's sweating bullets for a while.Cliff: Right. And I would submit that, at least in my experience, most companies which have tried to implement a Scrum-type process, what they're actually doing is they're running a two-week waterfall. Because a lot of times they've got a lot of technical debt, so the idea that you can ship things immediately might be a little bit shaky, and so what you end up doing is you have this iteration speed of, like, two weeks, and then you have to plan everything out for that, and then you have to go through testing, QA, acceptance. The whole cycle has to run in a two-week sprint. And it truly is a sprint in that case because it's too much work, it's too much stuff, and everything just falls apart. And then they wonder why they can't ship any software anymore. Well, it's because you adopted this process. [laugh].Corey: Oh, I've been in environments where we'll sit down and do quarterly or, God forbid, annual planning about what we're going to build this year, via Agile. It seems a little unlike what Agile professes to be. Now, other than the sheer aspect of hypocrisy surrounding all of this, you take it a step further and say that it in many ways causes harm.Cliff: Yeah. It causes harm in a couple of different ways. One of the ways that I think it is most harmful is the effect that it has on junior engineers, so people who are just starting their career, folks who are just coming out of college, and, you know, in most colleges, they don't really teach you software engineering processes, or software engineering practices beyond the nuts and bolts of the code, or the theory of the code, or whatever, but they don't teach you how to work in a professional environment. And so then you get a lot of folks just entering their first job, and they learn the way to do things at that job. And then they go on, they move to another job.And someone might have, you know… they might go through ten different companies in their career, maybe some more, but they learn a certain way of doing things, and then all of a sudden, it's like, “Yeah. I know how to do Agile. We did it at company X, Y, and Z.” And then they cargo-cult it and take it to the next place if they don't already have it. And so it's this sort of inculcation of younger engineers into this way of doing things that is completely harmful because most places, they don't sit you down and tell you why we're doing this because they don't necessarily know why we're doing it either.Like I said, a two-week sprint with Scrum, the system is shippable every two weeks, you have to go through testing, and yadda, yadda, yadda, this may actually make sense in some cases. And professionally, in my experience, I've designed certain processes that are similar to that. Longer timeframes, but they were designed towards both the product and the team, and, sort of, the interval that they had to ship on. But in most places I've been, no one's thinking about it from that perspective; they're not thinking, “We have to design our processes around the software, or customers, or whatever.” They just kind of do, either whatever the Agile consultant tells them or whatever they learned at the last place. And so it has this effect of replicating a cargo-cult mentality throughout the entire industry, which is sad.Corey: I've talked to a number of relatively Junior folks who have not heard of Agile or Scrum or any of these higher-level concepts about software development methodologies. They just walked into the workplace one day, and everyone's doing two-week planning sessions. I've had people ask me six months into their career, “Why is it called a sprint?” Or, “What is up with the swim-lane style things? It seems weird, but everyone I talk to is used to it. Is it this company thing, or is this an industry thing?” And, on some level, it's, “No, it's just a terrible thing [laugh] that's sort of like a mind virus that wind up taking root in an awful lot of people's minds.”Cliff: Yeah, absolutely. And so when we talk about the damage being done, I think that's the worst. When you think about new people getting into the industry, having a fresh perspective, and that perspective or having an opportunity to forge a new way to do things, that kind of gets ground out of them immediately and they have to do things this set way. And this especially goes for people who end up at large companies where it's just like, you're not going to change anything. You're going to get in there and you're going to do it their way and then that's it.In the rare case when someone comes out of college or comes out of a training program and then they go to a startup that doesn't have as much structure, those are really the only sorts of areas where you even have an opportunity to innovate in terms of the process of how we develop software. Because otherwise, it's just set in stone at this point.Corey: So, you're given a blank slate—or a blank whiteboard, as the case may be, or God forbid, a blank Jira board—how would you structure it instead? How would you advise companies to think about software development? Since I think it's pretty clear that an awful lot of what they're doing today either isn't working or is some weird bastardized hybrid of different methodologies that doesn't really have a name other than something cynical, like ScrumBut.Cliff: [laugh]. Yeah, so that's a great question. So, I think where I'll take this is, I can talk a little bit about—I mean, I've done this before, right, so I've been hired into several different startups as either, like, an engineering manager, or a director, or basically, like, hired management, and typically when a start-up hires an engineering manager or someone on that management chain, they only really do so when the pain has gotten so bad that they want to throw money at the problem. So, I specialized in that for a little bit; very thankless job, but it was interesting because what happens is that every team fails in its own unique and beautiful sort of way. [laugh].So, one of the first places where I did this, there was a person running product; he had learned his Agile methodology from being at Booz Allen Hamilton, which, I mean, it is a nightmare factory in every metric you can measure it on. But apparently, they specialize in Agile as well as the military-industrial complex, so great. [laugh]. He was running things on a one-week sprint. And it was a shippable system, so it had a cloud component but it also had a component that was forward-deployed into a customer network.So, he was running this where basically everyone would work on a one-week sprint; they would then do a bug-bash every Friday, and it was very much a case of, you keep doing the same thing and you keep getting the same results, and you keep doing it to see if you get different results. And they were very much in that kind of mode. So, they would do this every single week. You would have a bug-bash where the same bugs came up every single time; they wouldn't get fixed, no one would triage them. So, the same bug was in the system in maybe, like, 10 or 15 different tickets.No one was triaging it. And it was just a mess. And so when I got there, part of my job was to just kind of break apart this crazy structure that was happening, and again, try to design something that would actually work, again, for the product. You have to design something that has to work for how the product gets deployed. So, as I said earlier, if you have cloud services, they can deploy whenever so structuring them around some sort of timeframe doesn't really make a ton of sense.However, when you have something that gets deployed into a customer network, like an agent, or some kind of desktop software, or anything that's on machines that you do not have direct control over, you have to factor that into the speed at which you ship, you have to factor that into your engineering process. Because if you can only ship out that executable once every quarter, or—it's like, how fast can your customer actually consume these things, right? Most places, if you give them updates every two weeks, they're going to say, “What are you doing? Why are you making my life hard?” In a lot of places, the fastest—especially if you're selling to an enterprise—the fastest they can consume a forward-deployed component is once a month at the very fastest.Usually, they prefer on a quarterly or even a six-month basis. But if that's the case, you have to design your engineering process to account for that. Then the other part is that when you land in a place like this, you can't just pull the rug out from everybody immediately. It's similar to saying, “Oh, we got to do a rewrite.” It's like, well, you can't just do a rewrite of your engineering process either; you have to incrementally make changes to it so that people are not confused about what they're supposed to be doing, but you're making changes towards things running in a more smooth fashion.So, what I typically try to do is I try to design a process, and then get the team bought into it, and then hopefully get them moving faster. And the first time I tried this, it was a disaster. That was the company I was just talking about where they were running one-week sprints. I did not know what I was doing at the time; that was a very difficult situation. Landed at another place after that where this one was a two-week Scrum; similar problems around okay, frequency of testing, you have a component that gets deployed into a customer network, how fast can we deploy that?And similar sorts of problems, and so now that I could see what the pattern was, I could now develop a—I had a much better time developing a process that actually worked and helped the team ship with confidence. Which is really what you want the process to do is you want the process to be something that takes burden off of the engineering team, as opposed to something that makes your job as the engineering manager easier, which I think a lot of engineering managers approach it from that perspective of, “Oh, I can get a report at Jira and then I don't have to talk to everybody every day,” or whatever. If you're trying to make your job easier through the process, you are necessarily putting more burden onto your team.Corey: Your company might be stuck in the middle of a DevOps revolution without even realizing it. Lucky you! Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy in on DevOps practices? Well, download the 2021 State of DevOps report brought to you annually by Puppet since 2011 to explore the trends and blockers keeping evolution firms stuck in the middle of their DevOps evolution. Because they fail to evolve or die like dinosaurs. The significance of organizational buy in, and oh it is significant indeed, and why team identities and interaction models matter. Not to mention weither the use of automation and the cloud translate to DevOps success. All that and more awaits you. Visit: www.puppet.com to download your copy of the report now!Corey: It feels like this is almost the early version of a similar political machination playing out where, we see it now with—there are these large companies that, once upon a time, had these big mono repos, and they had 5000 developers, and every one wound up causing problems because a group of developers is collectively referred to as a merge conflict. Then they wound up building out, “Ah, we're going to break the monolith apart into microservices and it solves that political problem super well.” And then you wind up with a bunch of startups with five engineers working there, and they have 600 microservices running in their environment, and it feels like someone took an idea outside of the context in which it was designed for and applied it to a bunch of inappropriate areas and just bred an awful lot of complexity while actively making everything worse. Please don't email me if people disagree with that statement. But it feels like an echo of that, doesn't it?Cliff: Oh, absolutely, yeah. I mean, I think a lot of this is a reflection of our relative infancy as an industry. When you think about the amount of time software engineering has been around, and has been a going concern of itself, as opposed to other engineering disciplines, I think we are still very much in our infancy. Like I was saying, they don't really teach this sort of stuff in school, and certainly not the theory behind why you would structure things this way versus that way. In fact, most people who get promoted as a manager, you get promoted from being an engineer—someone who codes all day, or codes and does design work, but basically, someone who works as an individual contributor—then you get promoted to being a manager, and very few places give you any training or education or anything at all about how do you even do this job. And so you either sink or swim.Corey: It's an orthogonal skill set that basically bears little relation to what you were doing before?Cliff: Exactly. The thing that sort of gives you any sort of ability to swim in that type of job is having the clout or the respect of your former peers as you get promoted into that. And the people who do well with that, they basically learn on the job and rely on that inbuilt respect to basically screw up a lot until they can get the hang of it. But yeah, most of the time, you don't get an education and management or any of the other things that are not just specific to people management, but people management for software engineering, which I do think is its own discipline.Corey: And some, I guess, almost borderline ridiculous level, it feels like no one really knows what they're doing when it comes to management, especially in engineering. In other disciplines, it seems that management is treated as a distinct key skill, but very often—the way my management strategy evolved—and those people think I'm kidding whatever I say this, but I assure you I'm not—it came out of looking at what my terrible managers had done in the past and what didn't work for me, and what made me quit slash become demoralized slash convince others to quit, et cetera, et cetera; or, you know, steal office supplies. Whatever it is that—how it is that you act out, and then I just did the exact opposite of those things. And I've been told repeatedly, “Wow, you're a great manager.” Not really. I just don't do all the things I hated. It gets you surprisingly far.Cliff: Well, yeah, absolutely. But that gets you far with your own reports. There's a whole other side to being a manager, which is dealing with the outside world. And then that's, especially if you're in a large organization, even in some smaller ones as well, there's a whole dimension to the job that you as an individual engineer, you don't even see.That's the politics part of the job about how do you justify what you're doing? How do you advocate for your team? How do you operate as a quote-unquote, “Shit umbrella” for your team? And all these sorts of other things where you provide a safe harbor within the company for your team to operate, and then try to procure resources and make sure that the decision-makers above you understand the importance of what you're doing. And no one teaches you how to do that.Corey: Oh, never. You're absolutely right on this. I was mostly focused on managing my reports. I completely failed in those roles managing up and, in some cases, managing sideways as well, just because that was never clear to me when I was an independent contributor working on engineering problems. It's an evolution, on some level, of figuring out what it is that the role really is.And all this stuff is not that complicated to teach people, but for some reason, culturally, we don't do it. We take the Hacker News approach to things and try and figure out complex forms of interaction from first principles. And it really feels like there are some giants upon whose shoulders we could stand.Cliff: Yeah. I mean, I agree with that. I mean, there's definitely people in the industry who've written books and who are starting to try to put down that first layer of institutional knowledge to share with other folks. You got people like Camille Fournier and other folks who've written books specifically for engineering managers who work in the software world. Which I think is a really great first step.But yeah, when it comes down to it, it's like, “Okay, we're going to implement this process; we're going to do these things; I don't know why.” It's almost like no one got fired for buying IBM; no one got fired as an engineering manager for implementing Scrum. But if you try to go and do some other weird stuff, you're running the risk of getting fired, if you fail.Corey: There is the question of whether someone at IBM will get fired for buying Red Hat, but that's not the analogy that people always fall back on for the last 25 years. I think that there's also the idea that people will try and build their own thing where it makes sense for them. In complex engineering areas, that often makes sense, and sometimes it doesn't, but then they try and approach human interaction like it's an engineering problem, and that can lead to a lot of, frankly, disastrous outcomes, on some level. I feel like this does tie into the, I guess, almost unthinking adoption of Agile and similar methodologies or perversions of those methodologies in many large enterprises. Do you see a fix for this, or is this something that we all more or less have to live with, and watch people continue to make the same mistakes for another ten years?Cliff: I think, for the most part, you have to—I guess, change starts at home. [laugh]. What I would advocate for is that if you have problems or qualms with the process that your particular organization is following, and you have ways you think you could fix it or changes that you'd want to make to it, then start advocating for those. And you'd be surprised about how far you can get sometimes with just saying, “Hey, can we just stop doing this, or can we do this a different way?” But I would also say that, like—one of the things you just said sort of knocked something loose from my mind, which is that even when companies share, like, “Oh, we've done something amazing here. We've designed this amazing new process, it really works well for us.”And they write a big blog post about it, turns out if anyone ever follows up on that, they either never did it or it was never as described. And they certainly don't do it today. So, I think a good example of this would be like when Spotify put out there—this was a number of years ago—Spotify put out their big creed about, like, “Here's how Spotify develops software.” And they had this whole bespoke thing about they've got these pods of people, and you've got a matrix management, and they reinvented a whole bunch of stuff. And then you talk to anyone who was at Spotify during that time, and they're like, “Yeah, we tried that; it didn't work.” But they still put out the blog post. So. [laugh].Corey: And I think it's still up and hasn't been taken down yet. It's, “Yeah, did this work for other people?” “No, absolutely not. But it might work for us.”Cliff: [laugh]. Yeah, it's the same kind of trick that companies do with open-source, which is you open-source something to a bunch of fanfare and try to get people to adopt it when it hasn't even been adopted internally. And anyone who tries figures out it's not the right thing, and they don't even like it. And so, but it's like, “Oh, yeah, we can open-source it and then it comes with the imprimatur of whatever company it comes from.” I mean, this is a pretty classic joke. It's like that old movie, The Gods Must Be Crazy, you throw the Coke bottle out of the plane; someone on the ground picks it up, and eventually ruins your life, even though it's just a Coke bottle. Same thing with open-source; same thing with management processes.Corey: It seems like it's going to be one of those areas that continues to evolve whether we want it to or not. Or at least I hope because the failure is, it doesn't.Cliff: Yeah, I mean, hopefully it evolves. And like I said, I would say change starts at home. Try to advocate for changes on your own team and think outside of the box; try to figure out what you can get away with and try to figure out, I guess, ways to break down the walls and the rituals that the Agile consultants have set up.Corey: Ugh. [sigh]. I hope you're right. If people want to hear more about your thoughts on these and many other matters, where can they find you?Cliff: Yeah, so typically, I'm just usually tweeting. So, my Twitter account is @moonpolysoft, and that's usually where I'm doing most of my stuff. Yeah.Corey: And we will, of course, include a link to that in the [show notes 00:25:15]. Thank you so much for taking the time to chat with me today. I really appreciate it.Cliff: Yeah, Corey. It was great, and thanks for having me.Corey: Cliff Moon: absolutely everything except an Agile consultant. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment that you're going to continue to iterate on and update every two weeks, like clockwork.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
A Non-Traditional Path into the SRE Folds with Serena Tiede

Screaming in the Cloud

Play Episode Listen Later Aug 10, 2021 39:15


About Serena Serena Tiede is a SRE at Optum, a healthcare technology company that manages everything from the delivery of care to the management of patient data. Prior to becoming an SRE they were a Kafka operator for real time security logging and ingestion. In their off time, they moonlight as the proud admin of an incredibly over engineered Minecraft server.  Links: Optim: https://www.optum.com/ Twitter: https://twitter.com/SerenaTiede Personal Blog: https://blog.serenacodes.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Your company might be stuck in the middle of a DevOps revolution without even realizing it. Lucky you! Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy in on DevOps practices? Well, download the 2021 State of DevOps report brought to you annually by Puppet since 2011 to explore the trends and blockers keeping evolution firms stuck in the middle of their DevOps evolution. Because they fail to evolve or die like dinosaurs. The significance of organizational buy in, and oh it is significant indeed, and why team identities and interaction models matter. Not to mention weither the use of automation and the cloud translate to DevOps success. All that and more awaits you. Visit: www.puppet.com to download your copy of the report now!Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. A recurring theme of this show has been for a while, where does the next generation of cloud engineer come from because the path I walked of being a grumpy Unix admin isn't really as commonly available as it once was, and honestly, I wouldn't wish my path on anyone in good conscience. My guest today is Serena Tiede, who's a site reliability engineer at Optim and didn't start their career as a grumpy systems administrator. Serena, welcome to the show.Serena: Hey, thanks for having me. I'm so pumped to be here.Corey: Don't worry, that will soon pass. What I'm wondering is, you didn't come to be an SRE through a giant ops background of clawing your way up by dealing with hardware and data centers and driving at unsafe speeds in the middle of the night because someone tripped over a patch cable in the data center. You have a combination of traditional/non-traditional background. Tell me about that.Serena: Yeah. So, it's funny you mentioned hardware. So, I went to school for electrical engineering, went to University of Minnesota because you want to do engineering, you pretty much going to one of the big state schools in the Midwest. So, I grew up and was like, “I want to be a hardware designer.” I'm terrible at it. So terrible. [laugh].Corey: Wait, I didn't realize that you could want to be things you were bad at. If somebody told me that early on my career, it's, “Huh. This might have taken a very different turn, and far more productive one.” I just assumed if I wasn't good at something I should give up and never try it again.Serena: Oh, I took the courses and was like, “Whoa, this is circuit design? Not for me.” Then I ended up just taking a bunch of engineering math courses. So, I took communications, the digital signal processing, controls, and started programming. I was like, all right, let's do embedded systems. No one was hiring and then come internship time, there's this little company that I've never heard of called Optim. And they're like, “We want software engineers.” Well, I can write C. Does that count?Corey: Oh, question, of course, to really ask is, “Oh, can you really write C having gone through it?” The more I talk to people who've been writing C for their entire career, and you ask them, “Can you write C?” The answer is, “Not really slash reliably. I can basically type and sometimes it works.” And, “Oh, thank God they're mortal, too.” Was my response.Serena: Oh, my opinion: no one should learn C unless there are specific reasons why. And those reasons are: you're doing embedded systems where I had to learn how to write in assembly, for three weeks, and then my professor at the end said, “Hey, we're writing C. Be thankful; it's a high-level language.”Corey: That is terrifying. But let's get back to this idea of you going to school for electrical engineering, and you didn't just dabble in it; you graduated with a degree in electrical engineering, didn't you?Serena: Oh, yes, I did. I graduated. It was fun even though, unfortunately, it still had my dead name on the diploma. So, I refer to that as my… matrix, Mr. Smith moment. [laugh].Corey: They won't go back and edit and reissue it under your actual name?Serena: I haven't bothered to look, but I almost consider it just kind of hilarious and just keeping it that way.Corey: No. Again, I am not one to ever advise people how to deal with names. When I changed my name back in 2010 or so I wound up getting a whole lot of strange looks over it. And honestly, it is no one's business, except how you interact with a name. Not the direction that we need to go in on this. I'm more interested in understanding, on some level, how you got a degree as an electrical engineer and then immediately landed a job writing software. That one feels a little strange. Can you talk me through it?Serena: Oh, yeah. So, pretty much I took a bunch of operating systems classes and was like, “Wow, this computer science thing is cool.” But I was too far in the electrical engineering track to change degrees. So, I got the degree, ended up working at Optim. I originally started off in security, oddly enough, for my internship, then came back, did a—you know, we have a rotational program so I did security for six months and then… I wound up on this team for my second rotation where their literal job description, “Write RESTful APIs in streaming applications.”Corey: So, it wasn't even a software job that focused on the close-to-the-hardware stuff where you're doing embedded systems. Like, that would at least make a bit more intuitive sense to the way I see the world. No, this was full-on up-the-stack REST API stuff.Serena: Oh, yeah. I tried embedded, but in my market, it was all medical devices, and between all of us listening here, I don't do well with medicine. Get very squeaked out, very faint. So decided, all right, let's go up the stack, and turns out, it's, like, okay, Kafka Streams. And then we were trying to figure out, “Okay, why our services—like, how do we know if it's saturated?”I'm like, “Oh, well, we have this Prometheus thing. This sounds cool.” And it was deployed on, like, you know, a rudimentary Kubernetes cluster. “Oh, hey, there's this cool service discovery thing. Let's do that.” And then one thing led to another. Thanos was coming out, and before it had a release candidate, I decided my claim to fame at the company was like, “All right, let's do this Thanos thing because it seems really cool. I read about it on Reddit.” And the distinguished engineer in the room was like, “Oh, yeah, I heard about it on Hacker News. Do it.” I did; it was rough, but it was so cool. And then I come back, like, a year later because I went back to security for a wee bit, and the same monitoring stack is still there. And they were like, “Hey, can you do more monitoring things and pivot to observability?”Corey: Yeah, let's skip past the obvious joke that I could make about someone at a healthcare company saying, “Let's do it because I read about it last night on Hacker News,” because it's just too easy at some point. It's odd, though, because I always held the belief, somewhat publicly, that an SRE role was not going to be a junior role. It was something that required quasi-significant experience to wind up moving into it, it's always felt like a transition from traditional ops roles or folks who are deep in the weeds that have been doing software engineering at scale to a point where they see how these systems fail over time in production scenarios. It doesn't sound like that was your path at all. Not to delegitimize your path by any stretch of the imagination. This is more to do with me reevaluating how I view SRE, as a field that people get into and how they approach it.Serena: I just fell into it. And the reason why I bring up my digital signal processing background is a lot of the SRE stuff I look at all of our time-series metrics, and it's like, “Oh. Well, this is just a real-time stream of data that we scrape periodically.” And it's like, “Oh, cool. So, we can look at our averages, percentiles, I can eventually do some really cool fancy digital filtering.” And kind of was like, “Oh, wow. I, kind of, know the math behind a lot of this stuff and just have to just brute force apply it in places.”Corey: Tell me a little bit more about that because with my approach to SRE—which let's be clear, was fairly crap—the math that I tended to do was mostly addition and subtraction, and for the really deep stuff, I used the most common tool to manage anything at scale, Microsoft Excel, and that mostly handled even the addition and subtraction for me what math?Serena: So, for me, a lot of it comes down to—I actually have my signals book in the other room—the big concept behind all these systems is the concept of sampling. You're not going to, real-time, get memory and CPU data every second. Processors are running at gigahertz of speed, you would need double that to recreate your signal with full fidelity. That's the Nyquist sampling theorem. But you kind of can fudge the numbers a little bit and just say, “Ehh, do we need that granular detail?”We're not trying to reproduce what happens in the past, we're just trying to see what's going on now. So, I say okay, 15-second scrape interval, things are looking good and then rolling into what I'm doing later of applying, like, “All right, let's do some fun control loops,” because people wanted service-level objectives. People want service-level objectives; everyone loves them some SLOs and SLAs. No one wants to figure out, by hand, what their baseline is. But again, some fancy—this is more controls math—figure out what your baseline is just automatically and do some little magic in the frequency domain, courtesy of Laplace transforms, and that's it. I can just automate that for you and remove the human from the equation.Corey: I'm still somewhat astounded by the fact that people calculate these things out mathematically instead of, you know, dead reckoning and confident-sounding estimation.Serena: It's really just bringing that electr—like, controls background to software. Honestly, I'm kind of baffled that no one else is found this hack because I'm just thinking, “Oh, well, I can't be that unique. Someone else has to have done that.” And then I talk to the people in the room and it's like, “Oh, wait, no, I am the only person here.” [laugh].So, that's my whole thing. Everything is just applied math. And all of our human dead reckoning, it's great, but it doesn't scale well. You know, my boss wanted me to figure out how to do our SLOs for the entire team, and turns out realist—and when it came time to hire, realistically, cloning myself was not an option. [laugh]. So—Corey: For better or worse, it seems like it isn't. So, what was your first exposure to the SRE-style space? You started off in security, but looking at the timelines on this, it wasn't that long ago. It feels like you were probably not exposed, in many cases, to physical data centers as much as you would be cloud, or at least not having to image bare-metal systems. Were you up at the AMI level, or was it beyond that in having virtual machines that moved around into full-on containers, or serverless?Serena: So, I started my internship in 2016, and got my full-time offer in 2017. And we started having our—container platforms started becoming this up-and-coming thing. You know, my lead engineers were like, “All right, you've got to learn this thing called ‘Docker.'” And I have never heard of it, but I was just amazed that, “Wow, I can just run these little, little itty bitty pods anywhere on this hardware.” And later on, I did do some, like, virtual machine stuff, but I've had the luxury of all of these years of pain and toil, to be able to say, “Oh, yeah. I can just manage things with Ansible, create my Docker files, and do everything from a code deploy pipeline style. And it was awesome.” And I just can't fathom what it's like to work without those tools, but knowing… the past, it's kind of like, “Wow, we have gotten a lot farther. Things are abstracted. This is actually kind of nice.”Corey: It kind of is, on some level. I feel like my initial reticence towards containers—I gave a talk: “Heresy in the Church of Docker,” which sort of put me on the speaker map once upon a time—and it was about all the things that Docker as a technology didn't really have good answers for. Honestly, the reason that I gave the talk was I assumed that it did have answers and I was just unaware of them, and I just gave the talk so I could publicly become the idiot who didn't know what they were talking about and then get “well actually'd” to death by [ducks 00:12:40] slash Googlers. And it turns out that no, no, at that point in time, these things were not well understood or solved for. The observability stories, the metrics, the logging, the orchestration, the security story, the how you handle things like state, et cetera, et cetera, et cetera.And Kubernetes these days has largely solved a lot of those problems, but I don't dabble in those spaces just because of outright ornery. Back then it was a weird problem, but these problems have largely gotten solved in some ways. But I sort of just skipped over the whole Kubernetes slash container renaissance, and personally, I went directly into the serverless world. What's your take on that?Serena: Oh, so as someone who loved Kubernetes, I was a serverless skeptic, initially. I was like, “Well, I can just build my Docker file and write the deployment manifest. No big deal.” And then I started working on my side project. For, I think, better purposes, my iCloud account is tied to my credit card and I have to actually be on the hook for cloud bills. And I use GCP for my home lab and lo and behold, 1 million requests a month for free. And I love the sound of free when it's my money on the line.Corey: Oh, yeah, company money versus enterprise money, radically different scales. I mean, if you try and sell me personally a $50 hamburger, I'm going to tell you to go to hell. If you try to sell me, as representative of my company, a $50 hamburger, I'm going to need a receipt.Serena: Exactly. And then also, like I'm just running through, I was redoing one of my serverless functions and watching the deploy steps. And then one of my coworkers introduced me, he's like, “Hey, Serena, you hear this thing called ‘Buildpack?'” and I'm like, “No. What on earth is that thing?” And he's like, “Oh, well, you take your code, and then it just magically turns into a container.” I'm like, “Well, crap. Show me.” And lo and behold, code goes in one end, nice little container comes out the other. And that crap was magic.Corey: It really does change the world if you let it. I think. I know it sounds like a ridiculous, I guess, hype-driven thing to say, but for the right use case, it's great because it removes the entire administrative burden from running services. Now, critics are going to say that well, that means you're just putting all of your reliability in the hands of your cloud provider. Yeah, we're kind of all doing that already; serverless just, sort of, forces us to be a little bit more honest with ourselves about that.Serena: Oh, yeah. I mean, even if you self-host things, you're relying on your data center ops people to, like, make sure, oh, I don't know, your machines don't literally catch fire. We literally had a bug one time where it's like, “Why is this one node bad?” “Oh, actually—hey, did you increase the fan speed?” Someone had to literally go increase the fan speed for whatever servers, which, again, in the serverless and cloud provider world, I don't think about that. The cloud is just infinite to me. It's just computers and APIs as far as the eye can see. It's wonderful.Corey: It really is. It's amazing, and it's high level, and on some level, you went from getting a degree that required you to write assembly and super low-level stuff and figure it out hardware works into, let's be honest, writing in your primary language, which for all of us in SRE-land is, of course, YAML.Serena: Oh, I am a very spicy YAML engineer. YAML and a little bit of Go for what I need to make things go.Corey: You ever notice there's never a language called ‘Stay,' or ‘Stop,' or anything like that? It's always about moving to the next thing. And we in engineering always have sprint after sprint after sprint. Never a, “It's a marathon, not a sprint. Relax. Walk. Enjoy the journey.” Nope, nope, nope. Faster, further, sooner.Serena: Yeah, it is honestly weird because my relatively short career span, you know, it's 2021 and I graduated in 2017. The company is like, “Hey, you're a senior software engineer now.” Here's a program, here's a budget. Go forth.Corey: Oh, that's lucky. It must have been amazing to have an actual budget. When I started out, I was in one of those shops where it's, “Yeah, Palo Alto wants $4,000 for that appliance. That's okay. We have some crappy instances and pfSense, and you know, we could wind up spending eight weeks of your time to build something not as good. Get on it.”Serena: While the hilarious part is I'm stressing out about every single dollar I'm spending and then my boss is like, “Oh, you know, your budget is super small potatoes, right, compared to like our other stuff? Don't sweat it. It's fine.”Corey: I keep making this point to the cloud providers where they're somewhat parsimonious free tiers are damaging longer-term adoption because I look at building something myself, in my spare time in my dorm room or whatnot, and I'm spinning up some instances that talk to each other and I want to use a load balancer and I want to use a managed NAT gateway—God forbid—and at the end of the month, I get a bill for $300. And it's, what the hell is this? I thought I was on the free tier and it scares the living hell out of us. So, we learn not to use those services that are higher level and differentiated. And then when we start working in environments that have budgeting and are corporate, we still remember that, and, “Oh, don't use that thing. It's expensive.” And you'll inadvertently spend 80 times as much in what your employer is paying for your time, rather than using the high-level thing because they could not care less about a $500 a month charge. And it's this weird thing that really serves as a drag on adoption.Serena: It's super wei—I actually literally had this conversation with one of my engineers who wanted to, “Hey, we're trying to expose a GRPC thing.” And I had issues getting it to work with an ingress. And he's like, “Do you want me to take a crack at that?” And I'm like, “Look at the price of the load balancer.” And I'm like, “Unless you can figure it out in half an hour… it is literally more expensive for you to continue tilting at that windmill than for us to just leave it be.” [laugh]. And it's also weird. I have my personal stuff where I'm trying to keep my cloud bill to, you know, maybe a humble $100 a month max, versus, “Oh, the enterprise? Oh, yeah. That's just logging that you're paying for.” Which is baffling to me.Corey: I feel like as engineers, we always, always, always fall into this trap. And maybe I fall into it worse than others because my entire business is actually lowering the bill. But when I started as an independent consultant, my bill was something like seven bucks a month, which yeah, I'm pretty content with that. And I started looking at ways to golf it lower, which in most cases is never worth the time, but in my case, I should really understand every penny of the AWS bill or I'm going to have a problem someday. And now I look at it recently because we have a number of engineers building things here, and our bill was over $2,000 a month.And true story, by the way, it turns out that your AWS bill is not so much a function of how many customers you have; it's how many engineers you have. And I look at this and, “Oh, my God, we need to fix that immediately.” And I spent a little bit of time on it and knocked 500 bucks off, and, “Whew, that's better.” And it still bugs me to see a $1500 bill; it feels like it's an awful lot of money. I mean, think of what you can buy for 1500 bucks a month.And then in the context of the larger business picture, compared to payroll, compared to all the other nonsense we use, like Tableau, for example, it's nothing. It is a rounding error that gets lost in the weeds. I never understood that before having access to company budgets. When I was an employee, this was never explained to me, so I was always optimizing for absolutely the wrong thing in hindsight. It feels like this is part of the problem that we run into as a culture when we don't give our staff context to make the right decisions.Serena: Yeah, I actually do appreciate the way my company does things because I am, like—not personally, my bank account, but I am, like, responsible if someone should ask, “Hey, what's this charge for?” I have to say, “Oh, well, it's for all of these things, and we need that.” But for the most part, it's been really weird to, kind of, learn, like, one of the ways I, kind of, sped up my, like, “Okay, I need to learn how business works. What do I do?” Well, quite honestly, a lot of my cloud cost tips I have learned from your various podcasts. [laugh].Corey: Uh-oh, that's a problem.Serena: No, but like, all of a sudden, all this stuff and just hanging out on tech Twitter and hearing all the advice of people and then… it was, kind of a weird way of, like, yeah, years-wise, yeah, some people might look me askance and be, like, “You're really a senior engineer?” But then they hear me speak and it's all about like, “Oh, well, I”—again—“I stand on the shoulders of giants,” which is awesome, and I'm honestly just hoping that one day I will write something that is very cool and then someone will say, “Oh, well, they were right on these things, but not right on this. Let's edit this to make it a little bit better.” And the standing on the shoulders of giants trend continues.Corey: This episode is sponsored in part my Cribl Logstream. Cirbl Logstream is an observability pipeline that lets you collect, reduce, transform, and route machine data from anywhere, to anywhere. Simple right? As a nice bonus it not only helps you improve visibility into what the hell is going on, but also helps you save money almost by accident. Kind of like not putting a whole bunch of vowels and other letters that would be easier to spell in a company name. To learn more visit: cribl.ioCorey: I'm a little taken aback by the fact that you've learned a lot of this stuff from the podcast because I tend to envision when I'm telling stories about this, companies that show ads, or my mythical Twitter for Pets startup. I have to remember that there are banks, like, is one of the examples of serious businesses that I use all the time. But you're in healthcare. I'm sorry, that's more serious than finance, just because—I hate to say this because it sounds incredibly privileged and I don't even care—it's only money. What is money compared to the worth of someone's life?I don't think that you can ever draw an equivalent and I feel dirty every time I try. When you're working with things that impact people's ability to access healthcare, that is more important than showing banner ads. And a lot of the stories I tell about, “Maybe it's okay to have downtime.” Because yeah, if AWS takes a region down issue for an afternoon and you can't show ads to people or your website isn't working, yeah, that's kind of sad and it's obviously not great for your business, but at the same time, the stories in the news are always about Amazon's issue, not about your specific issue. If you're in an environment where there's a possibility that people will die if what you have built is not available, we're having a radically different conversation.Serena: Exactly. Fortunately, for me, I personally, not working in the, like, kind of, care delivery space, but the stuff I'm working on right now is supporting, you know, that lovely end-of-the-year where it's open enrollment, all the employers are saying, “Hey, time to re-up your benefits.” Yeah, it's kind of a big deal that our site doesn't go down. Because—Corey: Yeah. And open enrollment, to my understanding, changes based upon what plan you're on. I've known companies that have open enrollment in the summertime. I believe ours winds up coinciding pretty closely with the calendar year, but I've certainly worked in environments where that wasn't true. So, being able to say, “Oh, it's fine. It's April; no one's doing open enrollment now.” Is it actually true?Serena: So, it totally depends on which part of your business. If you're going through the healthcare exchanges, that's usually more in the fall. I think the Medicare plans, those are a little bit before the individual enrollments. And there's a ton of these things that even though I just work tangentially, that I'm just not even in the know for. And then, of course, we talk about open enrollment, but the thing that a lot of people don't really talk about is, so what happens when your plan goes live on January first of the next year? Yep. Our site's still got to be up. And it's a responsibility I take really seriously because it impacts so many people.Corey: It really does. And it shouldn't, to be clear. I try to avoid getting overly political on this podcast, but the state of healthcare in the United States as of the time of this recording is barbaric. And I really, really, really hope there comes a day where someone's listening to this and laughing because it's such an antiquated and outmoded story that isn't true anymore. But I'm terrified that it won't be.And yeah, having access to a website lets you sign up for healthcare during a limited period of availability, if you miss that window, you don't have healthcare, in many cases, until the following year when open enrollment opens again, or honestly, you wind up changing jobs because that is a qualifying event to change healthcare. “Well, I missed the open enrollment window, so I have to quit and take a job somewhere else,” is a terrifying thing. It's bad for the business for a variety of reasons, but that pales in comparison to the fact that people have to make life-altering career decisions based upon a benefit that is routed through an employer when it should not be. Okay, I'll climb off my soapbox.Serena: Oh, it's bizarre to me. Honestly, for better or worse—I argue worse—but I'm honestly optimistic. One of the weirdest things I saw that stuck out from the most recent stimulus bill was, “Oh, hey. We're having a special enrollment period during a pandemic.” And I'm like, “You know, it's not a hundred percent.Maybe we should just extend it to the whole year.” But it's better than what was the previous state, where it's like I can't make—I mean, even in my work life, I can't make everything perfect. I can't make outages go away, but I can make things just a touch better. And that's all I can do.Corey: Sometimes all we can do, and I wish there were better ways to handle that. I don't know what the future is going to hold, but I also think that there are bright areas. There are aspects that are promising as far as the future being brighter than today. The overall trend—I hope—is for humanity to uplift itself.Serena: Totally.Corey: Again, I do want to highlight that you went in a very strange direction where you went from software engineering—a generally pleasant job—to SRE, which is horrible and would not be recommended to anyone. What guidance would you have for people who are, for some godforsaken reason, trying to figure out what their career trajectory is going to be like, and thinking that they might want to become an SRE—even if they're not in tech yet—because for some reason they hear the stories and think there's some nobility in suffering or whatnot?Serena: Well, for starters, for me, it kind of came down to get real good with this great math. It's boring, but that's kind of the bread and butter of the concepts I've learned. Also for junior people, if you're also just curious—say you've written an app, go over to OpenTelemetry. Go, like, instrument your stuff and see how many requests you get in a day. Start getting your hands dirty with instrumentation.Look at how cool it is, and then maybe you want to start structuring your logs; maybe you start end up doing tracing. But at the end of the day, it's all, for me, I think best learning is just experiential, and you know, one of the things where how do you learn from production outages? Go to happy hour with some of the senior people and listen to the stories that they tell. With enough time they become funny, but they're also valuable learning things.Corey: The aspect I would push back on is the hard requirement around discrete math. I don't deny that it has been helpful for what you've done and how you do it. I don't know how any of that stuff works on paper; I have an eighth-grade education. That was never my path and never my strong suit. I would agree that knowing it would have made aspects of what I do easier, but the bulk of it I don't necessarily know that I would agree. I guess, my counterpoint slash pushback would be that if you thought you'd like this, but you don't want to deal with the math, it's not a hard requirement, and I don't think that I would frame it as one.Serena: Actually, that is a very good catch. It is not a hard requirement. I am not sitting here in my notebook, scribbling away at equations. But the concepts that I've learned from a while back, it's the concepts are way more important than the actual computation itself. Because computers do that, and a computer will absolutely run circles around me.Corey: Most of us do, unless, you know, the computer is an overheating processor from Intel. But that's a little bit of a low blow. Not that it stopped me. But it was a low blow.Serena: Well, I mean, your local science supply shop might have some liquid nitrogen. Maybe.Corey: So, what's next for you? You started off in security slash software engineering, transitioned on over to SRE work. What's the next step? What's the beyond for you?Serena: Ohh, great question. So, I don't really know. I'm enjoying the SRE thing. At some point, might write a book trying to make all the concepts I have learned from my electrical engineering degree, maybe a bit more accessible, be it a series of blog posts, maybe a book. I would love to get a book published. And honestly, just writing more because knowledge should be shared, and if someone learns something from my nonsense experiments on my home lab, then cool; it's all worth it.Corey: I'd agree with that. I'm a big fan of learning in public. One of the, I guess, magical things that I do, for lack of a better term, is that I will stumble my way through learning a new concept that I have no idea what I'm doing, and when I get lost, I call it out because invariably, I'm not the only person who runs into that problem. But for folks who don't have—I don't know if it's the platform, the seniority, the perceived gravitas, the very intentional misdirection where I fooled the entire world into thinking I know what the hell I'm doing, whatever that is, most people have a problem with admitting they don't know something and learning in public, so anytime I can take up that mantle or that burden, I love doing it, just because I don't have any technical credibility to lose from my point of view. I wish that were more accepted and more common. That's why I'm so intentional about being able to talk, on some level, about the things I don't understand or the things that I don't get.Serena: I love that. I used to read a bunch of philosophy books, way back when, and my big thing, this great quote—I always get it confused, Plato or Socrates, but it's, “I know that I know nothing,” and I just run with that because I mean, even though fortunately, for me, my corner of the internet, as a non-binary person, no one's really mean to me when I say, “Okay, I broke my DNS,” because, honestly, I knew DNS conceptually when I was setting up my Minecraft server for friends, but I never really got it until I, well, kind of, broke it, [laugh] and eventually fixed it. But I hope that over time, it becomes more acceptable to say, “I don't know things.” Within my team, I tell anyone that's working with me when they're asking me a question, say, “I don't know, but I have a feeling this rabbit hole, this trail of crumbs might lead us to an answer.” And then it's a fun little adventure.Corey: I miss the days when I could describe what I do is a fun little adventure. It's now, “Oh, dear Lord, it's this bullshit again.” [sigh]. That was my sign that I was burned out, in time, find other things to do than keeping sites up.Now, I have no on-call responsibilities because there's no real site to keep up. Thank you, serverless, I get to sleep at night again. But there are times I miss aspects of working in the trenches, of being able to dive deep into a problem on a very large scale architecture. The grass is always greener, somehow.Serena: The grass is always greener. In a weird way, I actually, I complain about my on-call weeks, but I actually kind of love them. There's a weird camaraderie about all of us dealing with a shared thing. And on my team, it's really cool because we do this whole thing where, you know, I have these junior people asking, “Oh, am I going to go on call?” And we're like, “Well, unfortunately, you're not quite fully baked yet. Not quite ready. Once you're here longer with us, then yeah, we'll go walk you through a game day and make sure you can do all the things. But being on-call, it should not be a punishment for people.” Honestly, it's just the greatest feedback mechanism that guides me because I say, “Wow, this stinks. This could be better.” And then try to make it better.Corey: If people want to learn more about what you're up to, how you think about these things, or potentially even reach out for advice, where can they find you?Serena: So, I am on Twitter at @Serena—S-E-R-E-N-A—Tiede—T-I-E-D-E. DMs are open; come bug me. I got my lovely blog. It's just blog.serenacodes.com. It's pretty bare-bones, but I'll have some new content up there hopefully pretty soon, once I get around to writing it. And say hi. I like meeting new people and learning new things. Adventures await.Corey: And we will, of course, put a link to that in the [show notes 00:34:30]. Thank you so much for taking the time to speak with me. I really appreciate it, Serena.Serena: Hey, thank you. I am so happy to be here. This was one of my life goals, and now I don't know what to do now that I've gone up here.Corey: That's the problem with achieving these bucket list items. It's, “Oh, well, I wake up the following day. Now, what do I do?” And when life eventually returns to normal, on some level. [laugh]. Thanks so much for your time. I really appreciate it.Serena: Thank you. Have a great day.Corey: Serena Tiede, site reliability engineer at Optim. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment saying that if you think that C is a high-level language, oh, just wait until you explore the beauty and majesty of Rust.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Security Unfiltered
Security Unfiltered Ep 26 - Aerobics For Cyber Espionage?

Security Unfiltered

Play Episode Listen Later Jul 28, 2021 37:14


In this episode we go over the below news articles that I found interesting this week in cyber security. Below are the links to those episodes so you can read them yourself. If you enjoy the podcast please leave a review. If you want to hear about a certain topic please reach out and let me know what you would like to hear about. Thanks everyone!https://www.wired.com/2017/08/the-hotel-hacker/https://thehackernews.com/2021/07/hackers-posed-as-aerobics-instructors.htmlhttps://thehackernews.com/2021/07/hackers-turning-to-exotic-programming.htmlRankings:https://blog.feedspot.com/cyber_security_podcasts/Support the show (https://www.buymeacoff.ee/secunf)

Software Social
It's Happening!

Software Social

Play Episode Listen Later Jul 27, 2021 40:40


Buy Michele's book! Paperback and Kindle: https://www.amazon.com/dp/173744660X (or search Deploy Empathy on Amazon)PDF/ePub: deployempathy.com/pdfMichele Hansen  0:00  This episode of Software Social is sponsored by Orbit. Orbit is mission control for your community, grow and measure your community across any platform with Orbit. Find out more at orbit.love.Colleen Schnettler  0:14  Good morning, Michele. Hey,Michele Hansen  0:17  Hey, how are you?Colleen Schnettler  0:19  Great. So I hear that you have some new book updates.Michele Hansen  0:24  Yeah. So we finalized the cover this week. And I just saw, like just today just submitted it to Amazon Kindle Direct Publishing and Ingram Spark, which is another self publishing print on demand platform and filed for the copyright. So things are happening.Colleen Schnettler  0:47  That's exciting.Michele Hansen  0:51  Yeah, you know, I was thinking about our conversation last week, and how you were talking about how you felt like you weren't getting anything getting anything done? And I was like, man, I feel the same way.Colleen Schnettler  1:03  Really? Has it just felt like for weeks,Michele Hansen  1:07  yeah, like, I feel it? Well, you know, it's kind of it's like this weird in between liminal space where like, the copy has basically been final for a month now. And it's just sort of been kind of waiting on other things. And, and then there's also the, there's sort of the fact that it's summer here. And like summer camps aren't really as much of a thing here as they are in the US. Which, you know, I guess if like, most people who work for other people get four weeks of vacation, and they have kids, it's not really a big deal. But if you're self employed, it kind of is sure. Um, and so I, you know, I'm just sort of working at night and whatever. Or maybe I wake up early and get a couple hours in and like, man, I don't I don't know how parents in Europe who are self employed, do it. Like, I really, I really don't know. And like, just for weeks now I've been I mean, like, yeah, like, today's the day, I'm going to start recording the audio book, private podcast, I'm super excited about doing that. Now that the copy is finalized, I'm, like, ready to go. And it just like that time just keeps not happening. And I feel like I'm not making any progress. Um, but this morning, I did submit it and then not now it has to be reviewed. And I wanted to get a proof copy. But I think I might have done something wrong when I configured that option. And it just says your book might be published in 72 hours.Colleen Schnettler  2:44  That's fast. Okay.Michele Hansen  2:45  I haven't even like I wanted to, like, look at it and make sure the, you know, the cover looked right. And like, you know, the pages aren't upside down and whatnot. So okay, so I'm alone? I don't know. So maybe if you search on Amazon next week, you'll actually find it even though I'm not gonna tell anybody.Colleen Schnettler  3:01  But it won't be a physical copy yet. That's justMichele Hansen  3:04  so that'll be that the physical copy? Yeah, who would be a physical copy on Amazon, Amazon printed like, book to Amazon. I know, they could upload a book to Amazon. And then they print it whenever somebody buys it. Really? I know I was going, I was like, they let just anybody do this, like this? Wait, this is soColleen Schnettler  3:25  easy. This is crazy. I had no idea. So so you submit to them your cover art and your book. And then when someone buys it, they print it on demand?Michele Hansen  3:34  There's some other stuff that happens. But basically, yes, that's cool. So I don't have to like go out and you know, buy, like, basically pay for a printer to print 500 copies or whatever, then mail them out myself, which I think is what you had to do before. Things like kind of KDP or Kindle on demand or Kindle on it was what they call it? Or, you know, sort of like Do you remember like cafe press in the 90s? Like, yes, people could make t shirts and then printed it whenever you bought one. It's basically like that for books. And then there's also in Ingram Spark, which is also print on demand. But I guess there's a lot of countries that Amazon doesn't serve. And also, I guess bookstores are more willing to work with Ingram spark than they are with Amazon because they can return books to Ingram spark because Ingram spark distributes a lot of non self published books to I'm learning all about this. So So yeah, so I uploaded it to them, and then they have to review it and like, I guess, make sure it looks good. Before it'll actually, I don't know, I don't know what's gonna happen next. So we're just, we're all going to find out together. I didn't really publish the ebook. I like, you know, Barnes and Noble and whatnot, like ebook platforms. I don't know. We will find out.Colleen Schnettler  4:58  That's exciting. So you are telling me in a matter of maybe five days, maybe less people will be able to purchase a physical copy of your book. I don't know, theoretically, probably, maybe we're gonna find out cheaper than this before. SoMichele Hansen  5:15  I, originally I was like trying to give people estimates. And I was like, Yeah, it looks to me, like end of June. And then I just realized, I have no idea what I'm doing. Well, I knew that all along. But I realized that I have no idea what I'm doing. And therefore I should not try to predict what is going to happen next. Because that is just an exercise in folly to try to predict a process that I have no past experience with.Colleen Schnettler  5:41  Sure. So does that mean from you will come out when it comes out? Does that mean from your perspective that it's finished? Like you're done?Unknown Speaker  5:51  Ah,Michele Hansen  5:52  I mean, yeah, like, like yesterday Mateus looks at me, he goes, you know, this is just the beginning. Right? What does that mean? It's like Kunkle in his IColleen Schnettler  6:01  started,Michele Hansen  6:02  because, I mean, after the book is like officially out, then there's there's the, the audio book to record, right. Like, I'm super excited about doing that as a podcast and recording it myself. You know, because then I can really make sure that the, the tone of voice is coming through and everything. And I just, you know, right. Yeah.Colleen Schnettler  6:25  Can I just say I'm super disappointed when authors don't read their own books.Unknown Speaker  6:30  Yeah,Colleen Schnettler  6:30  yeah. Like, that makes me sad. Like, there's a prominent bootstrapping book, which was great. But it was not read by the author. And I was sad. I don't know why. Like, I understand why people don't want to read their own books. Maybe they don't like to talk that much. Maybe they have an accent. And then yeah, me with it. I don't know.Michele Hansen  6:45  Yeah, exactly. I think people have different reasons for not recording their own book. But I am personally really excited to do it. And to do it as a podcast, too. Because, again, I feel like I never would have gotten the book out had I not written it as a newsletter, because for me, writing an email is a lot lower pressure and stress and just mentally, like cognitively easier than like sitting down staring at a blank cursor or thinking about writing a book. And I feel the same way about recording a podcast. Like it's like, oh, it's just a podcast. And actually, I don't even have to come up with anything to say I just read something like, great, versus the idea of sitting down to record an audio book for a 320 page book that feels daunting. But yeah, a bunch of podcast episodes for each chapter that feels easy. Feels written. They just have to be concatenated.Colleen Schnettler  7:35  Yeah, yeah. Yeah. So how is this been for you? You've been working on this four to six months.Michele Hansen  7:42  Since end of February, middle, middle end of February is when I started the new Okay,Colleen Schnettler  7:47  so four months. So how do you feel to me? Yeah, right. You just knock out a book and four months? Can I just say how ridiculous that is? By the way. That's not normal.Michele Hansen  7:59  I feel like it was all in my head already. I don't really do any original research.Colleen Schnettler  8:04  It's just funny because I feel like the arc of our podcast, like your story, and the arc of our podcast is we're chatting, we're chatting. I'm like, you should write a book.Unknown Speaker  8:12  You're like, Man,Colleen Schnettler  8:13  I'm like, you should write a book. You're like, yeah, and then you wrote it. And it's done. like four months later. It's like, wait, what happens?Michele Hansen  8:21  When I commit to doing something, I do it. And usually very quickly, so but it might take me a while to actually get around to doing it.Colleen Schnettler  8:31  How is this? Ben? Are you excited to have some time back? Do you feel like I mean, has it been quite stressful these past four to five months trying to work your full time job and write this book has been overwhelming.Michele Hansen  8:45  No, it's been funColleen Schnettler  8:46  because you love it. You love the material? Fine.Michele Hansen  8:48  Like it's a little it's a little side project. And I need a little side projects. It's, you know, it's, I mean, I guess this podcast started out as a side project. And then this podcast kind of spawned the book. So like, you know, just side projects beget side projects. But no, I mean, it's been good. It's been a really good outlet for me, like most of that newsletter, writing time was actually at night, like, you know, after, put our daughter to bed and just kind of sitting in bed with my laptop and just sort of enjoying writing things out. And as I said, sort of mentally cleaning out my closet and just hauling out all of these things that mentally felt like old pieces of furniture from my head that were collecting dust or, you know, where were things I was referencing often, but didn't really have a good place to send people to. So it was it was a relief in a way to write it. And then I had so much fun interviewing people who read the early drafts. I think a really pivotal moment was when I got it into a draft and then I put it on health this book, which is Rob Fitzpatrick, the author of the mom test his new platform for launching books, and he also wrote a book that sort of goes with the the platform called write helpful books. That is, I think it's coming out now. But I was given a link to that on his help this book. Page. And that helped, that was hugely helpful for me. And then, and then, but actually getting the draft in front of people and then, and then talking to them about how they're using it and, and what kinds of books they find useful. And like, you know, it was just, it was, it was so fun. Like, I love talking to people about talking to people. And that was really fun. And then it was a little frustrating, I think, towards the end, like, I felt like I did a read like a major whole book rewrite of the book every week, in May in June, like, just like, that was probably when I did the most work. Like I was probably like, 7525 book versus giuoco do which was not super great. Um, but that was kind of what what I needed at the time. But yeah, I think I guess from like, now going forward, it's going to be lower lift things, like, promoting it. And yeah, we're podcast podcast. Yeah, the audio book and whatnot.Colleen Schnettler  11:15  Well, that's super exciting. Congratulations.Michele Hansen  11:19  It's not out yet. So I'm not gonna like,Colleen Schnettler  11:21  have you have you sent it out? Or they hatch? I think your chickens have hatched? Yeah, whateverMichele Hansen  11:27  it's for, it's getting reviewed. It's it's things are happening, things are moving, you know.Colleen Schnettler  11:33  So very exciting. Yeah,Michele Hansen  11:35  I think you're a lot more excited than I,Colleen Schnettler  11:38  I'm just really impressed. And to your point, you had this stuff in your head already. So it wasn't like you had to spawn content for the book, you had all the content. But you turned out a book fast like you, when you started doing those newsletters. I mean, you were sending a lot of newsletters. This is a lot of information.Michele Hansen  12:01  When I get really into something I like I go all in to the point where it can be a bit of a firehose, you know, like, so yeah, Marie Marie poulan. and I were talking about this a couple of weeks ago, where like, we sink our teeth into something, and then we just don't give up until we're done. Even if we wanted to. Um, I definitely I definitely feel like this has been an exerciseUnknown Speaker  12:34  in that.Colleen Schnettler  12:35  Yeah. Well, I think it's really cool. I think you should be really proud of yourself for all the work you've put in, especially during the summer, that's hard. And you're working, you know, your normal job. And you wrote a book, super cool.Michele Hansen  12:48  You're so supportive, Colleen.Colleen Schnettler  12:51  That's what I'm here for.Michele Hansen  12:53  I need you in my voice. You know, that voice in my head being like, you should be proud of this. You've come a long way, when I'm like, sort of knee deep and like filing copyright applications and stuff like that, and sort of not really able to see over the wall.Colleen Schnettler  13:07  Yeah. Yeah.Michele Hansen  13:10  Should I do a little numbers updates? Well, I don't think I've done one andColleen Schnettler  13:15  we haven't done one in a while. Go ahead.Michele Hansen  13:18  So as of right now, I have sold 93 copies for the pre order nice. Which by the way, people can pre order the it's the you get the PDF, the notion and Google Drive script templates and access to the private forthcoming private podcast with the audiobook, the boy empathy.com. So 93 people have pre ordered it right now I know a bunch of people have said they want the print copy and like I'm there with you. I don't really buy a lot of ebooks, especially for something I might want to reference later. And I don't seem to be able to do a pre order for the print book. So Oh, but anyway, so 93 people have ordered and so just looking at sort of the overall revenue for that not including expenses or you know, processing fees or whatever. That is $2,697 and I added it up with expenses a couple days ago. And I believe that puts me around sort of 12 $100 in net revenue from that so my Sunday expensesColleen Schnettler  14:32  that's great for a book you can't that's not even available yet. I mean, I know it's available yeah order but that's pretty impressive considering it's not on Amazon yet.Michele Hansen  14:43  It's kind of I mean, so I've like you know, I've heard about building in public for a long time and of course you know, I'm a big advocate of including your your customers in the in the process, but I've never really like built from scratch in public. And like just kind of outlined every step of what I was doing, you know, the, the highs and the lows. Yeah. And the massive amount of confusion in between. And so it's been a really, really interesting, like, I don't think I would have gotten to this point had I not started it as a newsletter and had that level of just motivation, you know, even from the, you know, the first five people who subscribed and would reply and say, Hey, this was great. Thank you for writing that, like that kept me going. In a way that, that I just would not have, like, actually, I think I started the book, right around the time of when, when that container ship was stuck in the Suez.Colleen Schnettler  15:45  Yes, I remember,Michele Hansen  15:47  little, that little part that nobody had on their 2021 bingo card.Unknown Speaker  15:53  And I was reading a book.Michele Hansen  15:56  Or there's a book I picked up off my shelf that I had been meant to read for years, I finally did, because of that called the box, which is a history of container shipping, which is a really interesting book, by the way. Hey, Peter shipping, revolutionize the world. And it's pretty new to like, since the 60s anyway, okay. Not what this podcast is about. So, but so I opened that book, and like the beginning of the book is The acknowledgments from the author. And it like starts out with the author talking about how lonely the process for writing a book is, and especially on a very niche topic. Yeah. And I think I had had some little like Inklings in my head of like, whether I should write a book at that point. And I remember reading that and being like, Oh, God, like that sounds really awful. Like, and I felt really bad for the author as I was reading this, because you've I've heard writers talk about how lonely of a process it is. And I like, and I think that turned me off from it for such a long time. But then it kind of like, occurred to me later that like, I can write a book, but I can do it my way. I don't have to do it the lonely way. Right. Like I could write it in public, I could include readers in the process and make it a social process from the beginning. So I didn't feel like I was just, you know, closed off in a windowless room for six months, because I think that's why I really never wrote a book before, like I was wanted to, but I was like, I don't think I could deal with that amount of loneliness that writers talk about. So yeah, it's been good.Colleen Schnettler  17:37  That's awesome.Michele Hansen  17:38  How are you doing?Colleen Schnettler  17:39  I'm good. I'm good. Yeah. So in the spirit of our podcast last week, I'm, I took some notes, and I think I'm gonna break it up every week into like, what I did this week, what I'm struggling with and what I want to do next week, to keep myself focused and keep myself moving forward. Okay, my tangent is I listen to a podcast with Angela Duckworth. Do you know who she is? She's okay. So for those who don't know who she is, she's the MacArthur Genius Grant winner. She liked her coined the term grit. So I have this podcast I really like with her. And it's her and Stephen Dubner. And it's called no stupid questions. Anyway, this week, they were talking about the difference between urgency and importance. And they were talking about how, basically, that the summation was people don't do things that they don't consider urgent. So you can have these things on your to do list, like go to the gym, which is important. We all know, that's important. But without a sense of urgency. Like, I have to be at the gym at 6pm for my weightlifting class. Instead of instead of that, instead of being like, I'll go whenever I want. There's no urgency to it. So people just don't go, oh, that explainsMichele Hansen  18:55  so much.Colleen Schnettler  18:56  It's so good. Like, I'm gonna send you this episode. It was so good. But yeah, so it was this concept. So I started thinking about it. In terms of my business, because I have all these things that I feel are really important. But I have no urgency behind them, right. There's no timeline for me, I can just sit here and this thing makes me money. And yeah, the ones setting the deadlines, right. And they're fake. I mean, and I'm not really even setting up. I'm like, oh, if I get to it if it's convenient for me today. So I just really liked this whole concept of something being urgent versus important, and how will we'll even do the less important things if we feel that they are urgent. And I say that because I'm now every week until I get to a place that I'm pretty happy with. I'm going to share with you kind of my goals. And so to make them feel a little more urgent, so I feel like I actually will do that.Michele Hansen  19:48  So I like that.Colleen Schnettler  19:51  Yeah, let's try it. It was really good. So one of the things I'm really excited about is this week, I finally got my app on rails 6.1. That's improved. To me, because I was patching in all of the CDN stuff for images because rails 6.0 didn't include that. So basically what happened is I had my app on 6.0, all the stuff was pushed on the rails master to handle CDN. And so I cherry picked it off of rails master onto my stuff, but I incorporated it as a patch to my app, which doesn't make me very happy, because it just feels brittle. So I got up to rails 6.1. So that's like a huge deal. And all of the things I have been telling you, I wanted to do, I wanted to do this first. Like, I feel like this is now going to set the stage for me to actually move forward to do other useful things. So IMichele Hansen  20:42  feel good about that. It sounds like it's gonna help your development velocity,Colleen Schnettler  20:46  it will. And I feel like some of these development blockers are really frustrating for me, like there's a really simple one, which won't take that long to do API access, but I didn't want to, I could have added new features, and then gone back and got it on 6.1. But it's smarter, in my opinion, since I have the time to get it on 6.1 before, you know, adding all the API stuff. So I feel like now that that's done, development stuff will go faster. So I'm pumped about that. And that was something that's like really kind of boring to do. I don't know if boring is the right word. But you know, like, upgrading is always kind of likeMichele Hansen  21:26  it's not shiny, right? Like developer happiness and infrastructure stuff. And, like, security kind of falls in this category to have like, stuff that's like really important. But it's not shiny, there's no, you know, revenue number, like floating over your head if you do it, right. It's more of a like, it's more of like a cost thing. It's like last time, you know, lost energy, like, it could be lost revenue, if it's security issues. Like, I think when we went full time actually, like the first thing we prioritized was like, What can we do for infrastructure and developer happiness stuff so that when we are working on stuff, it's more enjoyable to work on, more resilient, less brittle?Colleen Schnettler  22:10  That's exactly that's exactly how I feel about it. So I said, it's transparent to my customers. But it feels really good to me. For exactly those reasons. My development time now going forward will go faster. I won't have to worry about writing something I'm later gonna have to rip out when I upgrade. It's good. So I was pumped about that. Something I'm struggling with this week. This is kind of funny. So you remember like a month ago, I told you, I hired my sister to help me do marketing. That's just been kind of an interesting challenge for us, because neither of us know what to do. And so I'm like trying to do my development stuff. She's asking me questions. I'm like, I don't know. So we're both kind of spinning around. Not quite sure what to do. Hmm. So what we did is we ended up having a call with one of our mutual friends who has his own podcast, his name is Josh Oh, and his podcast is searching for SAS. And he helped us lay out a SEO content, Google Search their Google Search Console strategy. Oh, yeah. So we are kind of excited to go down that path. What I originally had asked her to do was more traditional sales Safari. And it wasn't working. Hmm. Remember how Shawn came on the podcast? And he told us he spent 80 hours like doing sales Safari?Michele Hansen  23:44  Yeah,Colleen Schnettler  23:45  yes. So my sister was trying to do that for my product. And we just weren't really, we just weren't really getting anywhere. It felt like we just weren't getting any useful information. So we are going to starting this week try to tackle this more from a content SEO perspective.Michele Hansen  24:03  Hmm. You feel like the sales Safari kind of approach was?Colleen Schnettler  24:10  I don't know I guess you you kind of already built something that's that's what Josh said. He was like, you're already you're already paying for it.Michele Hansen  24:17  So it seems like you know, I mean, Salesforce is useful at many different stages. But it sounds like you need to get eyebrow eyeballs in front of this thing. And because there are people are willing to pay for it. There's clear there's a need a huge competitors went into the space, which tells you all the more that there's need for this. You just need to tell people you exist.Colleen Schnettler  24:40  Yeah, that was his point as well. And I think that's a better use of our time is to kind of lay out a content strategy. So we're gonna try to do that I'm such a bottleneck in this process, though. It's hard to find developers to write content technical. Here's a business idea. technical content rating is really hard. I have a mutual friend who has a business way more successful than mine. And he hired a technical content agency to write some articles. They're not very good. So I'm just saying, I think that this is like a real bottleneck is like really good technical content. I'm gonna go on a limb here and say, technical content for developers has to be written by developersMichele Hansen  25:27  or by technical writers, I know that we have at least two technical writers who listen to this podcast, okay, reading my book, and like they focus on writing documentation and for develop them to do the whole job. Yeah, to dm Colleen. Colleen. And actually, I mean, they get, you know, a lot of the work, they were telling me that they get frustrated, because, like, in big companies, they get really insulated from the customers, which inhibits their ability to write dry, good documentation. Yeah. Right. Because, you know, as you're talking about the challenges with getting your sister up to speed, like, it makes me wonder, like, has she gotten to sit in on any interviews with customers? Has she gotten to do any? Like? Has she got to hear from the customers directly about what you're solving and why it's important to them?Colleen Schnettler  26:26  No, we haven't done any new customer interviews yet.Michele Hansen  26:30  Get her in those? Yeah, I think that'll really help. And you might still be the person who's kind of guiding, you know, API documentation and whatnot. But if there's a difference between hearing about what something does, from somebody who built it, and hearing about what it does, from somebody who bought it, and is excited about it,Colleen Schnettler  26:53  yeah, those areMichele Hansen  26:54  two really different things. And for marketing, what she needs to communicate is, why you should buy it and why you should be excited about it. And the technical documentation is part of that. But she needs to be able to speak to what will get someone excited about it. Yes. And who better to hear that from than someone who is excited about that themselves, ie, a customer of yours?Colleen Schnettler  27:19  Yeah, we have a whole bunch of new customers. So I think in a couple, probably starting next week, once my life's a little more organized. We're going to start trying to do more customer interviews and get back on that bandwagon because I haven't done any since I did them with you, almost three months ago. So that is definitely a priority to get that to get that going. Yes, so content is challenging, because I would love to just churn out some content. But I am struggling to find the time myself or find people that are making the kind of content that I need. So that is challenging, but I did I don't know if I told you so Drew, who we interviewed together, who was a simple file upload customer is a developer and so I paid him to write a piece for me. Oh, no. I need Yeah, do this. I was like, Drew knows how this works. Maybe he will do? Yeah, so that's it's not Yeah,Michele Hansen  28:18  dude. Like hiring your own customers is really smart. Like, I think we talked about Chris from from webflow, our mutual friend we didn't realize was a mutual friend, a couple months ago. And his first support hire is one of his customers. And it worked out like amazingly well because like the person already understands the product. Yes, he knows how it works. He knows where it might go wrong. Like, that's like that is been in the back of my mind of you know, when we need to hire for something even just you know, for something on a contract. Like, who in our customer base could do that for us?Colleen Schnettler  28:58  Yeah, I thought like, I was so pumped. So I threw you know, he said he could do it. I was like, Yes. I mean, that's the best. That's the best of both worlds. Someone who knows what they're doing as a writer. And as technical it was, it was great. So I haven't actually published it yet. Because see all these other things I've been trying to do with my life. But it's it's a guide on how to use simple file upload with react. And that has been on my to do list for four months. So let me tell you how great it felt to give it to someone who could do it better than me. It felt great. And he just got it done in like three to four days. I was like, Oh, you're you're amazing. So that was really yeah, it felt really good because you know all those things you're supposed to do. They they kind of like weigh on you and your subconscious like the things you haven't done and that is literally been on my list for four months only I have to kind of learn react before I can write about how like I kind of sorta know react but this this partnership I feel worked out really well. So that really He inspired me, it went so well with Drew, it inspired me to hire more people to write for me. But I'm definitely having a bottleneck, like finding the right kind of people, especially for the rail stuff, because I feel like I can do that better than most people. So it's a trade off.Michele Hansen  30:18  Well, so. So first, I wonder if you could create some sort of pipeline where you create one piece of content, and it can be recycled in many different ways. And I wonder if even just that one piece of content from drew like if your sister can take that and with some understanding of what the customers are trying to solve, and where they're coming from and what the product does, and recycle that into many other pieces of content? What does that mean, risk can be used in other places to further improve your SEO?Colleen Schnettler  30:49  I literally don't know what you mean. Like you mean, put it on? Like, like, yeah, so like, heMichele Hansen  30:55  wrote up this, like, long guide? Yeah. Right. Yeah. So but then you can also have landing pages that are how to do this with react. And it's like taking like bits and pieces out of that. Like if she can read that and understand it, and then be like, Oh, we can use it in these other places. You can put bits and pieces of that on your homepage on other pages like, right and use that. You're probably trying to do this, like, Look, read that article, and then look at everything in Google Search Console and say, Okay, what are the similarities in terms here? What is the actual term that people are using per Google Search Console? What is the word we're using in this piece of content? Let's change that to the word that people are typing in? Are there five variations of it? Let's make sure in this article, we have headlines that use each one of those five different variations, like, use that on other parts of our site, like, so on and so forth.Colleen Schnettler  31:44  This is the stuff we don't understand. Like I hear the words coming out of your mouth. Okay, but I'm a little confused. I mean, like, okay, so I set up okay, Search Console. So go me, I get that. So you've got keywords, right? Yeah, yes. Yes, it did. Keywords?Michele Hansen  32:04  Yes. Okay. That is the most useful part about that for me, okay. Like before, until we started using h refs, that was what I used all the time. Okay. And so that tells you all of the different keywords that are leading people to your site, okay. It's very, it's very basic, but it's like, it's, it's enough. And I think you can sort it by volume, and you know, the number of clicks and stuff that you're getting right. And then basically taking that and so so in, like in that long article that drew wrote. So I was just, you know, publishing that as a web page, not as a PDF or anything. And then search engines pick up on the headlines. So if someone is typing in, you know how to do image upload, or file upload with react, for example, then your headlines need to be like step one, like, determine which files you want people to be able to upload with react, like with your react app, like step two, like do this thing with your react app, if you want to be able to have them, you know, import files, or like what like, use different variations of that. But like, use it in the headline. So like, we have a million of these things on our website. It actually if you go to geocoded I o. And then like in the Help menu, there's one that says tutorials, we've all these step by step guides, that are all in this format, which I actually learned from another friend of ours, who is a total SEO, like genius. And then each one is like bullet points of step one, determine which addresses you want to find the congressional district for step two, take the list of addresses that you want the congressional district for, and upload them to geocode, do step three, you know, like, and it's just using those same words over and over and over again, it's kind of like, you know, in the 90s, when you saw like, a huge block of like, tiny font text at the bottom of a web sites,Colleen Schnettler  33:55  yes,Michele Hansen  33:56  that is basically how this is done now, but use different versions of that of that text to because people might be typing in different things. Like we saw, for example, we'll see that people type in lat long to Congressional District, which is something I would not type in personally, like I think of address to congressional district. So we make sure that it says address to congressional district, it also says lat lon to congressional district to GPS coordinates to congressional district, like all of those, many permutations of it, and then having as many things in headlines as possible. So that that is what the you know, search engine picks up on.Colleen Schnettler  34:36  Okay. Okay, cool. Yeah, we can work in that direction. And you're right. I didn't think about that. We already have this piece of content. SoMichele Hansen  34:43  yeah, and then just use it in many other places.Colleen Schnettler  34:46  Okay, great. Awesome. Cool. That's exciting. Yes. That's something to to focus on a little bit. I mean, I think that's what's been challenging for us is we're just what do you do next? I have no idea. I mean, I told her I was like, we're both learning here, right? This is part of the fun. This is why we're doing it like this is part of the fun of the process. But it's definitely can be a little intimidating or confusing, and to what you said aboutMichele Hansen  35:12  important versus urgent. I feel like important projects that are nebulous, get shoved to them.Colleen Schnettler  35:19  Oh, yeah. Oh, yeah, for sure. Like, totally. So we, that's great. We'll work on that. And then what I really want to do this week, is get a test sandbox environment set up on my website. You and I actually talked about this ages ago. And then when I talked to Derek Rhymer a couple weeks ago, he said it again. And I was like, I should really do this. But all this rail 6.1 stuff was the reason I hadn't done it yet. So I'm hoping I'll be able to get something like that up in a week. And basically, that would be kind of test sandbox. Yeah. So you know, if you go on to upload Cara cloudinary website, there's a big button that says try it now. And you can literally just try and like that, you can see exactly what it does before you sign up for an account, and all of that stuff. So that is something I want to go. Okay. Yeah. And I think that would be great. Because that's going to give me higher quality leads. And I think it'll encourage more people to use the service because I think my service offers some things that these other these other services don't offer. SoMichele Hansen  36:21  show them what it does.Colleen Schnettler  36:22  Yeah, exactly. I mean, that's I try I have the video, which shows them what it does. But people like to, especially developers, like at least I do, I like to put my hands on thing, like you make it look easy. Is it actually that easy? So I feel like I think that's a pretty common feeling. Yeah.Michele Hansen  36:37  Don't tell me that it's easy. Let me experience how easy it exactlyColleen Schnettler  36:41  like I want to actually do it. So that's my goal for this week. That's a little ambitious, because there's a lot of moving parts in that. But once I get that set up, I think that's going to be great for marketing, and potential customers and stuff. SoMichele Hansen  36:54  yeah, what are some of those moving parts? Because maybe if there's five steps involved, if you get three out of five, by next week, that's still pretty good.Colleen Schnettler  37:02  Yeah. So the thing I have to do to do this, my plan, at least, first of all, if I have an open file uploader open to the world, I have to be really careful with security. And so I want to write a script that automatically deletes these uploaded files, like every 10 minutes. I don't know how to do that. I mean, I'm sure I can figure it out. But like, I've never done that before. So I have no idea. I don't just know how to do that. I, again, theoretically, it's easy, but I don't know. So I want to do that. And I guess I don't need a script, I can just do it in my app, but whatever. I also want to make sure those files go to a completely separate domain, like completely separate domain, then the files I'm serving for our production customers. Because if someone says it's open to the world, if someone were to upload an inappropriate file that could be that can be bad, right?Michele Hansen  37:58  I mean, it's files. I'm vaguely remember remembering somebody's like, warning you about like that. Yeah, it was like I think on Hacker News or something like this. It happened to somebody it happened to someone else app. And yeah,Colleen Schnettler  38:10  so there was, yeah, someone sent it to me on Twitter. And it was a there's this big Hacker News thread about it. Someone else who has a similar product didn't separate his domain. So he had everyone on the same domain. And so his whole site got blacklisted. Like he didn't even separate. I'm not saying he did, he didn't know. But he didn't even separate his app from his serving domain, like mine are already separate. So that's already good. But he had literally everything on the same domain. So when his site got blacklisted by Google, like, everything went down. Oh, yeah. And he said it. You know, the interesting thing, I read the Hacker News thread, and they didn't have problems for years. I mean, they had their file uploader open to the world for like, I think was like three years. And they didn't have any issues. And then one day, bam, everything, everything was shut down. So I've already taken many security steps. I have a wireless firewall, I have separate domains for my app and my serving domain. But if I'm going to open this to the world, I want a third domain for test files. So that's I already have that. I'm actually deleting the files.Michele Hansen  39:14  Yeah. is smart, too. I don't know if that other person did that. But that disincentivizes people from using it for malicious?Colleen Schnettler  39:21  Yeah, file. I mean, one of the good things is he wrote a really detailed what I learned I could just take all of that he's and that was one of the things is he was deleting the files, I think every 36 hours and he's like, that's not enough. Like you need to be deleting the files like every 20 minutes. Okay,Michele Hansen  39:38  that's a great he's got like a step by step,Colleen Schnettler  39:40  step by step. So what not to do, so. I want to make sure I hit all of those wickets before I open this up on my website. Absolutely. Yeah, but that would be a huge I'm really excited about that. Because I really think once I get that I really think I can I can push a little more and I really think that's going to help with my Yeah, so that's my goal for next week.Michele Hansen  40:06  Alright, so next week we will check in on whether the sandbox is live on your site and maybe possibly my book will be ready. Who knows? Stay tuned.Transcribed by https://otter.ai

Kickstart Commerce Podcast
From OneWord.Domains to Elegance.ai with Steven Tey

Kickstart Commerce Podcast

Play Episode Listen Later Jun 25, 2021 63:23


Welcome to the Kickstart Commerce podcast where we share search marketing and domain investing strategies to help grow your business. In today's episode, our guest is Steven Tey — a developer and founder of Elegance.ai, formerly known as OneWord.Domains.  Today Steven and I discuss: How pursuing education in Malaysia landed him in the United States, having recently graduated a double major in branding and data science from Minerva. We then discuss how a simple prototype to take the top 10,000 most commonly used English words and pair them with popular non .com domain extensions led to the birth of OneWord.Domains. And if that wasn't enough, Steven shares his lighting-in-a-bottle moment of OneWord.Domains going viral via ProductHunt and Hacker News. Steven also shares how he used OneWord.Domains to discover and launch his next business ideas, Elegance.ai and Recurrence.app. Last but not least, Steven gives us a glimpse of what is on the horizon for him personally and professionally. In closing, don't forget to subscribe as you enjoy this week's episode via iTunes, GooglePlay, Stitcher, or however you desire to listen.

Screaming in the Cloud
Data Center War Stories with Mike Julian

Screaming in the Cloud

Play Episode Listen Later Jun 15, 2021 32:36


About MikeBeside his duties as The Duckbill Group's CEO, Mike is the author of O'Reilly's Practical Monitoring, and previously wrote the Monitoring Weekly newsletter and hosted the Real World DevOps podcast. He was previously a DevOps Engineer for companies such as Taos Consulting, Peak Hosting, Oak Ridge National Laboratory, and many more. Mike is originally from Knoxville, TN (Go Vols!) and currently resides in Portland, OR.Links: Software Engineering Daily podcast: https://softwareengineeringdaily.com/category/all-episodes/exclusive-content/Podcast/ Duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored in part by our friends at Lumigo. If you've built anything from serverless, you know that if there's one thing that can be said universally about these applications, it's that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications. It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You've created more problems for yourself; make one of them go away. To learn more, visit lumigo.io.Corey: This episode is sponsored in part by ChaosSearch. As basically everyone knows, trying to do log analytics at scale with an ELK stack is expensive, unstable, time-sucking, demeaning, and just basically all-around horrible. So why are you still doing it—or even thinking about it—when there's ChaosSearch? ChaosSearch is a fully managed scalable log analysis service that lets you add new workloads in minutes, and easily retain weeks, months, or years of data. With ChaosSearch you store, connect, and analyze and you're done. The data lives and stays within your S3 buckets, which means no managing servers, no data movement, and you can save up to 80 percent versus running an ELK stack the old-fashioned way. It's why companies like Equifax, HubSpot, Klarna, Alert Logic, and many more have all turned to ChaosSearch. So if you're tired of your ELK stacks falling over before it suffers, or of having your log analytics data retention squeezed by the cost, then try ChaosSearch today and tell them I sent you. To learn more, visit chaossearch.io.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I spent the past week guest hosting the Software Engineering Daily podcast, taking listeners over there on a tour of the clouds. Each day, I picked a different cloud and had a guest talk to me about their experiences with that cloud.Now, there was one that we didn't talk about, and we're finishing up that tour here today on Screaming in the Cloud. That cloud is the obvious one, and that is your own crappy data center. And my guest is Duckbill Group's CEO and my business partner, Mike Julian. Mike, thanks for joining me.Mike: Hi, Corey. Thanks for having me back.Corey: So, I frequently say that I started my career as a grumpy Unix sysadmin. Because it isn't like there's a second kind of Unix sysadmin you're going to see. And you were in that same boat. You and I both have extensive experience working in data centers. And it's easy sitting here on the tech coast of the United States—we're each in tech hubs cities—and we look around and yeah, the customers we talked to have massive cloud presences; everything we do is in cloud, it's easy to fall into the trap of believing that data centers are a thing of yesteryear. Are they?Mike: [laugh]. Absolutely not. I mean, our own customers have tons of stuff in data centers. There are still companies out there like Equinix, and CoreSite, and DRC—is that them? I forget the name of them.Corey: DRT. Digital Realty [unintelligible 00:01:54].Mike: Digital Realty. Yeah. These are companies still making money hand over fist. People are still putting new workloads into data centers, so yeah, we're kind of stuck with him for a while.Corey: What's fun is when I talked to my friends over in the data center sales part of the world, I have to admit, I went into those conversations early on with more than my own fair share of arrogance. And it was, “[laugh]. So, who are you selling to these days?” And the answer was, “Everyone, fool.” Because they are.People at large companies with existing data center footprints are not generally doing fire sales of their data centers, and one thing that we learned about cloud bills here at The Duckbill Group is that they only ever tend to go up with time. That's going to be the case when we start talking about data centers as well. The difference there is that it's not just an API call away to lease more space, put in some racks, buy some servers, get them racked. So, my question for you is, if we sit here and do the Hacker News—also known as the worst website on the internet—and take their first principles approach to everything, does that mean the people who are building out data centers are somehow doing it wrong? Did they miss a transformation somewhere?Mike: No, I don't think they're doing it wrong. I think there's still a lot of value in having data centers and having that sort of skill set. I do think the future is in cloud infrastructure, though. And whether that's a public cloud, or private cloud, or something like that, I think we're getting increasingly away from building on top of bare metal, just because it's so inefficient to do. So yeah, I think at some point—and I feel like we've been saying this for years that, “Oh, no, everyone's missed the boat,” and here we are saying it yet again, like, “Oh, no. Everyone's missing the boat.” You know, at some point, the boat's going to frickin' leave.Corey: From my perspective, there are advantages to data centers. And we can go through those to some degree, but let's start at the beginning. Origin stories are always useful. What's your experience working in data centers?Mike: [laugh]. Oh, boy. Most of my career has been in data centers. And in fact, one interesting tidbit is that, despite running a company that is built on AWS consulting, I didn't start using AWS myself until 2015. So, as of this recording, it's 2021 now, so that means six years ago is when I first started AWS.And before that, it was all in data centers. So, some of my most interesting stuff in the data center world was from Oak Ridge National Lab where we had hundreds of thousands of square feet of data center floor space across, like, three floors. And it was insane, just the amount of data center stuff going on there. A whole bunch of HPC, a whole bunch of just random racks of bullshit. So, it's pretty interesting stuff.I think probably the most really interesting bit I've worked on was when I was at a now-defunct company, Peak Hosting, where we had to figure out how to spin up a data center without having anyone at the data center, as in, there was no one there to do the spin up. And that led into interesting problems, like you have multiple racks of equipment, like, thousands of servers just showed up on the loading dock. Someone's got to rack them, but from that point, it all has to be automatic. So, how do you bootstrap entire racks of systems from nothing with no one physically there to start a bootstrap process? And that led us to build some just truly horrific stuff. And thank God that's someone else's problem, now. [laugh].Corey: It makes you wonder if under the hood at all these cloud providers if they have something that's a lot cleaner, and more efficient, and perfect, or if it's a whole bunch of Perl tied together with bash and hope, like we always built.Mike: You know what? I have to imagine that even at AWS at a—I know if this is true at Facebook, where they have a massive data center footprint as well—there is a lot of work that goes into the bootstrap process, and a lot of these companies are building their own hardware to facilitate making that bootstrap process easier. When you're trying to bootstrap, say, like, Dell or HP servers, the management cards only take you so far. And a lot of the stuff that we had to do was working around bugs in the HP management cards, or the Dell DRACs.Corey: Or you can wind up going with some budget whitebox service. I mean, Supermicro is popular, not that they're ultra-low budget. But yeah, you can effectively build your own. And that leads down interesting paths, too. I feel like there's a sweet spot where working on a data center and doing a build-out makes sense for certain companies.If you're trying to build out some proof of concept, yeah, do it in the cloud; you don't have to wait eight weeks and spend thousands of dollars; you can prove it out right now and spend a total of something like 17 cents to figure out if it's going to work or not. And if it does, then proceed from there, if not shut it down, and here's a quarter; keep the change. With data centers, a lot more planning winds up being involved. And is there a cutover at which point it makes sense to evacuate from a public cloud into a physical data center?Mike: You know, I don't really think so. This came up on a recent Twitter Spaces that you and I did around, at what point does it really make sense to be hybrid, or to be all-in on data center? I made the argument that a large-scale HPC does not fit cloud workloads, and someone made a comment that, like, “What is large-scale?” And to me, large-scale was always, like—so Oak Ridge was—or is famous—for having supercomputing, and they have largely been in the top five supercomputers in the world for quite some time. A supercomputer of that size is tens of thousands of cores. And they're running pretty much constant because of how expensive that stuff is to get time on. And that sort of thing would be just astronomically expensive in a cloud. But how many of those are there really?Corey: Yeah, if you're an AWS account manager listening to this and reaching out with, “No, that's not true. After committed spend, we'll wind up giving you significant discounts, and a whole bunch of credits, and jump through all these hoops.” And, yeah, I know, you'll give me a bunch of short-term contractual stuff that's bounded for a number of years, but there's no guarantee that stuff gets renewed at that rate. And let's face it. If you're running those kinds of workloads today, and already have the staff and tooling and processes that embrace that, maybe ripping all that out in a cloud migration where there's no clear business value derived isn't the best plan.Mike: Right. So, while there is a lot of large-scale HPC infrastructure that I don't think particularly fits well on the cloud, there's not a lot of that. There's just not that many massive HPC deployments out there. Which means that pretty much everything below that threshold could be a candidate for cloud workloads, and probably would be much better. One of the things that I noticed at Oak Ridge was that we had a whole bunch of SGI HPC systems laying around, and 90% of the time they were idle.And those things were not cheap when they were bought, and at the time, they're basically worth nothing. But they were idle most of the time, but when they were needed, they're there, and they do a great job of it. With AWS and GCP and Azure HPC offerings, that's a pretty good fit. Just migrate that whole thing over because it'll cost you less than buying a new one. But if I'm going to migrate Titan or Gaia from Oak Ridge over to there, yeah, some AWS rep is about to have a very nice field day. That'd just be too much money.Corey: Well, I'd be remiss as a cloud economist if I didn't point out that you can do this stuff super efficiently in someone else's AWS account.Mike: [laugh]. Yes.Corey: There's also the staffing question where if you're a large blue-chip company, you've been around for enough decades that you tend to have some revenue to risk, where you have existing processes and everything is existing in an on-prem environment, as much as we love to tell stories about the cloud being awesome, and the capability increase and the rest, yadda, yadda, yadda, there has to be a business case behind moving to the cloud, and it will knock some nebulous percentage off of your TCO—because lies, damned lies, and TCO analyses are sort of the way of the world—great. That's not exciting to most strategic-level execs. At least as I see the world. Given you are one of those strategic level execs, do you agree? Am I lacking nuance here?Mike: No, I pretty much agree. Doing a data center migration, you got to have a reason to do it. We have a lot of clients that are still running in data centers as well, and they don't move because the math doesn't make sense. And even when you start factoring in all the gains from productivity that they might get—and I stress the word might here—even when you factor those in, even when you factor in all the support and credits that Amazon might give them, it still doesn't make enough sense. So, they're still in data centers because that's where they should be for the time because that's what the finances say. And I'm kind of hard-pressed to disagree with them.Corey: While we're here playing ‘ask an exec,' I'm going to go for another one here. It's my belief that any cloud provider that charges a penny for professional services, or managed services, or any form of migration tooling or offering at all to their customers is missing the plot. Clearly, since they all tend to do this, I'm wrong somewhere. But I don't see how am I wrong or are they?Mike: Yeah, I don't know. I'd have to think about that one some more.Corey: It's an interesting point because it's—Mike: It is.Corey: —it's easy to think of this as, “Oh, yeah. You should absolutely pay people to migrate in because the whole point of cloud is that it's kind of sticky.” The biggest indicator of a big cloud bill this month is a slightly smaller one last month. And once people wind up migrating into a cloud, they tend not to leave despite all of their protestations to the contrary about multi-cloud, hybrid, et cetera, et cetera. And that becomes an interesting problem.It becomes an area—there's a whole bunch of vendors that are very deeply niched into that. It's clear that the industry as a whole thinks that migrating from data centers to cloud is going to be a boom industry for the next three decades. I don't think they're wrong.Mike: Yeah, I don't think they're wrong either. I think there's a very long tail of companies with massive footprint staying in a data center that at some point is going to get out of a data center.Corey: For those listeners who are fortunate enough not to have to come up the way that we did. Can you describe what a data center is like inside?Mike: Oh, God.Corey: What is a data center? People have these mythic ideas from television and movies, and I don't know, maybe some Backstreet Boys music video; I don't know where it all comes from. What is a data center like? What does it do?Mike: I've been in many of these over my life, and I think they really fall into two groups. One is the one managed by a professional data center manager. And those tend to be sterile environments. Like, that's the best way to describe it. They are white, filled with black racks. Everything is absolutely immaculate. There is no trash or other debris on the floor. Everything is just perfect. And it is freezingly cold.Corey: Oh, yeah. So, you're in a data center for any length of time, bring a jacket. And the soulless part of it, too, is that it's well-lit with fluorescent lights everywhere—Mike: Oh yeah.Corey: —and it's never blinking, never changing. There are no windows. Time loses all meaning. And it's strange to think about this because you don't walk in and think, “What is that racket?” But there's 10,000, 100,000 however many fans spinning all the time. It is super loud. It can clear 120 decibels in there, but it's a white noise so you don't necessarily hear it. Hearing protection is important there.Mike: When I was at Oak Ridge, we had—all of our data centers, we had a professional data center manager, so everything was absolutely pristine. And to get into any of the data centers, you had to go through a training; it was very simple training, but just, like, “These are things you do and don't do in the data center.” And when you walked in, you had to put in earplugs immediately before you walked in the door. And it's so loud just because of that, and you don't really notice it because you can walk in without earplugs and, like, “Oh, it's loud, but it's fine.” And then you leave a couple hours later and your ears are ringing. So, it's a weird experience.Corey: It's awful. I started wearing earplugs every time I went in, just because it's not just the pain because hearing loss doesn't always manifest that way. It's, I would get tired much more quickly.Mike: Oh, yeah.Corey: I would not be as sharp. It was, “What is this? Why am I so fatigued?” It's noise.Mike: Yeah. And having to remember to grab your jacket when you head down to the data center, even though it's 95 degrees outside.Corey: At some point, if you're there enough—which you probably shouldn't be—you start looking at ways to wind up storing one locally. I feel like there could be some company that makes an absolute killing by renting out parkas at data centers.Mike: Yeah, totally. The other group of data center stuff that I generally run into is the exact opposite of that. And it's basically someone has shoved a couple racks in somewhere and they just kind of hope for the best.Corey: The basement. The closet. The hold of a boat, with one particular client we work with.Mike: Yeah. That was an interesting one. So, we had a—Corey and I had a client where they had all their infrastructure in the basement of a boat. And we're [laugh] not even kidding. It's literally in the basement of a boat.Corey: Below the waterline.Mike: Yeah below the waterline. So, there was a lot of planning around, like, what if the hold gets breached? And like, who has to plan for that sort of thing? [laugh]. It was a weird experience.Corey: It turns out that was—was hilarious about that was while they were doing their cloud migration into AWS, their account manager wasn't the most senior account manager because, at that point, it was a small account, but they still stuck to their standard talking points about TCO, and better durability, and the rest, and it didn't really occur to them to come back with a, what if the boat sinks? Which is the obvious reason to move out of that quote-unquote, “data center?”Mike: Yeah. It was a wild experience. So, that latter group of just everything's an absolute wreck, like, everything—it's just so much of a pain to work with, and you find yourself wanting to clean it up. Like, install new racks, do new cabling, put in a totally new floor so you're not standing on concrete. You want to do all this work to it, and then you realize that you're just putting lipstick on a pig; it's still going to be a dirty old data center at the end of the day, no matter how much work you do to it. And you're still running on the same crappy hardware you had, you're still running on the same frustrating deployment process you've been working on, and everything still sucks, despite it looking good.Corey: This episode is sponsored in part by ChaosSearch. As basically everyone knows, trying to do log analytics at scale with an ELK stack is expensive, unstable, time-sucking, demeaning, and just basically all-around horrible. So why are you still doing it—or even thinking about it—when there's ChaosSearch? ChaosSearch is a fully managed scalable log analysis service that lets you add new workloads in minutes, and easily retain weeks, months, or years of data. With ChaosSearch you store, connect, and analyze and you're done. The data lives and stays within your S3 buckets, which means no managing servers, no data movement, and you can save up to 80 percent versus running an ELK stack the old-fashioned way. It's why companies like Equifax, HubSpot, Klarna, Alert Logic, and many more have all turned to ChaosSearch. So if you're tired of your ELK stacks falling over before it suffers, or of having your log analytics data retention squeezed by the cost, then try ChaosSearch today and tell them I sent you. To learn more, visit chaossearch.io.Corey: The worst part is playing the ‘what is different here?' Game. You rack twelve servers: eleven come up fine and the twelfth doesn't.Mike: [laugh].Corey: It sounds like, okay, how hard could it be? Days. It can take days. In a cloud environment, you have one weird instance. Cool, you terminate it and start a new one and life goes on whereas, in a data center, you generally can't send back a $5,000 piece of hardware willy nilly, and you certainly can't do it same-day, so let's figure out what the problem is.Is that some sub-component in the system? Is it a dodgy cable? Is it, potentially, a dodgy switch port? Is there something going on with that node? Was there something weird about the way the install was done if you reimage the thing? Et cetera, et cetera. And it leads down rabbit holes super quickly.Mike: People that grew up in the era of computing that Corey and I did, you start learning tips and tricks, and they sound kind of silly these days, but things like, you never create your own cables. Even though both of us still remember how to wire a Cat 5 cable, we don't.Corey: My fingers started throbbing when you said that because some memories never fade.Mike: Right. You don't. Like, if you're working in a data center, you're buying premade cables because they've been tested professionally by high-end machines.Corey: And you still don't trust it. You have a relatively inexpensive cable tester in the data center, and when—I learned this when I was racking stuff the second time, it adds a bit of time, but every cable that we took out of the packaging before we plugged it in, and we tested on the cable tester just to remove that problem. And it still doesn't catch everything because, welcome to the world of intermittent cables that are marginal that, when you bend a certain way, stop working, and then when you look at them, start working again properly. Yes, it's as maddening as it sounds.Mike: Yeah. And then things like rack nuts. My fingers hurt just thinking about it.Corey: Think of them as nuts that bolts wind up screwing into but they're square and they have clips on them so they clip into the standard rack cabinets, so you can screw equipment into them. There are different sizes of them, and of course, they're not compatible with one another. And you have—they always pinch your finger and make you bleed because they're incredibly annoying to put in and out. Some vendors have quick rails, which are way nicer, but networking equipment is still stuck in the ‘90s in that context, and there's always something that winds up causing problems.Mike: If you were particularly lucky, the rack nuts that you had were pliable enough that you could pinch them and pull them out with your fingers, and hopefully didn't do too much damage. If you were particularly unlucky, you had to reach for a screwdriver to try to pry it out, and inevitably stab yourself.Corey: Or sometimes pulling it out with your fingers, it'll—like, those edges are sharp. It's not the most high-quality steel in some cases, and it's just you wind up having these problems. Oh, one other thing you learn super quickly, is first, always have a set of tools there because the one you need is the one you don't have, and the most valuable tool you'll have is a pair of wire cutters. And what you do when you find a bad cable is you cut it before throwing it away.Mike: Yep.Corey: Because otherwise someone who is very well-meaning but you will think of them as the freaking devil, will, “Oh, there's a perfectly good cable sitting here in the trash. I'll put it back with the spares.” So you think you have a failed cable you grab another one from the pile of spares—remember, this is two in the morning, invariably, and you're not thinking on all cylinders—and the problem is still there. Cut the cable when you throw it away.Mike: So, there are entire books that were written about these sorts of tips and tricks that everyone working [with 00:19:34] data center just remembers. They learned it all. And most of the stuff is completely moot now. Like, no one really thinks about it anymore. Some people are brought up in computing in such a way that they never even learned these things, which I think it's fantastic.Corey: Oh, I don't wish this on anyone. This used to be a prerequisite skill for anyone who called themselves a systems administrator, but I am astonished when I talk to my AWS friends, the remarkably senior engineers I talk to who have never been inside of an AWS data center.Mike: Yeah, absolutely.Corey: That's really cool. It also means you're completely divorced from the thing you're doing with code and the rest, and the thing that winds up keeping the hardware going. It also leads to a bit of a dichotomy where the people racking the hardware, in many cases, don't understand the workloads that are on there because if you have the programming insight, and ability, and can make those applications work effectively, you're probably going to go find a role that compensates far better than working in the data center.Mike: I [laugh] want to talk about supply chains. So, when you build a data center, you start planning about—let's say, I'm not Amazon. I'm just, like, any random company—and I want to put my stuff into a data center. If I'm going to lease someone else's data center—which you absolutely should—we're looking at about a 180-day lead time. And it's like, why? Like, that's a long time. What's—Corey: It takes that long to sign a real estate lease?Mike: Yeah.Corey: No. It takes that long to sign a real estate lease, wind up talking to your upstream provider, getting them to go ahead and run the thing—effectively—getting the hardware ordered and shipped in the right time window, doing the actual build-out once everything is in place, and I'm sure a few other things I'm missing.Mike: Yeah, absolutely. So yeah, you have all these things that have to happen, and all of them pay for-freaking-ever. Getting Windstream on the phone to begin with, to even take your call, can often take weeks at a time. And then to get them to actually put an order for you, and then do the turnup. The turnup alone might be 90 days, where I'm just, “Hey, I've bought bandwidth from you, and I just need you to come out and connect the [BLEEP] cables,” might be 90 days for them to do it.And that's ridiculous. But then you also have the hardware vendors. If you're ordering hardware from Dell, and you're like, “Hey, I need a couple servers.” Like, “Great. They'll be there next week.” Instead, if you're saying, “Hey, I need 500 servers,” they're like, “Ooh, uh, next year, maybe.” And this is even pre-pandemic sort of thing because they don't have all these sitting around.So, for you to get a large number of servers quickly, it's just not a thing that's possible. So, a lot of companies would have to buy well ahead of what they thought their needs would be, so they'd have massive amounts of unused capacity. Just racks upon racks of systems sitting there turned off, waiting for when they're needed, just because of the ordering lead time.Corey: That's what auto-scaling looks like in those environments because you need to have that stuff ready to go. If you have a sudden inrush of demand, you have to be able to scale up with things that are already racked, provisioned, and good to go. Sometimes you can have them halfway provisioned because you don't know what kind of system they're going to need to be in many cases, but that's some up-the-stack level thinking. And again, finding failed hard drives and swapping those out, make sure you pull the right or you just destroyed an array. And all these things that I just make Amazon's problem.It's kind of fun to look back at this and realize that we would get annoyed then with support tickets that took three weeks to get resolved in hardware, whereas now three hours in you and I are complaining about the slow responsiveness of the cloud vendor.Mike: Yeah, the amount of quick turnaround that we can have these days on cloud infrastructure that was just unthinkable, running in data centers. We don't run out of bandwidth now. Like, that's just not a concern that anyone has. But when you're running in a data center, and, “Oh, yeah. I've got an OC-3 line connected here. That's only going to get me”—Corey: Which is something like—what is an OC-3? That's something like, what, 20 gigabit, or—Mike: Yeah, something like that. It's—Corey: Don't quote me on that.Mike: Yeah. So, we're going to have to look that up. So, it's equivalent to a T-3, so I think that's a 45 megabit?Corey: Yeah, that sounds about reasonable, yeah.Mike: So, you've got a T-3 line sitting here in your data center. Like that's not terrible. And if you start maxing that out, well, you're maxed out. You need more? Again, we're back to the 90 to 180 day lead time to get new bandwidth.So, sucks to be you, which means you'd have to start planning your bandwidth ahead of time. And this is why we had issues like companies getting Slashdotted back in the day because when you capped the bandwidth out, well, you're capped out. That's it. That's the game.Corey: Now, you've made the front page of Slashdot, a bunch of people visited your site, and the site fell over. That was sort of the way of the world. CDNs weren't really a thing. Cloud wasn't a thing. And that was just, okay, you'd bookmark the thing and try and remember to check it later.We talked about bandwidth constraints. One thing that I think the cloud providers do—at least the tier ones—that are just basically magic is full line rate between any two instances almost always. Well, remember, you have a bunch of different racks, and at the top of every rack, there's usually a switch called—because we're bad at naming things—top-of-rack switches. And just because everything that you have plugged in can get one gigabit to that switch—or 10 gigabit or whatever it happens to be—there is a constraint in that top-of-rack switch. So yeah, one server can talk to another one in a different rack at one gigabit, but then you have 20 different servers in each rack all trying to do something like that and you start hitting constraints.You do not see that in the public cloud environments; it is subsumed away, you don't have to think about that level of nonsense. You just complain about what feels like the egregious data transfer charge.Mike: Right. Yeah. It was always frustrating when you had to order nice high-end switching gear from Cisco, or Arista, or take your pick of provider, and you got 48 ports in the top-of-rack, you got 48 servers all wired up to them—or 24 because we want redundancy on that—and that should be a gigabit for each connection, except when you start maxing it out, no, it's nowhere even near that because the switch can't handle it. And it's absolutely magical, that the cloud provider's like, “Oh, yeah. Of course, we handle that.”Corey: And you don't have to think about it at all. One other use case that I did want to hit because I know we'll get letters if we don't, where it does make sense to build out a data center, even today, is if you have regulatory requirements around data residency. And there's no cloud vendor in an area that suits. This generally does not apply to the United States, but there are a lot of countries that have data residency laws that do not yet have a cloud provider of their choice region, located in-country.Mike: Yeah, I'll agree with that, but I think that's a short-lived problem.Corey: In the fullness of time, there'll be regions everywhere. Every build—a chicken in every pot and an AWS availability zone on every corner.Mike: [laugh]. Yeah, I think it's going to be a fairly short-lived problem, which actually reminds me of even our clients that have data centers are often treating the data center as a cloud. So, a lot of them are using your favorite technology, Corey, Kubernetes, and they're treating Kubernetes as a cloud, running Kube in AWS, as well, and moving workloads between the two Kube clusters. And to them, a data center is actually not really data center; it's just a private cloud. I think that pattern works really well if you have a need to have a physical data center.Corey: And then they start doing a hybrid environment where they start expanding to a public cloud, but then they treat that cloud like just a place to run a bunch of VMs, which is expensive, and it solves a whole host of problems that we've already talked about. Like, we're bad at replacing hard drives, or our data center is located on a corner where people love to get drunk on the weekends and smash into the power pole and take out half of the racks here. Things like that great, yeah, cloud can solve that, but cloud could do a lot more. You're effectively worsening your cloud experience to improve your data center experience.Mike: Right. So, even when you have that approach, the piece of feedback that we give the client was, you have built such a thing where you have to cater to the lowest common denominator, which is the constraints that you have in the data center, which means you're not able to use AWS the way that you should be able to use it so it's just as expensive to run as a data center was. If they were to get rid of the data center, then the cloud would actually become cheaper for them and they would get more benefits from using it. So, that's kind of a business decision for how they've structured it, and I can't really fault them for it, but there are definitely some downsides to the approach.Corey: Mike, thank you so much for joining me here. If people want to learn more about what you're up to, where can they find you?Mike: You know, you can find me at duckbillgroup.com, and actually, you can also find Corey at duckbillgroup.com. We help companies lower their AWS bills. So, if you have a horrifying bill, you should chat.Corey: Mike, thank you so much for taking the time to join me here.Mike: Thanks for having me.Corey: Mike Julian, CEO of The Duckbill Group and my business partner. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and then challenge me to a cable-making competition.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Python Bytes
#238 A cloud-based file system for Python and a new GUI!

Python Bytes

Play Episode Listen Later Jun 15, 2021 47:07


Watch the live stream: Watch on YouTube About the show Sponsored by Sentry: Sign up at pythonbytes.fm/sentry And please, when signing up, click Got a promo code? Redeem and enter PYTHONBYTES Special guest: Julia Signell Brain #1: Practical SQL for Data Analysis Haki Benita Pandas is awesome, but … “In this article I demonstrate how to use SQL to perform fast and efficient data analysis.” First part of the article. SQL is faster than Pandas But they are great together Then tons of examples showing exactly how to best use SQL queries and Pandas in data analysis:: Basics including random data and sampling Descriptive statistics Subtotals including rollup and groupign sets Pivot tables, both conditional expressions and aggregate expressions Running and cumulative agregation Linear Regression Interpolation Super cheat sheet for useful SQL queries Michael #2: Git Blame in your Python Tracebacks via Ruslan Portnoy, by Ofer Koren Helpful Modules: traceback & linecache traceback uses linecache, and we can change linecache line's text They create a git blame bit of functionality to add to line's source Turns out this flows to things like PDB. Ripe for a proper package we can add to requirements-dev.txt Julia #3: fsspec: a unified file system library Martin Durant Other libraries conform to the interface so that each part of the analysis pipeline is like an interchangeable building block (for example s3fs, gcsfs) With the cloud providers competing to host data, fsspec makes it easy to swap out the read layer so that you can hop clouds. Brian #4: The need for slimmer containers or I'm even more confused now as to the usefulness of official base images on Docker Hub Ivan Velichko @iximiuz I read this article recently and it had me concerned. Then just yesterday read it again and there are some updates. I'm still concerned, but now also confused. So let's run it down. docker scan can be run on official Python images. It uses Snyk Container. We talked about one form of Snyk on Episode 227. Spoiler, all of the official Python containers have vulnerabilities except alpine. But. In an update, the author says that Alpine has a bunch of problems. The update includes some discussion on Hacker News vulnerability scanners tend to have lots of false positives official base images are rarely updated some people suggest adding an upgrade command in the beginning of every Dockerfile. but others object saying that the practice leads to unrepeatable builds So, I'm left with wondering if using official Python images are even worth it. Michael: Python's official image on docker hub Michael: PEP 656 -- Platform Tag for Linux Distributions Using Musl Michael: We dive a lot into this in our latest Talk Python recording (not out yet, but live stream is available) Some stats: Ubuntu: Found 32 vulnerabilities, 31 with upgrade. python:latest: Found 364 vulnerabilities, 353 with upgrade Ubuntu with source Python: 35 total, 28 low, 7 medium, several from intermediate tools such as wget, gcc, etc. Removing many dev tools SHOULD lower the count, but doesn't (e.g. wget, gcc) Switching from python:3-9 to python:3.9-slim-buster dropped the issues to 69. Michael #5: PandasGUI: A GUI for analyzing Pandas DataFrames Features View DataFrames and Series (with MultiIndex support) Interactive plotting Filtering Statistics summary Data editing and copy / paste Import CSV files with drag & drop Search toolbar Best way to see what it's about is to watch the video. Julia #6: xarray: pandas-like API for labeled N-dimensional data We've been talking a lot about the pandas API and how it's a common target for dataframe libraries. Xarray is not a dataframe library, it's for labeled N-dimensional data. People use it in geosciences, and in image processing where they don't have tabular data, but the axes mean something (lat, lon, time, band…) You can select, aggregate, resample, using the real dimension labels. It can be backed with dask arrays or numpy arrays (or other types of arrays). It supports plotting with .plot Extras Michael Python 3.10.0b2 is available (even windows store) Django security releases issued: 3.2.4, 3.1.12, and 2.2.24 Another method overloading library? Recently moved to pip-compile requirements.in style after last week I'm running PyCharm EAP Brian Someone responded to me the other day on twitter with an emoji that I was not clear on the meaning of. So I looked it up on emojipedia.org. Super useful for occasionally out of touch people like myself. pytestbook.com (redirects to pythontest.com/pytest-book/) has a facelift and a new home, to get ready for an announcement later this week. It's built on markdown, hugo, github, and Netlify, so changes can be done super quick with just a commit and push. I just needed a nice readable theme, and Pradyun's blog looked great, so I copied his choices. The blog will eventually also have writing, the legacy posts worth keeping from pythontesting.net, and probably transcripts from Test & Code. Julia GH CLI entrypoints - they are so cool! Example - with pandas you can plot with different backends not just matplotlib and the logic for those backends is contained in the plotting libraries not pandas. Joke From https://upjoke.com/programmer-jokes I asked a programmer what her New Year's resolution will be. She answered: 1920x1080. How does a programmer confuse a mathematician? x = x + 1 Why do Python programmers have low self esteem? They're constantly comparing their self to other.

The Bike Shed
296: Speedy Performance with Nate Berkopec

The Bike Shed

Play Episode Listen Later Jun 15, 2021 63:33


Nate Berkopec is the author of the Complete Guide to Rails Performance, the creator of the Rails Performance Workshop, and the co-maintainer of Puma. He talks with Steph about being known as "The Rails Speed Guy," and how he ended up with that title, publishing content, working on workshops, and also contributing to open source projects. (You could say he's kind of a busy guy!) Speedshop (https://www.speedshop.co/) Puma (https://github.com/puma/puma/commits/master?author=nateberkopec) The Rails Performance Workshop (https://www.speedshop.co/rails-performance-workshop.html) The Complete Guide to Rails Performance (https://www.railsspeed.com/) How To Use Turbolinks to Make Fast Rails Apps (https://www.speedshop.co/2015/05/27/100-ms-to-glass-with-rails-and-turbolinks.html) Sidekiq (https://sidekiq.org/) Follow Nate Berkopec on Twitter (https://twitter.com/nateberkopec) Visit Nate's Website (https://www.nateberkopec.com/) Sign up for Nate's Speedshop Ruby Performance Newsletter (https://speedshop.us11.list-manage.com/subscribe?u=1aa0f43522f6d9ef96d1c5d6f&id=840412962b) Transcript: STEPH: All right. I'll kick us off with our fancy intro. Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Steph Viccari. And this week, Chris is taking a break. But while he's away, I'm joined by Nate Berkopec, who is the owner of Speedshop, a Ruby on Rails performance consultancy. And, Nate, in addition to running a consultancy, you're the co-maintainer of Puma. You're also an author as you wrote a book called The Complete Guide to Rails Performance. And you run the workshop called The Rails Performance Workshop. So, Nate, I'm sensing a theme here. NATE: Yeah, make code go fast. STEPH: And you've been doing that for quite a while, haven't you? NATE: Yeah. It's pretty much been since 2015, or so I think. It all started when I actually wrote a blog post about Turbolinks that got a lot of pick up. My hot take at the time was that Turbolinks is actually a good thing. That take has since become uncontroversial, but it was quite controversial in 2015. So I got a lot of pick up on that, and I realized I liked working on performance, and people seem to want to hear about it. So I've been in that groove ever since. STEPH: When you started down the path of really focusing on performance, were you running your own consultancy at that point, or were you working for someone else? NATE: I would say it didn't really kick off until I actually published The Complete Guide to Rails Performance. So after that came out, which was, I think, March of 2016…I hope I'm getting that right. It wasn't until after that point when it was like, oh, I'm the Rails performance guy now. And I started getting emails inbound about that. I didn't really have any time when I was actually working on the CGRP to do that sort of thing. I just made that my full-time job to actually write, and market, and publish that. So it wasn't until after that that I was like, oh, I'm a performance consultant now. This is the lane I've driven myself into. I don't think I really had that as a strategy when I was writing the book. I wasn't like, okay, this is what I'm going to do. I'm going to build some reputation around this, and then that'll help me be a better consultant with this. But that's what ended up happening. STEPH: I see. So it sounds like it really started more as a passion and something that you wanted to share. And it has manifested to this point where you are the speed guy. NATE: Yeah, I think you could say that. I think when I started writing about it, I just knew...I liked it. I liked the work of performance. In a lot of ways, performance is a much more concrete discipline than a lot of other sub-disciplines of programming where I joke my job is number go down. It's very measurable, and it's very clear when you've made a difference. You can say, “Hey, this number was this, and now it's this. Look what I did.” And I always loved that concreteness of performance work. It makes it actually a lot more like a real kind of engineering discipline where I think of performance engineering as clarifying requirements and the limitations and then building a project that meets the requirements while staying within those limitations and constraints. And that's often not quite as clear for other disciplines like general feature work. It's kind of hard to say sometimes, like, did you actually make the user's life better by implementing such and such? That's more of a guess. That's more of a less clear relationship. And with performance, nobody's going to wake up ten years from today and wish that their app was slower. So we can argue about the relative importance of performance in an application, but we don't really argue about whether or not we made it faster because we can prove that. STEPH: Yeah. That's one area that working with different teams (as I tend to shift the clients that I'm working with every six months) where we often push hard around feature work to say, “How can we measure this? How can we know that we are delivering something valuable to users?” But as you said, that's really tricky. It's hard to evaluate. And then also, when you add on the fact that if I am leaving that project in six months, then I don't have the same insights to understand how something went for that team. So I can certainly appreciate the satisfaction that comes from knowing that, yes, you are delivering a faster app. And it's very measurable, given the time that you're there, whether it's a short time or if it's a long time that you're with that team. NATE: Yeah, totally. My consulting engagements are often really short. I don't really do a lot of super long-term stuff, and that's usually fine because I can point to stuff and say, “Yep. This thing was at A, and now it's at B. And that's what you hired me to do, so now it's done.” STEPH: I am curious; given that you have so many different facets where you are running your consultancy, you are also often publishing a lot of content and working on workshops and then also contributing to open source projects. What does a typical week look like for you? NATE: Well, right now is actually a decent example. I have client work two or three days a week. And I'm actually working on a new product right now that I'm calling Sidekiq in Practice, which is a course/workshop about scaling Sidekiq from zero to 1000 jobs per second. And I'll spend the other days of the week working on that. My content is...I always struggle with how much time to spend on blogging specifically because it takes so much time for me to come up with a post and publish that. But the newsletter that I write, which I try to write two once a week, I haven't been doing so well with it lately. But I think I got 50 newsletters done in 2020 or something like that. STEPH: Wow. NATE: And so I do okay on the per-week basis. And it's all content I've never published anywhere else. So that actually is like 45 minutes of me sitting down on a Monday and being like rant, [chuckles] slam keyboard and rant and then hit send. And my open source work is mostly 15 minutes a day while I'm drinking morning coffee kind of stuff. So I try to spread myself around and do a lot of different stuff. And a lot of that means, I think, pulling back in terms of thinking how much you need to spend on something, especially with newsletters, email newsletters, it was very easy to overthink that and spend a lot of time revising and whatever. But some newsletter is better than no newsletter. And especially when it comes to content and marketing, I've learned that frequency and regularity is more important than each and every post is the greatest thing that's ever come out since sliced bread. So trying to build a discipline and a practice around doing that regularly is more important for me. STEPH: I like that, some newsletter is better than no newsletter. I was listening to your chat with Brittany Martin on the Ruby on Rails podcast. And you said something very honest that I appreciated where you said, “Writing is really hard, and writing sucks.” And that made me laugh in the moment because even though I do enjoy writing, I still find it very hard to be disciplined, to sit down and make it happen. And then you go into that editor mode where you critique everything, and then you never really get it published because you are constantly fixing it. It sounds like...you've mentioned you set aside about 45 minutes on a Monday, and you crank out some work. How do you work through that inner critic? How do you get past it to the point where then you just publish? NATE: You have to separate the steps. You have to not do editing and first drafting at the same time. And the reason why I say it sucks and it's hard is because I think a lot of people don't do a lot of regular writing, maybe get intimidated when they try to start. And they're like, “Wow, this is really hard. This is not fun.” And I'm just trying to say that's everybody's experience and if it doesn't get any better, because it doesn't, [chuckles] there's nothing wrong with you, that's just writing, it's hard. For me, especially with the newsletter, I just have to give myself permission not to edit and to just hit send when I'm done. I try to do some spell checking,, and that's it. I just let it go. I'm not going back and reading it through again and making sure that I was very clear and cogent in all my points and that there's a really good flow through that newsletter. I think it comes with a little bit of confidence in your own ideas and your own experience and knowledge, believing that that's worth sharing and that's worth somebody's time, even if it's not a perfect expression of what's in your head. Like, a 75% expression is good enough, especially in a newsletter format where it's like 500 to 700 words. And it's something that comes once a week. And maybe not everyone's amazing, but some of them are, enough of them are that people stay subscribed. So I think a combination of separating editing and first drafting and just having enough confidence and the basis where you have to say, “It doesn't have to be perfect every single time.” STEPH: Yeah, I think that's something that I learned a while back to apply to my coding process where I had to separate those two steps of where I have to let the creator in me just create and write some code and make it work, and then come back to the editing process, and taking a similar approach with writing. As you may be familiar with thoughtbot, we're big advocates when it comes to sharing content and sharing things that we have learned throughout the week and different projects that we're working on. And often when people join thoughtbot, they're very excited to contribute to the blog. But it is daunting for that first post because you think it has to be this really grand novel. And it has to be something that is really going to appeal to everybody, and it's going to help everyone. And then over time, you learn it's like, oh well, actually it can be this very just small thing that I learned that maybe only helps 20 people, but it still helped those 20 people. And learning to publish more frequently versus going for those grand pieces is more favorable and often more helpful for people. NATE: Yeah, totally. That's something that is difficult for people at first. But everything in my experience has led me to believe that frequency and regularity is just as, if not more important than the quality of any individual piece of content that I put out. So that's not to say that...I guess it's weird advice to give because people will take it too far the other way and think that means he's saying quality doesn't matter. No, of course, it does, but I think just everyone's internal biases are just way too tuned towards this thing must be perfect. I've also learned we're just really bad judges internally of what is useful and good for people. Stuff that I think is amazing and really interesting sometimes I'll put that out, and nobody cares. [chuckles] And the other stuff I put out that's just like the 45-minute banging out newsletter, people email me back and say, “This is the most helpful thing anyone's ever read.” So that quality bias also assumes that you know what is good and actually we're not really good at that, knowing every time what our audience needs is actually really difficult. STEPH: That's totally fair. And I have definitely run into that too, where I have something that I'm very proud of and excited to share, and I realize it relates to a very small group of people. But then there's something small that I do every day, and then I just happen to tweet about it or talk about it, and suddenly that's the thing that everybody's really excited about. So yeah, you never know. So share it all. NATE: Yeah. And it's important to listen. I pay attention to what people get interested in from what I put out, and I will do more of that in the future. STEPH: You mentioned earlier that you are working on another workshop focused on Sidekiq. What can you tell me about that? NATE: So it's meant to be a guide to scaling Sidekiq from zero to 1000 requests per second. And it's meant to be a missing guide to all the things that happen, like the situations that can crop up operationally when you're working on an application that does a lot of work with Sidekiq. Whereas Mike Sidekiq, Wiki, or the docs are great about how do, you do this? What does this setting mean? And the basics of getting it just running, Sidekiq in practice, is meant to be the last half of that. How do you get it to run 1,000 jobs per second in a day-to-day application? So it's the collected wisdom and collected battle scars from five years of getting called in to fix people's Sidekiq installations and very much a product of what are the actual problems that people experience, and how do you fix and deal with those? So stuff about memory and managing Sidekiq memory usage, how to think about queues. Like, what should your queue structure be? How many should you have? Like, how do you organize jobs into queues, and how do you deal with problems like some client is dropping 10,000, 20,000 jobs into a queue. And now the other jobs I put in that queue have 20,000 jobs in front of them. And now this other job I've got will take three hours to get through that queue. How do you deal with problems like that? All the stuff that people have come to me over the years and that I've had to help them fix. STEPH: That sounds really great. Because yeah, I find that teams who are often in this space with Sidekiq we just let it run until there's a fire. And then suddenly, we start to care as to how it's processing, and we care about our queue structure and how many workers that we have that are pulling from that queue. So that sounds really helpful. When you're building a workshop, do you often go back to any of those customers and pull more ideas from them, or do you find that you just have enough examples from your collective work with clients that that itself creates a course? NATE: Usually, pretty much every chapter in the workshop I've probably implemented like three-plus times, so I don't really have to go back to any individual customer. I have had some interesting stuff with my current client, Gusto. And Gusto is going through some background job reorganization right now and actually started to implement a lot of the things that I'm advocating in the workshop actually without talking to me. It was a good validation of hey, we all actually think the same here. And a lot of the solutions that they were implementing were things that I was ready to put down into those workshops. So I'd like to see those solutions implemented and succeed. So I think a lot of the stuff in here has been pretty battle-tested. STEPH: For the Rails Performance Workshop, you started off doing those live and in-person with teams, and then you have since switched to now it is a CLI course, correct? NATE: That's correct. Yep. STEPH: I love that very much. When you've talked about it, it does feel very appropriate in terms of developers and how we like to consume content and learn. So that is really novel and also, it seems like a really nice win for you. So then other people can take this course, but you are no longer the individual that has to deliver it to their team, that they can independently take the course and go through it on their own. Are you thinking about doing the same thing for the Sidekiq course, or what are your plans for that one? NATE: Yeah, it's the exact same structure. So it's going to be delivered via the command line. Although I would say Sidekiq in practice has more text components. So it's going to be a combination of a very short manual or book, and some video, and some hands-on exercises. So, an equal blend between all three of those components. And it's a lot of stuff that I've learned over having to teach; I guess intermediate to advanced programming concepts for the last five years now that people learn at different paces. And one of the great things about this kind of format is you can pick it up, drop it off, and move at your own speed. Whereas a lot of times when I would do this in person, I think I would lose people halfway through because they would get stuck on something that I couldn't go back to because we only had four hours of the day. And if you deliver it in a class format, you're one person, and I've got 24 other people in this room. So it's infinitely pausable and replayable, and you can go back, or you can just skip ahead. If you've got a particular problem and you're like, hey, I just want to figure out how to fix such and such; you can do that. You can just come in and do a particular thing and then leave, and that's fine. So it's a good format that way. And I've definitely learned a lot from switching to pre-recorded and pre-prepared stuff rather than trying to do this all live in person. STEPH: That is one of the lessons that I've learned as well from the couple of workshops that I've led is that doing them in person, there's a lot of energy. And I really enjoy that part where I get to see people respond to the content. And then I get a lot of great feedback from people about what type of questions they have, where they are getting stuck. And that part is so important to me that I always love doing them live first. But then you get to the point, as you'd mentioned, where if you have a room full of 20 people and you have two people that are stuck, how do you help them but then still keep the class going forward? And then, if you are trying to tailor this content for a wide audience…so maybe beginners could take the Rails Performance Workshop, or they could also take the Sidekiq course. But you also want the more senior engineers to get something out of it as well. It's a very challenging task to make that content scale for everyone. NATE: Yeah. What you said there about getting feedback and learning was definitely something that I got out of doing the Rails Performance Workshop in person like three dozen times, was the ability to look over people's shoulders and see where they got stuck. Because people won't email me and say, “Hey, this thing is really confusing.” Or “It doesn't work the way you said it does for me.” But when I'm in the same room with them, I can look over their shoulder and be like, “Hey, you're stuck here.” People will not ask questions. And you can get past that in an in-person environment. Or there are even certain questions people will ask in person, but they won't take the time to sit down and email me about. So I definitely don't regret doing it in person for so long because I think I learned a lot about how to teach the material and what was important and how people...what were the problems that people would encounter and stuff like that. So that was useful. And definitely, the Rails Performance Workshop would not be in the place that it is today if I hadn't done that. STEPH: Yeah, helping people feel comfortable asking questions is incredibly hard and something I've gone so far in the past where I've created an anonymous way for people to submit questions. So during class, even if you didn't want to ask a question in front of everybody, you could submit a question to this forum, and I would get notified. I could bring it up, and we could answer it together. And even taking that strategy, I found that people wouldn't ask questions. And I guess it circles back to that inner critic that we have that's also preventing us from sharing knowledge that we have with the world because we're always judging what we're going to share and what we're going to ask in front of our peers who we respect. So I can certainly relate to being able to look over someone's shoulder and say, “Hey, I think you're stuck. We should talk. Let me walk you through this or help you out.” NATE: There are also weird dynamics around in-person, not necessarily in a small group setting. But I think one thing I really picked up on and learned from RailsConf2021 which was done online, was that in-person question asking requires a certain amount of confidence and bravado that you're not...People are worried about looking stupid, and they won't ask things in a public or semi-public setting that they think might make them look dumb. And so then the people that do end up asking questions are sometimes overconfident. They don't even ask a question. They just want to show off how smart they are about a particular issue. This is more of an issue at conferences. But the quality of questions that I got in the Q&A after RailsConf this year (They did it as Discord chats.) was way better. The quality of questions and discussion after my RailsConf talk was miles better than I've ever had at a conference before. Like, not even close. So I think experimenting with different formats around interaction is really good and interesting. Because it's clear there's no perfect format for everybody and experimenting with these different settings and different methods of delivery has been very useful to me. STEPH: Yeah, that makes a ton of sense. And I'm really glad then for those opportunities where we're discovering that certain forums will help us get more feedback and questions from people because then we can incorporate that and to future conferences where people can speak up and ask questions, and not necessarily be the one that's very confident and enjoys hearing their own voice. For the Rails Performance Workshop, what are some of the general things that you dive into for that workshop? I'm curious, what is it like to attend that workshop? Although I guess one can't attend it anymore. But what is it like to take that workshop? NATE: Well, you still can attend it in some sense because I do corporate bookings for it. So if you want to buy 20 seats, then I can come in and basically do a Q&A every week while everybody takes the workshop. Anyway, I still do that. I have one coming up in July, actually. But my overall approach to performance is to always start with monitoring. So the course starts with goals and monitoring and understanding where you want to go and where you are when it comes to performance. So the first module of the Rails Performance Workshop is actually really a group exercise that's about what are our performance requirements and how can we set those? Both high-level and low-levels. So what is our goal for page load time? How are we going to measure that? How are we going to use that to back into lower-level metrics? What is our goal for back-end response times? What is our goal for JavaScript bundle sizes? That all flows from a higher-level metric of how fast you want the page to load or how fast you want a route to change in a React app or something, and it talks about those goals. And then where should you even start with where those numbers should be? And then how are you going to measure it? What are the browser events that matter here? What tools are available to help you to get that data? Because without measurement, you don't really have a performance practice. You just have people guessing at what stuff is faster and what is not. And I teach performance as a scientific process as science and engineering. And so, in the scientific method, we have hypotheses. We test those hypotheses, and then we learn based on those tests of our hypotheses. So that requires us to A, have a hypothesis, so like, I think that doing X makes this faster. And I talk about how you generate hypotheses using profiling, using tools that will show you where all the time goes when you do this particular operation of your software—and then measuring what happens when you do that? And that's benchmarking. So if you think that getting rid of method X or changing method X will speed up the app, benchmarking tells you did you actually speed it up or not? And there are all sorts of little finer points to making sure that that hypothesis and that experiment is tested in a valid way. I spend a lot of time in the workshop yapping about the differences between development/local environments and production environments and which ones matter. Because what differences matter, it's not often the ones that we think about, but instead it's differences like actually in Rails apps the asset packaging and asset pipeline performs very differently in production than it does in development, works very differently. And it makes it one of the primary reasons development is slower than production, so making sure that we understand how to change those settings to more production-like settings. I talk a lot about data. It's the other primary difference between development and production is production has a million users, and development has 10. So when you call things like User.all, that behavior is very different in production than it is locally. So having decent production-like data is another big one that I like to harp on in the workshops. So it's a process in the workshop of you just go lesson by lesson. And it's a lot of video followed up by hands-on exercises that half of them are pre-baked problems where I'm like, hey, take a look at this Turbolinks app that I've given you and look at it in DevTools. And here's what you should see. And then the other half is like, go work on your application. And here are some pull requests I think you should probably go try on your app. So it's a combination of hands-on and videos of the actual experience going through it. STEPH: I love how you start with a smaller application that everyone can look at and then start to learn how performant is this particular application that I'm looking at? Versus trying to assess, let's say, their own application where there may be a number of other variables that they have to consider. That sounds really nice. You'd mentioned one of the first exercises is talking about setting some of those goals and perhaps some of those benchmarks that you want to meet in terms of how fast should this page load, or how quickly should a response from the API be? Do you have a certain set of numbers for those benchmarks, or is it something that is different for each product? NATE: Well, to some extent, Google has suddenly given us numbers to work with. So as of this month, I think, June 2021, Google has started to use what they're calling Core Web Vitals in their ranking of search results. They've always tried to say it's not a huge ranking factor, et cetera, et cetera, but it does exist. It is being used. And that data is based on Chrome user telemetry. So every time you go to a website in Chrome, it measures three metrics and sends those back to Google. And those three metrics are Largest Contentful Paint (LCP), First Input Delay (FID), and Cumulative Layout Shift (CLS). And First Input Delay and Cumulative Layout shift are more important for your single-page apps kind of stuff. It's hard to screw those up with a Golden Path Rails app that just does Turbolinks or Hotwire or whatever. But Largest Contentful Paint is an easy one to screw up. So Google's line in the sand that they've drawn is 2.5 seconds for Largest Contentful Paint. So that's saying that from clicking on your website in a Google search result, it should take 2.5 seconds for the page to paint the largest component of that new page. That's often an image or a video or a large H1 tag or something like that. And that process then will help you to...to get to 2.5 seconds in Largest Contentful Paint; there are things that have to happen along the way. We have to download and execute all JavaScript. We have to download CSS. We have to send and receive back-end responses. In the case of a simple Hotwire app, it's one back-end response. But in the case of a single-page app, you got to download the document and then maybe download several XHR fetches or whatever. So there's a chain of events that has to happen there. And you have to walk that back now from 2.5 seconds in Largest Contentful Paint. So that's the line that I'm seeing getting drawn in the sand right now with Google's Core Web Vitals. So pretty much any meaningful web application performance metric can be walked back from that. STEPH: Okay. That's super helpful. I wasn't aware of the Core Web Vitals and that particular stat that Google is using to then rank the sites. I was going to ask, this kind of blends in nicely into when do you start caring about performance? So if you have a new application that you are just starting to get to market, based on the fact that Google is going to start ranking you right away, you do have to care some right out of the gate. But I am curious, when do you start caring more about performance, and are there certain tools and benchmarking that you want to have in place from day one versus other things that you'll say, “Well, we can wait until we have X numbers of users or other conditions before we add more profiling?” NATE: I'd say as an approach, I teach people not to have a performance strategy of monitoring. So if your strategy is to have dashboards and look at them regularly, you're going to lose. Eventually, you're not going to look at that dashboard, or more often, you just don't understand what you're looking at. You just install New Relic or Datadog or whatever, and you don't know how to turn a dashboard into actual action. Also, it seems to just wear teams out, and there's no clear mechanism when you just have a dashboard of turning that into oh, well, this has to now be something that somebody on our team has to go work on. Contrast that with bugs, so teams usually have very defined processes around bugs. So usually, what happens is you'll get an Exception Notification through Sentry or Bugsnag or whatever your preferred Exception Notification service is. That gets read by a developer. And then you turn that into a Jira ticket or a Kanban board or whatever. And then that is where work is done and prioritized. Contrast that with performance; there's often no clear mechanism for turning metrics into stuff that people actually work on. So understanding at your organization how that's going to work and setting up a process that automatically will turn performance issues into actual work that people get done is important. The way that I generally teach people to do this is to focus instead of dashboards and monitoring, on alerts, on automated thresholds that get tripped and then sends somebody's an email or put something in the Kanban board or whatever. It just has to be something that automatically gets fired. Different tools have different ways of doing this. Datadog has pretty much built their entire product around monitoring and what they call monitors. That's a perfectly fine way to do it, whatever your chosen performance monitoring tool, which I would say is a required thing. I don't think there's really any good excuse in 2021 for not having a performance monitoring tool. There are a million different ways to slice it. You can do it yourself with OpenTelemetry and then like statsD, I don't know, or pay someone else like everyone else does for Datadog or New Relic or AppSignal or whatever. But you got to have one installed. And then I would say you have to have some sort of automated alerting. Now that alerting means that you've also decided on thresholds. And that's the hard work that doesn't get done when your strategy is just monitoring. So it's very easy to just install a dashboard and say, “Hey, I have this average page time load dashboard. That means I'm paying attention to performance.” But if you don't have a clear answer to what number is good and what number is bad, then that dashboard cannot be turned into real action. So that's why I push monitoring so hard is because it allows people to ignore performance is all that matters, and it forces you to make the decision upfront as to what number matters. So that is what I would say, install some kind of performance monitoring. I don't really care what kind. Nowadays, I also think there's probably no excuse to not have Real User Monitoring. So there's enough GDPR compliance Real User Monitoring now that I think everyone should be using it. So for industry terms, Real User Monitoring is just performance monitoring in the browser. So it's just users' browser APIs and sends those back to you or your third-party provider, so having that so you actually are collecting back-end and front-end performance metrics. And then making decisions around what is bad and what is good. Probably everybody should just start with a page load time monitor, Largest Contentful Paint monitor. And if you've got a single-page app, probably hooking up some stuff around route changes or whatever your app...because you don't actually have page loads on every single time you navigate. You have to instrument whatever those interactions are. So having those up and then just drawing some lines that say, “Hey, we want our React route changes to always be one second or less.” So I will set an alert that if the 95th percentile is one second or more, I'm going to get alerted. There's a lot of different ways to do that, and everybody will have different needs there. But having a handful of automated monitors is probably a place to start. STEPH: I like how you also focus on once you have decided those thresholds and have that monitoring in place, but then how do you make it actionable? Because I have certainly been part of teams where we get those alerts, but we don't necessarily...what you just mentioned, prioritize that work to get done until we have perhaps a user complaint about it. Or we start actually having pages that are timing out and not loading, and then they get bumped up in the priority queue. So I really like that idea that if we agree upon those thresholds and then we get alerted, we treat that alert as if it is a user that is letting us know that a page is too slow and that they are unable to use our application, so then we can prioritize that work. NATE: And it's not all that dissimilar to bugs, really. And I think most teams have processes around correctness issues. And so, all that my strategy is really advocating for is to make performance fail loudly in the same way that most exceptions do. [chuckles] Once you get to that point, I think a lot of teams have processes around prioritization for bugs versus features and all that. And just getting performance into that conversation at least tends to make that solve itself. STEPH: I'm curious, as you're joining teams and helping them with their performance issues, are there particular buckets or categories of performance issues that are the most common in terms of, let's say, 50% of issues are SQL-related N+1 issues? What tends to be the breakdown that you see? NATE: So, when it comes to why something is slow in a Ruby application, I teach a method that I call DRM. And that doesn't have anything to do with actual DRM. It's just memorable because it reminds me of things I don't like. DRM stands for Database Ruby and Memory and in that order. So the most common issue is database, the second most common issue is issues with your Ruby code. The least common issue is memory. Specifically, I'm talking about allocation of objects, creating lots of objects. So probably 80% of your issues are in some way database-related. In Rails, it's 50% of those are probably N+1. And then 30% of database issues are probably what I would call unnecessary SQL. So it's not necessarily N+1, but it's a SQL query for information that you already had, or you could do in a more efficient way. So a common thing for unnecessary SQL would be people will filter an ActiveRecord::Collection like ten different ways when they could have just loaded the whole collection, filtered it with Ruby in the ten different ways afterwards, and that works really well if the collection that you're loading is like 10, 20. Turning that into one database query, plus a bunch of calls to innumerable methods is often way faster than doing that as ten separate database queries. Also, that tends to be a more robust approach. This doesn't happen in most companies, but what could happen is the database is like a shared resource. It's a resource that everybody is affected by. So a performance degradation to the database is the worst possible scenario because everything is affected. But if you screw up what's happening at an individual Rails process, then only that Rails process is affected. The blast radius is tiny. It's just that one request. So doing less stuff in the database while it can actually seem like, oh, that doesn't feel right. I'm supposed to do a lot of stuff in the database. It actually can reduce blast radiuses of performance issues because you're not doing it on this database that everyone has to have access to. There are a lot of areas of gray here. And I talk a lot in all my other material like why -- There's a lot of nuance here. So database is the main stuff. Issues in how you write your Ruby code is probably the other one. Usually, that's just what I would call code that goes bump in the night. It's code that you don't know is running but actually is. Profilers are what help us figure that out. So oftentimes, I'll have someone open up a profiler on their controller action for the first time. And they're like, wait a minute, I had no idea that such and such was running during this controller action, and actually, we don't need to do that at all. So why is it here? So that's the second most common issue. And then the third issue that really doesn't actually come up all that often is object allocation, numbers of objects that get created. So primarily, this is a problem in index actions or actions transactions that deal with big collections. So in Ruby, we often get overly focused on garbage collection, but garbage collection doesn't take any time if you just don't create objects. And object creation itself takes time. So looking at code through the lens of what object does this code create? And trying to get rid of those object allocations can often be a pretty productive way to make stuff faster. STEPH: You said a lot of amazing things there. So I'm debating on which one to follow up on. I think the one that stuck out to me the most where I have felt pain around this is you mentioned identifying code that goes bump in the night or code that is running, but it doesn't need to be run. And that is something that I've run into with applications where we have a code path that seems important, but yet I can't prove that it's being executed and exactly why it's there and what flow it's supporting. And I'm curious, do you have any tips or tricks in how you've helped teams identify that this code path isn't used and it's something that we can remove and then that itself will help speed up the performance of that particular endpoint? NATE: Like, there's no performance cost to like 100 models in an application that never actually get used. There's really no performance downside to code in an app that doesn't actually ever get run. But instead, what happens is code gets added into callbacks that usually is probably the biggest offender that's like, always do this thing after you do X. But then, two years later, you don't always need to do that thing after you do X. So the callbacks always run, but sometimes requirements change, and they don't always need to be run. So usually, it's enough to just pop the profiler now on something. And I have people look at it, and they're like, “I don't know why any of this is happening.” Like, it's usually a pretty big Eureka moment once we look at a flame graph for the first time and people understand how to read those, and they understand what they're looking at. But sometimes there's a bit of a process where especially in a bigger app where it's like, “Such and such is running, and this was an entire other team that's working on this. I have no idea what this even does.” So on bigger apps, there's going to be more learning that has to get done there. You have to learn about other parts of the application that maybe you've never learned about before. But profiling helps us to not only see what code is running but also what that relative importance is. Like, okay, maybe this one callback runs, and you don't know what it does, and it's probably unnecessary. But if it only takes 1% of the total time to run this action, that's probably less important than something that takes 20% of total time. And so profilers help us to not only just see all the code that's being run but also to know where that time goes and what time corresponds to what parts of the code. STEPH: Yeah, that's often the code that makes me the most nervous is where it's code that I suspect is being run or maybe being run, but I don't understand why it's there and then figuring out if it can be removed and then figuring out ways to perhaps even log when a call is being made to that code to determine if it's truly in use or not or at least supported by a code path that a user is hitting. You have a blog post that I read recently that I really appreciated that talks about essentially gaming benchmarking where you talk about the importance of having context around benchmarks. So if someone says, “I've improved something where it is now 10% faster.” It's like, well, what is that 10% relative to? And if it's a tool that other people are using, what does that mean for them? Or did you improve something that was already very fast, and you made it 10% faster? Was that a really valuable use of your time? NATE: Yeah. You know, something that I read recently that made me think of that again was this Hacker News post that went viral. That was like, how I optimize an AWS EC2 instance to take 1.5 million requests per second on my JSON API. And out of the box, it was like 500 requests per second, and then he got it to 1.5 million. And the whole article was presented with relative numbers. So it was like, “I made this change, and things got 33% faster. And if you do the whole thing right, 500 to 1.5 million requests per second, it's like my app is three times faster now,” or whatever. And that's true, but it would probably be more accurate to say, “I've taken three-millionth of a second out of every request in my app.” That's two ways of saying the same thing because latency and throughput are just related that way. But it's probably more accurate and more useful to say the absolute number, but it doesn't make for great blog posts, so that doesn't tend to get said. The kinds of improvements that were discussed in this article were really, really low-level stuff. That was like if you turn off...I think it was like turn off iptables or something like that. And it's like, that shaves a microsecond off of every time we make a syscall or something. And that is useful if your performance goal is to serve 1.5 million requests per second Hello World responses off of my EC2 instance, which is what this person admittedly was doing. But there's a tendency to walk that back to if I do all things in this article, my application will be three times faster. And that's just not what the evidence says. It's not what you were told. So there's just a tendency to use relative numbers when absolute numbers would be more useful to giving you the context of like, oh, well, this will improve my app or it won't. We get this a lot in Puma. We get benchmarks that are like, hey, this thing is going to help us to do 50,000 requests per second in Puma instead of 10,000. And another way of saying that is you took a couple of nanoseconds off of the overhead of every single request to Puma. And most Puma applications have a hundred millisecond response time. So it's like, yeah, I guess it's cool that you took a nanosecond off, and I'm sure it's going to help us have cool benchmarks, but none of our users are going to care. No one that's used Puma is going to care that their requests are one nanosecond faster now. So what did we really gain here? STEPH: Yeah, it makes sense that people would want to share those more...I want to call them sparkly stats and something that catches your attention, but they're not necessarily something that's going to translate to us in the way that we hoped that they will in terms of it's not going to speed up our app 30% or have those same rewards or benefits. Speaking of Puma, how is it being a co-maintainer of Puma? And how do you balance that role with all of your other work? NATE: Actually, it doesn't take all that much of my time. I try to spend about 15 minutes a day on it. And that's really possible because of the philosophy I have around open-source maintenance. I think that open source projects are fundamentally about collaboration and about sharing our hard-fought extractions and fixes and knowledge together. And it's not about a single super contributor or super maintainer who is just out of the goodness of their heart releasing all of their incredible work and time into the public domain or into a free software license. Puma is a pretty popular piece of Ruby software, so a lot of people use it. And I have things on my back burner of if I ever got 20 hours to work on Puma, here's stuff I would do. But there are a lot of other people that have more time than me to work on Puma. And they're just as smart, and they have other tools they've got in their locker that I don't have. And I realized that it was more important that I actually find ways to recruit and then unblock those people than it was for me to devote as much time as I could to Puma. And so my work on Puma now is really just more like management than anything else. It's more trying to recruit new contributors and trying to give them what they need to help Puma. And contributing to open source is a really fraught experience for a lot of people, especially their first time. And I think we should also be really conscious of that. Like, 95% of software developers have really never contributed to open source in a meaningful way. And that's a huge talent pool of people that could be helping us that aren't. So I'm less concerned about the problems of the 5% that are currently contributing than I am about why there are 95% of us that don't do anything. So that's what gets me excited to work on Puma now, is trying to change that ratio. STEPH: I really like that mindset of where you are there to provide guidance but then essentially help unblock others as they're making contributions to the project but then still be there to have the history and full context and also provide a path forward of a good direction for Puma to head. In regards to encouraging more people to contribute to open source projects, I've often heard people say how challenging that is, where they have an open-source project that they would really love people to contribute to but finding people is really hard or just letting people know that they're interested in contributions. Have you had any strategies that have been successful for you in encouraging people to contribute? NATE: Yeah. So first thing, the easiest thing is we have a contributing.md file. So that's something I think more projects should adopt is have an actual file in your project that says everything about how to contribute. Like, what kinds of contributions do you want? Different projects have different things that they want. Like, Rails doesn't want to refactor PRs. Don't send a refactor PR to Rails because they'll reject it. Puma, I'm happy to accept those. So letting people know like, “Hey, here's how we work here. Here's the community we're creating, and here's how it works. Here's how to get involved.” And I think of it as hanging out the shingle and saying, “Yes, I want your contributions. Here's how to do it.” That alone puts you a step above other projects. The second thing I would say is you need to have contributor-only communication channels. So we have Matrix chat. So Matrix is like this successor to IRC. So we have a chat channel basically, but it's like contributors only. I don't enforce that, but I just don't want support requests in there. I don't want people coming in there and being like, “My Puma config doesn't work.” And instead, it's just for people that want to contribute to Puma, and that want to help out. If you have a question and come in there, anyone can answer it. And then finally, another thing that I've had success with is doing one-on-one stuff. So I will actually...I have a Calendly invite that I think is in contributing.md now that you can just book 30 minutes with me anytime about contributing to Puma. And I will get on a Zoom call with you and talk to you about what are your concerns? Where do I think you can help? And I give my time away that way. The way I see it is like if I do that 20 times and I create one super contributor to Puma, that is worth more than me spending 10 hours on Puma because that person can contribute 100, 200, 1,000 hours over their lifetime of contributing to Puma. So that's actually a much more higher leverage contribution, really from my perspective. It's actually helping other people contribute more. STEPH: Yeah, that's huge to offer people to say, “Hey, you can book time with me, and I will walk you through and let you know where you can start making an impactful contribution right away,” or “Here are some areas that I think you'd be interested, to begin with.” That seems like such a nice onboarding for someone who says, “I'm interested, but I'm nervous,” or “I'm just not sure about where to get started.” Also, I love your complaint department voice for the person who their Puma config doesn't work. That was delightful. [chuckles] NATE: I think it's a little bit part of my open-source philosophy that, especially at a certain scale like Puma is at that we really kind of over-prioritize users. And I'm not really here to do support; I'm here to make the project better. And users don't actually contribute to open source projects. Users use the thing, and that's great. That's the whole reason we're open-sourcing is so more people use it. But it's important not to prioritize that over people who want to make the project better. And I think a lot of times; people get caught up in this almost clout chasing of getting the most GitHub stars that they think they need and users they think they need. And you don't get paid for having users, and the product doesn't get any better either. So I don't prioritize users. I prioritize the quality of the project and getting contributors. And that will create a better project, which will then create more users. So I think it's easy to get sidetracked by people that ask for your time when they're not giving anything back to the project in return. And especially at Puma's scale, we have enough people that want my time or the time of other maintainers at Puma so that they can contribute to the project. And putting user support requests ahead of that is not good for the project. It's not the biggest, long-term value increase we could be making, so I don't prioritize them anymore. STEPH: Yep. That sounds like more the pursuit of sparkly stats and looking for all those GitHub stars or all of those likes. Well, Nate, if you're game, I have two listener questions that I'd like to run by you because I shared with some folks that you are going to be on The Bike Shed today. And they're very excited and have two questions that they'd like me to run by you. How does that sound? NATE: Yeah, all right. STEPH: So the first question is, are there any paradigms or trends in Rails that inherently hurt performance? NATE: Yeah. I get this question a lot, and I will preface it with saying that I'm the performance guy, and I'm not the software design guy. And I get a lot of questions about does such and such software design...how does that impact performance? And usually, there's like a way to do anything in a performant way. And I'm just here to help people to find the performant way and not to prescribe “You must always do X, Y, or Z,” or “ActiveRecord is bad. Never use it.” That's not my job here. And in my experience, there's a fast way to do almost anything. Now, one thing that I think is dying, I guess, or one approach or one common...I don't know what to call it. One common mistake that is clearly wrong is to not do any form of server-side rendering in a web application. So I am anti-client-side app. But there are ways to do that and to do it quickly. But rendering a basically blank document, which is what most of these applications will do when they're using Rails as a back-end…you'll serve this basically blank document or a document with maybe some Chrome in it. And then, the client-side app has to execute, compile JavaScript, make XHR requests, and then render the page. That is just by definition slower than serving somebody a server-side rendered page. Now, I am 100% agnostic on how you want to generate that server-side rendering. There are some people that are working on better ways to do that with Rails and client-side apps. Or you could just go the Hotwire Turbolinks way. And it's more progressive enhancement where the back-end is always just serving the server-side rendered response. And then you do some JavaScript on top of that. So I think five years from now, nobody will be doing this approach of serving blank documents and then booting client-side apps into that. Or at least it will be seen as outdated enough that you should never design a project that way anymore. It's one of those few things where it's like, yeah, just by definition, you're adding more steps into a rendering flow. That means, by necessity, it has to be slower. So I think everybody should be thinking about server-side rendering in their project. Again, I'm totally agnostic on how you want to implement that. With React, whatever front-end flavor of the month you want to go with, there's plenty of ways to do that, but I just think you have to be prioritizing that now. STEPH: All right. Well, I like that five-year projection of where we're headed. I have found that it's often the admin-side where people will still bring in a lot of JavaScript rendering, just to touch on a bit of what you're saying, in terms of let's favor the server-rendered HTML versus over-optimizing a space that one, probably isn't a profitable space in terms that we do want our admins to have a great experience for our product. But if they are not necessarily our users, then it also doesn't need to be anything that is over the top or fancy or probably uses a lot of JavaScript. And instead, we can start simple. And there's a number of times that I've been on projects where we have often walked the admin back to be more server-rendered because we got to a point where someone was very excited to make the admin very splashy and quick but then couldn't keep up with the requests because then they were having to prioritize the user experience first. So it was almost like optimizing the admin, but then it got left out in the cold. So then it's just sort of this poor experience. NATE: Yes. Shopify famously walked back their admin from I think it was Backbone to Turbolinks. And I think that that has now moved back to React is my understanding. But Shopify is a huge company, so they have plenty of time and resources to be able to do that. But I just remember that happening at the time where I was like, oh wow, they just rolled the whole thing back to Turbolinks again. And now, with the consolidation that's gone on in the React world, it's a little bit easier to pipe a server-side rendering into a React app. Whereas with Backbone, it was like no one knew what you were doing. So there was less knowledge about how to server-side render this stuff. Now it doesn't seem to be so much of a problem. But yeah, I mean, Rails is really good at CRUD apps, and admin is like 99% CRUD. And adhering as closely as possible to the Rails Golden Path there in an admin seems to be the most productive way to work on that kind of feature. STEPH: All right. Ready for your second question? NATE: Yes. STEPH: Okay. This one's a bit more in-depth. They also mentioned a particular project name. So I am going to swap it out with a different name. So on project cinnamon roll, we found a really gnarly time-consuming API endpoint that's getting hammered. And on a first pass, we addressed a couple of N+1 issues and tuned the performance, and felt pretty confident that they had addressed the issue. But it was still fairly slow. So then they took some additional incremental steps. So they swapped out to use OJ for serialization that shaved off an additional 10% but was still slow. They also went the route of going straight to Rails cache with a one-minute expiration. So that way, they could avoid mucking with cache busting because they confirmed with the client that data could be slightly stale. And this was great. It worked out well. So it dropped their average response time down to less than 70 milliseconds. With all that said, that journey took a few hours over a few days, and multiple production deploys. And had they gone straight to the cache, then they would have had a 15-minute fix with a single deploy. So this person's wondering, are there any other examples like that where, rather than taking these incremental seemingly obvious performance whims, there are situations where you want to be much more direct with your path? NATE: I guess I'd say that profiling can help you to understand and form better hypotheses about what will make things faster and what won't. Because a profiler can't really lie to you about where time goes, either you spent 20% of your time in this method, or you didn't. So I don't spend any time in any of my material talking about what JSON serializer you use. Because really that's actually never...that's really never anybody's bottleneck. It's never a huge proportion of people's total percentage of time. And I know that because I've looked at enough profiles that the issues are usually in other places. So I would say that if your hypotheses that you're generating are not working, it's because you're not generating good enough hypotheses. And profiling is the place to do that. So having profilers running in production is probably the biggest level-upscale-like that most teams could take. So having profilers that you can access as on production servers as a user is probably the biggest level up that you could make to generating the hypotheses because that'll have real production data, real production servers, real production environment. And it's pretty common now that pretty much every team that I work with either has that already, or we work on implementing it. It's something that I've seen in production at GitHub and Shopify. You can do it yourself with rack-mini-profiler. It's all about setting up the authorization, just making sure that only authorized users get to see every single SQL query generated in the flame graph and all that. But other than that, there's no reason you shouldn't do it. So I would say that if you're not generating the right hypotheses or you don't...if the last hypothesis out of 10 is the one that works, you need better hypotheses, and the best way to do that is better profiling. STEPH: Okay, better profiling. And yeah, it sounds like there's also a bit of experience in there in terms of things that you're used to seeing, that you've noticed that could be outliers in terms of that they're not necessarily the thing that you want to improve. Like you mentioned spending time on how you're serializing your JSON is not somewhere that you would look. But then there are other areas that you've gained experience that you know would be likely more beneficial to then focus on to form that hypothesis. NATE: Yeah, that's a long way of saying experience pays off. I've had six years of doing this every single day. So I'm going to be pretty good at...that's what I get paid for. [laughs] So if I wasn't very good at that, I probably wouldn't be making any money at it. STEPH: [laughs] All right. Well, thanks, Nate, so much for coming on the show today and talking so much about performance. On that note, I think it's a good place for us to wrap up. If people are interested in following along with what you're working on and they want to keep up with your latest and greatest workshops that are coming out, where can they find you on the internet? NATE: speedshop.co is my site. @nateberkopec on Twitter. And speedshop.co has a link to my newsletter, which is where I'm actively thinking every week and publishing stuff too. So if you want to get the drip of news and thoughts, that's probably the best place to go. STEPH: Perfect. All right. Well, thank you so much. NATE: No problem. STEPH: The show notes for this episode can be found at bikeshed.fm. CHRIS: This show is produced and edited by Mandy Moore. STEPH: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or a review in iTunes as it helps other people find the show. CHRIS: If you have any feedback for this or any of our other episodes, you can reach us @bikeshed on Twitter. And I'm @christoomey. STEPH: And I'm @SViccari. CHRIS: Or you can email us at hosts@bikeshed.fm. STEPH: Thanks so much for listening to The Bike Shed, and we'll see you next week. Together: Bye. Announcer: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

Screaming in the Cloud
Deserted Island DevOps with Austin Parker

Screaming in the Cloud

Play Episode Listen Later Jun 8, 2021 37:53


About AustinAustin makes problems with computers, and sometimes solves them. He's an open source maintainer, observability nerd, devops junkie, and poster. You can find him ignoring HN threads and making dumb jokes on Twitter. He wrote a book about distributed tracing, taught some college courses, streams on Twitch, and also ran a DevOps conference in Animal Crossing.Links: Lightstep: https://lightstep.com/ Lightstep Sandbox: https://lightstep.com/sandbox Desert Island DevOps: https://desertedislanddevops.com lastweekinAWS.com Resources: https://lastweekinAWS.com/resources Distributed Tracing in Practice: https://www.amazon.com/Distributed-Tracing-Practice-Instrumenting-Microservices/dp/1492056634 Twitter: https://twitter.com/austinlparker Personal Blog: https://aparker.io TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored by ExtraHop. ExtraHop provides threat detection and response for the Enterprise (not the starship). On-prem security doesn't translate well to cloud or multi-cloud environments, and that's not even counting IoT. ExtraHop automatically discovers everything inside the perimeter, including your cloud workloads and IoT devices, detects these threats up to 35 percent faster, and helps you act immediately. Ask for a free trial of detection and response for AWS today at extrahop.com/trial.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Austin Parker, who's a principal developer advocate at Lightstep. Austin, welcome to the show.Austin: Hey, it's great to be here.Corey: It really is. I love coming here. It's one of my favorite places to go. So, let's get the obvious stuff out of the way. You're a principal developer advocate at Lightstep. I know this because I said it a whole sentence ago, which is about the limit of my attention span. What is Lightstep? And what does your job mean?Austin: So, Lightstep is an observability platform. We take traces, and metrics, and logs, and all that good stuff, throw them together in a big old swamp of data, and then, kind of, give you some really cool workflows to help you make sense of it, figure out, hey, where is the slow SQL query? Where is the performance bad?Corey: The way to figure out, in most of my environments, where's the performance bad is git blame, figure out what part I wrote.Austin: But imagine there were, like, 1000, or 100,000 of you all working on this massive distributed system, and you didn't know half—Corey: It would snark itself to death before it ever got off the ground.Austin: Yeah. I mean, I think that's actually most large companies, right? We deliver shippable software only through inertia.Corey: Yeah. Just because at some point, it bounces off all the walls, there's nowhere else for it to go but to production.Austin: Yep. But yeah, you have thousands of people, hundreds of people, however many people, right? I think the whole distributed workforce thing that most people are dealing with now has really made observability rise to the top of your concern list because you don't have the luxury of just going and poking your head around the corner and saying, “Hey, Joanne. What the heck? Why did things break?” You can't just poke someone anymore. Or you can, but you never know what you're going to have to deal with.Corey: It feels weird to call them at home or bug their family members to poke them or whatnot. It just seems weird.Austin: It does. And until Amazon comes out with a minder drone that just, kind of like, hovers over your shoulder at all times, and pokes you, when someone is like, “Hey, you broke the build.” Then I think we're going to need observability so that people can sort of self-serve, figure out what's going on with their systems.Corey: Cool. One of the things I'm going to point out is that I've had a bunch of people attempt to explain what distributed tracing is and how observability works, and it never really stuck. And one of the things that I found that did help explain it—and we didn't even talk about this in the pre-show, while we figure out how to pronounce each other's names—but one of the things that has always stuck with me is the interactive sandbox on Lightstep, which used to be prominently featured on your page; now it's buried in the menu somewhere. But it's an interactive sandbox that sets up a scenario, problem you're trying to solve, gives you data—so it gets away from the problem of, “Step one, have a distributed application where it's all instrumented and reporting things in.” Because in a lot of shops, that's not exactly a small lift that you can do in an afternoon to start testing things like this out. It's genius. It shows what the product does, how it works, mapped to the type of problems people will generally encounter. And after I played with this, “Oh, my stars, I get it.”Austin: We actually just recently updated that to add some new stuff to it because we shipped a feature called ‘Change Intelligence' where you can take actual time-series metrics, and then overlay those on traces and say, “Hey, I saw a weird spike,” and highlight that, and then we go through, look at all the traces for that service and its related services during that time, and tell you, “Hey, we think it might be this. Here's things that are highly correlated in those time windows.” So, if you haven't checked it out recently, go back and check it out. It's—yeah, a little more hidden than it used to be, but I believe you can find it at lightstep.com/sandbox.Corey: Yeah. And there's no sign up to do this. It's free access. It asked for an email address, but that's okay, I just use yours. No, I'm not kidding. I actually did. And, yeah, it works; it shows exactly what it is. It even has, instead of ‘start' it says ‘play' because that's fundamentally what it is. If you're trying to wrap your head around distributed tracing, take a look at this.Austin: Yes, definitely. I have a long-standing Jira ticket to add achievements to that.Corey: Oh, that could be fun. You could bury some, too, like misusing services as databases—Austin: Ooh.Corey: —or most expensive query to get the right answer.Austin: Yeah. And then maybe, like, there's just one span, kind of, hidden there where it's ‘using Route 53 as a database.'Corey: I keep seeing that cropping up more and more places. That's something I get to own and that's an awful lot of fun. Speaking of gamification and playing in strange ways, one of the things you did last year that I wasn't paying attention to—because, you know, there was a pandemic on—was you were one of the organizers behind Desert Island DevOps which is a strange thing that I've only recently delved into—delven into—gone spelunking inside of. There we go.It wasn't instrumented for observability—buh-dum-tss. But it's fundamentally a DevOpsDays that takes place inside the animated world of Animal Crossing's New Horizon, which is apparently a Nintendo game, which is apparently a game company.Austin: Yeah.Corey: It is not really my space. I don't want to misspeak.Austin: No, you hit it. ‘Deserted.' Deserted Island [crosstalk 00:05:43].Corey: Oh, ‘Deserted.' Ah, got it. And don't spell it as ‘dessert' either, as in this would be a delicious game to play.Austin: I mean, it is a delicious and comforting sort of experience. If you aren't familiar with Animal Crossing, the short 30-second explanation is it is a life simulator, building game where, you as your character, you are on an island, and there are relatively adorable animal NPCs that are your villagers, and you can talk to them, and they will say funny things to you. You can go around and do chores like picking up fruit or fishing. And the purpose is, kind of, do these chores, get some in-game currency, and then go spend that in-game currency on furniture so that you can make a pretty house, or buy pretty clothing. And it came out at a perfect time last year because everyone was about to bundle inside for the—well, we're still inside—but everyone had to go inside. And suddenly, here's this like, “Oh, it's just this cute, sort of like, putz around and do whatever.”Corey: It was community-oriented. It was more of a building-oriented game than a destruction game.Austin: Yeah.Corey: It's the sort of thing that is a great way of taking your mind off your troubles. It is accessible to a bunch of people that aren't generally perceived as gamers when you think of that subculture. It really is an encompassing, warm, wonderful thing—by all accounts—and you looked at it and figured, “All right, how can we ruin something?” And the correct answer you got to is, “Let's pour DevOps on it.”Austin: Yeah. Let's use this as an event platform, and let's really just tech-bro this shit up.Corey: And it seems to work super well. At the time of this recording, I have submitted a talk that I live-streamed my submission around, and I have not heard in either direction. To be perfectly frank, I forget what I wound up submitting, which is always a bit of a challenge, just because I make so many throwaway random jokes that, cool. Well, we'll see how it plays out. I think you were even in the audience for that on the Twitch stream.Austin: Yeah. You found some bugs on the CFP form [laugh] that I had to fix.Corey: To be clear, the reason I do those things is not because it's a look how clever I am, but rather to instead talk about how it's not scary to submit a talk proposal. Everyone has a story that they can tell. And you don't need a big platform or decades of experience in this space to tell a story. And that was my goal, and I think I succeeded. You would have the numbers more than I do; I hope people wound up submitting based upon seeing that. I want to hear voices that, frankly, aren't ours all the time.Austin: I think in, like, a week, we basically got more submissions than we did for the entire CFP last year. One thing that I kind of think is interesting to bring up because you bring up, oh, we don't hear a variety of voices, right? One thing I tell people, and I know that it's not universally applicable advice, but I got into DevRel as a—not quite luck, but, like, everything in my life is luck, on some level. It always plays some level of importance. But I didn't go to school to get into DevRel, I didn't do a lot of things.I have actually been in tech, maybe—depending on how you want to count it—in terms of actually being in a software development job or primarily software development job, maybe, like, five or six years, give or take. And before that, I did a lot of stuff. I was a short-order cook; I worked at gas stations; I did tech support for Blackberry, and I did a lot of community organization. I was a union organizer for a little while. I like DevRel because it's like, oh, this kind of integrates a lot of things I'm interested in, right?I enjoy teaching, helping people, and helping people learn, but I also like talking; I like to go and be a public figure, and I like to build a platform and use that to get a message out. And I think what I did with Deserted Island, or what the impetus there was, we suddenly were in a situation where it's like, “Hey, there's a bunch of people that normally get together and they fly around the globe in decent airplane seats, and people come and see us talk.” Because why? Because they think we know what we're talking about, or because we have something that shows we know what we're talking about, or however you want to say it. But in a lot of cases, I think people are coming for that sort of community, they're coming because, “Hey, I can go to a room and I can sit in some weird little hotel, or conference center, or whatnot, and everyone I look at, everyone I see is someone that is doing what I'm doing, on some level. These are all people that are working in technology, they're building things, they're solving problems.”And that goes away really quickly when you get into this remote-first world, and when we can't travel, and we don't have that visual aspect. So, what I wanted to do with Deserted Island, what I thought what was important about it is, I was already sick of Zoom by the time, everyone went to Zoom; I was already sick of the idea of, oh my god, a year or two years of these sort of events and these community things just being, like, everyone's staring at a bunch of slides and a talking head. Didn't sound very appealing, so what if we try something different? What if we do something where it's like, look, we're going to take people out of their day; we're going to put them in somewhere else. And maybe that's somewhere else is just, hey, you're watching people run around on an Animal Crossing Island on a Twitch stream.But that sort of moment of just, like, this isn't what you would normally be doing, I think takes people's heads out of their normal routine and puts them in a place where they can learn, and they can feel community, and they can feel, like, a kinship. I also think it's really important because it's that whole stupid New Yorker joke of, “On the internet, nobody knows you're a dog.” We have this really cool opportunity to craft who we are as people, and how we present that to the world. And for a lot of people, you're stuck inside; you don't get that self-expression, so here's a way to be expressive, right? Here's a way to communicate who you are on a level that isn't just a profile picture or something, or things that don't work as well over Zoom.It's a way to help project your identity. And that, I think, gives more weight to what you're saying because when you feel like, “Hey, this is more of who I am,” or, “This is a representation of me. I can show something about who I am.” And that helps you speak. And that helps you deliver, I think, an effective talk. And that, again, builds community and builds these bonds.Corey: I want to talk to you about that, specifically because you are one of those people that aligns very much with my view of the world on developer marketing. But I don't want to lead you too much on this, so why don't you start? Take it away. Where do you stand on developer marketing? And what do people get wrong?Austin: I think the thing that a lot of people get wrong is that they try to monetize the idea of community. If you go and you search, insert major company name here; you search “Amazon community,” or you search “Microsoft community,” or you search “Google community,”—well, if you do that, you'll get no results, but whatever, right? You get the picture that marketers in a way have turned the idea of developer community into something that you can just throw a KPI or throw an OKR on and squeeze it for money. And I don't like that. I'm not very comfortable with that idea of community—because I think community in a lot of ways, it's like family. And the families that you like the best are the ones you choose. I think this is—Corey: The family you choose is an important concept.Austin: Right. And for the most part… so much of human experience activity is built around finding those people you choose, and those communities develop out of that. I use AWS sometimes, I don't necessarily know if I would put myself in a community with every other AWS user. I—Corey: Oh, I certainly wouldn't. This is the problem. Everyone thinks when you talk about community or a group of people doing something, they're ‘other people' that are in some level of otherness. And that's—like there are entire communities around AWS that I do not talk to, I do not see, I do not pretend to understand.Austin: Yeah, even at Lightstep. We're not a massive, massive company by any means, but we have a bunch of different users that are using our tool in different ways. And they all have different needs, and they all have different wants. So, I could say, “Oh, here's the Lightstep community.” But it's not a useful abstraction.It's not a useful way to abstract all of our users because any tool that's worth using is going to be this collection of other abstractions and building blocks. Like, you… I don't know, look at something like Notion, or look at something like Airtable, or the popularity of low or no-code stuff, where someone built a platform and then other people are building stuff on top of that platform, if you go to those user groups or you go to those forums, and it's just like, there's a million, million different varied use cases, and people are doing it in different ways, and some people are building this kind of application, or that kind of application, or whatever. So, the idea of, oh, there's a community and we can monetize that community somehow, I'm uncomfortable with that from, sort of, a base level. And I'm uncomfortable with the idea of the DevRel industry—or the developer marketing industry—kind of moving towards this idea of, like, we're going to become community marketers or whatever. I think you have to approach people as individuals.And individuals are motivated by a lot of things. They're motivated by, can you solve this problem? Do I like you? Are you funny? Whatever. And I believe that if you're a developer tool, and you are trying to attract developers, then [sigh] it works a lot better, I think, to have just individuals, to have people that can help influence the much broader—the superset of all developers that might have an interest in what you're doing by being different, I guess.Being something that's like, hey, this is entertaining, or this is informative, or this is interesting. The world is not a meritocracy. The world is governed by many, many different things. You're not going to win over the developer industry simply by going out and having the best white papers, or having one more ad read than your competitor. You need to do something to get people interested and excited in [sigh] a way that they can see themselves using it.It's like, why did Apple go and do ‘Think Different' ads? Because it's like, you using a Mac, that's kind of like being Einstein, or that's kind of like being Picasso. This is basic marketing stuff that I feel like a lot of technical marketers or developer marketers sort of leave at the door because they think the audience is too sophisticated for it, or their—Corey: I'll even soft-launch it here because I haven't at this point in time, talked about it in public, but if you go to lastweekinAWS.com/resources we wrote our own developer marketing guide because I got tired of explaining the same type of thing again, and again, and again. It asks for an email address and it sends it to you—I know, I'm as guilty as any. And I, of course, called it ‘Devreloper,' which is absolutely a problem with me and I talk about things. But I'm right.And it goes to an awful lot of what you're saying. An example that you just talked about of giving people something rather than trying to treat them as metrics, one of the best marketing things I've seen you do, for example, is you wrote O'Reilly's Distributed Tracing in Practice which means if someone has a question about distributed tracing and how it's supposed to work, well, that's not a half-bad resource. And okay, I've read it and I have some further questions. Let me track down the author and ask them. Oh, you work at a company that is in this space? Huh. Maybe I'll look into this. And it's a very long-tail story. And how do you attribute that as far as, did this lead come from someone who read your book or not will drive marketers crazy.Austin: Oh, it's super hard. And it does drive them crazy. [laugh].Corey: Yeah, my answer is, I don't know and I don't care. One of the early sponsors of this podcast sponsored for a month and then didn't continue because they saw no value. A month goes by, they bought out everything that held still long enough, and, “Thank you for your business.” “Can you explain to me what changed?” “Oh, we talked to some of our big customers and it turned out the two of them had heard about us for the first time on your show.”And that inspired them to start digging into it and reaching things out, but big companies, corporate games of telephone, there was no way to attribute that. My firm belief is, on some level, that if you get in front of an audience with a message that resonates and—and this is the part some people miss—is something that solves an actual problem that they have. It works. It's not necessarily predictable and it's hard to say that this thing is going to go big and this thing isn't. So, the solution, on some level is just keep publishing things that speak to your audience. But it works, long term. I'm living proof of this.Austin: Yeah. I think that it makes a lot more sense to… rather than to do, sort of, I don't want to say vanity metrics, but kind of vanity metrics around, like, oh, this many stars, or this many forks, or whatever. There's a lot of people, especially in this OSS proximate world. Where you have a lot of businesses that are implicitly or explicitly built on top of an open-source project, not everyone that is using your open-source project is going to, one, be capable of converting into a paid user, or two, be super interested in it. And I would rather spend time thinking about, well, what is the value someone gets out of this product?And even if that only thing is, is that, hey, we know what we're talking about because we've got a bunch of really smart people that are building this product that would solve their problem. If you want to go out and build your own internal observability solution using completely open-source tools Grafanas and Prometheuses of the world, great. Go for it. I'm not going to hold you back. And for a lot of people, if they come to me and say, “Well, this is what we got, and this what we're thinking about.”I'll say, “Yeah. Go for it. You don't need what we're offering.” But I can guarantee you that as it scales and as it grows, then you're going to have a moment where you have to ask yourself the question of, “Do I want to keep spending a bunch of time stitching together all these different data sources, and care and feeding of these databases, and this long term storage, and dealing with requests from end-users, or I just want to pay someone else to solve that problem for me? And if I'm going to pay someone else, shouldn't I pay the people who literally spend all day every day thinking about these problems and have had decades of experience solving these problems at really big companies that have a lot of time and effort to invest in this?”Corey: This episode is sponsored in part by our friends at Lumigo. If you've built anything from serverless, you know that if there's one thing that can be said universally about these applications, it's that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications. It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You've created more problems for yourself; make one of them go away. To learn more, visit lumigo.io.Corey: Oh, yeah. We're doing some new content experiments on our site, and what we're doing is we're having some folks write content for us. Now, when people hear that, what a lot of marketers will immediately do is dive down the path of, “Ah. I'm going to go ahead and hire some content farm.” Well, that doesn't work, I found that we wound up working with individual people that work super well.And these are people who are able to talk about these things because their day job is managing a team of 30 SREs or something like that, where they are very clearly experts in the space. And I want to be very clear, I'm not claiming credit for our content writers; they get their own bylines on these things.Austin: Yeah.Corey: And it turns out that that, over time, leads to good outcomes because it helps people what they need. There's the mystical SEO Juju that I don't pretend to understand, but okay, I'm told it's important, so fine, whatever. And it makes for an easier onboarding story, where there are now resources that I can trust and edit if I need to, as things change, that I can point people to, that isn't a rotating selection of sketchy sites.Austin: Mm-hm. I think that's one thing that I would love to see more of, just not in any one particular part of the tech industry, but overall, the one thing I've noticed, at least in the pandemic, during this whole work-from-home, whatever, whatever, we don't talk enough. And it sounds maybe weird, but I think this actually goes back to what you're saying earlier, about everyone having a story to tell. People don't feel comfortable, I think, putting their opinion out there or saying, “Hey, this is what worked. This is what didn't work.”And so if you want to go find that out—like, if I wanted to go write something about, hey, these are the five things you should do to ensure you have great observability, then that's going to involve a lot of me going around and sort of Sherlocking my way through StackOverflow posts, and forums, and reaching out to people individually for stories and comments and whatever. And I would love to see us get to a point where we're just like, “Actually, no. This isn't—we should just be sharing this. Let's write blogs about it.” If you're sitting there thinking no one's going to find this useful, right—like, you solve a problem, or you see something that could have worked better, and you're like, “Eh, no one else is going to find that valuable.”I can almost guarantee you that someone is going to find that valuable. Maybe not today, maybe not tomorrow, but go ahead and write about your experiences, write about the problems you've solved, write about the things that have vexed you, and put that on the internet because it's really easy to publish stuff on the internet.Corey: Yes. Which is a blessing and a curse. That is very much a double-edged sword.Austin: That very much is a double-edged sword. But I think that by biasing towards being more open, by biasing towards transparency and sharing what works, what doesn't work, and having that just kind of be the default state, I'm a big proponent of things like radical transparency in terms of incident reports, or outages, or hiring, or anything. The more information that you can put in the world is going to—it might not make it better, but it at least helps change the conversation, gives more data points. There was a whole blow-up on Twitter this week, where someone posted like, “Hey, this is a salary I'm looking for.” I think you—Corey: Oh, yeah. She's great.Austin: Yeah, she's worth it, right? And the thing that got everyone's bee in a bonnet was, like, she's saying, “Oh, I want $185k.” And it's like, “Well, why don't we just publish that information?” Why isn't everyone just very open and honest about their salary expectations? And I know why: because the paucity of information is a benefit to employers and it works against employees.There was a lady that left—gosh, where was it? [sigh] I forget the company, but she left because she found out she was systemically underpaid compared to their male peers. Having these sort of information imbalances don't really help the people at the bottom of the pyramid. They don't help the little guys. They really only help the people that are in the very large companies with a lot of clout and ability to control narratives.And they want it to stay that way; they don't necessarily want you to know what everyone's salary is because then it gives you, as someone trying to get a job, a better negotiating position because you know what someone with your level of experience is worth to them.Corey: It's important to understand the context behind these salary negotiations and how to go about getting interviews and the rest. The entire job-hunting process is heavily biased in favor of employers because, especially at large employers, they go through this multiple times a week, whereas we go through this, as employees basically, every time we change jobs. Which for most people is every couple of years and for me, because of my mouth, it's every three weeks.Austin: Yeah. I'm not saying it's a simple solution. I am advocating for, sort of, societal, or just cultural shifts, but I think that it all comes full circle in the sense that, hey, a big part of observability is the idea that you need to be able to ask arbitrary questions. You want to know about unknown unknowns. And maybe that's why I like it so much as a field, why I like tracing, why I like this idea.Because, yeah, a lot of things in the world would be interesting, and different, and maybe more equitable if we did have more observability about not just, hey, I use Kafka, I use these parameters on it, and that gives me better throughput, but what if you had observability for how HR runs? What if you had observability for how hiring is done? And that was something that you could see outside of the organization as well. What if we shared all this stuff more, and more, and more, and we treated a few less things as trade secrets? I don't know if that's ever going to happen in my lifetime, but it's my default position. Let's share more rather than less.Corey: Yes, absolutely. Especially those of us with inordinate amounts of privilege. And that privilege takes different forms; there's the usual stuff people are talking about in terms of the fact that we are over-represented in tech in many respects, but there are other forms of privilege, too. There's a privilege that comes with seniority in the space, there is a privilege with being a published author, in your case, there is privilege in having a broad audience, like I do. And it just becomes this incredibly nuanced story.The easiest part of it to lose sight of—at least for me—is I tell stories about what has worked for me and how I've done what I do, and I have to be constantly conscious of the fact that there is that privilege baked in and call it out where I can. I've gotten much better at that, but it's an ongoing process. Because what works for me does not work for other people across a wide variety of different axes. And I don't want people to feel bad based upon what I say.Austin: Oh, yeah, absolutely. I mean, I'm in the same boat. Like, I tend to be very irreverent and/or shitpost-y and I don't have much of an explanation other than, I learned at some point in my life, that it's just… [sigh] I would rather go through life shitposting on Twitter, rather than be employable. It's just who I am. There's—I'm sure some people think I come off as rude. I don't know. I also agree, you'd never punch down. You only punch up. But you never know how other people are going to take that, and I don't think that it always gets interpreted in the spirit it was meant. And I can always do better, right?Corey: As can we all. The hard part for, I think, a lot of us is to suppress that initial flash of defensiveness when someone says you didn't quite get there, and learn from the experience. One of the ways I do that, personally, is I walk away before responding, sometimes. I want to be a better version of myself, but when I get called out of—like, this tweet thread is the whitest thing I've seen since I redid my bathroom walls, and I get a flash of defensiveness, “Excuse me. That's not accurate.”And… and then I stop and I think, and then sanity prevails, where it's, yeah. There's a lot of privilege baked into my existence, and if I don't see it, that doesn't mean it's not there. I have made it a firm rule of not responding defensively to things like that, ever. And there are times when I get called out for aspects of how I present that I don't believe are justified, to be very honest. But that is a me thing; that is not them, and I welcome the feedback, regardless. If you make people feel like a jerk for giving you feedback, they stop giving you feedback. And then where are you?Austin: Yeah. Funny anecdote. I wrote a blog for my personal blog a little while ago about, oh, togetherness, community, something like that. But I wrote—the intro was something like, talking about why people love Sweet Caroline, right? Favorite song in the world.Corey: [sings].Austin: [joins in]. Yeah.Corey: Yeah. I'm not allowed to play with that song here at The Duckbill Group because one of our employees is named Caroline and, firm rule: don't make fun of people's names. They're sensitive about it, and let's not kid ourselves here, I own the company. Even if she says, “It's fine, I love it.” That doesn't help because I own the company. There is a power imbalance here.Austin: Yeah.Corey: I don't know that she would feel that she had the psychological safety to say, “That's not funny.” I absolutely hope she would because that's the culture that I spend significant effort on building, but I can't depend on that. So, I don't go down the path of making those jokes. But I—yes, I love the intro to the song. Please continue.Austin: It's great. Everyone loves it. So, the intro of my initial paragraph was ruminating on that. And this post went around enough that it got submitted to Hacker News a few times, and the only comment it got was some mendacious busybody Hacker News type going on about why I would be so racist against white people. [laugh]. And I was just like, “And this is why I don't come to this website at all.”Corey: Yeah. There are so many things on Twitter that are challenging and difficult and obnoxious, and it's still the best thing we have for a sense of community. This has replaced IRC for me, to be perfectly honest.Austin: Yeah. No, I used to be big on IRC, and then I left because [sigh], well, a couple reasons. One, I really liked being able to post gifs.Corey: Yeah, that is something where the IRC experience is substandard. I was Freenode network staff for years—Austin: Oh wow.Corey: —and that was the thing to do. Now, turns out that the open-source dialogue and the community dialogue have shifted form. And I still hang out there periodically for specific things, but by and large, it's not where the discourse is.Austin: Yeah, it is interesting. It's something that concerns me, kind of, in a long term sense about not only our identity but also, sort of, the actual organic communities we formed, we've put on to these extremely unaccountable privately held platforms whose goal is monetization and growth so that they can continue to make money. And for as much as anyone can rightfully say, “Hey, Twitter's missed the mark,” a lot of times, it is a hard balance to strike. They don't have simple questions to answer, and I don't necessarily know if the nuance of their solutions has really risen to the challenge of answering those well, but it's a hard thing for them to do. That said, I think we're in a really awkward position where suddenly you've got the world's collection of open-source software is being hosted on a platform that is run by Microsoft, and I am old enough to remember. “Embrace, extend, extinguish.”Corey: Oh, yeah. I made an entire personality out of hating Microsoft.Austin: Yeah. And I mean, a lot of people still do. I read MacRumors sometimes, and they're all posting there still. Or Slashdot.Corey: I wondered where they'd gone. I didn't think everyone had changed their mind.Austin: I had just a very out-of-body moment yesterday because someone replied to a comment on mine about Slashdot on it, and then the Slashdot Twitter account liked it. And there exists a photo of me from when I was a teenager, where I owned a Slashdot ballcap. And that picture is somewhere in the world. Probably not on the internet, though, for very good reason.Corey: I'm mostly just still reeling at the discovery that there's a Slashdot Twitter account. But I guess time does evolve.Austin: It does. It makes fools of us all.Corey: It really does. Well, Austin, thank you so much for taking the time to speak with me. If people want to learn more about what you're up to, how you view the world, et cetera, et cetera, et cetera. Where can they find you?Austin: So, you can find me on Twitter, mostly, at @austinlparker. You can find my blog with various musings that is updated frequently at aparker.io and you can learn more about Deserted Island DevOps 2021, coming on April 30th this year, at desertedislanddevops.com.Corey: Excellent. And we will put links to all of that in the [show notes 00:34:01]. Thank you so much for taking the time to speak with me. I appreciate it.Austin: Thank you for having me. This was a lot of fun.Corey: It really was. Austin Parker, principal developer advocate at Lightstep. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and then a giant series of comments that all reference one another and then completely lose track of how they all interrelate and be unable to diagnose performance issues.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.This has been a HumblePod production. Stay humble.

Software Social
Marketing an eBook

Software Social

Play Episode Listen Later Jun 1, 2021 54:16


Michele Hansen  00:00Welcome back to Software Social. This episode is sponsored by the website monitoring tool, Oh Dear. We use Oh Dear to keep track of SSL certificates. If an SSL certificate is about to expire, we get an alert beforehand. We have automated processes to renew them, so we use Oh Dear as an extra level of peace of mind. You can sign up for a ten day free trial with no credit card required at OhDear.app. Michele Hansen  00:28Hey, welcome back to Software Social. So today we're doing something kind of fun. We're leaning on the social part of Software Social, and we have invited our friend, Sean Fioritto, to join us today.Sean Fioritto  00:44Hey guys. Thanks for having me.  Colleen Schnettler  00:47Hi Sean. Thanks for being here. Michele Hansen  00:48So, and the reason why we asked Sean, in addition to being a great person, is that Sean wrote a book called Sketching With CSS, and as you all know, I am writing a book and figuring it out. And there is a lot of stuff I haven't figured out, especially when it comes to, like, actually selling the book. Like, I feel like that, I feel like the, writing the book is, like, I feel like I kind of got a handle on that. The whole selling the book thing, like, not so much. Um, so we thought it would be kind of helpful to have Sean come on, since like, he's done this successfully. Colleen Schnettler  021:36So Sean, I would love to start with a little bit of your background with the book. What inspired you to write it? How did you get started? Where did that idea come from?  Sean Fioritto  01:50Yeah, so I wanted to quit my job.  Colleen Schnettler  01:53Don't we all? Michele Hansen  01:55Honest goal. Sean Fioritto  01:56I always wanted to go on my own, be independent, run my own business. That's been a goal for a very long time. So, I tried various things, you know, in my spare time, with limited to no success for years and years before that, and I was just getting sick of, the plan was, you know, I'm like, okay, I have this job. And in my spare time, I'm gonna get something going and then, and that just wasn't working. So I was getting impatient. Anyway, I ended up signing up with Amy Hoy's 30x500 class. This was seven or eight years ago. So, I signed up for that class. Actually, wait, I'm getting my timeline a little mixed up. So, I started reading stuff by Amy Hoy. It's funny, I'd actually bought another book that she wrote, and she used her sort of process for that book. And I bought that for my, for my job earlier. And I was like, oh, this Amy Hoy person is interesting. And so I started reading her blog, and then she has these things she writes called ebombs. You guys are probably familiar with that term. But they're basically content that, it's educational content directed at her target, you know, customer, which she would call her audience. So I was just, she, at that point, she had started 30x500. I think it was actually called a Year of Hustle at that point. And so she had all this content, and I was just devouring it, because I was like, she gets me. She knows my problem, and this is awesome. So I was just reading everything that she could write, that she wrote, and, you know, finding any resource that she'd ever written about, like, what's her process, because she was talking about this mysterious process that she has, she, she would talk about it. And I was able to sort of reverse engineer part of her course, the main thing called Sales Safari. So I'm not, I'm at my job, coasting, doing a half-assed job, spending a lot of time doing Sales Safari, trying to figure out what, what product I should do. Not product, but that's not the way to think about it with Sales Safari, but trying to figure out like, what, who, what audience should I focus on? And what problems do they have, and what's the juiciest problem that makes sense for me to tackle? And then, and she would call them pains, by the way, not, not problems. So what's the juiciest pain that they have, for me, that was like, be the easiest for me to peel off, and, and work on. So I started digging, and it was like, alright, well, what audience makes sense for me? This is kind of the process, and it was like, you know, like web designers, web developers, because I was a web developer. And so like, what are the, you know, audiences that are close to audiences that I'm in is kind of ideal. So I started there, and then I just read and read and read. I probably put like, 80 hours of research time into that process.  Colleen Schnettler  05:05Wow. Michele Hansen  05:06That's a lot. Sean Fioritto  05:06Of just reading and reading and reading and reading, and taking notes. And really understanding and whittling down and figuring out my audience, and figuring out, so the thinking, the benefit of that amount of time spent deliberately going through a process like that is that at some point, I became so in-tune with the audience that I could identify, and this is gonna pay off for you, Michele, this, this little story, because this feeds into like, how do you sell it. At some point, it meant that I could tell when a thing that I was, like a piece of content marketing that I was working on, was going to resonate very strongly with my audience and be worth the effort, if that makes sense. And it didn't really take much. Like, after I got done with that much amount of research, it was sort of, like, pretty trivial for me to come up with ideas for content that I could write that I knew people were gonna just eat up. And so that's, that's how I started building my, building my mailing list. And then that's how I eventually, Colleen, to your question, I came up with Sketching With CSS, which it was a solution to a pain point that I'd identified in my audience, which at that point was web designers. Colleen Schnettler  06:37How big did your mailing list grow? Sean Fioritto  06:39I have 20,000 people on my mail list. Colleen Schnettler  06:4120,000? Michele Hansen  06:42Holy guacamole.  Sean Fioritto  06:46Yeah. So like I said, I got really good. No, no, no. Michele Hansen  06:51I've got like, 200 people on my mailing list, or like, 220. And like, for context, that's like, 200 more people than I ever expected to have on the mailing list, and hearing, like, 20,000 feels very far from, from 200. Sean Fioritto  07:10Yeah, well, let me say something that will hopefully be more reassuring. The, Amy and Alex, for example, they've been running