Screaming in the Cloud with Corey Quinn features conversations with domain experts in the world of Cloud Computing. Topics discussed include AWS, GCP, Azure, Oracle Cloud, and the "why" behind how businesses are coming to think about the Cloud.
About ChrisChris is a robotics engineer turned cloud security practitioner. From building origami robots for NASA, to neuroscience wearables, to enterprise software consulting, he is a passionate builder at heart. Chris is a cofounder of Common Fate, a company with a mission to make cloud access simple and secure.Links: Common Fate: https://commonfate.io/ Granted: https://granted.dev Twitter: https://twitter.com/chr_norm TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Let's face it, on-call firefighting at 2am is stressful! So there's good news and there's bad news. The bad news is that you probably can't prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It doesn't matter where you are on your journey in cloud—you could never have heard of Amazon the bookstore—and you encounter AWS and you spin up an account. And within 20 minutes, you will come to the realization that everyone in this space does. “Wow, logging in to AWS absolutely blows goats.”Today, my guest, obviously had that reaction, but unlike most people I talked to, decided to get up and do something about it. Chris Norman is the co-founder of Common Fate and most notably to how I know him is one of the original authors of the tool, Granted. Chris, thank you so much for joining me.Chris: Hey, Corey, thank you for having me.Corey: I have done podcasts before; I have done a blog post on it; I evangelize it on Twitter constantly, and even now, it is challenging in a few ways to explain holistically what Granted is. Rather than trying to tell your story for you, when someone says, “Oh, Granted, that seems interesting and impossible to Google for in isolation, so therefore, we know it's going to be good because all the open-source projects with hard to find names are,” what is Granted and what does it do?Chris: Granted is a command-line tool which makes it really easy for you to get access and assume roles when you're working with AWS. For me, when I'm using Granted day-to-day, I wake up, go to my computer—I'm working from home right now—crack open the MacBook and I log in and do some development work. I'm going to go and start working in the cloud.Corey: Oh, when I start first thing in the morning doing development work and logging into the cloud, I know. All right, I'm going to log in to AWS and now I know that my day is going downhill from here.Chris: [laugh]. Exactly, exactly. I think maybe the best days are when you don't need to log in at all. But when you do, I go and I open my terminal and I run this command. Using Granted, I ran this assume command and it authenticates me with single-sign-on into AWS, and then it opens up a console window in a particular account.Now, you might ask, “Well, that's a fairly standard thing.” And in fact, that's probably the way that the console and all of the tools work by default with AWS. Why do you need a third-party tool for this?Corey: Right. I've used a bunch of things that do varying forms of this and unlike Granted, you don't see me gushing about them. I want to be very clear, we have no business relationship. You're not sponsoring anything that I do. I'm not entirely clear on what your day job entails, but I have absolutely fallen in love with the Granted tool, which is why I'm dragging you on to this show, kicking and screaming, mostly to give me an excuse to rave about it some more.Chris: [laugh]. Exactly. And thank you for the kind words. And I'd say really what makes it special or why I've been so excited to be working on it is that it makes this access, particularly when you're working with multiple accounts, really, really easy. So, when I run assume and I open up that console window, you know, that's all fine and that's very similar to how a lot of the other tools and projects that are out there work, but when I want to open that second account and that second console window, maybe because I'm looking at like a development and a staging account at the same time, then Granted allows me to view both of those simultaneously in my browser. And we do that using some platform sort of tricks and building into the way that the browser works.Corey: Honestly, one of the biggest differences in how you describe what Granted is and how I view it is when you describe it as a CLI application because yes, it is that, but one of the distinguishing characteristics is you also have a Firefox extension that winds up leveraging the multi-container functionality extension that Firefox has. So, whenever I wind up running a single command—assume with a-c' flag, then I give it the name of my AWS profile, it opens the web console so I can ClickOps my heart's content inside of a tab that is locked to a container, which means I can have one or two or twenty different AWS accounts and/or regions up running simultaneously side-by-side, which is basically impossible any other way that I've ever looked at it.Chris: Absolutely, yeah. And that's, like, the big differentiating factor right now between Granted and between this sort of default, the native experience, if you're just using the AWS command line by itself. With Granted, you can—with these Firefox containers, all of your cookies, your profile, everything is all localized into that one container. It's actually it's a privacy features that are built into Firefox, which keeps everything really separate between your different profiles. And what we're doing with Granted is that we make it really easy to open a specific profiles that correspond with different AWS profiles that you're using.So, you'd have one which could be your development account, one which could be production or staging. And you can jump between these and navigate between them just as separate tabs in your browser, which is a massive improvement over, you know, what I've previously had to use in the past.Corey: The thing that really just strikes me about this is first, of course, the functionality and the rest, so I saw this—I forget how I even came across it—and immediately I started using it. On my Mac, it was great. I started using it when I was on the road, and it was less great because you built this thing in Go. It can compile and install on almost anything, but there were some assumptions that you had built into this in its early days that did not necessarily encompass all of the use cases that I use. For example, it hadn't really occurred to you that some lunatic would try and only use an iPad when they're on the road, so they have to be able to run this to get federated login links via SSHing into an EC2 instance running somewhere and not have it open locally.You seemed almost taken aback when I brought it up. Like, “What lunatic would do that?” Like, “Hi, I'm such a lunatic. Let's talk about this.” And it does that now, and it's awesome. It does seem to me though, and please correct me if I'm wrong on this assumption slash assessment that this is first and foremost aimed at desktop users, specifically people running Mac on the desktop, is that the genesis of it?Chris: It is indeed. And I think part of the cause behind that is that we originally built a tool for ourselves. And as we were building things and as we were working using the cloud, we were running things—you know, we like to think that we're following best practices when we're using AWS, and so we'd set up multiple accounts, we'd have a special account for development, a separate one for staging, a separate one for production, even internal tools that we would build, we would go and spin up an individual account for those. And then you know, we had lots of accounts. and to go and access those really easily was quite difficult.So, we definitely, we built it for ourselves first and I think that that's part of when we released it, it actually a little bit of cause for some of the initial problems. And some of the feedback that we had was that it's great to build tools for yourself, but when you're working in open-source, there's a lot of different diversity with how people are using things.Corey: We take different approaches. You want to try to align with existing best practices, whereas I am a loudmouth white guy who works in tech. So, what I do definitionally becomes a best practice in the ecosystem. It's easier to just comport with the ones that are already existing that smart people put together rather than just trying to competence your way through it, so you took a better path than I did.But there's been a lot of evolution to Granted as I've been using it for a while. I did a whole write-up on it and that got a whole bunch of eyes onto the project, which I can now admit was a nefarious plan on my part because popping into your community Slack and yelling at you for features I want was all well and good, but let's try and get some people with eyes on this who are smarter than me—which is not that high of a bar when it comes to SSO, and IAM, and federated login, and the rest—and they can start finding other enhancements that I'll probably benefit from. And sure enough, that's exactly what happened. My sneaky plan has come to fruition. Thanks for being a sucker, I guess. I mean—[laugh] it worked. I'm super thrilled by the product.Chris: [laugh]. I guess it's a great thing I think that the feedback and particularly something that's always been really exciting is just seeing new issues come through on GitHub because it really shows the kinds of interesting use cases and the kinds of interesting teams and companies that are using Granted to make their lives a little bit easier.Corey: When I go to the website—which again is impossible to Google—the website for those wondering is granted.dev. It's short, it's concise, I can say it on a podcast and people automatically know how to spell it. But at the top of the website—which is very well done by the way—it mentions that oh, you can, “Govern access to breakglass roles with Common Fate Cloud,” and it also says in the drop shadow nonsense thing in the upper corner, “Brought to you by Common Fate,” which is apparently the name of your company.So, the question I'll get to in a second is what does your company do, but first and foremost, is this going to be one of those rug-pull open-source projects where one day it's, “Oh, you want to log into your AWS accounts? Insert quarter to continue.” I'm mostly being a little over the top with that description, but we've all seen things that we love turn into molten garbage. What is the plan around this? Are you about to ruin this for the rest of us once you wind up raising a round or something? What's the deal?Chris: Yeah, it's a great question, Corey. And I think that to a degree, releasing anything like this that sits in the access workflow and helps you assume roles and helps you day-to-day, you know, we have a responsibility to uphold stability and reliability here and to not change things. And I think part of, like, not changing things includes not [laugh] rug-pulling, as you've alluded to. And I think that for some companies, it ends up that open-source becomes, like, a kind of a lead-generation tool, or you end up with, you know, now finally, let's go on add another login so that you have to log into Common Fate to use Granted. And I think that, to be honest, a tool like this where it's all about improving the speed of access, the incentives for us, like, it doesn't even make sense to try and add another login for to try to get people to, like, to say, login to Common Fate because that would make your signing process for AWS take even longer than it already does.Corey: Yeah, you decided that you know, what's the biggest problem? Oh, you can sleep at night, so let's go ahead and make it even worse, by now I want you to be this custodian of all my credentials to log into all of my accounts. And now you're going to be critical path, so if you're down, I'm not able to log into anything. And oh, by the way, I have to trust you with full access to my bank stuff. I just can't imagine that is a direction that you would be super excited about diving head-first into.Chris: No, no. Yeah, certainly not. And I think that the, you know, building anything in this space, and with what we're doing with Common Fate, you know, we're building a cloud platform to try to make IAM a little bit easier to work with, but it's really sensitive around granting any kind of permission and I think that you really do need that trust. So, trying to build trust, I guess, with our open-source projects is really important for us with Granted and with this project, that it's going to continue to be reliable and continue to work as it currently does.Corey: The way I see it, one of the dangers of doing anything that is particularly open-source—or that leans in the direction of building in Amazon's ecosystem—it leads to the natural question of, well, isn't this just going to be some people say stolen—and I don't think those people understand how open-source works—by AWS themselves? Or aren't they going to build something themselves at AWS that's going to wind up stomping this thing that you've built? And my honest and remarkably cynical answer is that, “You have built a tool that is a joy to use, that makes logging into AWS accounts streamlined and efficient in a variety of different patterns. Does that really sound like something AWS would do?” And followed by, “I wish they would because everyone would benefit from that rising tide.”I have to be very direct and very clear. Your product should not exist. This should be something the provider themselves handles. But nope. Instead, it has to exist. And while I'm glad it does, I also can't shake the feeling that I am incredibly annoyed by the fact that it has to.Chris: Yeah. Certainly, certainly. And it's something that I think about a little bit. I like to wonder whether there's maybe like a single feature flag or some single sort of configuration setting in AWS where they're not allowing different tabs to access different accounts, they're not allowing this kind of concurrent access. And maybe if we make enough noise about Granted, maybe one of the engineers will go and flick that switch and they'll just enable it by default.And then Granted itself will be a lot less relevant, but for everybody who's using AWS, that'll be a massive win because the big draw of using Granted is mainly just around being able to access different accounts at the same time. If AWS let you do that out of the box, hey, that would be great and, you know, I'd have a lot less stuff to maintain.Corey: Originally, I had you here to talk about Granted, but I took a glance at what you're actually building over at Common Fate and I'm about to basically hijack slash derail what probably is going to amount the rest of this conversation because you have a quick example on your site for by developers, for developers. You show a quick Python script that tries to access a S3 bucket object and it's denied. You copy the error message, you paste it into what you're building over a Common Fate, and in return, it's like, “Oh. Yeah, this is the policy that fixes it. Do you want us to apply it for you?”And I just about fell out of my chair because I have been asking for this explicit thing for a very long time. And AWS doesn't do it. Their IAM access analyzer claims to. Like, “Oh, just go look at CloudTrail and see what permissions it uses and we'll build a policy to scope it down.” “Okay. So, it's S3 access. Fair enough. To what object or what bucket?” “Guess,” is what it tells you there.And it's, this is crap. Who thinks this is a good user experience? You have built the thing that I wish AWS had built in natively. Because let's be honest here, I do what an awful lot of people do and overscope permissions massively just because messing around with the bare minimum set of permissions in many cases takes more time than building the damn thing in the first place.Chris: Oh, absolutely. Absolutely. And in fact, this—was a few years ago when I was consulting—I had a really similar sort of story where one of the clients that we were working with, the CTO of this company, he was needing to grant us access to AWS and we were needing to build a particular service. And he said, “Okay, can you just let me know the permissions that you will need and I'll go and deploy the role for this.” And I came back and I said, “Wait. I don't even know the permissions that I'm going to need because the damn thing isn't even built yet.”So, we went sort of back and forth around this. And the compromise ended up just being you know, way too much access. And that was sort of part of the inspiration for, you know, really this whole project and what we're building with Common Fate, just trying to make that feedback loop around getting to the right level of permissions a lot faster.Corey: Yeah, I am just so overwhelmingly impressed by the fact that you have built—and please don't take this as a criticism—but a set of very simple tools. Not simple in the terms of, “Oh, that's, like, three lines of bash, and a fool could write that on a weekend.” No. Simple in the sense of it solves a problem elegantly and well and it's straightforward—well, straightforward as anything in the world of access control goes—to wrap your head around exactly what it does. You don't tend to build these things by sitting around a table brainstorming with someone you met at co-founder dating pool or something and wind up figuring out, “Oh, we should go and solve that. That sounds like a billion-dollar problem.”This feels very much like the outcome of when you're sitting around talking to someone and let's start by drinking six beers so we become extraordinarily honest, followed immediately by let's talk about what sucks. What pisses you off the most? It feels like this is sort of the low-hanging fruit of things that upset people when it comes to AWS. I mean, if things had gone slightly differently, instead of focusing on AWS bills, IAM was next on my list of things to tackle just because I was tired of smacking my head into it.This is very clearly a problem space that you folks have analyzed deeply, worked within, and have put a lot of thought into. I want to be clear, I've thrown a lot of feature suggestions that you for Granted from start to finish. But all of them have been around interface stuff and usability and expanding use cases. None of them have been, “Well, that seems screamingly insecure.” Because it hasn't been.Chris: [laugh].Corey: It has been effective, start to finish, I think that from a security posture, you make terrific choices, in many cases better than ones I would have made a starting from scratch myself. Everything that I'm looking at in what you have built is from a position of this is absolutely amazing and it is transformative to my own workflows. Now, how can we improve it?Chris: Mmm. Thank you, Corey. And I'll say as well, maybe around the security angle, that one of the goals with Granted was to try and do things a little bit better than the default way that AWS does them when it comes to security. And it's actually been a bit of a source for challenges with some of the users that we've been working with with Granted because one of the things we wanted to do was encrypt the SSO token. And this is the token that when you sign in to AWS, kind of like, it allows you to then get access to all of the rest of the accounts.So, it's like a pretty—it's a short-lived token, but it's a really sensitive one. And you know, by default, it's just stored in plain text on your disk. So, we dump to a file and, you know, anything that can go and read that, they can go and get it. It's also a little bit hard to revoke and to lock people out. There's not really great workflows around that on AWS's side.So, we thought, “Okay, great. One of the goals for Granted can be that we will go and store this in your keychain in your system and we'll work natively with that.” And that's actually been a cause for a little bit of a hassle for some users, though, because by doing that and by storing all of this information in the keychain, it's actually broken some of the integrations with the rest of the tooling, which kind of expects tokens and things to be in certain places. So, we've actually had to, as part of dealing with that with Granted, we've had to give users the ability to opt out for that.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: That's why I find this so, I think, just across the board, fantastic. It's you are very clearly engaged with your community. There's a community Slack that you have set up for this. And I know, I know, too many Slacks; everyone has this problem. This is one of those that is worth hanging in, at least from my perspective, just because one of the problems that you have, I suspect, is on my Mac it's great because I wind up automatically updating it to whatever the most recent one is every time I do a brew upgrade.But on the Linux side of the world, you've discovered what many of us have discovered, and that is that packaging things for Linux is a freaking disaster. The current installation is, “Great. Here's basically a curl bash.” Or, “Here, grab this tarball and install it.” And that's fine, but there's no real way of keeping that updated and synced.So, I was checking the other day, oh wow, I'm something like eight versions behind on this box. But it still just works. I upgraded. Oh, wow. There's new functionality here. This is stuff that's actually really handy. I like this quite a bit. Let's see what else we can do.I'm just so impressed, start to finish, by just how receptive you've been to various community feedbacks. And as well—I want to be very clear on this point, too—I've had folks who actually know what they're doing in an InfoSec sense look at what you're up to, and none of them had any issues of note. I'm sure that they have a pile of things like, with that curl bash, they should really be doing a GPG check. Yes, yes, fine. Whatever. If that's your target threat model, okay, great. Here in reality-land for what I do, this is awesome.And they don't seem to have any problems with, “Oh, yeah. By the way, sending analytics back up”—which, okay, fine, whatever. “And it's not disclosing them.” Okay, that's bad. “And it's including the contents of your AWS credentials.”Ahhhh. I did encounter something that was doing that on the back-end once. [cough]—Serverless Framework—sorry, something caught in my throat for a second.Chris: [laugh].Corey: No faster way I can think of to erode trust in that. But everything you're doing just makes sense.Chris: Oh, I do remember that. And that was a little bit of a fiasco, really, around all of that, right? And it's great to hear actually around that InfoSec folks and security people being, you know, not unhappy, I guess, with a tool like this. It's been interesting for me personally. We've really come from a practitioner's background.You know, I wouldn't call myself a security engineer at all. I would call myself as a sometimes a software developer, I guess. I have been hacking my way around Go and definitely learning a lot about how the cloud has worked over the past seven, eight years or so, but I wouldn't call myself a security engineer, so being very cautious around how all of these things work. And we've really tried to defer to things like the system keychain and defer to things that we know are pretty safe and work.Corey: The thing that I also want to call out as well is that your licensing is under the MIT license. This is not one of those, “Oh, you're required to wind up doing a bunch of branding stuff around it.” And, like some people say, “Oh, you have to own the trademark for all of these things.” I mean, I'm not an expert in international trademark law, let's be very clear, but I also feel that trademarking a term that is already used heavily in the space such as the word ‘Granted,' feels like kind of an uphill battle. And let's further be clear that it doesn't matter what you call this thing.In fact, I will call attention to an oddity that I've encountered a fair bit. After installing it, the first thing you do is you run the command ‘granted.' That sets it up, it lets you configure your browser, what browser you want to use, and it now supports standard out for that headless, EC2 use case. Great. Awesome. Love it. But then the other binary that ships with it is Assume. And that's what I use day-to-day. It actually takes me a minute sometimes when it's been long enough to remember that the tool is called Granted and not Assume what's up with that?Chris: So, part of the challenge that we ran into when we were building the Granted project is that we needed to export some environment variables. And these are really important when you're logging into AWS because you have your access key, your secret key, your session token. All of those, when you run the assume command, need to go into the terminal session that you called it. This doesn't matter so much when you're using the console mode, which is what we mentioned earlier where you can open 100 different accounts if you want to view all of those at the same time in your browser. But if you want to use it in your terminal, we wanted to make it look as really smooth and seamless as possible here.And we were really inspired by this approach from—and I have to shout them out and kind of give credit to them—a tool called AWSume—they're spelled A-W-S-U-M-E—Python-based tool that they don't do as much with single-sign-on, but we thought they had a really nice, like, general approach to the way that they did the scripting and aliasing. And we were inspired by that and part of that means that we needed to have a shell script that called this executable, which then will export things back out into the shell script. And we're doing all this wizardry under the hood to make the user experience really smooth and seamless. Part of that meant that we separated the commands into granted and assume and the other part of the naming for everything is that I felt Granted had a far better ring to it than calling the whole project Assume.Corey: True. And when you say assume, is it AWS or not? I've used the AWSume project before; I've used AWS Vault out of 99 Designs for a while. I've used—for three minutes—the native AWS SSO config, and that is just trash. Again, they're so good at the plumbing, so bad at the porcelain, I think is the criticism that I would levy toward a lot of this stuff.Chris: Mmm.Corey: And it's odd to think there's an entire company built around just smoothing over these sharp, obnoxious edges, but I'm saying this as someone who runs a consultancy and have five years that just fixes the bill for this one company. So, there's definitely a series of cottage industries that spring up around these things. I would be thrilled, on some level, if you wound up being completely subsumed by their product advancements, but it's been 15 years for a lot of this stuff and we're still waiting. My big failure mode that I'm worried about is that you never are.Chris: Yeah, exactly, exactly. And it's really interesting when you think about all of these user experience gaps in AWS being opportunities for, I guess, for companies like us, I think, trying to simplify a lot of the complexity for things. I'm interested in sort of waiting for a startup to try and, like, rebuild the actual AWS console itself to make it a little bit faster and easier to use.Corey: It's been done and attempted a bunch of different times. The problem is that the console is a lot of different things to a lot of different people, and as you step through that, you can solve for your use case super easily. “Yeah, what do I care? I use RDS, I use some VPC nonsense, and I use EC2. The end.” “Great. What about IAM?”Because I promise you're using that whether you know it or not. And okay, well, I'm talking to someone else who's DynamoDB, and someone else is full-on serverless, and someone else has more money than sense, so they mostly use SageMaker, and so on and so forth. And it turns out that you're effectively trying to rebuild everything. I don't know if that necessarily works.Chris: Yeah, and I think that's a good point around maybe while we haven't seen anything around that sort of space so far. You go to the console, and you click down, you see that list of 200 different services and all of those have had teams go and actually, like, build the UI and work with those individual APIs. Yeah.Corey: Any ideas as far as what's next for features on Granted?Chris: I think that, for us, it's continuing to work with everybody who's using it, and with a focus of stability and performance. We actually had somebody in the community raise an issue because they have an AWS config file that's over 7000 lines long. And I kind of pity that person, potentially, for their day-to-day. They must deal with so much complexity. Granted is currently quite slow when the config files get very big. And for us, I think, you know, we built it for ourselves; we don't have that many accounts just yet, so working to try to, like, make it really performant and really reliable is something that's really important.Corey: If you don't mind a feature request while we're at it—and I understand that this is more challenging than it looks like—I'm willing to fund this as a feature bounty that makes sense. And this also feels like it might be a good first project for a very particular type of person, I would love to get tab completion working in Zsh. You have it—Chris: Oh.Corey: For Fish because there's a great library that automatically populates that out, but for the Zsh side of it, it's, “Oh, I should just wind up getting Zsh completion working,” and I fell down a rabbit hole, let me tell you. And I come away from this with the perception of yeah, I'm not going to do it. I have not smart enough to check those boxes. But a lot of people are so that is the next thing I would love to see. Because I will change my browser to log into the AWS console for you, but be damned if I'm changing my shell.Chris: [laugh]. I think autocomplete probably should be higher on our roadmap for the tool, to be honest because it's really, like, a key metric and what we're focusing on is how easy is it to log in. And you know, if you're not too sure what commands to use or if we can save you a few keystrokes, I think that would be the, kind of like, reaching our goals.Corey: From where I'm sitting, you definitely have. I really want to thank you for taking the time to not only build this in the first place, but also speak with me about it. If people want to learn more, where's the best place to find you?Chris: So, you can find me on Twitter, I'm @chr_norm, or you can go and visit granted.dev and you'll have a link to join the Slack community. And I'm very active on the Slack.Corey: You certainly are, although I will admit that I fall into the challenge of being in just the perfectly opposed timezone from you and your co-founder, who are in different time zones to my understanding; one of you is on Australia and one of you was in London; you're the London guy as best I'm aware. And as a result, invariably, I wind up putting in feature requests right when no one's around. And, for better or worse, in the middle of the night is not when I'm usually awake trying to log into AWS. That is Azure time.Chris: [laugh]. Yeah, no, we don't have the US time zone properly covered yet for our community support and help. But we do have a fair bit of the world timezone covered. The rest of the team for Common Fate is all based in Australia and I'm out here over in London.Corey: Yeah. I just want to thank you again, for just being so accessible and, like, honestly receptive to feedback. I want to be clear, there's a way to give feedback and I do strive to do it constructively. I didn't come crashing into your Slack one day with a, “You know what your problem is?” I prefer to take the, “This is awesome. Here's what I think would be even better. Does that make sense?” As opposed to the imperious demands and GitHub issues and whatnot? It's, “I'd love it if it did this thing. Doesn't do this thing. Can you please make it do this thing?” Turns out that's the better way to drive change. Who knew?Chris: Yeah. [laugh]. Yeah, definitely. And I think that one of the things that's been the best around our journey with Granted so far has been listening to feedback and hearing from people how they would like to use the tool. And a big thank you to you, Corey, for actually suggesting changes that make it not only better for you, but better for everybody else who's using Granted.Corey: Well, at least as long as we're using my particular byzantine workload patterns in some way, or shape, or form, I'll hear that. But no, it's been an absolute pleasure and I really want to thank you for your time as well.Chris: Yeah, thank you for having me.Corey: Chris Norman, co-founder of Common Fate, as well as one of the two primary developers originally behind the Granted project that logs you into AWS without you having to lose your mind. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, incensed, raging comment that talks about just how terrible all of this is once you spend four hours logging into your AWS account by hand first.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Full Description / Show Notes Steren and Corey talk about how Google Cloud Run got its name (00:49) Corey talks about his experiences using Google Cloud (2:42) Corey and Steven discuss Google Cloud's cloud run custom domains (10:01) Steren talks about Cloud Run's high developer satisfaction and scalability (15:54) Corey and Steven talk about Cloud Run releases at Google I/O (23:21) Steren discusses the majority of developer and customer interest in Google's cloud product (25:33) Steren talks about his 20% projects around sustainability (29:00) About SterenSteren is a Senior Product Manager at Google Cloud. He is part of the serverless team, leading Cloud Run. He is also working on sustainability, leading the Google Cloud Carbon Footprint product.Steren is an engineer from École Centrale (France). Prior to joining Google, he was CTO of a startup building connected objects and multi device solutions.Links Referenced: Google Cloud Run: https://cloud.run sheets-url-shortener: https://github.com/ahmetb/sheets-url-shortener snark.cloud/run: https://snark.cloud/run Twitter: https://twitter.com/steren TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined today by Steren Giannini, who is a senior product manager at Google Cloud, specifically on something called Google Cloud Run. Steren, thank you for joining me today.Steren: Thanks for inviting me, Corey.Corey: So, I want to start at the very beginning of, “Oh, a cloud service. What are we going to call it?” “Well, let's put the word cloud in it.” “Okay, great. Now, it is cloud, so we have to give it a vague and unassuming name. What does it do?” “It runs things.” “Genius. Let's break and go for work.” Now, it's easy to imagine that you spent all of 30 seconds on a name, but it never works that way. How easy was it to get to Cloud Run as a name for the service?Steren: [laugh]. Such a good question because originally it was not named Cloud Run at all. The original name was Google Serverless Engine. But a few people know that because they've been helping us since the beginning, but originally it was Google Serverless Engine. Nobody liked the name internally, and I think at one point, we wondered, “Hey, can we drop the engine structure and let's just think about the name. And what does this thing do?” “It runs things.”We already have Cloud Build. Well, wouldn't it be great to have Cloud Run to pair with Cloud Build so that after you've built your containers, you can run them? And that's how we ended up with this very simple Cloud Run, which today seems so obvious, but it took us a long time to get to that name, and we actually had a lot of renaming to do because we were about to ship with Google Serverless Engine.Corey: That seems like a very interesting last-minute change because it's not just a find and replace at that point, it's—Steren: No.Corey: —“Well, okay, if we call it Cloud Run, which can also be a verb or a noun, depending, is that going to change the meaning of some sentences?” And just doing a find and replace without a proofread pass as well, well, that's how you wind up with funny things on Twitter.Steren: API endpoints needed to be changed, adding weeks of delays to the launch. That is why we—you know, [laugh] announced in 2018 and publicly launched in 2019.Corey: I've been doing a fair bit of work in cloud for a while, and I wound up going down a very interesting path. So, the first native Google Cloud service—not things like WP Engine that ride on top of GCP—but my first native Google Cloud Service was done in service of this podcast, and it is built on Google Cloud Run. I don't think I've told you part of this story yet, but it's one of the reasons I reached out to invite you onto the show. Let me set the stage here with a little bit of backstory that might explain what the hell I'm talking about.As listeners of this show are probably aware, we have sponsors whom we love and adore. In the early days of this show, they would say, “Great, we want to tell people about our product”—which is the point of a sponsorship—“And then send them to a URL.” “Great. What's the URL?” And they would give me something that was three layers deep, then with a bunch of UTM tracking parameters at the end.And it's, “You do realize that no one is going to be sitting there typing all of that into a web browser?” At best, you're going to get three words or so. So, I built myself a URL redirector, snark.cloud. I can wind up redirecting things in there anywhere it needs to go.And for a long time, I did this on top of S3 and then put CloudFront in front of it. And this was all well and good until, you know, things happened in the fullness of time. And now holy crap, I have an operations team involved in things, and maybe I shouldn't be the only person that knows how to work on all of these bits and bobs. So, it was time to come up with something that had a business user-friendly interface that had some level of security, so I don't wind up automatically building out a spam redirect service for anything that wants to, and it needs to be something that's easy to work with. So, I went on an exploration.So, at first it showed that there were—like, I have an article out that I've spoken about before that there are, “17 Ways to Run Containers on AWS,” and then I wrote the sequel, “17 More Ways to Run Containers on AWS.” And I'm keeping a list, I'm almost to the third installation of that series, which is awful. So, great. There's got to be some ways to build some URL redirect stuff with an interface that has an admin panel. And I spent three days on this trying a bunch of different things, and some were running on deprecated versions of Node that wouldn't build properly and others were just such complex nonsense things that had got really bad. I was starting to consider something like just paying for Bitly or whatnot and making it someone else's problem.And then I stumbled upon something on GitHub that really was probably one of the formative things that changed my opinion of Google Cloud for the better. And within half an hour of discovering this thing, it was up and running. I did the entire thing, start to finish, from my iPad in a web browser, and it just worked. It was written by—let me make sure I get his name correct; you know, messing up someone's name is a great way to say that we don't care about them—Ahmet Balkan used to work at Google Cloud; now he's over at Twitter. And he has something up on GitHub that is just absolutely phenomenal about this, called sheets-url-shortener.And this is going to sound wild, but stick with me. The interface is simply a Google Sheet, where you have one column that has the shorthand slug—for example, run; if you go to snark.cloud/run, it will redirect to Google Cloud Run's website. And the second column is where you want it to go. The end.And whenever that gets updated, there's of course some caching issues, which means it can take up to five seconds from finishing that before it will actually work across the entire internet. And as best I can tell, that is fundamentally magic. But what made it particularly useful and magic, from my perspective, was how easy it was to get up and running. There was none of this oh, but then you have to integrate it with Google Sheets and that's a whole ‘nother team so there's no way you're going to be able to figure that out from our Docs. Go talk to them and then come back in the day.They were the get started, click here to proceed. It just worked. And it really brought back some of the magic of cloud for me in a way that I hadn't seen in quite a while. So, all which is to say, amazing service, I continue to use it for all of these sponsored links, and I am still waiting for you folks to bill me, but it fits comfortably in the free tier because it turns out that I don't have hundreds of thousands of people typing it in every week.Steren: I'm glad it went well. And you know, we measure tasks success for Cloud Run. And we do know that most new users are able to deploy their apps very quickly. And that was the case for you. Just so you know, we've put a lot of effort to make sure it was true, and I'll be glad to tell you more about all that.But for that particular service, yes, I suppose Ahmet—who I really enjoyed working with on Cloud Run, he was really helpful designing Cloud Run with us—has open-sourced this side project. And basically, you might even have clicked on a deploy to Cloud Run button on GitHub, right, to deploy it?Corey: That is exactly what I did and it somehow just worked and—Steren: Exactly.Corey: And it knew, even logging into the Google Cloud Console because it understands who I am because I use Google Docs and things, I'm already logged in. None of this, “Oh, which one of these 85 credential sets is it going to be?” Like certain other clouds. It was, “Oh, wow. Wait, cloud can be easy and fun? When did that happen?”Steren: So, what has happened when you click that deploy to Google Cloud button, basically, the GitHub repository was built into a container with Cloud Build and then was deployed to Cloud Run. And once on Cloud Run, well, hopefully, you have forgotten about it because that's what we do, right? We—give us your code, in a container if you know containers if you don't just—we support, you know, many popular languages, and we know how to build them, so don't worry about that. And then we run it. And as you said, when there is low traffic or no traffic, it scales to zero.When there is low traffic, you're likely going to stay under the generous free tier. And if you have more traffic for, you know, Screaming in the Cloud suddenly becoming a high destination URL redirects, well, Cloud Run will scale the number of instances of this container to be able to handle the load. Cloud Run scales automatically and very well, but only—as always—charging you when you are processing some requests.Corey: I had to fork and make a couple of changes myself after I wound up doing some testing. The first was to make the entire thing case insensitive, which is—you know, makes obvious sense. And the other was to change the permanent redirect to a temporary redirect because believe it or not, in the fullness of time, sometimes sponsors want to change the landing page in different ways for different campaigns and that's fine by me. I just wanted to make sure people's browser cache didn't remember it into perpetuity. But it was easy enough to run—that was back in the early days of my exploring Go, which I've been doing this quarter—and in the couple of months this thing has been running it has been effectively flawless.It's set it; it's forget it. The only challenges I had with it are it was a little opaque getting a custom domain set up that—which is still in beta, to be clear—and I've heard some horror stories of people saying it got wedged. In my case, no, I deployed it and I started refreshing it and suddenly, it start throwing an SSL error. And it's like, “Oh, that's not good, but I'm going to break my own lifestyle here and be patient for ten minutes.” And sure enough, it cleared itself and everything started working. And that was the last time I had to think about any of this. And it just worked.Steren: So first, Cloud Run is HTTPS only. Why? Because it's 2020, right? It's 2022, but—Corey: [laugh].Steren: —it's launched in 2020. And so basically, we have made a decision that let's just not accept HTTP traffic; it's only HTTPS. As a consequence, we need to provision a cert for your custom domain. That is something that can take some time. And as you said, we keep it in beta or in preview because we are not yet satisfied with the experience or even the performance of Cloud Run custom domains, so we are actively working on fixing that with a different approach. So, expect some changes, hopefully, this year.Corey: I will say it does take a few seconds when people go to a snark.cloud URL for it to finish resolving, and it feels on some level like it's almost like a cold start problem. But subsequent visits, the same thing also feel a little on the slow and pokey side. And I don't know if that's just me being wildly impatient, if there's an optimization opportunity, or if that's just inherent to the platform that is not under current significant load.Steren: So, it depends. If the Cloud Run service has scaled down to zero, well of course, your service will need to be started. But what we do know, if it's a small Go binary, like something that you mentioned, it should really take less than, let's say, 500 milliseconds to go from zero to one of your container instance. Latency can also be due to the way the code is running. If it occurred is fetching things from Google Sheets at every startup, that is something that could add to the startup latency.So, I would need to take a look, but in general, we are not spinning up a virtual machine anytime we need to scale horizontally. Like, our infrastructure is a multi-tenant, rapidly scalable infrastructure that can materialize a container in literally 300 milliseconds. The rest of the latency comes from what does the container do at startup time?Corey: Yeah, I just ran a quick test of putting time in front of a curl command. It looks like it took 4.83 seconds. So, enough to be perceptive. But again, for just a quick redirect, it's generally not the end of the world and there's probably something I'm doing that is interesting and odd. Again, I did not invite you on the show to file a—Steren: [laugh].Corey: Bug report. Let's be very clear here.Steren: Seems on the very high end of startup latencies. I mean, I would definitely expect under the second. We should deep-dive into the code to take a look. And by the way, building stuff on top of spreadsheets. I've done that a ton in my previous lives as a CTO of a startup because well, that's the best administration interface, right? You just have a CRUD UI—Corey: [unintelligible 00:12:29] world and all business users understand it. If people in Microsoft decided they were going to change Microsoft Excel interface, even a bit, they would revert the change before noon of the same day after an army of business users grabbed pitchforks and torches and marched on their headquarters. It's one of those things that is how the world runs; it is the world's most common IDE. And it's great, but I still think of databases through the lens of thinking about it as a spreadsheet as my default approach to things. I also think of databases as DNS, but that's neither here nor there.Steren: You know, if you have maybe 100 redirects, that's totally fine. And by the way, the beauty of Cloud Run in a spreadsheet, as you mentioned is that Cloud Run services run with a certain identity. And this identity, you can grant it permissions. And in that case, what I would recommend if you haven't done so yet, is to give an identity to your Cloud Run service that has the permission to read that particular spreadsheet. And how you do that you invite the email of the service account as a reader of your spreadsheet, and that's probably what you did.Corey: The click button to the workflow on Google Cloud automatically did that—Steren: Oh, wow.Corey: —and taught me how to do it. “Here's the thing that look at. The end.” It was a flawless user-onboarding experience.Steren: Very nicely done. But indeed, you know, there is this built-in security which is the principle of minimal permission, like each of your Cloud Run service should basically only be able to read and write to the backing resources that they should. And by default, we give you a service account which has a lot of permissions, but our recommendation is to narrow those permissions to basically only look at the cloud storage buckets that the service is supposed to look at. And the same for a spreadsheet.Corey: Yes, on some level, I feel like I'm going to write an analysis of my own security approach. It would be titled, “My God, It's Full Of Stars” as I look at the IAM policies of everything that I've configured. The idea of least privilege is great. What I like about this approach is that it made it easy to do it so I don't have to worry about it. At one point, I want to go back and wind up instrumenting it a bit further, just so I can wind up getting aggregate numbers of all right, how many times if someone visited this particular link? It'll be good to know.And I don't know… if I have to change permissions to do that yet, but that's okay. It's the best kind of problem: future Corey. So, we'll deal with that when the time comes. But across the board, this has just been a phenomenal experience and it's clear that when you were building Google Cloud Run, you understood the assignment. Because I was looking for people saying negative things about it and by and large, all of its seem to come from a perspective of, “Well, this isn't going to be the most cost-effective or best way to run something that is hyperscale, globe-spanning.”It's yes, that's the thing that Kubernetes was originally built to run and for some godforsaken reason people run their blog on it instead now. Okay. For something that is small, scales to zero, and has long periods where no one is visiting it, great, this is a terrific answer and there's absolutely nothing wrong with that. It's clear that you understood who you were aiming at, and the migration strategy to something that is a bit more, I want to say robust, but let's be clear what I mean when I'm saying that if you want something that's a little bit more impressive on your SRE resume as you're trying a multi-year project to get hired by Google or pretend you got hired by Google, yeah, you can migrate to something else in a relatively straightforward way. But that this is up, running, and works without having to think about it, and that is no small thing.Steren: So, there are two things to say here. The first is yes, indeed, we know we have high developer satisfaction. You know, we measure this—in Google Cloud, you might have seen those small satisfaction surveys popping up sometimes on the user interface, and you know, we are above 90% satisfaction score. We hire third parties to help us understand how usable and what satisfaction score would users get out of Cloud Run, and we are constantly getting very, very good results, in absolute but also compared to the competition.Now, the other thing that you said is that, you know, Cloud Run is for small things, and here while it is definitely something that allows you to be productive, something that strives for simplicity, but it also scales a lot. And contrary to other systems, you do not have any pre-provisioning to make. So, we have done demos where we go from zero to 10,000 container instances in ten seconds because of the infrastructure on which Cloud Run runs, which is fully managed and multi-tenant, we can offer you this scale on demand. And many of our biggest customers have actually not switched to something like Kubernetes after starting with Cloud Run because they value the low maintenance, the no infrastructure management that Cloud Run brings them.So, we have like Ikea, ecobee… for example ecobee, you know, the smart thermostats are using Cloud Run to ingest events from the thermostat. I think Ikea is using Cloud Run more and more for more of their websites. You know, those companies scale, right? This is not, like, scale to zero hobby project. This is actually production e-commerce and connected smart objects production systems that have made the choice of being on a fully-managed platform in order to reduce their operational overhead.[midroll 00:17:54]Corey: Let me be clear. When I say scale—I think we might be talking past each other on a small point here. When I say scale, I'm talking less about oh tens or hundreds of thousands of containers running concurrently. I'm talking in a more complicated way of, okay, now we have a whole bunch of different microservices talking to one another and affinity as far as location to each other for data transfer reasons. And as you start beginning to service discovery style areas of things, where we build a really complicated applications because we hired engineers and failed to properly supervise them, and that type of convoluted complex architecture.That's where it feels like Cloud Run increasingly, as you move in that direction, starts to look a little bit less like the tool of choice. Which is fine, I want to be clear on that point. The sense that I've gotten of it is a great way to get started, it's a great way to continue running a thing you don't have to think about because you have a day job that isn't infrastructure management. And it is clear to—as your needs change—to either remain with the service or pivot to a very close service without a whole lot of retooling, which is key. There's not much of a lock-in story to this, which I love.Steren: That was one of the key principles when we started to design Cloud Run was, you know, we realized the industry had agreed that the container image was the standard for the deployment artifact of software. And so, we just made the early choice of focusing on deploying containers. Of course, we are helping users build those containers, you know, we have things called build packs, we can continuously deploy from GitHub, but at the end of the day, the thing that gets auto-scaled on Cloud Run is a container. And that enables portability.As you said. You can literally run the same container, nothing proprietary in it, I want to be clear. Like, you're just listening on a port for some incoming requests. Those requests can be HTTP requests, events, you know, we have products that can push events to Cloud Run like Eventarc or Pub/Sub. And this same container, you can run it on your local machine, you can run it on Kubernetes, you can run it on another cloud. You're not locked in, in terms of API of the compute.We even went even above and beyond by having the Cloud Run API looks like a Kubernetes API. I think that was an extra effort that we made. I'm not sure people care that much, but if you look at the Cloud Run API, it is actually exactly looking like Kubernetes, Even if there is no Kubernetes at all under the hood; we just made it for portability. Because we wanted to address this concern of serverless which was lock-in. Like, when you use a Function as a Service product, you are worried that the architecture that you are going to develop around this product is going to be only working in this particular cloud provider, and you're not in control of the language, the version that this provider has decided to offer you, you're not in control of more of the complexity that can come as you want to scan this code, as you want to move this code between staging and production or test this code.So, containers are really helping with that. So, I think we made the right choice of this new artifact that to build Cloud Run around the container artifact. And you know, at the time when we launched, it was a little bit controversial because back in the day, you know, 2018, 2019, serverless really meant Functions as a Service. So, when we launched, we little bit redefined serverless. And we basically said serverless containers. Which at the time were two worlds that in the same sentence were incompatible. Like, many people, including internally, had concerns around—Corey: Oh, the serverless versus container war was a big thing for a while. Everyone was on a different side of that divide. It's… containers are effectively increasingly—and I know, I'll get email for this, and I don't even slightly care, they're a packaging format—Steren: Exactly.Corey: —where it solves the problem of how do I build this thing to deploy on Debian instances? And Ubuntu instances, and other instances, God forbid, Windows somewhere, you throw a container over the wall. The end. Its DevOps is about breaking down the walls between Dev and Ops. That's why containers are here to make them silos that don't have to talk to each other.Steren: A container image is a glorified zip file. Literally. You have a set of layers with files in them, and basically, we decided to adopt that artifact standard, but not the perceived complexity that existed at the time around containers. And so, we basically merged containers with serverless to make something as easy to use as a Function as a Service product but with the power of bringing your own container. And today, we are seeing—you mentioned, what kind of architecture would you use Cloud Run for?So, I would say now there are three big buckets. The obvious one is anything that is a website or an API, serving public internet traffic, like your URL redirect service, right? This is, you have an API, takes a request and returns a response. It can be a REST API, GraphQL API. We recently added support for WebSockets, which is pretty unique for a service offering to support natively WebSockets.So, what I mean natively is, my client can open a socket connection—a bi-directional socket connection—with a given instance, for up to one hour. This is pretty unique for something that is as fully managed as Cloud Run.Corey: Right. As we're recording this, we are just coming off of Google I/O, and there were a number of announcements around Cloud Run that were touching it because of, you know, strange marketing issues. I only found out that Google I/O was a thing and featured cloud stuff via Twitter at the time it was happening. What did you folks release around Cloud Run?Steren: Good question, actually. Part of the Google I/O Developer keynote, I pitched a story around how Cloud Run helps developers, and the I/O team liked the story, so we decided to include that story as part of the live developer keynote. So, on stage, we announced Cloud Run jobs. So now, I talked to you about Cloud Run services, which can be used to expose an API, but also to do, like, private microservice-to-microservice communication—because cloud services don't have to be public—and in that case, we support GRPC and, you know, a very strong security mechanism where only Service A can invoke Service B, for example, but Cloud Run jobs are about non-request-driven containers. So, today—I mean, before Google I/O a few days ago, the only requirement that we imposed on your container image was that it started to listen for requests, or events, or GRPC—Corey: Web requests—Steren: Exactly—Corey: It speaks [unintelligible 00:24:35] you want as long as it's HTTP. Yes.Steren: That was the only requirement we asked you to have on your container image. And now we've changed that. Now, if you have a container that basically starts and executes to completion, you can deploy it on a Cloud Run job. So, you will use Cloud Run jobs for, like, daily batch jobs. And you have the same infrastructure, so on-demand, you can go from zero to, I think for now, the maximum is a hundred tasks in parallel, for—of course, you can run many tasks in sequence, but in parallel, you can go from zero to a hundred, right away to run your daily batch job, daily admin job, data processing.But this is more in the batch mode than in streaming mode. If you would like to use a more, like, streaming data processing, than a Cloud Run service would still be the best fit because you can literally push events to it, and it will auto-scale to handle any number of events that it receives.Corey: Do you find that the majority of customers are using Cloud Run for one-off jobs that barely will get more than a single container, like my thing, or do you find that they're doing massively parallel jobs? Where's the lion's share of developer and customer interest?Steren: It's both actually. We have both individual developers, small startups—which really value the scale to zero and pay per use model of Cloud Run. Your URL redirect service probably is staying below the free tier, and there are many, many, many users in your case. But at the same time, we have big, big, big customers who value the on-demand scalability of Cloud Run. And for these customers, of course, they will probably very likely not scale to zero, but they value the fact that—you know, we have a media company who uses Cloud Run for TV streaming, and when there is a soccer game somewhere in the world, they have a big spike of usage of requests coming in to their Cloud Run service, and here they can trust the rapid scaling of Cloud Run so they don't have to pre-provision things in advance to be able to serve that sudden traffic spike.But for those customers, Cloud Run is priced in a way so that if you know that you're going to consume a lot of Cloud Run CPU and memory, you can purchase Committed Use Discounts, which will lower your bill overall because you know you are going to spend one dollar per hour on Cloud Run, well purchase a Committed Use Discount because you will only spend 83 cents instead of one dollar. And also, Cloud Run and comes with two pricing model, one which is the default, which is the request-based pricing model, which is basically you only have CPU allocated to your container instances if you are processing at least one request. But as a consequence of that, you are not paying outside of the processing of those requests. Those containers might stay up for you, one, ready to receive new requests, but you're not paying for them. And so, that is—you know, your URL redirect service is probably in that mode where yes when you haven't used it for a while, it will scale down to zero, but if you send one request to it, it will serve that request and then it will stay up for a while until it decides to scale down. But you the user only pays when you are processing these specific requests, a little bit like a Function as a Service product.Corey: Scales to zero is one of the fundamental tenets of serverless that I think that companies calling something serverless, but it always charges you per hour anyway. Yeah, that doesn't work. Storage, let's be clear, is a separate matter entirely. I'm talking about compute. Even if your workflow doesn't scale down to zero ever as a workload, that's fine, but if the workload does, you don't get to keep charging me for it.Steren: Exactly. And so, in that other mode where you decide to always have CPU allocated to your Cloud Run container instances, then you pay for the entire lifecycle of this container instances. You still benefit from the auto-scaling of Cloud Run, but you will pay for the lifecycle and in that case, the price points are lower because you pay for a longer period of time. But that's more the price model that those bigger customers will take because at their scale, they basically always receive requests, so they already to pay always, basically.Corey: I really want to thank you for taking the time to chat with me. Before you go, one last question that we'll be using as a teaser for the next episode that we record together. It seems like this is a full-time job being the product manager on Cloud Run, but no Google, contrary to popular opinion, does in fact, still support 20% projects. What's yours?Steren: So, I've been looking to work on Cloud Run since it was a prototype, and you know, for a long time, we've been iterating privately on Cloud Run, launching it, seeing it grow, seeing it adopted, it's great. It's my full-time job. But on Fridays, I still find the time to have a 20% project, which also had quite a bit of impact. And I work on some sustainability efforts for Google Cloud. And notably, we've released two things last year.The first one is that we are sharing some carbon characteristics of Google Cloud regions. So, if you have seen those small leaves in the Cloud Console next to the regions that are emitting the less carbon, that's something that I helped bring to life. And the second one, which is something quite big, is we are helping customers report and reduce their gross carbon emissions of their Google Cloud usage by providing an out of the box reporting tool called Google Cloud Carbon Footprint. So, that's something that I was able to bootstrap with a team a little bit on the side of my Cloud Run project, but I was very glad to see it launched by our CEO at the last Cloud Next Conference. And now it is a fully-funded project, so we are very glad that we are able to help our customers better meet their sustainability goals themselves.Corey: And we will be talking about it significantly on the next episode. We're giving a teaser, not telling the whole story.Steren: [laugh].Corey: I really want to thank you for being as generous with your time as you are. If people want to learn more, where can they find you?Steren: Well, if they want to learn more about Cloud Run, we talked about how simple was that name. It was obviously not simple to find this simple name, but the domain is https://cloud.run.Corey: We will also accept snark.cloud/run, I will take credit for that service, too.Steren: [laugh]. Exactly.Corey: There we are.Steren: And then, people can find me on Twitter at @steren, S-T-E-R-E-N. I'll be happy—I'm always happy to help developers get started or answer questions about Cloud Run. And, yeah, thank you for having me. As I said, you successfully deployed something in just a few minutes to Cloud Run. I would encourage the audience to—Corey: In spite of myself. I know, I'm as surprised as anyone.Steren: [laugh].Corey: The only snag I really hit was the fact that I was riding shotgun when we picked up my daughter from school and went through a dead zone. It's like, why is this thing not loading in the Google Cloud Console? Yeah, fix the cell network in my area, please.Steren: I'm impressed that you did all of that from an iPad. But yeah, to the audience give Cloud Run the try. You can really get started connecting your GitHub repository or deploy your favorite container image. And we've worked very hard to ensure that usability was here, and we know we have pretty strong usability scores. Because that was a lot of work to simplicity, and product excellence and developer experience is a lot of work to get right, and we are very proud of what we've achieved with Cloud Run and proud to see that the developer community has been very supportive and likes this product.Corey: I'm a big fan of what you've built. And well, of course, it links to all of that in the show notes. I just want to thank you again for being so generous with your time. And thanks again for building something that I think in many ways showcases the best of what Google Cloud has to offer.Steren: Thanks for the invite.Corey: We'll talk again soon. Steren Giannini is a senior product manager at Google Cloud, on Cloud Run. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice. If it's on YouTube, put the thumbs up and the subscribe buttons as well, but in the event that you hated it also include an angry comment explaining why your 20% project is being a shithead on the internet.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Full Description / Show Notes Gafnit explains how she found a vulnerability in RDS, an Amazon database service (1:40) Gafnit and Corey discuss the concept of not being able to win in cloud security (7:20) Gafnit talks about transparency around security breaches (11:02) Corey and Gafnit discuss effectively communicating with customers about security (13:00) Gafnit answers the question “Did you come at the RDS vulnerability exploration from a perspective of being deeper on the Postgres side or deeper on the AWS side? (18:10) Corey and Gafnit talk about the risk of taking a pre-existing open source solution and offering it as a managed service (19:07) Security measures in cloud-native approaches versus cloud-hosted (22:41) Gafnit and Corey discuss the security community (25:04) About GafnitGafnit Amiga is the Director of Security Research at Lightspin. Gafnit has 7 years of experience in Application Security and Cloud Security Research. Gafnit leads the Security Research Group at Lightspin, focused on developing new methods to conduct research for new cloud native services and Kubernetes. Previously, Gafnit was a lead product security engineer at Salesforce focused on their core platform and a security researcher at GE Digital. Gafnit holds a Bs.c in Computer Science from IDC Herzliya and a student for Ms.c in Data Science.Links Referenced: Lightspin: https://www.lightspin.io/ Twitter: https://twitter.com/gafnitav LinkedIn: https://www.linkedin.com/in/gafnit-amiga-b1357b125/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. We've taken a bit of a security bent to the conversations that we've been having on this show and over the past year or so and, well, today's episode is no different. In fact, we're going a little bit deeper than we normally tend to. My guest today is Gafnit Amiga, who's the Director of Security Research at Lightspin. Gafnit, thank you for joining me.Gafnit: Hey, Corey. Thank you for inviting me to the show.Corey: You sort of burst onto the scene—and by ‘scene,' I of course mean the cloud space, at least to the level of community awareness—back, I want to say in April of 2022 when you posted a very in-depth blog post about exploiting RDS and some misconfigurations on AWS's side to effectively display internal service credentials for the RDS service itself. Now, that sounds like it's one of those incredibly deep, incredibly murky things because it is, let's be clear. At a high level, can you explain to me exactly what it is that you found and how you did it? Gafnit: Yes, so, RDS is database service of Amazon. It's a managed service where you can choose the engine that you prefer. One of them is Postgres. There, I found the vulnerability. The vulnerability was in the extension in the log_fdw—so it's for—like, stands for Foreign Data Wrapper—where this extension is, therefore reading the logs directly of the engine, and then you can query it using SQL queries, which should be simpler and easy to use.And this extension enables you to provide a path. And there was a path traversal, but the traversal happened only when you dropped a validation of the wrapper. And this is how I managed to read local files from the database EC2 machine, which shouldn't happen because this is a managed service and you shouldn't have any access to the underlying host.Corey: It's always odd when the abstraction starts leaking, from an AWS perspective. I know that a friend of mine was on Aurora during the beta and was doing some high-performance work and suddenly started seeing SQL errors about /var/temp filling up, which is, for those who are not well versed in SQL, and even for those who are, that's not the sort of thing you tend to expect to show up on there. It feels like the underlying system tends to leak in—particularly in RDS sense—into what is otherwise at least imagined to be a fully-managed service.Gafnit: Yes because sometimes they want to give you an informative error so you will be able to realize what happened and what caused to the error, and sometimes they prefer not to give you too many information because they don't want you to get to the underlying machine. This is why, for example, you don't get a regular superuser; you have an RDS superuser in the database.Corey: It seems to me that this is sort of a problem of layering different security models on top of each other. If you take a cloud-native database that they designed, start to finish, themselves, like DynamoDB, the entire security model for Dynamo, as best I can determine, is wrapped up within IAM. So, if you know IAM—spoiler, nobody knows IAM completely, it seems—but if you have that on lock you've got it; there's nothing else you need to think about. Whereas with RDS, you have to layer on IAM to get access to the database and what you're allowed to do with it.But then there's an entirely separate user management system, in many respects, of local users for other Postgres or MySQL or any other systems that were using, to a point where even when they started supporting IRM for authentication to RDS at the database user level. It was flagged in the documentation with a bunch of warnings of, “Don't do this for high-volume stuff; only do this in development style environments.” So, it's clear that it has been a difficult marriage, for lack of a better term. And then you have to layer on all the other stuff that if God forbid, you're in a multi-cloud style environment or working with Kubernetes on top of all of this, and it seems like you're having to pick and choose between four or five different levels of security modeling, as well as understand how all of those things interplay together. How come we don't see things like this happening four times a day as a result?Gafnit: Well, I guess that there are more issues being found, but not always published but I think that this is what makes it more complex for both sides. Creating managed services with resources and third parties that everybody knows. To make it easy for them to use requires a deep understanding of the existing permission models of the service where you want to integrate it with your permission model and how the combination works. So, you actually need to understand how every change is going to affect the restrictions that you want to have. So, for example, if you don't want the database users to be able to read-write or do a network activity, so you really need to understand the permission model of the Postgres itself. So, it makes it more complicated for development, but it's also good for researchers because they already know Postgres and they have a good starting point.Corey: My philosophy has always been when you're trying to secure something, you need to have at least a topical level of understanding of the entire system, start to finish. One of the problems I've had with the idea of microservices as is frequently envisioned is that there's separation, but not real separation, so you have to hand-wave over a whole bunch of the security model. If you don't understand something, I believe it's very difficult to secure it. And let's be honest, even if you do understand [laugh] something, it can be very difficult to secure it. And the cloud vendors with IAM and similar systems don't seem to be doing themselves any favors, given the sheer complexity and the capabilities that they're demanding of themselves, even for having one AWS service talk to another one, but in the right way.And it's finicky, and it's nuanced, and debugging it becomes a colossal pain. And finally, at least those of us who are bad at these things, finally say, “The hell with it,” and they just grant full access from Service A to Service B—in the confines of a test environment. I'm not quite that nuts myself, most days. And then it's the biggest lie we always tell ourselves is once we have something overscoped like that, usually for CI/CD, it's, “Oh, todo: I'll go back and fix that later.” Yeah, I'm looking back five years ago and that's still on my todo list.For some reason, it's never been the number one priority. And in all likelihood, it won't be until right after it really should have been my number one priority. It feels like in cloud security particularly, you can't win, you can only not lose. I always found that to be something of a depressing perspective and I didn't accept it for the longest time. But increasingly, these days, it started to feel like that is the state of the world. Am I wrong on that? Am I just being too dour?Gafnit: What do you mean by you cannot lose?Corey: There's no winning in security from my perspective because no one is going to say, “All right. We won the security. Problem solved. The end.” Companies don't view security as a value-add. It is only about a downside risk mitigation play.It's, “Yay, another day of not getting breached.” And the failure mode from there is, “Okay, well, we got breached, but we found out about it ourselves immediately internally, rather than reading about it in The New York Times in two weeks.” The winning is just the steady-state, the status quo. It's just all different flavors of losing beyond that.Gafnit: So, I don't think it's quite the case because I can tell that they do do always an active work on securing the services and their structure because I went over other extensions before reaching to the log foreign data wrapper, and they actually excluded high-risk functionalities that could help me to achieve privileged access to the underlying host. And they do it with other services as well because they do always do the security review before having it integrated externally. But you know, it's an endless zone. You can always have something. Security vulnerabilities are always [arrays 00:09:06]. So everyone, whenever they can help and to search and to give their value, it's appreciated.Corey: I feel like I need to clarify a bit of nuance. When your blog post first came out talking about this, I was, well let's say a little irritated toward AWS on Twitter and other places. And Twitter is not a place for nuance, it is easy to look at that and think, “Oh, I was upset at AWS for having a vulnerability.” I am not, I want to be very clear on that. Now, it's certainly not good, but these are computers; that is the nature of how they work.If you want to completely secure computer, cut the power to it, sink it in concrete and then drop it in the ocean. And even then, there are exceptions to all of that. So, it's always a question of not blocking all risk; it's about trade-offs and what risk is acceptable. And to AWS is credit, they do say that they practice defense-in-depth. Being able to access the credentials for the running RDS service on top of the instance that it was running on, while that's certainly not good, isn't as if you'd suddenly had keys to everything inside of AWS and all their security model crumbles away before you.They do the right thing and the people working on these things are incredibly good. And they work very hard at these things. My concern and my complaint is, as much as I enjoy the work that you do and reading these blog posts talking about how you did it, it bothers me that I have to learn about a vulnerability in a service for which I pay not small amounts of money—RDS is the number one largest charge in my AWS bill every month—and I have to hear about it from a third-party rather than the vendor themselves. In this case, it was a full day later, where after your blog post went up, and they finally had a small security disclosure on AWS's site talking about it. And that pattern feels to me like it leads nowhere good.Gafnit: So, transparency is a key word here. And when I wrote the post, I asked if they want to add anything from their side, and they told that they already reached out to the vulnerable customers and they helped them to migrate to their fixed version. So, from their side, it didn't felt it's necessary to add it over there. But I did mention the fact that I did the investigation and no customer data was hurt. Yeah, but I think that if there will be maybe a more organized process for any submission of any vulnerability that where all the steps are aligned, it will help everyone and anyone can be informed with everything that happens.Corey: I have always been extraordinarily impressed by people who work at AWS and handle a lot of the triaging of vulnerability reports. Zack Glick, before he left, was doing an awful lot of that Dan [Erson 00:12:05] continues to be a one of the bright lights of AWS, from my perspective, just as far as customer communication and understanding exactly what the customer perspective is. And as individuals, I see nothing but stars over at AWS. To be clear, ‘Nothing but Stars' is also the name of most of my IAM policies, but that's neither here nor there.It seems like, on some level, there's a communications and policy misalignment, on some level, because I look at this and every conversation I ever have with AWS's security folks, they are eminently reasonable, they're incredibly intelligent, and they care. There's no mistaking that they legitimately care. But somewhere at the scale of company they're at, incentives get crossed, and everyone has a different position they're looking at these things from, and it feels like that disjointedness leads to almost a misalignment as far as how to effectively communicate things like this to customers.Gafnit: Yes, it looks like this is the case, but if more things will be discovered and published, I think that they will have eventually an organized process for that. Because I guess the researchers do find things over there, but they're not always being published for several reasons. But yes, they should work on that. [laugh].Corey: And that is part of the challenge as well, where AWS does not have a public vulnerability disclosure program. [unintelligible 00:13:30] hacker one, they don't have a public bug bounty program. They have a vulnerability disclosure email address, and the people working behind that are some of the hardest working folks in tech, but there is no unified way of building a community of researchers around the idea of exploring this. And that is a challenge because you have reported vulnerabilities, I have reported significantly fewer vulnerabilities, but it always feels like it's a hurry up and wait scenario where the communication is not always immediate and clear. And at best, it feels like we often get a begrudging, “Thank you.”Versus all right, if we just throw ethics completely out the window and decide instead that now we're going to wind up focusing on just effectively selling it to the highest bidder, the value of, for example, a hypervisor escape on EC2 for example, is incalculable. There is no amount of money that a bug bounty program could offer for something like that compared to what it is worth to the right bad actor at the right time. So, the vulnerabilities that we hear about are already we're starting from a basis of people who have a functioning sense of ethics, people who are not deeply compromised trying to do something truly nefarious. What worries me is the story of—what are the stories that we aren't seeing? What are the things that are being found where instead of fighting against the bureaucracy around disclosure and the rest, people just use them for their own ends? And I'm gratified by the level of response I see from AWS on the things that they do find out about, but I always have to wonder, what aren't we seeing?Gafnit: That's a good question. And it really depends on their side if they choose to expose it or not.Corey: Part of the challenge too, is the messaging and the communication around it and who gets credit and the rest. And it's weird, whenever they release some additional feature to one of their big headline services, there are blog posts, there are keynote speeches, there are customer references, they go on speaking tours, and the emails, oh, God, they never stopped the emails talking about how amazing all of these things are. But whenever there's a security vulnerability or a disclosure like this—and to be fair, AWS's response to this speaks very well of them—it's like you have to go sneak down into the dark sub-basement, with the filing cabinet behind the leopard sign and the rest, to even find out that these things exist. And I feel like they're not doing themselves any favors by developing that reputation for lack of transparency around these things. “Well, while there was no customer impact, so why would we talk about it?”Because otherwise, you're setting up a myth that there never is a vulnerability on the side of—what is it that you're building as a cloud provider. And when there is a problem down the road—because there always is going to be; nothing is perfect—people are going to say, “Hey, wait a minute. You didn't talk about this. What else haven't you talked about?”And it rebounds on them with sometimes really unfortunate side effects. With Azure as a counterexample here, we see a number of Azure exploits where, “Yeah, turned out that we had access to other customers' data and Azure had no idea until we told them.” And Azure does it statements about, “Oh, we have no evidence of any of this stuff being used improperly.” Okay, that can mean that you've either check your logs and things are great or you don't have logging. I don't know that necessarily is something I trust.Conversely, AWS has said in the past, “We have looked at the audit logs for this service dating back to its launch years ago, and have validated that none of that has never been used like this.” One of those responses breeds an awful lot of customer trust. The other one doesn't. And I just wish AWS knew a little bit more how good crisis communication around vulnerabilities can improve customer trust rather than erode it.Gafnit: Yes, and I think that, as you said, there will always be vulnerabilities. And I think that we are expecting to find more, so being able to communicate as clearly as you can and to expose things about maybe the fakes and how the investigation is being done, even in a high level, for all the vulnerabilities can gain more trust from the customer side.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: You have experience in your background specifically around application security and cloud security research. You've been doing this for seven years at this point. When you started looking into this, did you come at the RDS vulnerability exploration from a perspective of being deeper on the Postgres side or deeper on the AWS side of things?Gafnit: So, it was both. I actually came to the RDS lead from another service where there was something [about 00:18:21] in the application level. But then I reached to an RDS and thought, well, it will be really nice to find thing over here and to reach the underlying machine. And when I entered to the RDS zone, I started to look at it from the application security eyes, but you have to know the cloud as well because there are integrations with S3, you need to understand the IAM model. So, you need a mix of both to exploit specifically this kind of issue. But you can also be database experts because the payload is a pure SQL.Corey: It always seems to me that this is an inherent risk in trying to take something that is pre-existing is an open-source solution—Postgres is one example but there are many more—and offer it as a managed service. Because I think one of the big misunderstandings is that when—well, AWS is just going to take something like Redis and offer that as a managed service, it's okay, I accept that they will offer a thing that respects the endpoints and then acts as if it were Redis, but under the hood, there is so much in all of these open-source projects that is built for optionality of wherever you want to run this thing, it will run there; whatever type of workload you want to throw at it, it can work. Whereas when you have a cloud provider converting these things into a managed service, they are going to strip out an awful lot of those things. An easy example might be okay, there's this thing that winds up having to calculate for the way the hard drives on a computer work and from a storage perspective.Well, all the big cloud providers already have interesting ways that they have solved storage. Every team does not reimplement that particular wheel; they use in-house services. Chubby's file locking, for example, over on Google side is a classic example of this that they've talked about an awful lot so every team building something doesn't have to rediscover all of that. So, the idea that, oh, we're just going to take up this open-source thing, clone it off a GitHub, fork it, and then just throw it into production as a managed service seems more than a little naive. What's your experience around seeing, as you get more [laugh] into the weeds of these things than most customers are allowed to get, what's your take on this?Do you find that this looks an awful lot like the open-source version that we all use? Or is it something that looks like it has been heavily customized to take advantage of what AWS is offering internally as underlying bedrock services?Gafnit: So, from what I saw until now, they do want to save the functionality so you will have the same experience as you're working with the same service that not on AWS because you're you are used to that. So, they are not doing dramatic changes, but they do want to reduce the risk in the security space. So, there will be some functionalities that they will not let you to do. And this is because of the managed party in areas where the full workload is deployed in your account and you can access it anyway, so they will not have the same security restrictions because you can access the workload anyway. But when it's managed, they need to prevent you from accessing the underlying host, for example. And they do the changes, but they're really picked to the specific actions that can lead you to that.Corey: It also feels like RDS is something of a, I don't want to call it a legacy service because it is clearly still very much actively developed, but it's what we'll call it a ‘classic service.' When I look at a new AWS launch, I tend to mentally bucket them into two things. There's the cloud-native approach, and we've already talked about DynamoDB. That would be one example of this. And there's the cloud-hosted model where you have to worry about things like instances and security groups and the networking stuff, and so on and so forth, where it's basically feels like they're running their thing on top of a pile of EC2 instances, and that abstraction starts leaking.Part of me wonders if looking at some of these older services like RDS, they made decisions in the design and build out of these things that they might not if they were to go ahead and build it out today. I mean, Aurora is an example of what that might look like. Have you found as you start looking around the various security foibles of different cloud services, that the security posture of some of the more cloud-native approaches is better or worse or the same as the cloud-hosted world?Gafnit: Well, so for example, in the several issues that were found, and also here in the RDS where you can see credentials in a file, this is not a best practice in security space. And so, definitely there are things to improve, even if it's developed on the provider side. But it's really hard to answer this question because in a managed area where you don't have any access, it's hard to tell how it's configured and if it's configured properly. So, you need to have some certification from their side.Corey: This is, on some level, part of the great security challenge, especially for something that is not itself open-source, where they obviously have terrific security teams, don't get me wrong. At no point do I want to ever come across a saying, “Oh, those AWS people don't know how security works.” That is provably untrue. But there is something to be said for the value of having a strong community in the security space focusing on this from the outside of looking at these things, of even helping other people contextualize these things. And I'm a little disheartened that none of the major cloud providers seem to have really embraced the idea of a cloud security community, to the point where the one that I'm most familiar with, the cloud security forum Slack team seems to be my default place where I go for context on things.Because I dabble. I keep my hand in when it comes to security, but I'm certainly no expert. That's what people like you are for. I make fun of clouds and I work on the billing parts of it and that's about as far as it goes for me. But being able to get context around is this a big deal? Is this description that a company is giving, is it accurate?For example, when your post came out, I had not heard of Lightspin in this context. So, reaching out to a few people I trusted, is this legitimate? The answer was, “Yes. It's legitimate and it's brilliant. That's a company that keep your eye on.” Great. That's useful context and there's no way to buy that. It has to come from having those conversations with people in the [broader 00:24:57] sense of the community. What's your experience been looking at the community side of the world of security?Gafnit: Well, so I think that the cloud security has a great community, and this is one of the things that we at Lightspin really want to increase and push forward. And we see ourselves as a security-driven company. We always do the best to publish a post, even detailed posts, not about vulnerabilities, about how things works in the cloud and how things are being evaluated, to release open-source tools where you can use them to check your environment even if you're not a customer. And I think that the community is always willing to explain and to investigate together. And it's a welcome effort, but I think that the messaging should be also for all layers, you know, also for the DevOps and the developers because it can really help if it will start from this point from their side, as well.Corey: It needs to be baked in, from start to finish.Gafnit: Yeah, exactly.Corey: I really want to thank you for taking the time out of your day to speak with me today. If people want to learn more about what you're up to, where's the best place for them to find you?Gafnit: So, you can find me on Twitter and on LinkedIn, and feel free to reach out.Corey: We will, of course, put links to that in the [show notes 00:26:25]. Thank you so much for being so generous with your time today. I appreciate it.Gafnit: Thank you, Corey.Corey: Gafnit Amiga, Director of Security Research at Lightspin. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, and if it's on the YouTubes, smash the like and subscribe buttons, which I'm told are there. Whereas if you've hated this podcast, same story, like and subscribe and the buttons, leave a five-star review on a various platform, but also leave an insulting, angry comment about how my observation that our IAM policies are all full of stars is inaccurate. And then I will go ahead and delete that comment later because you didn't set a strong password.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Full Description / Show Notes Marie talks about Oki Doki's primary product, Notion Mastery (2:38) Corey and Marie talk ADHD diagnosis and how it has impacted their lives and work (4:26) Marie and Corey discuss techniques they've developed for coping with ADHD (11:22) Corey and Marie talk about workarounds for people with ADHD who want to adopt something like Notion (16:13) Marie discusses the importance of being excited about the tools you're employing (18:54) Corey and Marie talk about finding tools that work for you (26:43) Marie and Corey discuss the unique challenge of teaching skills versus dumping knowledge (30:35) About Marie PoulinMarie teaches business owners to level up their digital systems, workflow, and knowledge management processes using Notion.She's the co-founder of Oki Doki and creator of Notion Mastery, an online program and community that helps creators, entrepreneurs and small teams tame their work + life chaos by building life and business management systems with Notion.Diagnosed with ADHD at age 37, Marie is especially passionate about helping folks customize their workflows and workspaces to meet their unique needs and preferences.She believes that Notion is especially powerful for neurodivergent folks who have long struggled to adhere to traditional or rigid project management processes, and may need a little extra customization and flexibility.When she's not tinkering in Notion or doing live trainings, you can find her in the garden, playing video games, or cooking up some delicious vegetarian tacos.Links Referenced: Oki Doki: https://weareokidoki.com/ Personal website: https://mariepoulin.com Notion Mastery: https://notionmastery.com Twitter: https://twitter.com/mariepoulin TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. that's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today I'm joined by Marie Poulin, the CEO of Oki Doki. Marie, thank you for joining me.Marie: Thank you for having me. I'm excited.Corey: So, let's start at the very beginning. What does Oki Doki do? And for folks listening that is O-K-I D-O-K-I, so you might want to have to think about that if you're doing the Google approach of, “What is this thing?”Marie: Well, at the moment, the majority of our products and services are surrounded by helping people learn how to use Notion to manage their life and business. So, it's only a pivot that we took in the last couple of years, and so our signature program is a course called Notion Mastery. So, there's four full-time employees now and that's what we do. We design live trainings, we have a forum, we have a curriculum. It's all products and services related to Notion.Corey: That is an interesting pivot that you can wind up going through. Please tell me I'm not the first person to make the observation that you called it Oki Doki and you've turned yourself around.Marie: [laugh]. You are the first, Corey? [laugh].Corey: Oh, good. I am broken like that, so that's kind of awesome. So, you've been more or less doing—I don't know the best way to frame this, so my apologies if I'm getting it wrong—but the idea of well, what are you selling? Knowledge. You're selling an understanding of how to improve things, you're selling a better outcome.And it's easy to look at that and say, “Oh, you're selling education.” No, you're selling understanding. Education is the way that you get there because at least at the moment, you can't just jack gigabytes of data directly into people's head without going to prison for it. Or raising a whole boatload of VC money.Marie: [laugh]. I mean, you can also say you're kind of selling an outcome, right? You're selling this future version of who someone wants to be. And so, we talk a lot about—you know, on our sales page, we get a lot of compliments on our sales page, but just speaking to the scattered mind, you know, feeling like a shitshow, feeling like you don't really have all your data in one place. You know, it's learning how to improve your workflow at work but also in life as well.And so, a lot of our language speaks to the sort of future version of yourself. Like, stop feeling scattered, stop feeling stretched thin. Let's actually get it so that you turn things into a well-oiled machine. So, you could say we're selling a dream. [laugh].Corey: This is an interesting direction to take this conversation in because I don't normally talk about this. But why not; we'll give it a shot. It's been sufficiently long since the last time. Last year—you've been very public about this—you were diagnosed with ADHD. I periodically talk about the fact that I was diagnosed with it myself—back when it was called ADD—when I was five years old.So, growing up I always knew that there was something neurodivergent about me. And the lesson I took away from this, as someone growing up with a lot of the limitations—yes, there are advantages but at the time, all I saw were limitations—about, “Well, what is ADHD?” It's like, oh, okay. They sat down and explained it to me. And it's not what they said, but it was, “See, this is the medical reason why you suck.”And that was not the most constructive way of framing it. In adulthood, talking to other people who have been diagnosed with this, especially later in life. There's a—it's a spectrum disorder. It winds up impacting an awful lot of people differently, but the universal experience that I hear is, wait, you mean there's a reason that I am the way that I am? It's not that I'm lazy. It's not that I'm shitty at things. It's not that I'm—Marie: Yeah.Corey: —careless. And that is one of those things that just is transformative. I didn't realize at the time how fortunate I was to be diagnosed that early on because trying to try to figure out why am I getting fired all the time? Why do I get bored doing the same thing too many days in a row, so I start causing problems for other people? What is going on with this? Why do I have this incredible opposition to anything that remotely resembles authority, et cetera, et cetera?Not all of this might be ADHD traits, but here I am. And my only solution after, you know, deciding that I didn't really want to set a world record for number of times getting fired was, well, I guess I'll start my own company because that at least to get fired, it's going to take some work. You figured this out while you were already self-employed.Marie: Yes.Corey: What was that like?Marie: What was it like to find out that I finally had an answer or reason for, maybe, past behaviors? [laugh].Corey: Right. Because it's the simultaneous, “Oh, my God, there's a reason that I am like I am,” and then followed immediately by, “I still am the way that I am. Huh. Okay.” It feels like it helps things, but it also doesn't help things. But it does, and it comes back around. What was your experience with it?Marie: Yeah, it started because I was doing research to understand my sister better because she had been diagnosed with ADHD for a couple years. It made so much sense once I kind of understood and started researching a little bit more about it. And then, of course, doing my deep-dive research. I'm hearing all these traits that I'm like, “Oh. Wait, that does really sound like me.” The not being able to wake—Corey: What do you [mean 00:07:01]—Marie: —up in the morning—Corey: ADHD trait? Everyone does that. Wait.Marie: [laugh]. Yeah. When you said that enough times, you're like, “Oh, wait. Maybe this is not normal.” Or you don't really know what is—what is normal anyway, right? So, in doing that research, trying to connect with her, trying to understand her experience better, I just started learning about more and more of these traits.I also knew a shit ton of people in our course, had mentioned that they had ADHD in their intake form, and I was like, what is it about people that ADHD that are actually drawn to my YouTube videos or my way of explaining things? And I started to learn a little bit more; it's quite common for folks with ADHD to be drawn to one another, probably because of our communication styles, even the sort of mild interrupting, or kind of the way we banter together. There's different styles of communicating that I think often folks with ADHD are maybe drawn to one another or have an easier time understanding one another. So, listening to some of these symptoms, I was like, “Wait a second.” Because my sister and I are so different in the way our symptoms present.I thought, “Well, that's what ADHD looks like.” It's pure unbridled chaos and unfiltered. And I just had this idea of what it looked like because she was one of the few examples that I had. Meanwhile, I'm skipping grades, I'm in the gifted program, I'm off, you know, doing my own thing. It looked very different.I thought, “Oh, people with ADHD don't thrive in university,” or whatnot. So, I had a lot of assumptions that I had to unpack. And I think the one, sort of, I don't know, symptom that kind of twinged something in my brain was extreme difficulty getting up in the morning and even sort of waking up your brain in the morning. This has been a problem with jobs, it's been a problem was school, getting to school on time, getting to work on time. Similar to you, it has caused job loss, it has caused tension with partners. They don't understand, like, why can't you get out of bed and seize the day?And I just thought, “There's something weird going on there with my body.” But I can be, you know, wide awake at 7 p.m. and I'm, like, ready to go. And I can hyperfocus for days on end. So, just noticing some of these symptoms and kind of unpacking it a bit, I thought, “Okay, there's something to go a little deeper in here.”Corey: I have trouble getting up, but I'm almost never late. That one does not hit me in quite the same way. In fact—Marie: Well—Corey: —my first consulting clients, and I'd been building—I was independent for two weeks at that point, and I was in an in-person meeting in San Francisco and one day, I showed up 20 minutes late, and he just stared at me. “You're never late. What's the deal here?” And it's like, “Yeah, I had trouble getting up this morning.” That was a lie.I was able to tell him about three or four months later, that morning, I found out I was going to be a father. And that was an—you know, it turns out that I was going to be okay being late, but it was so early, you didn't want to tell anyone, yet. But it was—yeah, it's one of those things where that was more important than—Marie: Absolutely.Corey: —doing the work thing. But I still remember, yeah, I feel like I'm always about to be late but apparently my reputation is, I never am, so okay. I'll take it. That is a—again, it is a spectrum disorder. I also—Marie: Absolutely.Corey: —further there want to call out for viewers, listeners, et cetera, a couple of things. One, this is not mental health advice. If any of the stories we're telling resonate, talk to a qualified mental health professional. Secondly, I want to be clear as well here, Marie, that you and I both have significant advantages when it comes to dealing with these things. We both run our own companies, we can effectively restructure the way that we work in ways that are more accommodating for what we do.It turns out that in my employment days, that was never really a solution where, “Yeah, I decided I'm not going to wind up doing the on-call checklist every day. It doesn't resonate with me.”Marie: “Just not feeling like it.”Corey: “It's doing the same thing too many days in a row. And yeah, I'm not going to check the backups, either. What do you mean ‘I'm fired?'” yeah, it turns out, you're not able to—you're empowered to make those kinds of sweeping changes in the same way.Marie: Exactly.Corey: So, this is not advice for people. This is simply a pair of experience reports, the way I view it.Marie: Absolutely. I sort of feel like self-employment wasn't necessarily a choice, in a way. It just felt like that's the only way I'm going to be able to operate in this world. I need some more sense of control and say in how I structure my days, how I structure my work, being able to switch things up, being able to pivot quickly. I knew that I was going to need more control over that. So yeah, pretty unemployable over here. [laugh].Corey: So, once you wound up with the diagnosis, what happened next? What changes did you make that wound up resonating for you, things that were actionable? And, yeah, you've been very public about it as well. I want to highlight that. I'm not, for the most part.And part of that is because I internalized growing up that it was somehow a shameful thing that we don't talk about. And the other part of it, too, on some level, was I didn't want to turn it into a part of my brand identity, where, “Oh, yeah, Corey is very hard to describe.” So, people thrash around and look for labels to slap on me. ‘Shitposter' seems to have stuck rather well. Because as soon as people feel that they have a label for something, it becomes easier to classify and then dismiss it.It's aspects of my personality. It's who I am. I don't think of it as a disorder so much as it is part and parcel of who and what I am. And it turns out that being me is not—yet—a medically recognized diagnosis. So, I'm cautious to avoid the labeling aspect of it.But you have very publicly not, if not going for the label, you at least embraced it as an aspect of who you are, and you've been very vocal about your experiences and telling people how you have overcome aspects of this. It's admirable. I wish I did more of it, honestly.Marie: I think it's kind of essential, I think, in the nature of what we're teaching. Like, when we're teaching people to become more organized and we know that executive dysfunction is one of the signs or, you know, issues with ADHD, to me it sort of recontextualized why I became so freakin' obsessed with systems and organization: because I never felt organized. I always felt the sense of what is the stuff come so easy to other people? Why is it taking me so much longer? Why am I spending nights, evenings, taking courses about systems like I'm trying to understand how to give my life structure?And so, in a way, the way I have become organized was trial by fire, just teaching myself, learning, you know, getting coaches. Like, I literally had a systems coach to teach myself how to get my business organized. So, I had kind of obsessed over it, like a hyperfocus. And so, realizing that other people are struggling with this and there's a reason that people with ADHD are coming to the course seeking that sense of control. And so, learning that I had it, I was like, oh, this actually [laugh] does explain, in a way, my obsession with this or my curiosity about this, of, like, why does this come easy to some other people? Why do some people need to study this and learn this? Like, what is it about that?And so, I sort of felt like it would be doing a disservice if I didn't kind of name it and talk about it and say, well, this actually colors a lot of my opinions. This actually influences the way I approach organization or even productivity, not from a timing perspective, but from an energy management perspective. I didn't realize that was something that I'm doing. I'm not managing time, we're managing Marie's energy. And even my team is learning how to do that, too.So, I was like, “Oh, that actually makes a ton of sense.” And it also makes sense why some people won't resonate with this energy management thing or might think I'm going way too far down a rabbit hole on something and they're like, “Why can't people just do what they say?” Like, you don't understand, some of us need to trick ourselves into being productive. And this is how I've learned to do that. So, it was just kind of a funny recontextualizing or uncovering, oh, our brains operate very differently. And even within ADHD, people's brains operate differently, so how do we get people moving toward progress, but knowing that we kind of need different ways of doing that. So, it's just been kind of an interesting process.Corey: There's a fairly common experience report from folks who have ADHD that when they're kids, their memory is generally very good with a number of expressions of it, so we form our self-image in a lot of those times. And then for the rest of our lives, we tell ourselves the same lie, regardless of how many times it has proven to be a lie. And that lie is, “I don't need to write this down. I'll remember it.”Marie: Oh yes.Corey: “No, Corey, you will not remember it. You need to write it down. I promise.” And, for example, right now—I finally gave in and technology leapt ahead to the point where my entire life is run by Google Calendar—specifically three or four of them—that all route through Fantastical—which is the app I use—but it winds up grabbing my attention at the right time. It tells me what I need to do, when, and how, and it's wonderful.Because if it's not on my calendar, it does not happen.Marie: Yes.Corey: Like, I will forget our anniversary, my kids' birthdays, to pick my children up from school. We are talking about, if it is not on my calendar, it does not happen. That is the one system that has been forced on me that worked. Then we—let's talk about Notion for a minute because I looked at it briefly a few years ago, and it is one in the long, long, long list of tools or approaches or systems that I have played with and then discarded to act as basically an auxiliary brain pack. I used Evernote for a while and that sort of worked because I just would do different notes all the time and I'd wind up with 3000 of those things, and then the app gets bloaty and I move on to something else.For the last five years or so I've been using Drafts, a Mac slash iOS app, that only does text, which makes image management and attaching things kind of hard, but okay. And that's great, and now I have 5000 of those in my [back 00:16:25] folder, not categorized or organized anyway, so I focus instead on well, search for terms and hope I use the term I thought I did at the time. And so, every time I've tried to use something like Notion, it's yeah, this requires a way of thinking that I know I will get excited about if I look at it, and in a month, I'll be right back to where I am now. So, there's only so many times you go on the same ride before you know how it ends. How do y—like, that feels like a very common experience. How did you fix it?Marie: I think at the core though, you kind of have to be excited about the tool that you're using. And so, I don't think—Notion is not going to be an exciting fun tool for everyone. Some people are going to be like, “I don't want to frickin' build my productivity system. Are you kidding me? Like, just give me something that works out of the box.” Absolutely.But I think there's something about the visual components of Notion. Like, I am a designer; I went to design school. I think I'm—it's almost like something doesn't click until I see it in the way that I need to see it. And that's something I've learned about my brain is just, sometimes the same information can be presented to me, but if it's not in a visual way, or whether it's not spaced in the right way, my brain just kind of ignores it or it gets overwhelmed by it. And so, for me that visual aspect actually helps me learn.I'm priming my brain, I'm making my goals front and center. The fact that I can design it the way I need my brain to see it is part of its appeal to me. But I also recognize that's not something everyone gets excited about. They're not drawn to it. I'm all for using the tool that works the way that your brain is going to work.I get excited about making databases. I get excited about building glossaries of information to help me learn things. Like, for me, that's part of my learning and part of my process and it's just kind of what I'm used to, but I fully acknowledge, like, that stuff does not get everybody excited.[midroll 00:18:03]Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: There's something very key you're talking about here, which is the idea of having to be excited about what it is that you do. I look at the things that I do professionally, and if I didn't deeply enjoy them, they would not get done, and I would have pivoted long ago to something else. People wonder why—Marie: Absolutely.Corey: —I make fun of so many things in the tech ecosystem. The honest answer is because if I just tell the dry, boring version of it, I will get bored because it's a fairly boring field. Whereas instead, okay, someone releases a new thing. Great. How do I keep it interesting for me? How do I find a way to tell that story?How do I find a way to, in turn, build that into something that, in turn, I can start dragging in different directions and opening up to new ways of talking without going too far? It's always a razor's edge, it's always a bit of a mind puzzle, and it's always different. I love that. That's why I do it. It's not for the audience so much as it is for myself. Because if I'm not engaged, no one else is going to care what I have to say.Marie: Absolutely. And I think that's a huge part of ADHD as well which is that interest-based nervous system, right? It's like we have to [laugh] trick ourselves into finding the excitement in it or whatever that looks like for each of us. But just if I'm not motivated, if I'm not excited about it—writing email newsletters doesn't get me excited; I'm like, “Okay, do I need to hire someone to do this?” Or how can I find a way to do it, whether it's—if making a video is more fun or easy, great.How can I, you know, make content do double-duty in that way? So yeah, I'm always trying to find ways to incentivize myself to do the things that need to get done, even though they may not be the most exciting. But step one is actually run a business that is based on something that you love doing. Which not everyone, maybe, has the privilege to do, but I think everything about the way I've designed my business model and the services that we offer is, don't offer services you don't really want to offer. Don't make products that you don't want to maintain, you're not excited about. So, it's definitely a core part of kind of how we design our whole business model.Corey: For me, a big part of it has always been just trying to make sure that I'm doing the things that engage me. And this is where that whole idea of being in a very privileged position enters into it. Take this podcast slash video right now, as a terrific example. I'm having this conversation, I have an entire system when I wind up sending a link to someone, it fires off Calendly, that hides webhooks and gets a whole bunch of other things set up. I show up, we have a conversation before the show to figure out just this is the general ebb and flow of the show. Here's the generalized topics we want to talk about. Let's dive in.And we finish the recording session. Great, I wind up closing the window and that's the last time I generally think about it. Because everything else has been automated. If anything other than me having this conversation with you does not need to be me, I there is no differentiated value in me being the person that does the audio engineering. It turns out, I can pay people who are world's better than I am at that, who actually enjoy it as opposed to viewing it as unnecessary chore, and I can do things that I find more appealing, like shitposting about a $1.108 trillion—Marie: Exactly.Corey: Company. It comes down to find the thing, the differentiation point, and find ways to make sure you don't have to do the other parts of it. But that is not a path that's available to everyone in every context. And again, I'm talking about this in a professional sense. I still have to do a whole bunch of stuff as I go through the course of my life that is not differentiated, but I can't very well hire someone to get me dressed in the morning. Well, I can but I feel like that becomes a little bit out of the scope of the lived human experience most of the [crosstalk 00:22:29].Marie: [laugh]. Absolutely. I feel like that's one thing I sort of regret not doing earlier is hiring someone to work with. So, the very first hire that I made was my chief of operations, and oh my gosh, the things that she took on that I used to do that I'm like, how on earth did I do that before? Because now that you do that, and you do it way faster, I just got to wonder, like, how the heck did I ever convince myself to do those activities?I don't want to do touch spreadsheets, I don't want to [laugh] deal with that stuff. I don't want to, you know, email reminders, or whatever it is. There's so many activities that she handles that I just… I would be happy to never touch again. And so, I sort of wish I had explored that earlier, but I was in that lone wolf, like, I got this. I'm going to run my own business solo forever.And, you know, I just sort of thought it's difficult to work with me or because of the way that I work, I don't know how to delegate. Like, it's all in your head. I just didn't really know how to do that. So, that process, I think, takes a while. That first hire when you're going from solo person to okay, now we're two; how do we work together? Okay, who else can we hire? What other activities can I get other people to do? So, that's been a process, for sure.Corey: Mike Julian, my business partner who you know, is a very process-driven person. He is very organized. His love language is Microsoft Excel, as I frequently tease him with. And one of the—not the only factor by a landslide, but one of the big early factors of what would—okay, I know what I'd do. What would Mike do here?Part of it is the never-ending litany of mail I get from the state around things like taxes, business registration, the rest. And normally my response when I get those, is I look at it, and it's like, “Welp, I'm going to fucking prison. That's the end of it. The end.” Because it's not that I don't have the money to pay my taxes, I assure you. What, I don't have it—because I—financial planning is kind of part and parcel of how we think about cloud economics.But no, it's the fact that I'm not going to sit there, fill out the form, put a stamp on it—or God forbid, fax it somewhere—and the rest. It's not the paying of the taxes that bothers me it is the paperwork and the process and the heavy lift associated with getting the executive function necessary to do it. So, it never gets done and deadlines slide by. And Mike was good at that for a time, and then he took the more reasonable approach about this of, “Huh. Seems to me like a lot of this stuff is not differentiated value that I need to be doing either.”So, we have a CFO who handles a lot of that stuff now and other operational folks. And it turns out that yeah, wow, there's a lot—I can—the quality of what I put out is a lot better because I get to focus on things instead of having to deal with the ebb and flow minutia of running payroll myself every week.Marie: Oh, yeah. All of that is very relatable. And this is why I can't do paper in the office. I think this is why I just moved my entire brain online. It's like if there's paper, stamps, anything related to having to go [laugh] to a post office to mail something. I think I still have the stack of thank you cards from our wedding from, you know, five years ago. So, yeah. [laugh].Corey: That you haven't sent out yet. Of course.Marie: Yes, exactly.Corey: Exact same—sorry, people 13—11 years ago, whenever it was.Marie: I'm so sorry.Corey: Yeah, one of these years. Yeah, and see, that's exactly how I treat things like Drafts or Notion, if I were to use it, or something else is great, it's still going to be the digital equivalent of a giant pile of paper. The thing is that computers can search through the contents of that paper a hell of a lot faster than I can, even with my own, at times, uncanny reading speed. There's some value to that. So, understanding how the systems work and having them bend to accommodate you, rather than trying to fool yourself in half to work within the confines of an existing system, that seems to be the direction that you're taking Notion in, specifically in the context of it is not prescriptive.And, on some level, that's kind of the problem I have with it. Whenever I try the getting started for us, it's, “Great, you can build your own system.” It's like, “Isn't that your job? What am I missing here?” Because the scariest thing I ever see when it's time for you to write a blog post or whatnot is an empty editor. It's, where do I get started? Where's the rest?I even built a template that I wind up sometimes using text expander to autofill, that gets me started. And it's just get—once I get started, it's great. It's hard to get me started; it's hard to get me to stop, in case no one has been aware of that. But it's been understanding how I work and how that integrates with it. I'm curious, given that you do talk to people who are trying to build these systems for a living for themselves? How common is my perspective on this? Am I out there completely, this unique, beautiful Snowflake? Is it yeah, that's basically everyone? Or somewhere in between?Marie: Oh, I definitely don't think you're alone with that. And again, I often will dissuade people from taking on Notion. I'm like, “Oh, if you're just looking for a note-taker, or you're just looking for something else,” or, “Your tools are already working for you, great. Keep using them.” So, I think it's quite common. I don't think Notion is the right tool for everyone.I think it's great for very visual people like myself, people that it matters how you are seeing your information, and how much information you're seeing, and you want more control over that, that's great. For me, I like the integration. I know that as soon as I'm bouncing around to different tools, like, I just already feel kind of scattered, so I was like, how can I pull everything that I need into these, sort of, singular dashboards. So, my approach is very dashboard-focused. Okay, Marie is going into content mode, it's time to write a blog. Go to the content hub. On the content hub is your list of most recent ideas, your templates for how to write a blog post. There's resources for creating video. It's already there for me; I'm not having to start from scratch like you said.But again, it took time to build that up for myself. So, I think you're not alone, and I think some people get excited about that building process; other people get irritated by it, and I don't think there's a right or wrong answer. It's just how do our brains work? Know thyself. And, yeah, I've sort of—I think also in a way, something that's a little different, maybe, about the way that I use Notion is I think of it as a personal development tool.It is a tool for making me better in different ways. It's for exploring my interests, it's for feeding my curiosity, it's for looking at change over time. I track my feelings every day. I've been journaling for 1300 days in a row, which is probably the only thing I've done consistently in my life [laugh] in the last couple of years. But now I can look and I can see trends over time in a really beautiful and visual way. And I just, to me, it's like a curiosity tool, to see, like, where am I going? Where have I been? What do I want more of?Corey: I need to look into this a bit more because my idea of a well-designed user interface is—I'm very opinionated on this—but it comes down to the idea of where do you use nouns versus verbs in command-line arguments to things you're running in the terminal. Because I was a grumpy Unix sysadmin for the first part of my career—because there's no other kind of Unix sysadmin—and going down that path was great. Okay, everything I'm interacting with is basically a text file piped together to do different things. And it took a while for me to realize, you know, maybe—just spitballing here—there's a better way to convey information than a wall of text, sometimes. Blasphemy.And no, no, it turns out that just because it's hard using the tools I'm used to doesn't mean that's the best way to convey information. And even now, these days, I'm spending more time getting the color theme and the font choices and typeface choices of what I'm doing in the terminal to represent something that's a bit more aesthetically pleasing. Does it actually account for anything? I don't know, but it feels better and there's almost a Feng Shui element of it. Similar to work in a—Marie: Yes.Corey: Clean office versus a messy one.Marie: A hundred percent. I think that's kind of how I think of an approach. I am much more likely to get the things done. If, when I come in and I open Notion, it's like, “Here's what's on today, Marie.” And it's like speaking nicely to me, there's little positive messages, there's beautiful imagery.It just makes me feel good when I'm starting my day. And knowing that how I feel is going to very much influence what I'm likely to accomplish in the day, again, I'm constantly tricking myself into getting [laugh] more excited and amped up about what's on the schedule for the day. So, I really liked that about it. It feels beautiful to me.Corey: I'm going to have to take another look at it at some point. I think that there's a lot of interesting directions to go into on this. I also have the privilege of having known you for a little while, back when you were more or less just getting started. One of the things that you said at the time that absolutely resonated with me was the idea of, wait, you mean build a business around teaching people how to use Notion? Like an info product or a training approach?And a lot of your concerns are the ones that I've harbored for a while, too, which is the idea of there's a proliferation of info products in technical and other spaces, and an awful lot of them—without naming any names or talking in any particular direction—are not the highest quality. People are building these courses while learning the thing themselves. And when they tell stories about it, it's all about, “And this is how I'm making money quickly.” I don't find that admirable; I don't necessarily want to learn how to do a thing from someone who does not have themselves at least a decent understanding themselves of what they're working on so they can address questions that go a bit off into the weeds. And so mu—again, knowing how to do a thing and knowing how to teach a thing are orthogonal concepts. And very often a lot of these info products are being created by people who don't really know how to do either, as best I can tell.Marie: Yes. So, I think you've nailed a point to that, knowing a thing deeply and then knowing how to teach that thing really well are two totally different skills. And I definitely bumped up against that myself. I'm like, I know, Notion inside and out. Like, you know, name something, I can make it, I can optimize it, I can, you know, build a system out of thin air really fast, no problem. I'm a problem solver that way.But to teach someone else how to do that requires very different skills. And I knew [laugh] as I was starting to teach people stuff, I'm like, “You could do this. You could do that.” And I'm like kind of bouncing around and I'm all over the place because I'm so excited about the possibilities. But wait a second.Beginners that are just learning how to use Notion don't need to know every frickin' possible way that you could use it. So, knowing that instructional design, curriculum design is a whole other skill, and I care about student results, it's like, this is a gap that I have, and I want to be an excellent teacher. It matters to me. I actually do want to become a better teacher. I want to have higher quality YouTube videos, I want to make sure that I'm not losing people along the way.I don't just care about making a shit ton of money with an info product; I care about peoples' experience and kind of having that, I don't know, that prestige element. Like, that's something that does matter in terms of producing quality products. So, I hired experts to help me do that because again, it's a not necessarily a strength of mine. So, I think I hired three different people in the course of six months to various consultants and people who understand learning design and that sort of thing. And I think that's something a lot of info product creators. They think of it as just packaging a blog and selling it, right?It's different. When you're teaching a course, for example, your formatting matters, how you display information matters, how you design activities matters. What separates a course from a passive income product or blog, right? We need to think about those things, and I think a lot of people are just like, what's the quickest, you know, buck that I can make on these products and just kind of turn them out. And I don't think every course creator has maybe done the extra legwork to really understand what makes students actually follow through and complete a course. It's hard. It's really hard.Corey: And these are also very different products. There's what you are teaching, which is here's how to contextualize these things and how to build a system around it. There's another offering out there that would be something that would also be very compelling from my perspective where, cool, I appreciate the understanding and the deep systems design approach that goes into this. Can I just give you a brain dump of all the problems that I have with this? You go away and build a system that accounts for all of that.And again, it's the outcome that I care about. There's this belief that oh we want consultants to build by the hour and work hard. No. I don't care. If you listen to this, nod and do the great customer service thing, the Zoom call, and just like, “Okay, that's template number three with three one-line changes. Done. Now, we're going to sit on it for a week so it looks hard.”Which we've all got that as consultants in the early days. And then you turn that around because it's the outcome that I really care about. But that's a different business, that is a different revenue model, that is different—Marie: Yes.Corey: That is not nearly so much a one-to-many, like an info product. That is a one-to-one or one-to-few.Marie: And I did that for the whole first year that the course was being developed and was out there. I was simultaneously consulting with people one-on-one all the time, with teams, with individuals. So, I'm learning about what are all those common challenges that keep popping up over and over again? What are the unique challenges? What are the common ones?And in my experience, what I bumped up against is people think they want to just pay someone to solve that, but then when you give someone a very fleshed out, organized system that they didn't participate in the building, it's a lot harder to get somebody to use it, to plug into a ready-made system. So, in our experience, there's a sort of back and forth. It has to happen in tandem; we do it over time. And you know, in my partner's case, Ben does consulting with companies as well, so he'll meet with them on a weekly basis and working with the different members of the team. So, there is some element of we built you a thing. Let's have you use it, notice where there's gaps, friction, whatever, because it's not a one-and-done process.It's not like, “You gave me all the info. We're good to go.” It's not until people are using it that you're like, “Oh, okay, that's close, but I'm finding myself doing this, or avoiding this, or clicking around too much.” And so, to me, it's a really organic process. But that's not something that I'm as keen to do. And maybe it's because I did it for, like, two years and kind of burnt out on it. I'm like, “I'm done. Like, I'd rather teach folks to do it themselves.” But so a partner does the consulting; I'm doing more of the teaching.Corey: That's what happened to an awful lot of our consulting work here at The Duckbill Group where it was exciting and fun for me for years, and at some point it turned into, I am interested in teaching how to do this a little bit more and systematizing it because I'm starting to get bored with aspects of it. And I was thinking, “Well, do I build a course?” It's, “Well, no. As it turns out that if you have the right starting point, I can hire people who I can teach how to do AWS bill analysis if they have the right starting point.” And it turns out that a lot of those people—read as all of them—are going to be way better at doing the systemic deep-dive across the board, rather than just finding the things that they find personally interesting and significant, and then, “Well, there you go. I did a consulting engagement.” And the output is basically three bullet points scrawled on the back of an envelope.Yeah, turns out that that's not quite the level of professionalism clients expect. Great, so our product is better, we're getting better insight into it, and I get to scratch my itch of teaching people how to do things internally without becoming a critical path blocker.Marie: Yeah, absolutely.Corey: I mean, I have shitposting to get back to. Come on.Marie: Yeah exactly. [laugh]. The important things. Love it.Corey: I really want to thank you for taking so much time to speak with me about all of these things. If people want to learn more—Marie: Absolutely.Corey: —where's the best place to find you?Marie: Yeah, you can find me at mariepoulin.com is where my personal blog, or weareokidoki.com, or notionmastery.com. You can also catch me on Twitter.Corey: And we will put links to—Marie: That's where I am most active. Yeah.Corey: Oh, of course. And all the links wind up going into the [show notes 00:37:42], as always. Thank you so much for your time. I appreciate it.Marie: Thanks for having me, Corey. It was awesome.Corey: Marie Poulin, CEO of Oki Doki. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice—and if it's on the YouTubes smash the like and subscribe buttons—whereas if you've hated this podcast episode, great, same thing, five-star review on whatever platform, smash the two buttons, but also leave an insulting comment and then turn that comment into an info product that you wind up selling to a whole bunch of people primarily to boost your own Twitter threads about how successful you are as a creator.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Full Description / Show Notes Guillermo talks about how he came to work at OCI and what it was like helping to pioneer Oracle's cloud product (1:40) Corey and Guillermo discuss the challenges and realities of multi-cloud (6:00) Corey asks about OCI's dedicated region approach (8:27) Guillermo discusses the problem of awareness (12:40) Corey and Guillermo talk cloud providers and cloud migration (14:40) Guillermo shares about how OCI's cost and customer service is unique among cloud providers (16:56) Corey and Guillermo talk about IoT services and 5G (23:58) About Guillermo RuizGuillermo Ruiz gets into trouble more often than he would like. During his career Guillermo has seen many horror stories while building data centers worldwide. In 2007 he dreamed with space-based internet and direct routing between satellites, but he could only reach “the Cloud”. And there he is, helping customer build their business in someone else servers since 2011.Beware of his sense of humor...If you ever see him in a tech event, run, he will get you in problems.Links: Twitter: https://twitter.com/IaaSgeek, https://twitter.com/OracleStartup LinkedIn: https://www.linkedin.com/in/gruizesteban/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I've been meaning to get a number of folks on this show for a while and today is absolutely one of those episodes. I'm joined by Guillermo Ruiz who is the Director of OCI Developer Evangelism, slash the Director of Oracle for Startups. Guillermo, thank you for joining me, and is Oracle for Startups an oxymoron because it kind of feels like it in some weird way, in the fullness of time.Guillermo: [laugh]. Thanks, Corey. It's a pleasure being in your show.Corey: Well, thank you. I enjoy having you here. I've been trying to get you on for a while. I'm glad I finally wore you down.Guillermo: [laugh]. Thanks. As I said, well, startup, I think, is the future of the industry, so it's a fundamental piece of our building blocks for the next generation of services.Corey: I have to say that I know that you folks at Oracle Cloud have been a recurring sponsor of the show. Thank you for that, incidentally. This is not a promoted guest episode. I invited you on because I wanted to talk to you about these things, which means that I can say more or less whatever I damn well want. And my experience with Oracle Cloud has been one of constantly being surprised since I started using it a few years ago, long before I was even taking sponsorships for this show. It was, “Oh, Oracle has a cloud. This ought to be rich.”And I started kicking the tires on it and I came away consistently and repeatedly impressed by the technical qualities the platform has. The always-free tier has a model of cloud economics that great. I have a sizable VM running there and have for years and it's never charged me a dime. Your data egress fees aren't, you know, a 10th of what a lot of the other cloud providers are charging, also known as, you know, you're charging in the bounds of reality; good for that. And the platform continues to—although it is different from other cloud providers, in some respects, it continues to impress.Honestly, I keep saying one of the worst problems that has is the word Oracle at the front of it because Oracle has a 40-some-odd-year history of big enterprise systems, being stodgy, being difficult to work with, all the things you don't generally tend to think of in terms of cloud. It really is a head turn. How did that happen? And how did you get dragged into the mess?Guillermo: Well, this came, like, back in five, six years ago, when they started building this whole thing, they picked people that were used to build cloud services from different hyperscalers. They dropped them into a single box in Seattle. And it's like, “Guys, knowing what you know, how you would build the next generation cloud platform?” And the guys came up with OCI, which was a second generation. And when I got hired by Oracle, they showed me the first one, that classic.It was totally bullshit. It was like, “Guys, there's no key differentiator with what's there in the market.” I didn't even know Oracle had a cloud, and I've been in this space since late-2010. And I had to sign, like, a bunch of NDAs a lot of papers, and they show me what they were cooking in the oven, and oh my gosh, when I saw that SDN out of the box directly in the physical network, CPUs assign, it was [BLEEP] [unintelligible 00:03:45]. It was, like, bare metal. I saw that the future was there. And I think that they built the right solution, so I joined the company to help them leverage the cloud platform.Corey: The thing that continually surprises me is that, “Oh, we have a cloud.” It has a real, “Hello fellow kids,” energy. Yes, yeah, so does IBM; we've seen how that played out. But the more I use it, the more impressed I am. Early on in the serverless function days, you folks more or less acquired Iron.io, and you were streets ahead as far as a lot of the event-driven serverless function style of thing tended to go.And one of the challenges that I see in the story that's being told about Oracle Cloud is, the big enterprise customer wins. These are the typical global Fortune 2000s, who have been around for, you know—which is weird for those of us in San Francisco, but apparently, these companies have been around longer than 18 months and they've built for platforms that are not the latest model MacBook Pro running the current version of Chrome. What is that? What is that legacy piece of garbage? What does it do? It's like, “Oh, it does about $4 billion a quarter so maybe show some respect.”It's the idea of companies that are doing real-world things, and they absolutely have cloud power. Problems and needs that are being met by a variety of different companies. It's easy to look at that narrative and overlook the fact that you could come up with some ridiculous Twitter for Pets-style business idea and build it on top of Oracle Cloud and I would not, at this point, call that a poor decision. I'm not even sure how it got there, and I wish that story was being told a little bit better. Given that you are a developer evangelist focusing specifically on startups and run that org, how do you see it?Guillermo: Well, the thing here is, you mentioned, you know, about Oracle, many startup doesn't even know we have a cloud provider. So, many of the question comes is like, how we can help on your business. It's more on the experience, you know, what are the challenges, the gaps, and we go in and identify and try to use our cloud. And even though if I'm not able to fill that gap, that's why we have this partnership with Microsoft. It's the first time to cloud providers connect both clouds directly without no third party in between, router to router.It's like, let's leverage the best of these clouds together. I'm a truly believer of multi-cloud. Non-single cloud is perfect. We are evolving, we're getting better, we are adding services. I don't want to get to 500 services like other guys do. It's like, just have a set of things that really works and works really, really well.Corey: Until you have 40 distinct managed database services and 80 ways to run containers, are you're really a full cloud provider? I mean, there's always that question that, at some point, the database Java, the future is going to have to be disambiguating between all the different managed database services on a per workload basis, and that job sounds terrible. I can't let the multi-cloud advocacy pass unchallenged here because I'm often misunderstood on this, and if I don't say something, I will get emails, and nobody wants that. I think that the idea of building a workload with the idea that it can flow seamlessly between cloud providers is a ridiculous fantasy that basically no one achieves. The number of workloads that can do that are very small.That said, the idea of independent workloads living on different cloud providers as is the best fit for placement for those is not just a good idea, it is the—whether it's a good idea or not as irrelevant because that's the reality in which we all live now. That is the world we have to deal with.Guillermo: If you want distributed system, obviously you need to have multiple cloud providers in your strategy. How you federate things—if you go down to the Kubernetes side, how you federate multi-clusters and stuff, that's a challenge out there where people have. But you mentioned that having multiple apps and things, we have customers that they've been running Google Cloud, for example, and we build [unintelligible 00:07:40] that cloud service out there. And the thing is that when they run the network throughput and the performance test, they were like, “Damn, this is even better than what I have in my data center.” It's like, “Guys, because we are room by room.” It's here is Google, here it's Oracle; we land in the same data center, we can provide better connectivity that what you even have.So, that kind of perception is not well seen in some customers because they realize that they're two separate clouds, but the reality is that most of us have our infrastructure in the same providers.Corey: It's kind of interesting, just to look at the way that the industry is misunderstanding a lot of these things. When you folks came out with your cloud at customer initiatives—the one that jumps out to my mind is the dedicated region approach—a lot of people started making fun of that because, “What is this nonsense? You're saying that you can deploy a region of your cloud on site at the customer with all of the cloud services? That's ridiculous. You folks don't understand cloud.”My rejoinder to that is people saying that don't understand customers. You take a look at for example… AWS has their Outpost which is a rack or racks with a subset of services in them. And that, from their perspective, as best I can tell, solves the real problem that customers have, which is running virtual machines on-premises that do not somehow charge an hourly cost back to AWS—I digress—but it does bring a lot of those services closer to customers. You bring all of your services closer to customers and the fact that is a feasible thing is intensely appealing to a wide variety of customer types. Rather than waiting for you to build a region in a certain geographic area that conforms with some regulatory data requirement, “Well, cool, we can ship some racks. Does that work for you?” It really is a game-changer in a whole bunch of respects and I don't think that the industry is paying close enough attention to just how valuable that is.Guillermo: Indeed. I've been at least hearing since 2010 that next year is the boom; now everybody will move into the cloud. It has been 12 years and still 75% of customers doesn't have their critical workloads in the cloud. They have developer environments, some little production stuff, but the core business is still relying in the data center. If I come and say, “Hey, what if I build this behind your firewall?”And it's not just that you have the whole thing. I'm removing all your operational expenses. Now, you don't need to think about hardware refresh, upgrade staff, just focus on your business. I think when we came up with a dedicated region, it was awesome. It was one of the best thing I've seen their Outpost is a great solution, to be honest, but if you lose the one connectivity, the control plane is still in the cloud.In our site, you have the control plane inside your data center so you can still operate and manage your services, even if there is an outage on your one site. One of the common questions we find on that area is, like, “Damn, this is great, but we would like to have a smaller size of this dedicated region.” Well, stay tuned because maybe we come with smaller versions of our dedicated regions so you guys can go and deploy whatever you need there.Corey: It turns out that, in the fullness of time, I like this computer but I want it to be smaller is generally a need that gets met super well. One thing that I've looked into recently has been the evolution of companies, in the fullness of time—which this is what completely renders me a terrible analyst in any traditional sense; I think more than one or two quarters ahead, and I look at these things—the average tenure of a company in the S&P 500 index is 21 years or so. Which means that if we take a look at what's going on 20 years or so from now in the 2040s, roughly half—give or take—of the constituency of the S&P 500 may very well not have been founded yet. So, when someone goes out and founds a company tomorrow as an idea that they're kicking around, let's be clear, with a couple of very distinct exceptions, they're going to build it on Cloud. There's a lot of reasons to do that until you hit certain inflection points.So, this idea that, oh, we're going to rent a rack, and we're going to go build some nonsense, and yadda, yadda, yadda. It's just, it's a fantasy. So, the question that I see for a lot of companies is the longtail legacy where if I take that startup and found it tomorrow and drive it all the way toward being a multinational, at what point did they become a customer for whatever these companies are selling? A lot of the big E enterprise vendors don't have a story for that, which tells me long-term, they have problems. Looking increasingly at what Oracle Cloud is doing, I have to level with you, I viewed Oracle as being very much in that slow-eroding dinosaur perspective until I started using the platform in some depth. I am increasingly of the mind that there's a bright future. I'm just not sure that has sunk into the industry's level of awareness these days.Guillermo: Yeah, I can agree with you in that sense. Mainly, I think we need to work on that awareness side. Because for example, if I go back to the other products we have in the company, you know, like the database, what the database team has done—and I'm not a database guy—and it's like, “Guys, even being an infrastructure guy, customers doesn't care about infrastructure. They just want to run their service, that it doesn't fail, you don't have a disruption; let me evolve my business.” But even though they came with this converged database, I was really impressed that you can do everything in a single-engine rather than having multiple database implemented. Now, you can use the MongoDB APIs.It's like, this is the key of success. When you remove the learning curve and the frictions for people to use your services. I'm a [unintelligible 00:13:23] guy and I always say, “Guys, click, click, click. In three clicks, I should have my service up and running.” I think that the world is moving so fast and we have so much information today, that's just 24 hours a day that I have to grab the right information. I don't have time to go and start learning something from scratch and taking a course of six months because results needs to be done in the next few weeks.Corey: One thing that I think that really reinforces this is—so as I mentioned before, I have a free tier account with you folks, have for years, whenever I log into the thing, I'm presented with the default dashboard view, which recommends a bunch of quickstarts. And none of the quickstarts that you folks are recommending to me involve step one, migrate your legacy data center or mainframe into the cloud. It's all stuff like using analytics to predict things with AI services, it's about observability, it's about governance of deploy a landing zone as you build these things out. Here's how to do a low-code app using Apex—which is awesome, let's be clear here—and even then launching resources is all about things that you would tend to expect of launch database, create a stack, spin up some VMs, et cetera. And that's about as far as it goes toward a legacy way of thinking.It is very clear that there is a story here, but it seems that all the cloud providers these days are chasing the migration story. But I have to say that with a few notable exceptions, the way that those companies move to cloud, it always starts off by looking like an extension of their data center. Which is fine. In that phase, they are improving their data center environment at the expense of being particularly cloudy, but I don't think that is necessarily an adoption model that puts any of these platforms—Oracle Cloud included—in their best light.Guillermo: Yeah, well, people was laughing to us, when we released Layer 2 in the network in the cloud. They were like, “Guys, you're taking the legacy to the cloud. It's like, you're lifting the shit and putting the shit up there.” Is like, “Guys, there are customers that cannot refactor and do anything there. They need to still run Layer 2 there. Why not giving people options?”That's my question is, like, there's no right answers to the cloud. You just need to ensure that you have the right options for people that they can choose and build their strategy around that.Corey: This has been a global problem where so many of these services get built and launched from all of the vendors that it becomes very unclear as a customer, is this thing for me or not? And honestly, sometimes one of the best ways to figure that out is to all right, what does it cost because that, it turns out, is going to tell me an awful lot. When it comes to the price tag of millions of dollars a year, this is probably not for my tiny startup. Whereas when it comes to a, oh, it's in the always free tier or it winds up costing pennies per hour, okay, this is absolutely something I want to wind up exploring and seeing what happens. And it becomes a really polished experience across the board.I also will say this is your generation two cloud—Gen 2, not to be confused with Gentoo, the Linux distribution for people with way more time on their hands than they have sense—and what I find interesting about it is, unlike a lot of the—please don't take this the wrong way—late-comers to cloud compared to the last 15 years of experience of Amazon being out in front of everyone, you didn't just look at what other providers have done and implement the exact same models, the exact same approaches to things. You've clearly gone in your own direction and that's leading to some really interesting places.Guillermo: Yeah, I think that doing what others are doing, you just follow the chain, no? That will never position you as a top number one out there. Being number one so many years in the cloud space as other cloud providers, sometimes you lose the perception of how to treat and speak to customers you know? It's like, “I'm the number one. Who cares if this guy is coming with me or not?” I think that there's more on the empathy side on how we treat customers and how we try to work and solve.For example, in the startup team, we find a lot of people that hasn't have infrastructure teams. We put for free our architects that will give you your GitHub or your GitLab account and we'll build the Terraform modules and give that for you. It's like now you can reuse it, spin up, modify whatever you want. Trying to make life easier for people so they can adopt and leverage their business in the cloud side, you know?[midroll 00:14:45]Corey: There's so much that we folks get right. Honestly, one of the best things that recommends this is the always free tier does exactly what it says on the tin. Yeah, sure. I don't get to use every edge case service that you've built across the board, but I've also had this thing since 2019, and never had to pay a penny for any of it, whereas recently—as we're recording this, it was a week or two ago—that I saw someone wondering what happened to their AWS account because over the past week, suddenly they went from not using SageMaker to being charged $270,000 on SageMaker. And it's… yeah, that's not the kind of thing that is going to endear the platform to frickin' anyone.And I can't believe I'm saying this, but the thing says Oracle on the front of it and I'm recommending it because it doesn't wind up surprising you with a bill later. It feels like I've woken up in bizarro world. But it's great.Guillermo: Yep. I think that's one of the clever things we've done on that side. We've built a very robust platform, really cool services. But it's key on how people can start learning and testing the flavors of your cloud. But not only what you have in the fleet here, you have also the Ampere instances.We're moving into a more sustainable world, and I think that having, like, the ARM architectures in the cloud and providing that on the free space of people can just go and develop on top, I think that was one of the great things we've done in the last year-and-a-half, something like that. Definitely a full fan of a free tier.Corey: You also, working over in the Developer Evangelist slash advocacy side of the world—devrelopers, as I tend to call it much to the irritation of basically everyone who works in developer relations—one of the things that I think is a challenge for you is that when I wind up trying to do something ridiculous—I don't know maybe it's a URL shortener; maybe it is build a small app that does something that's fairly generic—with a lot of the other platforms. There's a universe of blog posts out there, “Here's how I did it on this platform,” and then it's more or less you go to GitHub—or gif-UB, and I have mispronounced that too—and click the button and I wind up getting a deploy, whereas in things that are rapidly emerging with the Oracle Cloud space, it feels like, on some level, I wind up getting to be a bit of a trailblazer and figure some of these things out myself. That is diminishing. I'm starting to see more and more content around this stuff. I have to assume that is at least partially due to your organization's work.Guillermo: Oh, yeah, but things have changed. For example, we used to have our GitHub repository just as a software release, and we push to have that as a content management, you know, it's like, I always say that give—let people steal the code. You just put the example that will come with other ideas, other extensions, plug-in connectors, but you need to have something where you can start. So, we created this DevRel Quickstart that now is managed by the new DevRel organization where we try to put those examples. So, you just can go and put it.I've been working with the community on building, like, a content aggregator of how people is using our technology. We used to have ocigeek.com, that was a website with more than 1000 blog and, like, 500 visits a day looking after what other people were doing, but unfortunately, we had to, because of… the amount of X reasons we have to pull it off.But we want to come with something like that. I think that information should be available. I don't want people to think when it comes to my cloud is like, “Oh, how you use this product?” It's like no, guys how I can build with Angular, React the content management system? You will do it in my cloud because that example I'm doing, but I want you to learn the basics and the context of running Python and doing other things there rather than go into oh, no, this is something specific to me. No, no, that will never work.Corey: That was the big problem I found with doing a lot of the serverless stuff in years past where my first Lambda application took me two weeks to build because I'm terrible at programming. And now it takes me ten minutes to build because I'm terrible at programming and don't know what tests are. But the problem I ran into for that first one was, what is the integration format? What is the event structure? How do I wind up accessing that?What is the thing that I'm integrating with expecting because, “Mmm, that's not it; try again,” is a terrible error message. And so, much of it felt like it was the undifferentiated gluing things together. The only way to make that stuff work is good documentation and numerous examples that come at the problem from a bunch of different ways. And increasingly, Oracle's documentation is great.Guillermo: Yeah, well, in my view, for example, you have the Three-Tier Oracle. We should have a catalog of 100 things that you can do in the free tier, even though when I propose some of the articles, I was even talking about VMware, and people was like, “[unintelligible 00:22:34], you cannot deploy VMware.” It's like, “Yeah, but I can connect my [crosstalk 00:22:39]—”Corey: Well, not with that attitude.Guillermo: Yeah. And I was like, “Yeah, but I can connect to the cloud and just use it as a backup place where I can put my image and my stuff. Now, you're connecting to things: VMware with free tier.” Stuff like that. There are multiple things that you can do.And just having three blocks is things that you can do in the free tier, then having developer architectures. Show me how you can deploy an architecture directly from the command line, how I can run my DevOps service without going to the console, just purely using SDKs and stuff like that. And give me the option of how people is working and expanding that content and things there. If you put those three blocks together, I think you're done on how people can adopt and leverage your cloud. It's like, I want to learn; I don't want to know the basics of I don't know, it's—I'm not a database guy, so I don't understand those things and I don't want to go into details.I just they just need a database to store my profiles and my stuff so I can pick that and do computer vision. How I can pick and say, “Hey, I'm speaking with Corey Quinn and I have a drone flying here, he recommends your face and give me your background from all the different profiles.” That's the kind of solutions I want to build. But I don't want to be an expert on those areas.Corey: Because with all the pictures of me with my mouth open, you wouldn't be able to under—it would make no sense of me until I make that pose. There's method to—Guillermo: [laugh].Corey: —my insane madness over here.Guillermo: [laugh] [unintelligible 00:23:58].Corey: Yeah. But yeah, there's a lot of value as you move up the stack on these things. There's also something to be said, as well, for a direction that you folks have been moving in recently, that I—let me be fair here—I think it's clown shoes because I tend to think in terms of software because I have more or less the hardware destruction bunny level of aura when it comes to being near expensive things. And I look around the world and I don't have a whole lot of problems that I can legally solve with an army of robots.But there are customers who very much do. And that's why we see sort of the twin linking of things like IoT services and 5G, which when I first started seeing cloud providers talking about this, I thought was Looney Tunes. And you folks are getting into it too, so, “Oh, great. The hype wound up affecting you too.” And the thing that changed my mind was not anything cloud providers have to say—because let's be clear, everyone has an agenda they're trying to push for—but who doesn't have an agenda is the customers talking about these things and the neat things that they're able to achieve with it, at which point I stopped making fun, I shut up and listen in the hopes that I might learn something. How have you seen that whole 5G slash IoT slash internet of Nonsense space evolving?Guillermo: That's the future. That's what we're going to see in the next five years. I run some innovation sessions with a lot of customers and one of the main components I speak about is this area. With 5G, the number of IoT devices will exponentially grow. That means that you're going to have more data points, more data volume out there.How can you provide the real value, how you can classify, index, and provide the right information in just 24 hours, that's what people is looking. Things needs to be instant. If you say to the kids today, they cannot watch a football match, 90 minutes. If you don't get the answer in ten, they move to the next thing. That's how this society is moving [unintelligible 00:25:50].Having all these solutions from a data perspective, and I think that Oracle has a great advantage in that space because we've been doing that for 43 years, right? It's like, how we do the abstraction? How I can pick all that information and provide added value? We build the robot as a service. I can configure it from my browser, any robot anywhere in the world.And I can do it in Python, Java. I can [unintelligible 00:26:14] applications. Two weeks ago, we were testing on connecting IoT devices and flashing the firmware. And it was working. And this is something that we didn't do it alone. We did it with a startup.The guys came and had a sandbox already there, is like, “let's enable this on [unintelligible 00:26:28]. Let's start working together.” Now, I can go to my customers and provide them a solution that is like, hey, let's connect Boston Dynamics, or [unintelligible 00:26:37] Robotics. Let's start doing those things and take the benefits of using Oracle's AI and ML services. Pick that, let's do computer vision, natural language processing.Now, you're connecting what I say, an end-to-end solution that provides real value for customers. Connected cars, we turn our car into a wallet. I can go and pay on the petrol station without leaving my car. If I'm taking the kids to takeaway, I can just pay these kind of things is like, “Whoa, this is really cool.” But what if I [laugh] get that information for your insurance company.Next year, Corey, you will pay double because you're a crazy driver. And we know how you drive in the car because we have all that information in place. That's how the things will roll out in the next five to ten years. And [unintelligible 00:27:24] healthcare. We build something for emergencies that if you have a car crash, they have the guys that go and attend can have your blood type and some information about your car, where to cut the chassis and stuff when you get prisoner inside.And I got people saying, “Oh gee, GDPR because we are in Europe.” It's like, “Guys, if I'm going to die, I don't care if they have my information.” That's the point where people really need to balance the whole thing, right? Obviously, we protect the information and the whole thing, but in those situations is like hey, there's so many things we can do. There are countless opportunities out there.Corey: The way that I square that circle personally has always been it's about informed consent, when if people are given a choice, then an awful lot of those objections that people have seemed to melt away. Provided, of course, that is an actual choice and it's not one of those, “Well, you can either choose to”—quote-unquote—“Choose to do this, or you can pay $9,000 a month extra.” Which is, that's not really a choice. But as long as there's a reasonable way to get informed consent, I think that people don't particularly mind, I think it's when they wind up feeling that they have been spied upon without their knowledge, that's when everything tends to blow up. It turns out, if you tell people in advance what you're going to do with their information, they're a lot less upset. And I don't mean burying it deep and the terms and conditions.Guillermo: And that's a good example. We run a demo with one of our customers showing them how dangerous the public information you have out there. You usually sign and click and give rights to everybody. We found in Stack Overflow, there was a user that you just have the username there, nothing else. And we build a platform with six terabytes of information grabbing from Stack Overflow, LinkedIn, Twitter, and many other social media channels, and we show how we identify that this guy was living in Bangalore in India and was working for a specific company out there.So, people was like, “Damn, just having that name, you end up knowing that?” It's like there's so much information out there of value. And we've seen other companies doing that illegally in other places, you know, Cambridge Analytics and things like that. But that's the risk of giving your information for free out there.Corey: It's always a matter of trade-offs. There is no one-size-fits-all solution and honestly, if there were it feels like we wouldn't have cloud providers; we would just have the turnkey solution that gives the same thing that everyone needs and calls it good. I dream of such a day, but it turns out that customers are different, people are different, and there's no escaping that.Guillermo: [laugh]. Well, you mentioned dreamer; I dream direct routing between satellites, and look where I am; I'm just in the cloud, one step lower. [laugh].Corey: You know, bit by bit, we're going to get there one way or another, for an altitude perspective. I really want to thank you for taking so much time to speak with me today. If people want to learn more, where's the right place to find you?Guillermo: Well, I have the @IaaSgeek Twitter account, and you can find me on LinkedIn gruizesteban there. Just people wants to talk about anything there, I'm open to any kind of conversation. Just feel free to reach out. And it was a pleasure finally meeting you, in person. Not—well in person; through a camera, at least being in the show with you.Corey: Other than on the other side of a Twitter feed. No, I hear you.Guillermo: [laugh].Corey: We will, of course, put links to all of that in the [show notes 00:30:43]. Thank you so much for your time. I really do appreciate it.Guillermo: Thanks very much. So, you soon.Corey: Guillermo Ruiz, Director of OCI Developer Evangelism. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an insulting comment, to which I will respond with a surprise $270,000 bill.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About AlyssaAlyssa Miller, Business Information Security Officer (BISO) for S&P Global, is the global executive leader for cyber security across the Ratings division, connecting corporate security objectives to business initiatives. She blends a unique mix of technical expertise and executive presence to bridge the gap that can often form between security practitioners and business leaders. Her goal is to change how security professionals of all levels work with our non-security partners throughout the business.A life-long hacker, Alyssa has a passion for technology and security. She bought her first computer herself at age 12 and quickly learned techniques for hacking modem communications and software. Her serendipitous career journey began as a software developer which enabled her to pivot into security roles. Beginning as a penetration tester, her last 16 years have seen her grow as a security leader with experience across a variety of organizations. She regularly advocates for improved security practices and shares her research with business leaders and industry audiences through her international public speaking engagements, online content, and other media appearances.Links Referenced: Cybersecurity Career Guide: https://alyssa.link/book A-L-Y-S-S-A dot link—L-I-N-K slash book: https://alyssa.link/book Twitter: https://twitter.com/AlyssaM_InfoSec alyssasec.com: https://alyssasec.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Optimized cloud compute plans have landed at Vultr to deliver lightning-fast processing power, courtesy of third-gen AMD EPYC processors without the IO or hardware limitations of a traditional multi-tenant cloud server. Starting at just 28 bucks a month, users can deploy general-purpose, CPU, memory, or storage optimized cloud instances in more than 20 locations across five continents. Without looking, I know that once again, Antarctica has gotten the short end of the stick. Launch your Vultr optimized compute instance in 60 seconds or less on your choice of included operating systems, or bring your own. It's time to ditch convoluted and unpredictable giant tech company billing practices and say goodbye to noisy neighbors and egregious egress forever. Vultr delivers the power of the cloud with none of the bloat. Screaming in the Cloud listeners can try Vultr for free today with a $150 in credit when they visit getvultr.com/screaming. That's G-E-T-V-U-L-T-R dot com slash screaming. My thanks to them for sponsoring this ridiculous podcast.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the problems that many folks experience in the course of their career, regardless of what direction they're in, is the curse of high expectations. And there's no escaping for that. Think about CISOs for example, the C-I-S-O, the Chief Information Security Officer.It's generally a C-level role. Well, what's better than a C in the academic world? That's right, a B. My guest today is breaking that mold. Alyssa Miller is the BISO—B-I-S-O—at S&P Global. Alyssa, thank you for joining me to suffer my slings and arrows—Alyssa: [laugh].Corey: —as we go through a conversation that is certain to be no less ridiculous than it has begun to be already.Alyssa: I mean, I'm good with ridiculous, but thanks for having me on. This is awesome. I'm really excited to be here.Corey: Great. What the heck's BISO?Alyssa: [laugh]. I never get that question. So, this is—Corey: “No one's ever asked me that before.” [crosstalk 00:03:38]—Alyssa: Right?Corey: —the same thing as, “Do you know you're really tall?” “No, you're kidding.” Same type of story. But I wasn't clear. That means I'm really the only person left wondering.Alyssa: Exactly. I mean, I wrote a whole blog on it the day I got the job, right? So, Business Information Security Officer, Basically what it means is I am like the CISO but for my division, the Ratings Division at S&P Global. So, I lead our cyber security efforts within that division, work closely with our information security teams, our corporate IT teams, whatever, but I don't report to them; I report into the business line.I'm in the divisional CTO's org structure. And so, I'm the one bridging that gap between that business side where hey, we make all the money and that corporate InfoSec side where hey, we're trying to protect all the things, and there's usually that little bit of a gap where they don't always connect. That's me building the bridge across that.Corey: Someone who speaks both security and business is honestly in a bit of rare supply these days. I mean, when I started my Thursday newsletter podcast nonsense Last Week in AWS: Security, the problem I kept smacking into was everything I saw was on one side of that divide or the other. There was the folks who have the word security in their job title, and there tends to be this hidden language of corporate speak. It's a dialect I don't fully understand. And then you have the community side of actual security practitioners who are doing amazing work, but also have a cultural problem that more or less distills down to being an awful lot of shitheads in them there waters.And I wanted something that was neither of those and also wasn't vendor captured, which is why I decided to start storytelling in that space. But increasingly, I'm seeing that there's a significant problem with people who are able to contextualize security in the context of business. Because if you're secure enough, you can stop all work from ever happening, whereas if you're pure business side and only care about feature velocity and the rest, like, “Well, what happens if we get breached?” It's, “Oh, don't worry, I have my resume up to date.” Not the most reassuring answer to give people. You have to be able to figure out where that line lies. And it seems like that figuring out where that line is, is more or less your entire stock-in-trade.Alyssa: Oh absolutely, yeah. I mean, I can remember my earliest days as a developer, my cynical attitude towards security myself was, you know, their Utopia would be an impenetrable room full of servers that have no connections to anything, right? Like that would be wildly secure, yet completely useless. And so yeah, then I got into security and now I was one of them. And, you know, it's one of those things, you sit in, say a board meeting sometime and you listen to a CISO, a typical CISO talk to the board, and they just don't get it.Like, there's so much, “Hey, we're implementing this technology and we're doing this thing, and here's our vulnerability counts, and here's how many are overdue.” And none of that means anything. I mean, I actually had a board member ask me once, “What is a CISO?” I kid you not. Like, that's where they're at.Like, so don't tell them what you're doing, but tell them why connected back to, like, “Hey, the business needs this and this, and in order to do it, we've got to make sure it's secure, so we're going to implement these couple of things. And here's the roadmap of how we get from where we are right now to where we need to be so they can launch that new service or product,” or whatever the hell it is that they're going to do.Corey: It feels like security is right up there with accounting, in the sense of fields of endeavor where you don't want someone with too much personality involved. Because if the CISO's sitting there talking to the board, it's like, “So, what do you do here, exactly?” And the answer is the honest, “Hey, remember last month how we were in The New York Times for that giant data breach?” And they do a split take, “No, no, I don't.” “Exactly. You're welcome.” On some level, it is kind of honest, but it also does not instill confidence when you're that cavalier with the description of what it is you do here.Alyssa: Oh there's—Corey: At least there's some corners. I prefer—Alyssa: —there's so much—Corey: —places where that goes over well, but that's me.Alyssa: Yeah. But there's so much of that too, right? Like, here's the one I love. “Well, you know, it's not if you get breached, it's when. Oh, by the way, give me millions and millions of dollars, so I can make sure we don't get breached.”But wait, you just told me we're going to get breached no matter what we do. [laugh]. We do that in security. Like, and then you wonder why they don't give you funding for the initiative. Like, “Hello?” You know?And that's the thing that gets me it's like, can we just sit back and understand, like, how do you message to these people? Yeah I mean, you bring up the accounting thing; the funny thing is, at least all of them understand some level of accounting because most of them have MBAs and business degrees where they had to do some accounting. They didn't go through cyber security in their MBA program.So, one of my favorite questions on Twitter once was somebody asked me, you know, if I want to get into cyber security leadership, what is the one thing that I should focus on or what skills should I study? I said, “Go study MBA concepts.” Like, forget all the cyber security stuff. You probably have plenty of that technolog—go understand what they learn in MBA programs. And if you can start to speak that language, that's going to pay dividends for bridging that gap.Corey: So, you don't look like the traditional slovenly computer geek showing up at those meetings who does not know how to sound as if they belong in the room. Like, it's unfair, on some level, and I used to have bitter angst about that. Like, “Why should how I dress matter how people perceive me?” Yeah, in an absolute sense you're absolutely right, however, I can talk about the way the world is or the way I wish it were and there has to be a bit of a divide there.Alyssa: Oh, for sure. Yeah. I mean, you can't deny that you have to be prepared for the audience you're walking into. Now, I work in big conservative financial services on Wall Street. You know, and I had this conversation with a prominent member of our community when I started the job.I'm like, “Boy, I guess I can't really put stickers on my laptop. I'm going to have to get, you know, a protector or something to put stickers on.” Because the last thing I want to do is go into a boardroom with my laptop and whip out a bunch of hacker stickers on the backside of my laptop. Like, in a lot of spaces that will work, but you can't really do that when you're, you know, at, you know, the executive level and you're in a conservative, financial [unintelligible 00:10:16]. It just, I would love to say they should deal with that, I should be able to have pink hair, and you know, face tattoos and everything else, but the reality is, yeah, I can do all that, but these are still human beings who are going to react to that.And it's the same when talking about cyber security, then. Like, I have to understand as a security practitioner that all they know about cyber security is it's big and scary. It's the thing that keeps them up at night. I've had board members tell me exactly that. And so, how do I make it a little less scary, or at least get them to have some confidence in me that I'll, like, carry the shield in front of them and protect them. Like, that's my job. That's why I'm there.Corey: When I was starting my consultancy five years ago, I was trying to make a choice between something in the security cloud direction or the cost cloud direction. And one of the things that absolutely tipped the balance for me was the fact that the AWS bill is very much a business-hours-only problem. No one calls me at two in the morning screaming their head off. Usually. But there's a lot of alignment between those two directions in that you can spend all your time and energy fixing security issues and/or reducing the bill, but past a certain point, knock it off and go do the thing that your company is actually there to do.And you want to be responsible to a point on those things, but you don't want it to be the end-all-be-all because the logical outcome of all of that, if you keep going, is your company runs out of money and dies because you're not going to either cost optimize or security optimize your business to its next milestone. And weighing those things is challenging. Now, too many people hear that and think, “See, I don't have to worry about those things at all.” It's, “Oh, you will sooner or later. I promise.”Alyssa: So, here's the fallacy in that. There is this assumption that everything we do in security is going to hamper the business in some way and so we have to temper that, right? Like, you're not wrong. And we talked about before, right? You know, security in a traditional sense, like, we could do all of the puristic things and end up just, like, screeching the world to a halt.But the reality is, we can do security in a way that actually grows the business, that actually creates revenue, or I should say enables the creation of revenue in that, you know, we can empower the business to do more things and to be more innovative by how we approach security in the organization. And that's the big thing that we miss in security is, like, look, yes, we will always be a quote-unquote, “Cost center,” right? I mean, we in security don't—unless you work for a security organization—we're not getting revenue attributed to us, we're not creating revenue. But we are enabling those people who can if we approach it right.Corey: Well, the Red Team might if they go a little off-script, but that's neither here nor there.Alyssa: I—yeah, I mean, I've had that question. “Like, couldn't we just sell resell our Red Team services?” No. No. That's not our core [crosstalk 00:13:14]Corey: Oh, I was going the other direction. Like, oh, we're just going to start extorting other businesses because we got bored this week. I'm kidding. I'm kidding. Please don't do an investigation, any law enforcement—Alyssa: I was going to say, I think my [crosstalk 00:13:22]—Corey: —folks that happen to be listening to this.Alyssa: [crosstalk 00:13:24] is calling me right now. They're want to know what I'm [laugh] talking about. But no—Corey: They have some inquiries they would like you to assist them with and they're not really asking.Alyssa: Yeah, yeah, they're good at that. No, I love them, though. They're great. [laugh]. But no, seriously, like, I mean, we always think about it that way because—and then we wonder why do we have the reputation of, you know, the Department of No.Well, because we kind of look at it that way ourselves; we don't really look at, like how can we be a part of the answer? Like, when we look at, like, DevSecOps, for instance. Okay, I want to bring security into my pipeline. So, what do we say? “Oh, shared responsibility. That's a DevOps thing.” So, that means security is everybody's responsibility. Full stop.Corey: Right. It's a—Alyssa: Well—Corey: And there, I agree with you wholeheartedly. Cost is—Alyssa: But—Corey: —aligned with this. It has to be easier to do it the right way than to just go off half-baked and do it yourself off the blessed path. And that—Alyssa: So there—Corey: —means there's that you cannot make it harder to do the right thing; you have to make it easier because you will not win against human psychology. Depending on someone when they're done with an experiment to manually go in and turn things off. It will not happen. And my argument has been that security and cost are aligned constantly because the best way to secure something and save money on at the same time is to turn that shit off. You wouldn't think it would be that simple, but yet here we are.Alyssa: But see, here's the thing. This is what kills me. It's so arrogant of security people to look at it and say that right? Because shared responsibility means shared. Okay, that means we have responsibilities we're going to share. Everybody is responsible for security, yes.Our developers have responsibilities now that we have to take a share in as well, which is get that shit to production fast. Period. That is their goal. How fast can I pop user stories off the backlog and get them to deployment? My SRE is on the ops side. They're, like, “We just got to keep that stuff running. That's all we that's our primary focus.”So, the whole point of DevOps and DevSecOps was everybody's responsible for every part of that, so if I'm bringing security into that message, I, as security, have to be responsible for site's stability; I, in security, have to be responsible for efficient deployment and the speed of that pipeline. And that's the part that we miss.Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: I think you might be the first person I've ever spoken to that has that particular take on the shared responsibility model. Normally, when I hear it, it's on stage from an AWS employee doing a 45-minute song-and-dance about what the secured responsibility model is, and generally, that is interpreted as, “If you get breached, it's your fault, not ours.”Alyssa: [laugh].Corey: Now, you can't necessarily say it that directly to someone who has just suffered a security incident, which is why it takes 45 minutes and slides and diagrams and excel sheets and the rest. But that is what it fundamentally distills down to, and then you wind up pointing out security things that they've had that [unintelligible 00:17:11] security researchers have pointed out and they are very tight-lipped about those things. And it's, “Oh, it's not that you're otherworldly good at security; it's that you're great at getting people to shut up.” You know, not me, for whatever reason because I'm noisy and obnoxious, but most people who actually care about not getting fired from their jobs, generally don't want to go out there making big cloud companies look bad. Meanwhile, that's kind of my entire brand.Alyssa: I mean, it's all about lines of liability, right?Corey: Oh yeah.Alyssa: I mean, where am I liable, where am I not? And yeah, well, if I tell you you're responsible for security on all these things, and I can point to any part of that was part of the breach, well, hey, then it's out of my hands. I'm not liable. I did what I said I would; you didn't secure your stuff. Yeah, it's—and I mean, and some of that is to be fair.Like, I mean, okay, I'm going to host my stuff on your computer—the whole cloud is just somebody else's computer model is still ultimately true—but, yeah, I mean, I'm expecting you to provide me a stable and secure environment and then I'm going to deploy stuff on it, and you are expecting me to deploy things that are stable and secure as well. And so, when they say shared model or shared responsibility model, but it—really if you listen to that message, it's the exact opposite. They're telling you why it's a separate responsibility model. Here's our responsibilities; here's yours. Boom. It's not about shared; it's about separated.Corey: One of the most formative, I guess, contributors to my worldview was 13 years ago, I went on a date and met someone lovely. We got married. We've been together ever since, and she's an attorney. And it is been life-changing to understand a lot of that perspective, where it turns out when you're dealing with legal, they are not—and everyone says, “Oh, and the lawyers insisted on these things.”No, they didn't. A lawyer's entire role in a company is to identify risk, and then it is up to the business to make a decision around what is acceptable and what is not. If your lawyers ever insist on something, what that actually means in my experience is, you have said something profoundly ignorant that is one of those, like—that is—they're doing the legal equivalent of slapping the gun out of the toddler's hand of, “No, you cannot go and tweet that because you'll go to prison,” level of ridiculous nonsense where it is, “That will violate the law.” Everything else is different shades of the same answer: it depends. Here's what to consider.Alyssa: Yes.Corey: And then you choose—and the business chooses its own direction. So, when you have companies doing what appeared to be ridiculous things, like Oracle, for example, loves to begin every keynote with a disclaimer about how nothing they're about to say is true, the lawyers didn't insist on that—though they are the world's largest law firm, Kirkland Ellison. But instead, it's this entire story of given the risk and everything that we know about how we say things onstage and people gunning for us, yeah, we are going to [unintelligible 00:20:16] this disclaimer first. Most other tech companies do not do that exact thing, which I've got to say when you're sitting in the audience ready to see the new hotness that's about to get rolled out and it starts with a disclaimer, that is more or less corporate-speak for, “You are about to hear some bullshit,” in my experience.Alyssa: [laugh]. Yes. I mean and that's the thing, like, [clear throat], you know, we do deride legal teams a lot. And you know, I can find you plenty of security people who hate the fact that when you're breached, who's the first call you make? Well, it's your legal team.Why? Because they're the ones who are going to do everything in their power to limit the amount that you can get sued on the back-end for anything that got exposed, that you know, didn't meet service levels, whatever the heck else. And that all starts with legal privilege.Corey: They're reporting responsibilities. Guess who keeps up on what those regulatory requirements are? Spoiler, it's probably not you, whoever's listening to this, unless you're an attorney because that is their entire job.Alyssa: Yes, exactly. And, you know, work in a highly regulated environment—like mine—and you realize just how critical that is. Like, how do I know—I mean, there are times there's this whole discussion of how do you determine if something is a material impact or not? I don't want to be the one making that, and I'm glad I don't have to make that decision. Like, I'll tell you all the information, but yes, you lawyers, you compliance people, I want you to make the decision of if it's a material impact or not because as much as I understand about the business, y'all know way more about that stuff than I do.I can't say. I can only say, “Look, this is what it impacted. This is the data that was impacted. These are the potential exposures that occurred here. Please take that information now and figure out what that means, and is there any materiality to that that now we have to report that to the street.”Corey: Right, right. You can take my guesses on this or you can get it take an attorney's. I am a loud, confident-sounding white guy. Attorneys are regulated professionals who carry malpractice insurance. If they give wrong advice that is wrong enough in these scenarios, they can be sanctioned for it; they can lose their license to practice law.And there are challenges with the legal profession and how much of a gatekeeper the Bar Association is and the rest, but this is what it is [done 00:22:49] for itself. That is a regulated industry where they have continuing education requirements they need to certify in a test that certain things are true when they say it, whereas it turns out that I don't usually get people even following up on a tweet that didn't come true very often. There's a different level of scrutiny, there's a different level of professional bar it raises to, and it turns out that if you're going to be legally held to account for things you say, yeah, turns out a lot of your answers to are going to be flavors of, “It depends.”Alyssa: [laugh].Corey: Imagine that.Alyssa: Don't we do that all the time? I mean, “How critical is this?” “Well, you know, it depends on what kind of data, it depends on who the attacker is. It depends.” Yeah, I mean, that's our favorite word because no one wants to commit to an absolute, and nor should we, I mean, if we're speaking in hyperbole and absolutes, boy, we're doing all the things wrong in cyber.We got to understand, like, hey, there is nuance here. That's how you run—no business runs on absolutes and hyperbole. Well, maybe marketing sometimes, but that's a whole other story.Corey: Depends on if it's done well or terribly.Alyssa: [laugh]. Right. Exactly. “Hey, you can be unhackable. You can be breached-proof.” Oh, God.Corey: Like, what's your market strategy? We're going to paint a big freaking target in the front of the building. Like, I still don't know how Target the company was ever surprised by a data breach that they had when they have a frickin' bullseye as their logo.Alyssa: “Come get us.”Corey: It's, like, talk about poking the bear. But there we are.Alyssa: [unintelligible 00:24:21] no. I mean, hey, [unintelligible 00:24:23] like that was so long ago.Corey: It still casts a shadow.Alyssa: I know.Corey: People point to that as a great example of, like, “Well, what's going to happen if we get breached?” It's like, well look at Target because they wound up—like, their stock price a year later was above where it had been before and it seemed to have no lasting impact. Yeah, but they effectively replaced all of the execs, so you know, let's have some self-interest going on here by named officers of the company. It's, “Yeah, the company will be fine. Would you like to still be here what it is?”Alyssa: And how many lawsuits do you think happened that you never heard about because they got settled before they were filed?Corey: Oh, yes. There's a whole world of that.Alyssa: That's what's really interesting when people talk about, like, the cost of breach and stuff, it's like, we don't even know. We can't know because there is so much of that. I mean, think about it, any organization that gets breached, the first thing they're trying to do is keep as much of it out of the news as they can, and that includes the lawsuits. And so, you know, it's like, all right, well, “Hey, let's settle this before you ever file.”Okay, good. No one will ever know about that. That will never show up anywhere. It is going to show up on a balance sheet anywhere, right? I mean, it's there, but it's buried in big categories of lots of other things, and how are you ever going to track that back without, you know, like, a full-on audit of all of their accounting for that year? Yeah, it's—so I always kind of laugh when people start talking about that and they want to know, what's the average cost of a breach. I'm like, “There's no way to measure that. There is none.”Corey: It's not cheap, and the reputational damage gets annoying. I still give companies grief for these things all the time because it's—again, the breach is often about information of mine that I did not consciously choose to give to you and the, “Oh, I'm going to blame a third-party process.” No, no, you can outsource work, but not responsibility. You can't share that one.Alyssa: Ah, third-party diligence, uh, that seems to be a thing. You know, I think we're supposed to make sure our third parties are trustworthy and doing the right things too, right? I mean, it's—Corey: Best example I ever saw that was an article in the Wall Street Journal about the Pokemon company where they didn't name the vendor, but they said they declined to do business with them in part based upon their lax security policy around S3 buckets. That is the first and so far only time I have had an S3 Bucket Responsibility Award engraved and sent to their security director. Usually, it's the ignoble prize of the S3 Bucket Negligence Award, and there are oh so many of those.Alyssa: Oh, and it's hard, right? Because you're standing—I mean, I'm in that position a lot, right? You know, you're looking at a vendor and you've got the business saying, “God, we want to use this vendor. All their product is great.” And I'm sitting there saying, but, “Oh, my God, look at what they're doing. It's a mess. It's horrible. How do I how do we get around this?”And that's where, you know, you just have to kind of—I wish I could say no more, but at the end of the day, I know what that does. That just—okay, well, we'll go file an exception and we'll use it anyway. So, maybe instead, we sit and work on how to do this, or maybe there is an alternative vendor, but let's sort it out together. So yeah, I mean, I do applaud them. Like that's great to, like, be able to look at a vendor and say, “No, we ain't touching you because what you're doing over there is nuts.” And I think we're learning more and more how important that is, with a lot of the supply chain attacks.Corey: Actually, I'm worried about having emailed you, you're going to leak my email address when your inbox inevitably gets popped. Come on. It's awful stuff.Alyssa: Yeah, exactly. So, I mean, it's we there's—but like everything, it's a balance again, right? Like, how can we keep that business going and also make sure that their vendors—so that's where it just comes down to, like, okay, let's talk contracts now. So, now we're back to legal.Corey: We are. And if you talk to a lawyer and say, “I'm thinking about going to law school,” the answer is always the same. “No… don't do it.” Making it clear that is apparently a terrible life and professional decision, which of course, brings us to your most recent terrible life and professional decision. As we record this, we are reportedly weeks away from you having a physical copy in your hands of a book.And the segue there is because no one wants to write a book. Everyone wants to have written a book, but apparently—unless you start doing dodgy things and ghost-writing and exploiting people in the rest—one is a necessary prerequisite for the other. So, you've written a book. Tell me about it.Alyssa: Oof, well, first of all, spot on. I mean, I think there are people who really do, like, enjoy the act of writing a book—Corey: Oh, I don't have the attention span to write a tweet. People say, “Oh, you should write a book, Corey,” which I think is code for them saying, “You should shut up and go away for 18 months.” Like, yeah, I wish.Alyssa: Writing a book has been the most eye-opening experience of my life. And yeah, I'm not a hundred percent sure it's one I'll ever—I've joked with people already, like, I'll probably—if I ever want another book, I'll probably hire a ghostwriter. But no, I do have a book coming out: Cybersecurity Career Guide. You know, I looked at this cyber skills gap, blah, blah, blah, blah, blah, we hear about it, 4 million jobs are going to be left open.Whatever, great. Well, then how come none of these college grads can get hired? Why is there this glut of people who are trying to start careers in cyber security and we can't get them in?Corey: We don't have six months to train you, so we're going to spend nine months trying to fill the role with someone experienced?Alyssa: Exactly. So, 2020 I did a bunch of research into that because I'm like, I got to figure this out. Like, this is bizarre. How is this disconnect happening? I did some surveys. I did some interviews. I did some open-source research. Ended up doing a TED Talk based off of that—or TEDx Talk based off of that—and ultimately that led into this book. And so yeah, I mean, I just heard from the publisher yesterday, in fact that we're, like, in that last stage before they kick it out to the printers, and then it's like three weeks and I should have physical copies in my hands.Corey: I will be getting one when it finally comes out. I have an almost, I believe, perfect track record of having bought every book that a guest on this show has written.Alyssa: Well, I appreciate that.Corey: Although, God help me if I ever have someone, like, “So, what have you done?” “I've written 80 books.” Like, “Well, thank you, Stephen King. I'm about to go to have a big—you're going to see this number of the company revenue from orbit at this point with that many.” But yeah, it's impressive having written a book. It's—Alyssa: I mean, for me, it's the reward is already because there are a lot of people have—so my publisher does really cool thing they call it early acc—or electronic access program, and where there are people who bought the book almost a year ago now—which is kind of, I feel bad about that, but that's as much my publisher as it is me—but where they bought it a year ago and they've been able to read the draft copy of the book as I've been finishing the book. And I'm already hearing from them, like, you know, I'm hearing from people who really found some value from it and who, you know, have been recommending it other people who are trying to start careers and whatever. And it's like, that's where the reward is, right?Like, it was, it's hell writing a book. It was ten times worse during Covid. You know, my publisher even confirmed that for me that, like, look, yeah, you know, authors around the globe are having problems right now because this is not a good environment conducive to writing. But, yeah, I mean, it's rewarding to know that, like, all right, there's going to be this thing out there, that, you know, these pages that I wrote that are helping people get started in their careers, that are helping bring to light some of the real challenges of how we hire in cyber security and in tech in general. And so, that's the thing that's going to make it worthwhile. And so yeah, I'm super excited that it's looking like we're mere weeks now from this thing being shipped to people who have bought it.Corey: So, now it's racing, whether this gets published before the book does. So, we'll see. There is a bit of a production lag here because, you know, we have to make me look pretty and that takes a tremendous amount of effort.Alyssa: Oh, stop. Come on now. But it will be interesting to see. Like, that would actually be really cool if they came out at about the same time. Like, you know, I'm just saying.Corey: Yeah. We'll see how it goes. Where's the best place for people to find you if they want to learn more?Alyssa: About the book or in general?Corey: Both.Alyssa: So—Corey: Links will of course be in the [show notes 00:32:49]. Let's not kid ourselves here.Alyssa: The book is real easy. Go to Alyssa—A-L-Y-S-S-A, back here behind me for those of you seeing the video. Um—I can't point the right direction. There we go. That one. A-L-Y-S-S-A dot link—L-I-N-K slash book. It's that simple. It'll take you right to Manning's site, you can get in.Still in that early access program, so if you bought it today, you would still be able to start reading the draft versions of it. If you want to know more about me, honestly, the easiest way is to find me on Twitter. You can hear all the ridiculousness of flight school and barbecue and some security topics, too, once in a while. But at @alyssam_infosec. Or if you want to check out the website where I blog, every rare occasion, it's alyssasec.com.Corey: And all of that will be in the [show notes 00:33:41]. Thank you—Alyssa: There's a lot. [laugh].Corey: I'm looking forward to seeing it, too. Thank you so much for taking the time to deal with my nonsense today. I really appreciate it.Alyssa: Oh, that was nonsense? Are you kidding me? This was a great discussion. I really appreciate it.Corey: As have I. Thanks again for your time. It is always great to talk to people smarter than I am—which is, let's be clear, most people—Alyssa Miller, BISO at S&P Global. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice—or smash the like and subscribe button if this is on the YouTubes—whereas if you've hated the podcast, same thing, five-star review, platform of choice, smash both of the buttons, but also leave an angry comment, either on the YouTube video or on the podcast platform, saying that this was a waste of your time and what you didn't like about it because you don't need to read Alyssa's book; you're going to get a job the tried and true way, by printing out a copy of your resume and leaving it on the hiring manager's pillow in their home.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About SharoneI'm Sharone Zitzman, a marketing technologist and open source community builder, who likes to work with engineering teams that are building products that developers love. Having built both the DevOps Israel and Cloud Native Israel communities from the ground up, today I spend my time finding the places where technology and people intersect and ensuring that this is an excellent experience. You can find my talks, articles, and employment experience at rtfmplease.dev. Find me on Twitter or Github as @shar1z.Links Referenced: Personal Twitter: https://twitter.com/shar1z Website: https://rtfmplease.dev LinkedIn: https://www.linkedin.com/in/sharonez/ @TLVCommunity: https://twitter.com/TLVcommunity @DevOpsDaysTLV: https://twitter.com/devopsdaystlv TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: DoorDash had a problem as their cloud native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, competence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud/C-H-R-O-N-O-S-P-H-E-R-E.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance. To learn more visit: snark.cloud/deployandgoCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn and I have been remiss by not having today's guest on years ago because back before I started this ridiculous nonsense that, well, whatever it is you'd call what I do for a living, I did other things instead. I did the DevOps, which means I was sad all the time. And the thing that I enjoyed was the chance to go and speak on conference stages. One of those stages, early on in my speaking career, was at DevOpsDays Tel Aviv.My guest today is Sharone Zitzman, who was an organizer of DevOpsDays Tel Aviv, who started convincing me to come back. And today is in fact, in the strong tradition here of making up your own job titles in ways that make people smile, she is the Chief Manual Reader at RTFM Please Ltd. Sharone, thank you for joining me.Sharone: Thank you for having me, Corey. Israelis love the name of my company, but Americans think it has a lot of moxie and chutzpah. [laugh].Corey: It seems a little direct and aggressive. It's like, oh, good, you are familiar with how this is going to go. There's something to be said for telling people what you do on the tin upfront. I've never been a big fan of trying to hide that. I mean, the first iteration of my company was the Quinn Advisory Group because I thought, you know, let's make it look boring and sedate and like I can talk to finance people. And yeah, that didn't last more than ten seconds of people talking to me.Also, in hindsight, the logo of a big stylized Q. Yeah, I would have had to change that anyway, for the whole QAnon nonsense because I don't want to be mistaken for that particular brand of nuts.Sharone: Yeah, I decided to do away with the whole formalities and upfront, just go straight [laugh]. For the core of who we are, Corey; you are very similar in that. So, yes. Being a dev first company, I thought the developers would appreciate such a title and name for my company. And I have to give a shout out here to Avishai Ish-Shalom, who's my friend from the community who you also know from the DevOpsDays community.Corey: Oh, yeah @nukemberg on Twitter—Sharone: Yes exactly.Corey: For those who are not familiar.Sharone: [laugh]. Yep. He coined the name.Corey: The problem that I found is that people when they start companies or they manage their careers, they don't bias for the things that they're really good at. And it took me a long time to realize this, I finally discovered, “Ah, what am I the best at? That's right, getting myself fired for my personality, so why don't I build a business where that stops being a liability?” So, I started my own company. And I can tell this heroic retcon of what happened, but no, it's because I had nowhere else to go at that point.And would you hire me? Think about this for a minute. You, on the other hand, had options. You are someone with a storied history in community building, in marketing to developers without that either coming across as insincere or that marked condescending accent that so many companies love to have of, “Oh, you're a developer. Let me look at you and get down on my hands and knees like we're going camping and tell a story in ways that actively and passively insult you.”No, you have always gotten that pitch-perfect. The world was your oyster. And for some godforsaken reason, you looked around and decided, “Ah, I'm going to go out independently because you know what I love? Worrying.” Because let's face it, running your own company is an exercise in finding new and exciting things to worry about that 20 minutes ago, you didn't know existed. I say this from my own personal experience. Why would you ever do such a thing?Sharone: [laugh]. That's a great question. It was a long one, but a good one. And I do a thing where I hit the mic a lot because I also have. I can't control my hand motions.Corey: I too speak with my hands. It's fine.Sharone: [laugh]. Yeah, so it's interesting because I wanted to be independent for a really long time. And I wasn't sure, you know, if it was something that I could do if I was a responsible enough adult to even run my own company, if I could make it work, if I could find the business, et cetera. And I left the job in December 2020, and it was the first time that I hadn't figured out what I was doing next yet. And I wanted to take some time off.And then immediately, like, maybe a week after I started to get a lot of, like, kind of people reaching out. And I started to interview places and I started to look into possibly being a co-founder at places and I started to look at all these different options. And then just, I was like, “Well…. This is an opportunity, right? Maybe I should finally—that thing that's gnawing at the back of my head to see if, like, you know if I should go for this dream that I've always wanted, maybe now I can just POC it and see if, you know, it'll work.”And it just, like, kind of exploded on me. It was like there was so much demand, like, I just put a little, like, signal out to the world that this is something that I'm interested in doing, and everyone was like, “Ahh, I need that.” [laugh]. I wanted to take a quarter off and I signed my first clients already on February 1st, which was, like, a month after. I left in December and that—it was crazy. And since then, I've been in business. So, yeah. So, and since then, it's also been a really crazy ride; I got to discover some really exciting companies. So.Corey: How did you get into this? I found myself doing marketing-adjacent work almost entirely by accident. I started the newsletter and this podcast, and I was talking to sponsors periodically and they'd come back with, “Here's the thing we want you to talk about in the sponsor read.” And it's, “Okay, you want to give people a URL to go to that has four sub-directories and entire UTM code… okay, have you considered, I don't know, not?” And because so much of what they were talking about did not resonate.Because I have the engineering background, and it was, I don't understand what your company does and you're spending all your time talking about you instead of my painful problem. Because as your target market, I don't give the slightest of shits about you, I care about my problem, so tell me how you're going to solve my problem and suddenly I'm all ears. Spend the whole time talking about you, and I could not possibly care less and I'll fast-forward through the nonsense. That was my path to it. How did you get into it?Sharone: How did I get into it? It's interesting. So, I started my journey in typical marketing, enterprise B2B marketing. And then at GigaSpaces, we kickstarted the open-source project Cloudify, and that's when I found myself leading this project as the open-source community team leader, building, kind of, the community from the ground floor. And I discovered a whole new world of, like, how to build experience into your marketing, kind of making it really experiential and making sure that everyone has a really, really easy and frictionless way of using your product, and that the product—putting the product at the center and letting it speak for itself. And then you discover this whole new world of marketing where it's—and today, you know, it has more of a name and a title, PLG, and people—it has a whole methodology and practice, but then it was like we were—Corey: PLG? I'm unfamiliar with the acronym. I thought tech was bad for acronyms.Sharone: Right? [laugh]. So, product-led growth. But then, you know, like, kind of wasn't solidified yet. And so, a lot of what we were doing was making sure that developers had a really great experience with the product then it kind of sold itself and marketed itself.And then you understood what they wanted to hear and how they wanted to consume the product and how they wanted it to be and to learn about it and to kind of educate themselves and get into it. And so, a lot of the things that I learned in the context of marketing was very guerilla, right, from the ground up and kind of getting in front of people and in the way they wanted to consume it. And that taught me a lot about how developers consume technology, the different channels that they're involved in, and the different tools that they need in order to succeed, and the different, you know, all the peripheral experience, that makes marketing really, really great. And it's not about what you're selling to somebody; it's making your product shine and making the experience shine, making them ensure that it's a really, really easy and frictionless experience. You know, I like how [Donald Bacon 00:08:00] says it; he calls it, like, mean time to hello world, and that to me is the best kind of marketing, right? When you enable people to succeed very, very quickly.Corey: Yeah, there's something to be said for the ring of authenticity and the rest. Periodically I'll promote guest episodes on this, where it's a sponsored episode where people get up and they talk about what they're working on. And they're like, “Great. So, here's the sales pitch I want to give,” and it's no you won't because first, it won't work. And secondly, I'm sorry, whether it's a promoted episode or not, I will not publish something that isn't good because I have a reputation to uphold here.And people run into challenges an awful lot when they're trying to effectively tell their story. If you have a startup that was founded by an engineer, for example, as so many of these technical startups were, the engineer is often so deeply and profoundly in love with this problem space and the solution and the rest, but if they talk about that, no one cares about the how. I mean, I fix AWS bills, and people don't care—as a general rule—how I do that at all if they're in my target market. They don't care if it's through clever optimization, amazing tooling, doing it on-site, or taking hostages in Seattle. They care about their outcome much more than they ever do about the how.The only people who care about the how are engineers who very often are going to want to build it themselves, or work for you, or start a competitor. And it doesn't resonate in quite the same way. It's weird because all these companies are in slightly different spaces; all of them tend to do slightly different things—or very different things—but so many of the challenges that I see in the way that they're articulating what they do to customers rhymes with one another.Sharone: Yeah. So, I agree completely that developers will talk often about how it works. How it works. How does it work under the hood? What are the bits and bytes, you know?Like, nobody cares about how it works. People care about how will this make my life better, right? How will this improve my life? How will this change my life? [laugh]. As an operations engineer, if I'm, you know, crunching through logs, how will this tool change that? What my days look like? What will my on-call rotation look like? What will—you know, how are you changing my life for the better?So, I think that that's the question. When you learn how to crystallize the answer to that question and you hit it right on the mark—you know, and it takes a long time to understand the market, and to understand the buying persona, and t—and there's so much that you have to do in the background, and so much research you have to do to understand who is that person that needs to have that question answered? But once you do and you crystallize that answer, it lands. And that's the fun part about marketing, really trying to understand the person who's going to consume your product and how you can help them understand that you will make their life better.Corey: Back when I was starting out as a consultant myself, I would tell stories that I had seen in the AWS billing environment, and I occasionally had clients reach out to me, “Hey, why don't you tell our story in public?” It's, “Because that wasn't your story. That was something I saw on six different accounts in the same month. It is something that everyone is feeling.” It's, people think that you're talking about them.So, with that particular mindset on this, without naming specific companies, what themes are you seeing emerging? What are companies getting wrong when they are attempting and failing to market effectively to developers?Sharone: So, exactly what we're talking about in terms of the product pitch, in that they're talking at developers from this kind of marketing speak and this business language that, you know, developers often—you know, unless a company does a really, really good job of translating, kind of, the business value—which they should do, by the way—to engineers, but oftentimes, it's a little bit far from them in the chain, and so it's very hard for them to understand the business fluff. If you talk to them in bits and bytes of this is what my day-to-day developer workflow looks like and if we do these things, it'll cut down the time that I'm working on these things, it'll make these things easier, it'll help streamline whatever processes that are difficult, remove these bottlenecks, and help them understand, like I said, how it improves their life.But the things that I've seen breakdown is also in the authenticity, right? So obviously, the world is built on a lot of the same gimmicks and it's just a matter of whether you're doing it right or not, right? So, there's so much content out there and webcasts and webinars, and I don't know what and podcasts and whatever it is, but a lot of the time, people, their most valuable asset is their time. And if you end up wasting their time, without it being, like, really deeply valuable—if you're going to write content, make sure that there is a valuable takeaway; if you're going to create a webinar, make sure that somebody learned something. That if they're investing their time to join your marketing activities, make sure that they come away with something meaningful and then they'll really appreciate you.And it's the same idea behind the whole DevOpsDays movement with the law of mobility and open spaces that people if they find value, they'll join this open space and they'll participate meaningfully and they'll be a part of your event, and they'll come back to your event from year to year. But if you're not going to provide that tangible value that somebody takes away, and it's like, okay, well, I can practically apply this in my specific tech stack without using your tool, without having to have this very deterministic or specific kind of tech stack that they're talking about. You want to give people something—or even if it is, but even how to do it with or without, or giving them, like, kind of practical tools to try it. Or if there's an open-source project that they can check out first, or some kind of lean utility that gives them a good indication of the value that this will give them, that's a lot more valuable, I think. And practically understandable to somebody who wants to eventually consume your product or use your products.Corey: The way that I see things, at least in the past couple of years, the pandemic has sharpened an awful lot of the messaging that needs to happen. Because in most environments, you're sitting at a DevOpsDays in the front row or whatnot, and it's time for the sponsor talks and someone gets up and starts babbling and wasting your time, most people are not going to get up and leave. Okay, they will in Israel, but in most places, they're not going to get up and leave, whereas in pandemic land, it's you are one tab away from something I actually want—Sharone: Exactly.Corey: To be doing, so if you become even slightly boring, it's not going to go well. So, you have to be on message, you have to be on point or no one cares. People are like, “Oh, well what if we say the wrong thing and people wind up yelling about us on Twitter?” It's like unless it is for something horrifying, you should be so lucky because people are then talking about you. The failure mode isn't that people don't like your product, it's no one talks about it.Sharone: Yeah. No such thing as bad publicity [crosstalk 00:14:32] [laugh]—Corey: Oh, there very much is such a thing is bad publicity. Like, “I could be tweeting about your product most days,” is apparently a version of that, according to some folks. But it's a hard problem to solve for. And one of the things that continually surprises me is the things I'm still learning about this entire industry. The reason that people sponsor this show—and the rates they pay, to be direct—have little bearing to the actual size of the audience—as best we can tell; lies, damn lies, and podcast statistics; if you're listening to this, let me know. I'd love to know if anyone listens to this nonsense—but when you see all of that coming out, why are we able to charge the rates that we do?It's because the long-term value of someone who is going to buy a long-term subscription or wind up rolling out something like ChaosSearch or whatnot that is going to be a fundamental tenet of their product, one prospect becoming a customer pays for anything, I can sell a company, it will sponsor—they can pay me to sponsor for the next ten years, as opposed to the typical mass-market audience where well, I'm here to sling Casper mattresses today or something. It's a different audience and there's a different perception there. People are starting to figure out the value of—in an age where tracking is getting harder and harder to do and attribution will drive you nuts, instead of go where your audience is. Go where the people who care about the problem that you have and will experience that problem are going to hang out. And it always is wild to me to see companies missing out on that.It's, “Okay, so you're going to do a $25 million billboard ad in spotted in airports around the world talking about your company… but looking at your billboard, it makes no sense. I don't understand what it's there for.” Even as a brand awareness play, it fails because your logo is tiny in the corner or something. It's you spent that much money on ads, and maybe a buck on messaging because it seems like with all that attention you just bought, you had nothing worthwhile to say. That's the cardinal sin to me at least.Sharone: Yeah. One thing that I found—and back to our community circuit and things that we've done historically—but that's one thing that, you know, as a person comes from community, I've seen so much value, even from the smaller events. I mean, today, like with Covid and the pandemic and everything has changed all the equilibrium and the way things are happening. But some meetups are getting smaller, face-to-face events are getting smaller, but I've had people telling me that even from small, 30 to 40 people events, they'll go up and they'll do a talk and great, okay, a talk; everybody does talks, but it's like, kind of, the hallway track or the networking that you do after the talk and you actually talk to real users and hear their real problems and you tap into the real community. And some people will tell me like, I had four concrete leads from a 30-person meet up just because they didn't even know that this was a real challenge, or they didn't know that there was a tool that solves this problem, or they didn't understand that this can actually be achieved today.Or there's so many interesting technologies and emerging technologies. I'm privileged to be able to be at the forefront of that and discover it all, and I if I could, I would drop names of all of the awesome companies that work for me, that I work with, and just give them a shout out. But really, there's so many amazing companies doing, like, developer metrics, and all kinds of troubleshooting and failure analysis that's, like, deeply intelligent—and you're going to love this one: I have a Git replacement client apropos to your closing keynote of DevOpsDays 2015—and tapping into the communities and tapping into the real users.And sometimes, you know, it's just a matter of really understanding how developers are working, what processes look like, what workflows look like, what teams look like, and being able to architect your products and things around real use cases. And that you can only discover by really getting in front of actual users, or potential users, and learning from them and feedback loops, and that's the little core behind DevRel and developer advocacy is really understanding your actual users and your consumers, and encouraging them to you know, give you feedback and try things, and beta programs and a million things that are a lot more experiential today that help you understand what your users need, eventually, and how to actually architect that into your products. And that's the important part in terms of marketing. And it's a whole different marketing set. It's a whole different skill set. It's not talking at people, it's actually… ingesting and understanding and hearing and implementing and bringing it into your products.Corey: And it takes time. And you have to make yourself synonymous with a painful problem. And those problems are invariably very point-in-time specific. I don't give a crap about log aggregation today, but in two weeks from now, when I'm trying to chase down 18 different Lambdas function trying to figure out what the hell's broken this week, I suddenly will care very much about log aggregation. Who was that company that's in that space that's doing interesting things? And maybe it's Cribl, for example; they do a lot of stuff in that space and they've been a good sponsor. Great.I start thinking about those things in that light because it is—when I started having these problems, it sticks in your head and it resonates. And there's value and validity to that, but you're never going to be able to attribute that either, which is where people often lose their minds. Because for anything even slightly complicated—you're going to be selling things to big bank—great, good on you. Most of those customers are not going to go and spin up a trial in the dead of night. They're going to hear about you somewhere and think, “Ohh, this is interesting.”They're going to talk about a meeting, they're going to get approval, and at that point, you have long since lost any tracking opportunity there. So, the problem is that by saying it like this, as someone who is a publisher, let's be very clear here, it sounds like you're trying to justify your entire business model. I feel like that half the time, but I've been reassured by people who are experts in doing these things, like, oh, yeah, we have data on this; it's working. So, the alternative is either I accept that they're right or I sit here and arrogantly presume I know more about marketing than people who've devoted their entire careers to it. I'm not that bold. I am a white guy in tech, but not that much.Sharone: Yeah, I mean, the DevRel measurement problem is a known problem. We have people like [unintelligible 00:20:21] who have written about it. We have [Sarah Drasner 00:20:23], we have a million people that have written really, really great content about how do you really measure DevRel and the quality. And one of the things that I liked, Philipp Krenn, the dev advocate at Elastic once said in one of his talks that, you know, “If you're measuring your developer advocates on leads, you're a marketing organization. If you're measuring them on revenue, you're a sales organization. It's about reach, engagement, and awareness, and a lot of things that it's much, much harder to measure.”And I can say that, like, once upon a time, I used to try and attribute it at Cloudify. Like, I remember thinking, like, “Okay, maybe I could really track this back to, you know, the first touch that I actually had with this user.” It's really, really difficult, but I do remember, like, when we used to go out into the events and we were really active in the OpenStack community, in the DevOps community, and many other things, and I remember, like, even after events, like, you get all those lead gen emails. All I would say now is, like, “Hey, if you missed us at the booth, you know, and you want still want a t-shirt, you know, reach out and I'll ship it to you.” And some of those eventually, after we continued the relationship, and we, you know, when we were friends and community friends, six months later, when they moved to their next role at their next job, they were like, “Oh, now I have an opportunity to use Cloudify and I'm going to check it out.”And it's very long relationship that you have to cultivate. It has to be, you know, mutual. You have to be, you have to give be giving something and eventually is going to come back to you. Good deeds come back to you. So, I—that's my credo, by the way, good deeds come back to you. I believe in that and I try to live by that.Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: So, I have one last question for you and it is pointed and the reason I buried it this deep in the episode is so that if I open with it, I will get letters and I'm hoping to get fewer of them. But I met you, again, at DevOpsDays Tel Aviv, and it was glorious. And then you said, “This is fun. Come help me organize it next year.”And I, like an idiot said, “Sure, that sounds awesome because I love going to conferences and it's great. So, what's involved?” “Oh, a whole bunch of meetings.” “Okay, great.” “And planning”—things I'm terrible at—“Okay.” And then the big day finally arrives where, “Great, when do we get to get on stage and tell a story?” Like, “That's the neat part. We don't.” So, I have to ask, given that it is all behind-the-scenes work that is fairly thankless unless you really screw it up because then it's very visible, what is the point of being so involved in the community?Sharone: Wow, that's a big question, Corey.Corey: It really is.Sharone: [laugh].Corey: Because you've been involved in community for a long time and you're very good at it.Sharone: It's true. It's true. Appreciate it, thank you. So, for me, first of all, I enjoy, kind of, the people aspect of it, absolutely. And that people aspect of it actually has played out in so many different ways.Corey: Oh, you mean great people, and also me.Sharone: [laugh]. Particularly you, Corey, and we will bring you back. [laugh]. And we will make sure you chop wood and carry water because eventually it'll fill your soul, you'll see. [laugh] one of the things that really I have had the privilege and honor, and having come out of, like, kind of all my community work is really the network I've built and the people that I've met.And I've learned so much and I've grown so much, but I've also had the opportunity to connect people, connect things that you wouldn't imagine, un—seemingly-related things. So, there are so many friends of mine that have grown up with me in this community, it's been already ten years now, and a lot of folks have now been going on to new adventures and are looking to kickstart their new startup and I can connect them to this investor, I can connect them to this other person who is maybe a good, you know, partner for their startup, and hiring opportunities, and something—I've had this, like, privilege of kind of being able to connect Israel to the outer world and other things and the global kind of community, and also bring really intelligent folks into the community. And this has just created this amazing flywheel of opportunity that I'm really happy to be at the center of. And I think I've grown as a person, I think our community has grown, has learned, and there's a lot of value in that, I think, yeah. We got to meet wonderful folks like you, Corey. [laugh].Corey: It has its moments. Again, you're one of those rarities in that it's almost become a trope in VC land where VCs always like, “How may I be useful?” And it's this self-serving transparent thing. Every single time you have deigned to introduce me to someone, it's been a productive conversation and I'm always glad I took the meeting. That is no small thing.A lot of people say, “I'm good at community,” which is sort of cover for, “I'm not good at anything,” but in your case, it—Sharone: [laugh]. [I'm an entrepreneur 00:24:48].—Corey: Is very much not true. Oh, yeah. I'm a big believer that ‘entrepreneur' and ‘hero' and other terms like that are things people call you; you don't call yourself that. It always feels weird for, “Oh, he's an entrepreneur.” It's like, that's a pretty lofty word for shitposting, but okay, we'll roll with it.It doesn't work that way. You've clearly invested long-term in a building reputation for yourself by building a name for yourself in the space, and I know that whenever you reach out to me as a result, you are not there to waste my time or shill some bullshit. It is always something that is going to, even if I don't love every aspect of it or agree with the core of the message you're sending, great, it is never not going to be worth my time, which is why I'm so glad I got the chance to talk to you this show.Sharone: I appreciate that. It's something that I really believe in, I don't want to waste people's time and I really only will connect folks or only really will reach out to someone if I do think that there's something meaningful for both sides. It's never only what's in it for me, also. I also want to make sure that there's something in it for the other person and it's something that makes sense and it's meaningful for both sides. I've had the opportunity of meeting such interesting folks, and sometimes it's just like, “You must meet. [laugh]. You will love each other.” You will have so much to do together or it's so much collaboration opportunity.And so yeah, I really am that type of person. And I'll even say from a personal perspective, you know, I know a lot of people, and I've even been asked from the flip side, “Okay, is this a toxic manager? Or is this a, you know, a good hire? Is this”—and I tried to provide really authentic input so people make the right decisions, or make, you know, the right contacts, or make—and that's something I really value. And I managed to build trust with a lot of really great folks—Corey: And also me—Sharone: —and it's come back to me, also. And—[laugh] and particularly you, again. [laugh].Corey: If people want to learn more about how you see the world and the space and otherwise bask in your wisdom, where's the best place to find you?Sharone: So, I'm on Twitter as @shar1z, which is SharoneZ. Basically, everyone thinks it's such a smart, or I don't know what, like, or an esoteric screen name. And I'm like, no, it's just my name, I just—the O-N-E is… the one. [laugh].So yes, shar1z on Twitter, but also my website, rtfmplease.dev, you can reach out, there's a contact form there. You can find me on the web anywhere—LinkedIn. Reach out, I answer almost all my DMs when I can. It's very rare that I don't answer DMs. Maybe there'll be a slight lag, but I do. And I really do like when folks reach out to me. I do like it when people try and make contact.Corey: And you can also be found, of course, wherever find DevOps products are sold, on stage apparently.Sharone: [laugh]. The DevOps community, that's right. @TLVCommunity, @DevOpsDaysTLV—don't out me. All those are—yes, those are also handles that I run on Twitter, it's true.Corey: Excellent.Sharone: So, when you see them all retweeting the same tweet, yes, it's happening within same five minutes, it's me.Corey: Oh, that would have made it way easier to go viral. My God, I should have just thought of that earlier.Sharone: [laugh].Corey: Thank you so much for your time. I appreciate it.Sharone: Thank you, Corey, for having me. It's been a privilege and honor being on your show and I really do think that you are doing wonderful things in the cloud space. You're teaching us, and we're all learning, and you—keep up the good work.Corey: Well, thank you. I appreciate that.Sharone: I also want to add that on proposed marketing and whatever, I do actually listen to all of your openings of all of your shows because they're not fluffy and I like that you do, like, kind of a deep explanation, a deep technical explanation of what your sponsoring product does, and it gives a lot more insight into why is this important. So, I think you're doing that right. So, anybody who's sponsoring this show, listen. Corey knows what he's doing.Corey: Well, thank you. I appreciate that. Yay, “I know what I'm doing.” That one's going in the testimonial kit. My God.Sharone: [laugh]. That's the name of this episode, “Corey knows what he's doing.”Corey: We're going to roll with it, you know. No take-backsies. Sharone Zitzman, Chief Manual Reader at RTFM Please. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review of your podcast platform of choice, or if it's on the YouTubes smash the like and subscribe buttons, whereas if you've hated this show, exact same thing—five-star review wherever you happen to find it, smash both the buttons—but also leave an insulting comment telling me that I'm completely wrong which then devolves into an 18-page diatribe about exactly how your nonsense, bullshit product is built and works.Sharone: [laugh].Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.