About SterenSteren is a Group Product Manager at Google Cloud. He is part of the serverless team, leading Cloud Run. He is also working on sustainability, leading the Google Cloud Carbon Footprint product.Steren is an engineer from École Centrale (France). Before joining Google, he was CTO of a startup building connected objects and multi device solutions.Links Referenced: previous episode: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/google-cloud-run-satisfaction-and-scalability-with-steren-giannini/ Google Cloud Region Picker: https://cloud.withgoogle.com/region-picker/ Google Cloud regions: https://cloud.google.com/sustainability/region-carbon TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today was recently on the show. Steren Giannini is the product lead for Google Cloud Run, and we talked about that in a previous episode. If you haven't listened to it, you might wish to go back and listen to it, but it's not a prerequisite for what we're about to talk about today. Because apparently Google still does it's 20% time, and one of the things that Steren decided to do—because, you know, everyone needs a hobby—you decided to go ahead and start the Google Cloud Carbon Footprint, which is—well, Steren, thanks for coming back. What the hell is that?Steren: Thanks for having me back on the show. So yes, we started with Cloud Carbon Footprint, and this is a product that now has launched publicly, available to every Google Cloud customer right out of the box of the Google Cloud Console.Corey: I should also point out, because people always wonder and it's the first thing I always check, yes, this one is free. I'm trying to imagine a scenario which you charge for this and I wasn't incensed by it, and I can't. So, good work, you aren't charging anything for it. Good job. Please continue.Steren: So, Google Cloud Carbon Footprint helps a Google Cloud customer understand and reduce their gross carbon emissions linked to their Google cloud usage. So yeah, what do we mean by carbon emission? Just so that we are all on the same page, these are the greenhouse gases that are emitted due to the activity of using Google Cloud that are notably responsible for climate change. And we report them in equivalent of carbon dioxide—CO2—and you know, the shortcut is just to say ‘carbon.' Corey: Now, I'm going to start with something relatively controversial. It's an opinion I have around this sort of thing. And I should also disclaim, I am not in any way, shape, or form, disputing the climate change as caused by humans is real. It is. If you don't believe that, please go listen to something else, maybe Infowars. I don't know and I don't care. I just don't want you around.Now, the problem that I have with this is, on some level, it feels like a cloud provider talking to its customers about their carbon footprint is, on some level, shifting the onus of responsibilities in some way away from the cloud provider and onto the customer. Now, I freely admit that this is a nuanced topic, but how do you view this?Steren: What I mentioned is that we are exposing to customer their gross carbon emissions, but what about their net carbon emissions? Well, Google Cloud customers, net operational carbon emissions are simply zero. Why? Because if you open Google's environmental report, you will see that Google is purchasing as much renewable energy globally for the year as it is using. So, that means that on a yearly basis worldwide, every kilowatt hour of electricity has been matched with renewable energy.And you know, this Google has been doing since 2017. Since 2007, Google was already matching its carbon footprint with carbon offsets. But 2017, Google went beyond and is matching the purchase of the electricity with renewable energy. So, in a sense, your net operational emissions are zero.Now, that's not sufficient for our customers. They have some reporting obligations; they need to know before this renewable matching, what were their gross emissions? And they also need to know what are their emissions coming from, not only the electricity usage, but maybe the data center or manufacturing. And this is all of what we expose in Google Cloud Carbon Footprint. They are before offset, before renewable energy matching.And you're right also to say that this is not only the customer's problem, and indeed, Google itself has set a goal to get to a hundred percent carbon-free electricity for every hour in every location. The big goal for 2030 is that at every hour, every location, the electricity comes from carbon-free sources. This is very ambitious and never done before, of course, at the scale of Google, but this is the next goal for Google.Corey: The challenge that I have—in the abstract—with cloud providers, more or less, shaming customers—not to say that's what you're doing here—about their carbon usage and their carbon footprint is, okay, I appreciate that this is everyone's problem, and yes, it is something that we should be focusing on extensively. The counterargument is that I don't recall ever getting a meeting invite to a Google or Amazon or Microsoft or Oracle negotiation with any of your power bills or power companies or power sourcing. I have no input whatsoever as a customer on those things. And, on some level, it's “Ooh, you're causing a particular amount of carbon to be used by your usage of these services.” Like, well, at some level, it feels like that is more of a you thing than a me thing.And I want to be clear, I'm speaking more in the abstract to the industry rather than the specifics of Google Cloud, not to unfairly put you in the position of having to speak for everyone.Steren: No, but you're right. If you were to do nothing, Google is constantly working hard to sign more power purchase agreements with some renewable energy sources or optimizing its data centers. Google Cloud data centers are one of the most optimized data centers in the industry with a power usage effectiveness of 1.1, which is basically saying that the energy that is used to power the facility over the energy used to actually power the server is 1.1. So, not that much loss in between.So, all of that to say, Google Cloud and Google are working very hard anyway to reduce Google Cloud's carbon footprints and the carbon footprint of Google Cloud customers. So, if you were to do nothing, the charts that you're seeing on Google Cloud Carbon Footprint should trend to zero. But in the meantime, you know, that's not the case, so that's why we show the data. And, like, many customers want to know or have the obligation to report on this data.Corey: One of the challenges that I see—and I believe this might even be related to the carbon footprint tool you have built out on top of Google Cloud—is when I am looking at… at where to place something—first, let me just say the region experience is wildly different than I'm used to with AWS. In the AWS universe, every region is basically its own island; it is very difficult to get a holistic view between regions. Google Cloud does not have that approach. There are advantages and disadvantages to both. I'm not passing any particular value judgment—for once—on this topic in this context. But where do I want to spin something up? And I have a dropdown of the regions that I can put it in. And some of these now have a green leaf next to them and others do not. I'm going to go out on a limb and assume you had a hand in this somewhere.Steren: Exactly. That's something I worked on with the team. So, you will see on the Google Cloud Console location selectors on the Google Cloud location page, on the Google Cloud documentation, you will see a small low CO2 indicator next to some regions. And this indicator is basically saying that this region meets some criteria of high carbon-free energy percentage or low grid carbon intensity. So, you don't need to go into the details; you just need to know that if you see this small leaf, that means that for a given workload, the emissions in that particular region will be way lower than on another region which doesn't have the leaf.Often at Google, when we do a change we A/B test it. We A/B tested those small low CO2 indicators because, you know, that's a console-wide change so we want to make sure that it's worth is. And well, it turns out that for people who were in the experiment—so people will be seeing the leaf—among new Google Cloud users, they were 50% more likely to pick a low-carbon region when the leaf was displayed. And among all users, it was 19%. So, you see how just by surfacing the information, we were able to significantly influence customers' behavior towards reducing their carbon emissions.And, you know, if you ask me, I think picking the cleanest region is probably one of the simplest action you can take—if possible, of course—to reduce your gross carbon emissions because, you know, they don't require to change your architecture or your infrastructure; it just requires you to make the right choice in the first place. And just by letting people know that some regions are emitting much less carbon than others we basically allow them to reduce their footprint.Corey: A question I have is that as you continue to move up the stack, one of the things that Google has done extraordinarily well is the global network. And we talked previously about how I run the snark.cloud URL shortener in Google Cloud. That is homed out of us-central1 as far as regions go. But given that thing is effectively stateless, it just talks to Google Sheets for its source of truth, but then just runs a Docker invocation on every request, cool, I can see a scenario in which that becomes much more of a global service.In other words, if you can run that in pops in every region around the world on some level, there is no downside, from my perspective, on doing that. What I'm wondering then, as a result of that, is as you start seeing the evolution of services becoming more and more global, instead of highly region-specific, does that change the way that we should be thinking potentially about carbon footprint and regional selection? Or is that too much of a niche edge case to really be top of radar right now?Steren: Oh, there are many things to talk about here. The first one is that you might be hinting at something that Google is already doing, which is location shifting of workloads in order to optimize power usage, and, you know of course, carbon emissions. So, Google itself is already doing that. For example, I guess, to process YouTube videos, that can be done, not necessarily right away and that can be done in the location in which, for example, the sun is shining. So, there are some very interesting things that can be done if you allow the workloads to be run in not necessarily a specific region.Now, that being said, I think there are many other things that people consider when they pick a region. First, well, maybe they have some data locality constraints, right? This is very much the case in European countries where the data must stay in a given region, by law. Second, well, maybe they care about the price. And as you probably know, [laugh] the price of cloud providers is not the same in every region.Corey: I've noticed that and in fact, I was going to get into that as our next transition, but you've just opened Pandora's Box early. It's great to have the carbon-friendly indicator next to the region, but I also want number of dollar signs next to it as well. Like in AWS-land, do you have the tier one regions where everything is the lowest price: us-east-1, us-west-2, and a few others escaped me from time to time, where Managed NAT Gateways are really expensive. And then you go under some others and they get even more expensive, somehow. Like, talk about pushing the bounds of cloud economics. It's astonishing to me.Steren: Yes. And so—Corey: Because I want that display, on some level—Steren: Exactly.Corey: —as a customer, in many cases.Steren: So, there is price, there is carbon, but of course, you know, if you are serving web requests, there is probably also latency that you care about, right? Even if—for example, Finland is very low carbon. You might not host your workloads in Finland if you want to serve US customers. So, in a sense, there are many dimensions to optimize when you pick a region. And I just sent you a link to something that I built, which is called Google Cloud Region Picker.It's basically a tool with three sliders. First one is carbon footprint; you tell us how much you care about that. Hopefully, you put it to the right. The second one is lower price. So, how much do you want the tool to optimize to lower your bill? And third one is latency, and then you tell us where your users are coming from and if you care about latency.Because some workloads are not subject to latency requirements. Like, if you do batch jobs, well, that doesn't serve a user request, so that can be done asynchronously at a later time or in a different place. And what this tool does is that it takes your inputs and it basically tells you which Google Cloud region is the best fit for you. And if you use it, you will see it has very small symbols like three dollars for the most expensive regions, one dollar for the least expensive ones, three leaves for the greenest regions, and zero leaves for the non-green one.Corey: This is awesome. I'm a little bit disappointed that I hadn't seen this before. This is a thing of beauty.Steren: Yeah. Again, done by me as a 20%. [laugh]. And, you know, the goal is to educate, like, of course, it's way more complex. Like, you know that price optimization is way more complex than a slider, but the goal of this tool is to educate and to give a first result. Like, okay, if you are in France and care about carbon, then go here. If you are in Oregon, go here. And so, many parameters that this tool help you optimize in a very simple way.Corey: One of the challenges I think I get into when I look at this across the board, is that you have a couple of very different ends on a confusing spectrum, by which I mean that one of the things I would care about from a region picker, for example, is there sufficient capacity in that region for the things I want to run. At my scale of things where right now on Google Cloud I run a persistent VM that hangs out all the time, and I run some Google Cloud Run stuff. Great. If you have capacity problems with either one of those, are you really a cloud?But then we have other folks who are spinning up tens or hundreds of thousands of a very particular instance type in a very specific region. That's the sort of thing that requires a bit more in the way of capacity planning and the rest. So, I have to imagine for those types of use cases, this tool is insufficient. The obvious reason, of course, if you're spinning up that much of anything, for God's sake, reach out and talk to your account manager before trying to do it willy-nilly but yes.Steren: That's exactly right. So, as I said, this tool is simplified tool to give, like, the vast majority of users a sense of where to put their workloads. But of course, if you're a very big enterprise customer who is going to sign a very big deal with Google Cloud, talk to your account manager because if you do need a lot of capacity, Google Cloud might need to plan for it. And not every regions have the same capacity and we are always working with our customers to make sure we direct them in the right place and have enough capacity. A real-life example of a very high profile Google Cloud customer was that they were selecting a region without knowing its carbon impact, and when we started to disclose the carbon characteristics of Google Cloud regions—which is another link we can send to the audience—this customer realized that the region they selected—you know, maybe because it was close to their user base—was really not the most carbon friendly.So, they decided to switch to another one. And if we take an example, if you take Las Vegas, it has a carbon-free energy percentage of 20%. So, that basically means that on average, 20% of the time, the electricity comes from carbon-free sources. If you are to move to Oregon, this same workload, Oregon has a carbon-free energy percentage of 90%. So, you can see how just by staying on the West Coast, moving from Las Vegas to Oregon, you have drastically reduced your carbon emissions. And your bill, by the way because it turns out Oregon is one of the cheapest Google Cloud Data Center. So, you see how just being aware of those numbers led some very important customers who care about sustainability to make some fundamental choices when it comes to the regions they select.Corey: I guess that leads to my big obvious question, where I wind up pulling up my own footprint in Google Cloud—again, I don't run much there—and apparently over the last year, I've had something on the order of two kilograms of carbon. Great. It feels like for this scale, I almost certainly burn more carbon than that just searching Google for carbon-related questions about where to place things. So, for my use case, this is entirely academic. You can almost run my workloads based upon, I don't know, burning baby seals or something, and the ecological footprint does not materially change.Then we go to the other extreme end of the spectrum with the hundreds of thousands of instances, where this stuff absolutely makes a significant and massive difference. My question is, when should people begin thinking about the carbon footprint of their cloud workload at what point of scale?Steren: So, as you said, a good order of magnitude is one transatlantic flight is a thousand kilogram of equivalent CO2. So, you see how just by flying once, you're already, like, completely overshadowing your Google Cloud carbon footprint. But that's because you are not using a lot of Google Cloud resources. Overall, you know, I think your question is basically the same as when should individuals try to optimize reducing their carbon footprint? And here I always recommend there are tons of things you can optimize.Start by the most impactful ones. And impactful means an action will have a lot of impact in reducing the footprint, but also the footprint reduction will be significant by itself. And two kilograms of CO2, yes indeed, it is very low, but if you start reaching out into the thousands of kilograms of CO2 that starts to represent, like, one flight, for example. So, you should definitely care about it. And as I said, some actions might be rather easy, like picking the right region might be something you can do pretty easily for your business and then you will see your carbon emissions being divided by, you know, sometimes five.This episode is sponsored in part by our friends at Lambda Cloud. They offer GPU instances with pricing that's not only scads better than other cloud providers, but is also accessible and transparent. Also, check this out, they get a lot more granular in terms of what's available. AWS offers NVIDIA A100 GPUs on instances that only come in one size and cost $32/hour. Lambda offers instances that offer those GPUs as single card instances for $1.10/hour. That's 73% less per GPU. That doesn't require any long term commitments or predicting what your usage is gonna look like years down the road. So if you need GPUs, check out Lambda. In beta, they're offering 10TB of free storage and, this is key, data ingress and egress are both free. Check them out at lambdalabs.com/cloud. That's l-a-m-b-d-a-l-a-b-s.com/cloud.Corey: I want to challenge your assertion, incidentally. You say that I'm not using a whole lot of Google Cloud resources. I disagree. I use roughly a dozen different Google Cloud resources tied together for some of these things, but they're built on serverless design patterns, which means that they scale to nothing. I'm not sitting there with an idle VM—except that one—that is existing on a persistent basis.For example, I look at the things that show up on the top five list. Compute Engine is number one, Cloud Run, Cloud Logging, Cloud Storage, and App Engine are the rest that are currently being used. I think there's a significant untold story around the idea of building in a serverless way for climate purposes.Steren: Yes. So, maybe for those who are not aware of what you are seeing on the dashboard, so when you open this Google Cloud Carbon Footprint tool on the Cloud Console, you saw a breakdown of your yearly carbon footprint and monthly carbon footprint across a few dimensions. The first one is the regions because as we said, this matters a lot; like, the regions have a lot of impact. The second one are the month; of course, you can see over time, how you're trending. The third one is a concept called Google Cloud Project, which is, for those who are not aware, it's a way to group Google Cloud resources into buckets.And the third one is Google Cloud services. So, what you described here is, which of your services emits the most and therefore which ones should you optimize first? Like, again, to go back to impactful actions. And to your point, yes, it is very interesting that if you use products which auto-scale, basically, the carbon attributed to you, the customer, will really follow this auto-scaling behavior. Compare that to a virtual machine that is always on, burning some CPU for almost nothing because you have a server that doesn't process requests. That is wasting, in a sense, resources.So, what you describe here is very interesting, which is basically the most optimized products you're going to pick, the less waste you're going to have. Now, I also want to be careful because comparing one CPU hour of Cloud Run and one CPU hour of Compute Engine is not comparing apples to apples. Why? Because when you use Cloud Run, I'm not sure if you know, but you are using a regional product. So, a product which has built-in redundancy, which is safe in case of one zone going down in a region.But that means the Cloud Run infrastructure has to provision a little bit more machines than if it was a zonal product. While Compute Engine, your virtual machine lives in one zone and there is only one machine for you. So, you see how we should also be careful comparing products with other products because fundamentally, they are not offering the same value and they are not running on the same infrastructure. But overall, I think you are correct to say that, you know, avoiding waste, using auto-scaling products, is a good way to reduce your footprint.Corey: I do want to ask—and this is always a delicate topic because you're talking about cultural things—how much headwind did you have internally at Google when you had the idea to start exposing this? How difficult was it to bring this to fruition?Steren: I think we are lucky that our leadership cares about reducing carbon emissions and understood that our customers needed our help to understand their cloud emissions. Like, many customers before we had this tool, we're trying to some kind of estimate their cloud emissions. And it was—you know, Google Cloud was a black box for them. They did not have access to what you said, to some data that only Google has access to.And you know, to build that tool, we are using energy measurement of every machine in every data center. We are using, you know, customer-wide resource usage. And that is something that we use to divide the footprint among customers. So, there is some data used to compute those numbers that only Google Cloud has access to. And indeed, you're correct; it required some executive approval which we received because many of our leaders believe that, you know, this is the right thing to do, and this is helping customers towards the same goal as Google, which is being net-zero and carbon-free.Many of our customers have made some sustainability commitments, and they need our help to meet those goals. So yeah, we did receive approval, first to share the per-region characteristics. This was already, you know, a first in the industry where a cloud provider disclosed that not every region is equal and some are emitting more carbon than others. And second, another approval which was to disclose a per customer carbon footprint, which is broken down by service, project, region, using some, you know, if you touch a little bit on the methodology, you know, it uses energy consumption, resource usage, and carbon intensity coming from a partner of ours to compute, basically, a per customer footprint.Corey: My question for you is, on some level, given that Google is already committed to being net-zero across the board for all of its usage, why do customers care? Why should they care? Effectively, haven't you made that entirely something that is outside of their purview? You've solved the problem, either way.Steren: This is where we should explore it a bit more the kinds of carbon emissions that exist. For a customer, their emissions linked to the cloud usage is all considered the indirect emissions. This, in the Greenhouse Gas Protocol Standard, this is called Scope 3. So, our Google Cloud emissions are the customers' Scope 3 emissions; they are all indirect for them. But those indirect emissions, what I mentioned as being net-zero are the emissions coming from electricity usage.So, to power those data centers, those data centers are located in certain electricity grids. Those electricity grids might be using energy sources that emit more or less carbon, right? Simply put, if in a given place, the electricity comes from coal, it will be emitting a lot of carbon compared to when electricity comes from solar, for example. So, you see how the location itself determines the carbon intensity. And these are the emissions coming from electricity usage, right?So, these are neutralized by Google purchasing as much renewable energy. But there are also types of emissions. For example, when a data center loses connection to the grid, we startup diesel generators. Those diesel generators will directly emit carbon. This is called Scope 1 emissions.And finally, there is the carbon emissions that are coming from the manufacturing of those servers, of those data centers. And this is called Scope 3 emissions. And the goal of Google is for the emissions coming from electricity to be always coming from carbon-free sources. So, this is a change that we've recently released to Google Cloud Carbon Footprint, which is now we also break down your emissions by scope. So, they are all Scope 3 for you, the customer, they are all indirect emissions for you, the customer, but now those indirect emissions, you can see how much is coming from diesel generators, how much is coming from electricity consumption, and how much is coming from manufacturing of the data center, and other, like, upstream, downstream activities. And yeah, overall, this is something that customers do need to report on.Corey: I think that's very fair. I do want to thank you for taking so much time to speak with me. And instead of the usual question I'd like to ask here of where can people go to find out more because we have a bunch of links for that, instead, I want to ask something a little bit different here, which is, what are the takeaways that customers or prospective customers should really have around their carbon footprint when it comes to cloud?Steren: So, I would recommend our audience to consider carbon emissions in your cloud infrastructure decisions. And my advice is, first, move to the cloud. Like, we've talked that Google Cloud has very well-optimized data centers. Like, your cloud gross carbon emissions are anyway going to be much lower than any on-premise carbon emissions. And by the way, if you use Google Cloud, your net operational emissions are zero.Second action is pick the region with the lowest carbon impact. Like we discussed that this is probably a low-effort action, if possible, that will have a lot of impact on your gross carbon emissions. And you know, if you want to go further, try to schedule those workloads when the electricity is the greenest, you know, when the sun is shining, the wind is blowing, for example, or try to schedule those workloads in regions which have the lowest impact. And yeah, Google Cloud gives you all the tools to do that, the tools to optimize your region selection, and the tools to report and reduce your gross carbon emissions. We haven't talked about it, but Google Cloud Carbon Footprint will even send you some proactive recommendations of things to do to reduce your emissions.For example, if you have a project, a machine that you forgotten, Google Cloud Carbon Footprint, will recommend you to delete it and we'll tell you how much carbon you would save by deleting it, as well as dollar, of course.Corey: It's funny because I feel like there's a definite alignment between my view of cloud economics and the carbon perspective on this, which is step one, everyone wins if you turn things off when you're not using them. What a concept. I sometimes try and take it too far of, ‘turn off all of production because your company's terrible.' Yeah, it turns out, that doesn't work super well. But the idea of step one, turn it off, especially when you're not using it. And if you're never using it, why would you want to pay for it? That becomes a very clear win for everyone involved. I think that in the fullness of time, economics are what are going to move the needle on driving further adoption about this. I have to guess that you see the same thing from where you are?Steren: Yes, very often working to reduce your carbon footprint is also working to reduce your bill. And we've also observed—not always—but some correlation between regions that have the lowest carbon impact and regions that are the cheapest. So, in a sense, this region selection, optimizing for price and carbon is often optimizing for the same thing. It's not always true, but it is often true.Corey: I really want to thank you for spending so much time to talk with me about this. This has definitely giving me a lot of food for thought, and I have to imagine that this will not be our last conversation around the topic.Steren: Well, thanks for having me. And I'm very happy to talk to you in the podcast, of course.Corey: Steren Giannini, product lead for Google Cloud Carbon Footprint and Google Cloud Run. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry screed about how climate change isn't real as you sit there wondering why it's 120 degrees in March.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About AlexAlex Su is a lawyer who's currently the Head of Community Development at Ironclad, the #1 contract lifecycle management technology company that's backed by Accel, Sequoia, Y Combinator, and other leading investors. Prior to joining Ironclad, Alex sold cloud software to legal departments and law firms on behalf of early stage startups. Alex maintains an active presence on social media, with over 180,000 followers across Twitter, LinkedIn, Instagram, and TikTok. Links Referenced: Ironclad: https://ironcladapp.com/ LinkedIn: https://www.linkedin.com/in/alexander-su/ Twitter: https://twitter.com/heyitsalexsu Instagram: https://www.instagram.com/heyitsalexsu/ TikTok: https://www.tiktok.com/@legaltechbro TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it's still somebody's problem. And a big part of that responsibility is app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you're actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That's S-N-Y-K.co/screamCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I've been off the beaten path from the traditional people building things in cloud by the sweat of their brow and the snark on their Twitters. I'm joined today by Alex Su, who's the Head of Community Development at Ironclad, and also relatively well-renowned on the TikToks, as the kids say. Alex, thank you for joining me.Alex: Thank you so much for having me on the show.Corey: It's always been an interesting experience because I joined TikTok about six months or so ago, due to an escalatingly poor series of life choices that continue to fail me, and I have never felt older in my life. But your videos consistently tend to show up there. You are @legaltechbro, which sounds like wow, I hate all of those things, and yet your content is on fire.How long have you been doing the public dance thing, for lack of a better term? I don't even know what they call it. I know how to talk about Twitter. I know how to talk about LinkedIn—sad. LinkedIn is sad—but TikTok is still something I'm trying to wrap my ancient brain around.Alex: Yeah, I felt out of place when I first made my first TikTok. And by the way, I'm known for making funny skits. I have actually never danced. I've always wanted to, but I don't think I have that… that talent. I started posting TikToks in, I will call it—let's call it the fall of 2020. So, after the pandemic.Before that, I had been posting consistently on LinkedIn for, gosh, ever since 2016, when I got into legal tech. And during the pandemic, I tried a bunch of different things including making funny skits. I'd seen something somewhere online if somebody's making fun of the doctor life. And so, I thought, hey, I could do that for legal too. And so, I made one with iMovie. You know, I recorded it on Zoom.And then people started telling me, “Hey, you should get on this thing called TikTok.” And so, I resisted it for a while because I was like, “This is not for me.” But at some point, I said, “I'll try this out. The editing seems pretty easy.” So, I made a couple of videos poking fun at the life of a law firm lawyer or a lawyer working for a corporate legal department.And on my fourth video, I went massively viral. Like, unexpected went viral, like, millions of—I think two million or so views. And I found myself with a following. So, I thought, “Hey, I guess this is what I'm doing now.” And so, it's been, I don't know, a year-and-a-half since then, and I've been continuously posting these skits.Corey: It's like they say the worst thing can happen when you go into a casino and play for the first time is you win.Alex: [laugh].Corey: You get that dopamine hit, and suddenly, well now, guess what you're doing for the rest of your life? There you go. It sounds like it worked out for you in a lot of fun ways. Your skits about big law of life definitely track. My wife used to work in that space, and we didn't meet till she was leaving that job because who has time to date in those environments?But I distinctly remember one of our early dates, we went out to meet a bunch of her soon-to-be-former coworkers at something like eight or nine o'clock in Los Angeles on a Friday night. And at the end of it, we went back to one of our places, and they went back to work. Because that is the lifestyle, apparently, of being in big law. I don't have the baseline prerequisites to get into law school, to let alone get the JD and then go to work in big law, and looking at that lifestyle, it's, “Yeah, you know, I don't think that's for me.” Of course, I say that, and then three days later, I was doing a middle of the night wake up because the pager went off.Like, “Oh, are you a doctor?” And the pager is like, “Holy shit. This SSL certificate expires in 30 days.” It's, yeah. Again, life has been fun, but it's always been one of those things that was sort of, I guess, held in awe. And you're putting a very human face on it.Alex: Yeah. You know, I never expected to be in big law either, Corey. Like, I was never good at school, but as I got older, I found a way to talk my way into, like, a good school. I hustled my way into a job at a firm that I never imagined I could get a job at. But once I got in, that's when I was like, “Okay, I don't feel like I fit in.”And so, I struggled but I still you know grinded it out. I stayed at the job for a couple of years. And I left because I was like, “This is not right for me.” But I never imagined that all of those experiences in big law ended up being the source material for my content, like, eight years after I'd left. So, I'm very thankful that I had that experience even if it wasn't a good fit for me. [laugh].Corey: And on some level, it feels like, “Where do you get your material from?” It's, “Oh, the terrible things that happened to me. Why do you ask?”Alex: That's basically it. And people ask me, they say, you know, “You haven't worked in that environment for eight years. It's probably different now, right?” Well, no. You know, the legal industry is not like the tech industry. Like, things move very slowly there.The jokes that made people laugh back then, you know, 10 years ago, even 20 years ago, people still laugh at today because it's the same way things have always worked. So, again, I'm very thankful that that's been the case. And, you know, I feel like, the reason why my content is popular is because a lot of people can resonate with it. Things that a lot of people don't really talk about publicly, about the lifestyle, the culture, how things work in a large firm, but I make jokes about it, so people feel comfortable laughing about it, or commenting and sharing.Corey: I want to get into that a little bit because when you start seeing someone pop up again and again and again on TikTok, you're one of those, “Okay, I should stalk this person and figure out what the hell their story is.” And I didn't have to look very far in your case because you're very transparent about it. You're the head of community development at a company called Ironclad, and that one threw me for a little bit of a loop. So, let's start with the easy question, I suppose. What is Ironclad?Alex: We're a digital contracting technology that helps accelerate business contracts. Companies deal with contracts of all types; a lot of times it gets bogged down in legal review. We just help with that process to make that process move faster. And I never expected I'd be in this space. You know, I always thought I was going to be a trial lawyer.But I left that world, you know, maybe six years ago to go into the legal technology space, and I quickly saw that contracts was kind of a growing challenge, contracting, whether it's for sales or for procurement. So, I found myself as a salesperson in legal tech selling, first e-discovery software, and then contracting software. And then I found my way to Ironclad as part of the community team, really to talk about how we can help, but also speaking up about the challenges of the legal profession, of working at a law firm or at a legal department. So, I feel like it's all been the culmination of all my experiences, both in law and technology.Corey: In the world in which I've worked, half of my consulting work has been helping our clients negotiate their large-scale AWS contracts and the other half is architectural nonsense of, “Hey, if you make these small changes, that cuts your bill in half. Maybe consider doing them.” But something that I've learned that is almost an industry-wide and universal truism, is that you want to keep the salespeople and the lawyers relatively separate just due to the absolute polar opposites of incentives. Salespeople are incentivized to sell anything that holds still long enough or they can outrun, whereas lawyers are incentivized to protect the company from risk. No, is the easy answer and everything else is risk that has to be managed. You are one of those very rare folks who has operated successfully and well by blending the two. How the hell did that happen?Alex: I'm not sure to this day how it happened. But I think part of the reason why I left law in the first place was because I don't think I fit in. I think there's a lot of good about having a law degree and being part of the legal profession, but I just wanted to be around people, I wanted to work with people, I didn't want to always worry about things. And so, that led me to technology sales, which took me to the other extreme. And so, you know, I carried a sales quota for five years and that was such an interesting experience to see where—to both sell technology, but also to see where legal fit into that process.And so, I think by having the legal training, but also having been part of a sales team, that's given me appreciation for what both teams do. And I think they're often at tension with one another, but they're both there to serve the greater goals of the company, whether it's to generate revenue or protect against risk.Corey: I think that there's also a certain affinity that you may have—I'm just spitballing wildly—one of the things that sales folks and attorneys tend to have in common is that in the public imagination, as those roles are not, shall we call it, universally beloved. There tend to be a fair number of well, jokes, in which case, both sides of that tend to be on the receiving end. I mean, at some level, all you have to do is become an IRS auditor and you've got the holy trifecta working for you.Alex: [laugh]. I don't know why I gravitated to these professions, but I do think that it's partly because both of these roles hold a significant amount of power. And if you look at just contracting in general, a salesperson at a company, they're really the driver of the sales process. Like, if there's no sale to be made, there's no contract. On the flip side, the law person, the lawyer, knows everything about what's inside of the contract.They understand the legal terms, the jargon, and so they hold an immense amount of power over advising people on what's going to happen. And so, I think sometimes, salespeople and legal people take it too far and either spend too much time reviewing a contract and lording it over the business folks, or maybe the salesperson is too blase about getting a deal done and maybe bypasses legal and doesn't go through the right processes. By the way, Corey, these are jokes that I make in my TikToks all the time and they always go viral because it's so relatable to people. But yeah, that's probably why people always make jokes about lawyers and salespeople. There's probably some element of ridiculing people with a significant amount of power within a company to determine these transactions.Corey: Do you find that you have a better affinity for the folks doing contract work on the seller side or the buyer side? Something they don't tell you when you run companies is, yeah, you're going to spend a lot of time working on contracts, not just when selling things, but also when buying things and going back and forth. Aspects of what you're talking about so far in this conversation have resonated, I guess, with both sides of that for me. What do you have the affinity for?Alex: I think on the sales side, just because of my experience, you know, I think when you go through a transaction and you're trying to convince someone to doing something, and this is probably why I wanted to go to law school in the first place. Like I watched those movies, right? I watched A Few Good Men and I thought I'd be standing up in court convincing a jury of something. Little did I know that that sort of interest [crosstalk 00:10:55]—Corey: Like, Perry Mason breakthrough moment.Alex: That moment where—the gotcha moment, right? I found that in sales. And so, it was really a thrill to be able to, like, talk to someone, listen to them, and then kind of convince them that, based on what challenges they're facing, for them to buy some technology. I love that. And I think that was again, tied to why I went to law school in the first place.I didn't even know sales was a possible profession because I grew up in an immigrant community that was like, you just go to school, and that'll lead to your career. But there's a lot of different careers that are super interesting that don't require formal schooling, or at least the seven years of schooling you need for law. So, I always identify with the sales side. And maybe that's just how I am, but obviously, the folks who deal with the buy side, it's a pretty important job, too.Corey: There's a lot of surprise when I start talking to folks in the engineering world. First, they're in for a rough awakening at times when they learn exactly how much qualified enterprise salespeople can make. But also because being a lawyer without, you know, the appropriate credentials to tie into that, you're going to have a bad time. There are regulatory requirements imposed on lawyers, whereas to be a salesperson, forget the law degree, forget the bachelor's, forget the high school diploma, all you really need to be able to do from an academic credential standpoint is show up.The rest of it is, can you actually sell? Can you have the conversations that convince people to see the outcome that benefits everyone? And I don't know what that it's possible, or advised necessarily, to be able to find a way to teach that in some formalized way. It almost feels like folks either have that spark or they don't. Do you think it's one of those things that can be taught? Do you think it's something that people have to have a pre-existing affinity for?Alex: It's both, right, because part of it is some people will just—they don't have the personality to really sell. It's also like their interest; they don't want to do that. But what I found that's interesting is that what I thought would make a good salesperson didn't end up being true when I looked at the most effective sellers. Like, in my head, I thought, “Oh, this is somebody who's very boisterous, very extroverted,” but I found that in my experience in B2B SaaS that the most effective sellers are very, very much active listeners. They're not the people showing up and talking at you. They are asking you about your day-to-day asking about processes, understanding the context of your situation, before making a small suggestion about what you might want to do.I was very impressed the first time I saw one of these enterprise sellers who was just so good at that. Like, I saw him, and he looked nothing like what I imagined an effective sales guy to look like. And he was really kind and he just, kind of, just talked to me, like, I was a human being, and listened to my answers. So, I do think that there is some element of nature, your talent when it comes to that, but it can also be trained because I think a lot of folks who have sales talent, they don't realize that they could be good at it. They think that they've got to be this extroverted, happy hour, partying, storyteller, where —Corey: The Type A personality that interrupts people as they're having the conversation.Alex: Yeah, yeah.Corey: Yeah.Alex: So anyways, I think that's why it's a mix of both.Corey: The conversations that I've learned the most from when I'm talking to prospects and clients have been when I asked the quote-unquote, dumb question that I already know the answer to, and then I shut up and I listen. And wow, I did not expect that answer. And when you dig a little further, you realize there's nuance that—at least in my case—that I've completely missed to the entire problem space. I think that is really one of the key differentiators to my mind, that separate people who are good at this role from folks who just misunderstand what the role is based upon mass media, or in other cases—same problem with lawyers—the worst examples, in some cases, of the profession. The pushy used car salesperson or the lawyer they see advertising on the back of a bus for personal injury cases. The world is far more nuanced than that.Alex: Absolutely. And I think you hit the nail on the head when you said, you know, you ask those questions and let them talk. Because that's an entire process within the sales process. It's called discovery, and you're really asking questions to understand the person's situation. More broadly, though, I think pitching at people doesn't seem to work as well as understanding the situation.And you know, I've kind of done that with my content, my TikToks because, you know, if you look at LinkedIn, a lot of people in our space, they're always prescribing solutions, giving advice, posting content about teaching people things. I don't do that. As a marketer, what I do is I talk about the problems and create discussions. So, I'll create a funny video—Corey: I think you're teaching a whole generation that maybe law school isn't what they want to be doing, after all there is that.Alex: There is that. There is that. It's a mix of things. But one of the things I think I focus on is talking about the challenges of working with a sales team if you're an in-house lawyer. And I don't prescribe technology, I don't prescribe Ironclad, I don't say this is what you need to do, but by having people talk about it, they realize, right—and I think this is why the videos are popular—as opposed to me coming out and saying, “I think you need technology because of XYZ.” I think, like, facilitating the conversation of the problem space, that leads people to naturally say, “Hey, I might need something. What do you guys do, by the way?”Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: It sounds ridiculous for me to say that, “Oh, here's my entire business strategy: step one, I shitpost on the internet about cloud computing; step two, magic happens here; and step three people reach out to talk about their AWS bills.” But it's also true. Is that the pattern that you go through: step one, shitpost on TikTok; step two, magic happens here; and step three people reach out asking to learn more about what your company does? Or is there more nuance to do it?Alex: I'm still figuring out this whole thing myself, but I will say shitposting is incredibly effective. Because I'm active on Twitter. Twitter is where I start my shitposts. TikTok, I also shitpost, but in video format, I think the number one thing to do is figure out what resonates with people, whether it's the whole contracting thing or if it's frustrations about law school. Once you create something that's compelling, the conversation gets going and you start learning about what people are thinking.And I think that what I'm trying to figure out is how that can lead to a deeper conversation that can lead to a business transaction or lead to a sale. I haven't figured it out, right, but I didn't know that when I started creating content that spoke to people when I was a quota-carrying salesperson, people reached out to me for demo requests, for sales conversations. There is something that is happening in this quote-unquote, “Dark funnel,” that I'm sure you're very familiar with. There's something that's happening that I'm trying to understand, and I'm starting to see.Corey: This is probably a good thing to the zero in on a bit because to most people's understanding of the sales process, it would seem that you going out and making something of a sensation out of yourself on the internet, well what are you doing that for? That's not sales work? How is that sales? That's just basically getting distracted and going to do something fun. Shouldn't you be picking up the phone and cold calling people or mass-emailing folks who don't want to hear from you because you trick them into having a badge scanned somewhere? I don't necessarily think that is accurate. How do you see the interplay of what you do and sales?Alex: When you're selling something like makeup or clothing, it's a pretty transactional process. You create a video; people will buy, right? That's B2C. In B2B, it's a much more complex processes. There's so many touchpoints. The start of a sales conversation and when they actually buy may take six months, 12 months, years. And so, there's got to be a lot of touch points in between.I remember when I was starting out in my content journey, I had this veteran enterprise sales leader, like, your classic, like, CRO. He said to me, “Hey, Alex, your content's very funny, but shouldn't you be making cold calls and emails? Like, why are you spending your time doing this?” And I said, “Hey, listen, do you notice that I'm actually sourcing more outbound sales calls than any other sales rep? Like, have you noticed that?”And he's like, “Actually, yeah, I did notice that. You know, how are you doing it?” And I was like, “Do you not see that these two are tied? These are not people I just started calling. They are people who have seen my content over time. And this is how it works.”And so, I think that the B2B world is starting to wise up to this. I think, for example, Ironclad is leading the way on creating a community team to create those conversations, but plenty of B2B companies are doing the same thing. And so, I think by inserting themselves in a conversation—a two-way conversation—during that process, that's become incredibly effective, far more so than, like, cold-calling a lawyer or a developer who doesn't want to be bothered by some pushy salesperson.Corey: Busy, expensive professionals generally don't want to spend all their time doing that. The cold outreach emails that drive me nuts are, “Hey, can we talk for half an hour?” Yeah, I don't tend to think in terms of billable hours because that's not how I do anything that I do, but there is an internal rate that I used to benchmark and it's what you want me just reach into my pocket and give you how much money for a random opportunity to pitch me on something that you haven't even qualified whether I need or not? It's like, asking people for time is worse, in some ways, than asking for money because they can always make more money, but no one can make more time.Alex: Right, right. That's absolutely right.Corey: It's the lack of awareness of understanding the needs and motivations of your target market. One thing that I found that really aided me back when I was working for other folks was trying to find a company or a management structure that understood and appreciated this. Easy example, when I was setting out as an independent consultant after a few months I'd been doing this and people started to hear about me. But you know, it turns out that there are challenges to running a business that are not recommended for most people. And I debated, do I take a job somewhere else?So, I interviewed at a few places, and I was talking to one company that's active in the cloud costing space at the time and they wanted me to come aboard. But discussions broke down because they thought I was, quote, “More interested in thought leadership than I was and actually fixing the bills themselves.” And looking at this now, four years later or so, yeah, they were right. And amazing how that whole thing played out, but that the lack of vision around, there's an opportunity here, if we can chase it, at least in the places I was at, was relatively hard to come by. Did you luck out in finding a role that works for you in this way or did you basically have to forge it for yourself from the sweat of your brow and the strength of your TikTok account?Alex: It was uphill at first, but eventually, I got lucky. And you know, part of it was engineered luck. And I'll explain what I mean. When I first started out doing this, I didn't expect this to lead to any jobs. I just thought it would support my sales career.Over time, as the content got more popular, I never wanted to do anything else because I was like, I don't want to be a marketer. I'm not a—I don't know anything about demand gen. All I know is how to make funny videos that get people talking. The interesting that happened was that these videos created this awareness, this energy in our space, in the legal space. And it wasn't long before Ironclad found me.And you know, Ironclad has always been big on community, has always done things like—like, our CEO, our founder, he said that he used to host these dinners, never talking about Ironclad, but just kind of talking about law school and law with potential clients. And it would lead to business. Like, it's almost the same concept of, like, not pushing sales on people. And so, Ironclad has always had that in its DNA. And one of our investors, our board members, Jessica Lee from Sequoia, she is a huge believer in community.I mean, she was the CEO of another company that leveraged community, and so there's this community element all throughout the DNA of Ironclad. Now, had I not put myself out there with this content, I may not have been discovered by Ironclad. But they saw me, they found me, and they said, “We don't think about these things like many other companies. We really want to invest in this function.” And so, it's almost like when you put yourself out there, yes, sometimes some people will say, “What are you doing? Like, this makes no sense. Like, stop doing that.” But there's going to be some true believers who come out and seek you out and find you.And that's been my experience here, like, at Ironclad. Like, people were like, “When you go there, are they going to censor you? Is your content going to be less edgy?” No. Like, they pulled me aside multiple times and said, “Keep being yourself. This is what we want.” And I think that is so special and unique. And part of it is very much lucky, but it's also when you put yourself out there kind of in a big way, like-minded people will seek you out as well.Corey: I take the position that part of marketing, part of the core of marketing, is you've got to have an opinion. But as soon as you have an opinion, people are going to disagree with you. They're going to, effectively, forget the human on the other side of it and start taking you for a drag on social media and whatnot. So, the default reaction a lot of people have is oh, I shouldn't venture opinions forward.No. People are always going to dislike you for something and you may as well have it be for who you are and what you want to be doing rather than who you're pretending to be. That's always been my approach. For me, the failure mode was not someone on Twitter is going to get mad about what I wrote. No one's going to read it. That's the failure mode. And the way to avoid that is make it interesting.Alex: That is a hundred percent relatable to me because I think when I was younger, I was scared. I did worry that I would get in trouble for what I posted. But I realized these people I was worried about, they weren't going to help me anyways. These are not people who are going to seek me out and help me but then say, “Oh, I saw your content, so now I can't help you.” They were not going to help me anyways.But by being authentic to myself and putting things out there, I attracted my own tribe of people who have helped me, right? A lot of my early results from content came not because I reached my target customers; it was because somebody resonated with what I put out there and they carried my message and said, “Hey, you should talk to Alex.” Something special happens when you kind of put yourself out there and say an opinion or share a perspective that not everyone agrees with because that tribe you build ends up helping you a lot. And meanwhile, these other people that might not like it, they probably weren't going to help you either.Corey: I maintain that one of the most valuable commodities in the universe is attention. And so, often there's so much information overload that's competing for our attention every minute of every day that trying to blend in with the rest of it feels like the exact wrong approach. I'm not a large company here. I don't have a full marketing department to wind up doing ad buys, and complicated campaigns, and train a team of attacking interns to wind up tackling people to scan their badges at conferences. I've got to work with what I've got.So, the goal I've always had is trigger the Rolodex moment where someone hears about a problem in the AWS billing space—ideally—and, “Oh, my God, you need to talk to Corey about that.” And it worked, for better or worse. And a lot of it was getting lucky, let's be very clear here, and people doing me favors that they had no reason to do and I'll never be able to repay. But being able to be in that space really is what made the difference. Now, the downside, of course, when you start doing that is, how do you go back to what happened before?If you decide okay, well, it's been a fun run for you and Ironclad. And yeah, TikTok. Turns out that is, in fact, for kids; time to go somewhere else. Like, I don't know that you would fit into your old type of job.Alex: Yeah. No, I wouldn't. But very early on, I realized, I said, “If I'm going to find meaningful work, it's okay to be wrong.” And when I went to big law, I realized this is not right for me. That's okay. I'm just not going to get another big law job.And so, when people ask me, “Hey, now that you've put yourself out there, you probably can't get a job at a big firm anymore.” And that's okay to me because I wasn't going to go back anyways. But what I have found, Corey, is that there's this other universe of people, whether it's a entrepreneur, smaller businesses, technology companies, they would be interested in working with me. And so, by being myself, I may have blocked out a certain level of opportunities or a safety net, but now I'm kind of in this other world where I feel very confident that I won't have trouble finding a job. So, I feel very lucky to have that, but that's why I also don't worry about the possibility of not going back.Corey: Yeah, I've never had to think about the idea of, well, what if I go have to get a job again? Because at that point, it means well, it's time to let every one at the company who is depending on the go, and that's the bigger obstacle because, let's be honest, I'm a white guy in tech, and I look like it. My failure mode is basically a board seat and a book deal because of inherent bias in the system.Alex: [laugh]. Oh, my god.Corey: That's the outcome that, for me personally, I will be just fine. It's the other people took a chance on me. I'm terrified of letting them down. So far, knock on wood, I haven't said anything too offensive in public is going to wind up there. That's also not generally my style.But it is the… it is something that has weighed on me that has kept me from I guess, thinking about what would my next job be? I'm convinced this is the last job I'll ever have, if for no other reason that I've made myself utterly unemployable.Alex: [laugh]. Well, I think many of us aspire to find that perfect intersection of what you love doing and what pays the bills. Sounds like you've found it, I really do feel like I found it, too. I never imagined I'd be doing what I do now. Which is also sometimes hard to describe.I'm not making TikToks for a living; I'm just on the community team, doing events—I'm getting to work with people. I'm basically doing the things that I wanted to do that led me to quit that job many years ago, that big law job many years ago. So, I feel very blessed and for anybody who's, like, looking for that type of path, I do think that at some point, you do need to kind of shed the safety nets because if you always hang on to the safety nets, whether it's a big tech job or a big law job, there's going to be elements of that that don't fit in with your personality, and you're never going to be able to find that if you kind of stay there. But if you venture out—and, you know, I admire you for what you've done; it sounds like you're very successful at what you do and get to do what you love every day—I think great things can happen.Corey: Yeah, I get to insult Amazon for a living. It's what I love. It's what I would do if I weren't being paid. So, here we are. Yeah—Alex: [laugh].Corey: I have no sense of self-preservation. It's kind of awesome.Alex: I love it.Corey: But you're right. It's… there's something to be said for finding the thing that winds up resonating with you and what you want to be doing.Alex: It really does. And you know, I think when I first made the move to technology, to sales, there was no career path. I thought I would—maybe I thought I might be a VP of Sales. But the thing is, when you put yourself out there, the opportunities that show up might not be the ones that you had always seen from the beginning. Like if you ask a lawyer, like, “What can I do if I don't practice law?” They're going to give you these generic answers. “Work here. Work there. Work for that company. I've seen a lot of people do this.”But once you put yourself out there in the wilderness, these opportunities arise. And I've been very lucky. I mean, I never imagined I'd be a TikTokker. And by the way, I also make memes on Twitter. Couldn't imagine I'd be doing that either. I learned, like, Mematic, these tools. Like, you know, like, I'm immersed in this internet culture now.Corey: It is bizarre to me and I never saw it coming either. For better or worse, though, here we are, stuck at it.Alex: [laugh].Corey: I really want to thank you for taking so much time to speak with me today. If people want to learn more about what you're up to and follow along for the laughs, if nothing else, where's the best place for them to find you?Alex: The best way to find me is on LinkedIn; just look up Alex Su. But I'm around and on lots of social media platforms. You can find me on Twitter, on Instagram, and on TikTok, although I might be a little bit embarrassed of what I put on TikTok. I put some crazy gnarly stuff out there. But yeah, LinkedIn is probably the best place to find me.Corey: And we will put links to all of it in the show notes, and let people wind up making their own decisions. Thanks so much for your time, Alex. I really appreciate it.Alex: Corey, thank you so much for having me. This was so much fun.Corey: Alex Su, Head of Community Development at Ironclad. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry insipid comment talking about how unprofessional everything we talked about is that you will not be able to post for the next six months because it'll be hung up in legal review.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About JonA husband, father of 3 wonderful kids who turned Podcaster during the pandemic. If you told me in early 2020 I would be making content or doing a podcast, I probably would have said "Nah, I couldn't see myself making YouTube videos". In fact, I told my kids, no way am I going to make videos for YouTube. Well, a year later I'm over 100 uploads and my subscriber count is growing.Links Referenced: LinkedIn: https://www.linkedin.com/in/jon-myer/ Twitter: https://twitter.com/_JonMyer jonmyer.com: https://jonmyer.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. Every once in a while I get to talk to a guest who has the same problem that I do. Now, not that they're a loud, obnoxious jerk, but rather that describing what they do succinctly is something of a challenge. It's not really an elevator pitch anymore if you have to sabotage the elevator before you start giving it. I'm joined by Jon Myer. Jon, thank you for joining me. What the hell do you do?Jon: Corey, thanks for that awesome introduction. What do I do? I get to talk into a microphone. And sometimes I get to stare at myself on camera, whether it makes a recording or not. And either I talk to myself or I talk to awesome people like you. And I get to interview and tell other people's stories on my show; I pull out the interesting parts and we have a lot of freaking fun doing it.Corey: I suddenly feel like I've tumbled down the rabbit hole and I'm in the wrong side of the conversation. Are we both trying to stand in the same part of the universe? My goodness.Jon: Is this your podcast or mine? Maybe I should do an introduction right now to introduce you onto it and we'll see how this works.Corey: The dueling podcast banjo. I liked the approach quite a bit. So, you have done a lot of very interesting things. For example, once upon a time, you worked at AWS. But you have to go digging to figure that out because everything I'm seeing about you in your professional bio and the rest is forward-looking, as opposed to Former Company A, Former Company B, and this one time I was an early investor in Company C, which means, that's right, one of the most interesting things about me is that I wrote a check once upon a time, which is never something I ever want to say about myself, ever. You're very forward-looking, and I strive to do the same. How do you wind up coming at it from that position?Jon: When I first left AWS—it's been a year ago, so I served my time—and I actually used to have ex-Amazonian on it and listed on it. But as I continuously look at it, I used to have a podcast called The AWS Blogger. And it was all about AWS and everything, and there's nothing wrong with them. And what I would hear—Corey: Oh, there's plenty wrong with them, but please continue.Jon: [laugh]. We won't go there. But anyway, you know, kind of talking about it and thinking about it ex-Amazonian, yeah, that's great, you put it on your resume, put it on your stuff, and it, you know, allows you that foot in the door. But I want to look at and separate myself from AWS, in that I am my own independent voice. Yes, I worked for them; great company, I've learned so much from them, worked with some awesome people there, but my voice in the community has become very engaging and trustworthy. I don't want to say I'm no longer an Amazonian; I still have some of the guidelines, some of the stuff that's instilled in me, but I'm independent. And I want that to speak for itself when I come into a room.Corey: It's easy as hell, by the way, for me to sit here and cast stones at folks who, “Oh, you're going to talk about this big company you worked for, even though you don't work there anymore.” Yeah, I really haven't worked anywhere that most people would recognize unless they're, you know, professionally sad all the time. So, I don't have that luxury; I had to wind up telling a story that was forward-looking just because I didn't really have much of a better option. You have that option and decided to go in a direction where it presents, honestly as your viewpoint is that your best days are yet to come. And I want to be clear that for folks who are constantly challenged in our space to justify their existence there, usually because they don't look like our wildly over-represented selves, Jon, they need that credibility.And when they say that it's necessary for them, I am not besmirching that. I'm speaking from my own incredibly privileged position that you share. That is where I'm coming from on this, so I don't want people to hear this as shaming folks who are not themselves wildly over-represented. I'm not talking about you fine folks, I assure you.Jon: You can have ex-Amazonian on your resume and be very proud of it. You can remove it and still be very proud of the company. There's nothing wrong with either approach. There are some conversations that I'll be in, and I'll be on with AWS folks and I'll say, “I completely understand where you're coming from. I'm an ex-Amazonian.” And they're like, “Oh, you get us. You get the process. You get the everything.”I just want to look forward that I will be that voice in the community and that I have an understanding of what AWS is and will continuously be. And I have so much that I'm working towards that I'm very proud of where I've come from, but I do want to look forward.Corey: One of these days, I really feel like I should hang out with some Amazonians or ex-Amazonians who don't know who I am—which is easier to find than you think—and pretend that I used to work there and wonder how long I can keep the ruse going. Just because I've been told a few times that I am suspiciously Amazonian for someone who's never worked there.Jon: You have a lot of insights on the AWS processes and understanding. I think you could probably keep it going for quite a while. You will have to get that orange lanyard though, when you go to, like—Corey: I got one once when I was at a New York Summit a couple years ago. My affiliation then, before I started The Duckbill Group, was Last Week in AWS, and apparently, someone saw that and thought that I was the director of Take-this-Job-and-Shove-it, but I'll serve out my notice until Friday. So, cool; employee lanyard, it was. And I thought this is going to be awesome because I'll be able to walk around and I'll get the inside track if people think I work there. And they treated me like crap until I put the customer lanyard back on. It's, “Oh, it's better to be a customer at an AWS event than it is to be an employee.” I learned that when the fun way.Jon: There is one day that I hope to get the press or analyst lanyard. I think it would be an accomplishment for me. But you get to experience that firsthand, and I hate to switch the tables because I know it's your podcast recording, not mine, but—Corey: Having the press analyst lanyard is interesting because a lot of people are not allowed to speak to you unless they've gone through training. Which, okay, great. I will say that it is a lot nicer walking the expo floor because most of the people working the booths know that means that person is press, generally—they're not quite as familiar with analysts—but they know that regardless that they're not going to sell you a damn thing, so they basically give you a little bit of breathing room, which is awesome, especially in these pandemic times. But the challenge I have with it is that very often I want to talk to folks who are AWS employees who may not have gone through press training. And I've never gotten anyone in trouble or taken advantage of things that I hear in those conversations and write about them.Everything I write about is what I've experienced in public or as a customer, not based upon privileged inside information. I have so many NDAs at this point, I can't keep track, so I just make sure everything I talked about publicly cited I have that already.Jon: Corey, I got to flip the script real quick. I got to give you a shout-out because everybody sees you on Twitter and sees, like, “Oh, my God, he's saying this negative, that negative towards AWS.” You and I had, I don't know, it was a 30, 45 minute at the San Francisco Summit, and I think every Summit, we try to connect for a little bit. But that was really the premise I kicked off a lot of our conversations when you joined my podcast. No, this is not my podcast, this is Corey's, but anyway—Corey: And just you remember that. Please continue.Jon: [laugh]. But you know, kind of going off it you have so much insight, so much value, and you kind of really understand the entire processes and all the behind the scenes and everything that's going on that I was like, “Corey, I got to get your voice out there and show the other side of you, that you're not there trying to get people in trouble, you never poke fun of an AWS employee. I heard there was some guy named Larry that you do, but we won't jump into that.”Corey: One of the things that I think happened is, first and foremost, there is an algorithmic bias towards outrage. When I say nice things about AWS or other providers, which I do periodically, they get basically no engagement. When I say something ridiculous, inflammatory, and insulting about a company, oh, goes around the internet three times. One of the things that I'm slowly waking up to is that when I went into my Covid hibernation, my audience was a quarter of the size it is now. People don't have the context of knowing what I've been up to for the last five or six years. All they see is a handful of tweets.And yeah, of course, you wind up taking some of my more aggravated moment tweets and put a few of those on a board, and yeah, I start to look a fair bit like a jerk if you're not aware of what's going on inside-track-wise. That's not anyone else's fault, except my own, and I guess understanding and managing that perception does become something of a challenge. I mean, it's weird; Amazon is a company that famously prides itself on being misunderstood for long periods of time. I guess I never thought that would apply to me.Jon: Well, it does. Maybe that's why most people think you're an Amazonian.Corey: You know, honestly, I've got to say, there are a lot of worse things people can and do call me. Amazon has a lot to recommend it in different ways. What I find interesting now is that you've gone from large companies to sort of large companies. You were at Spot for a hot minute, then you were doing the nOps thing. But one thing that you've been focusing on a fair bit has been getting your own voice and brand out there—and we talked about this a bit at the Summit when we encountered each other which is part of what sparked this conversation—you're approaching what you're doing next in a way that I don't ever do myself. I will not do it justice, but what are you working on?Jon: All right. So Corey, when we talked at the New York Summit, things are actually moving pretty good. And some of the things that I am doing, and I've actually had a couple of really nice engagements kind of kick off is, that I'm creating highly engageable, trustworthy content for the community. Now, folks, you're asking, like, what is that? What is that really about? You do podcasts?Well, just think about some of the videos that you're seeing on customer sites right now. How are they doing? How's the views? How's the engagement? Can you actually track those back to, like, even a sales engagement in utilizing those videos?Well, as Jon Myer—and yes, this is highly scalable because guess what I am in talks with other folks to join the crew and to create these from a brand awareness portion, right? So, think about it. You have customers that you want to get engaged with: you have products, you have demos, you have reviews that you want to do, but you can't get them turned around in a quick amount of time. We take the time to actually dive into your product and pull out the value prop of the exact product, a demo, maybe a review, all right? We do sponsors as well; I have a number of them that I can talk about, so Veeam on AWS, Diabolical Coffee, there's a couple of other I cannot release just yet, but don't worry, they will be hitting out there on social pretty soon.But we take that and we make it an engaging kind of two to three-minute videos. And we say, “Listen, here's the value of it. We're going to turn this around, we're going to make this pop.” And putting this stuff, right, so we'll take the podcast and I'll put it on to my YouTube channel, you will get all my syndication, you'll get all my viewers, you'll get all my views, you'll get my outreach. Now, the kicker with that is I don't just pick any brand; I pick a trusted brand to work with because obviously, I don't want to tarnish mine or your brand. And we create these podcasts and we create these videos and we turn them around in days, not weeks, not months. And we focus on those who really need to actually present the value of their product in the environment.Corey: It sounds like you're sort of the complement to the way that I tend to approach these things. I'll periodically do analyst engagements where I'll kick the tires on a product in the space—that's usually tied to a sponsorship scenario, but not always—where, “Oh, great. You want me to explain your product to people. Great, could I actually kick the tires on it so I understand at first? Otherwise, I'm just parroting what may as well be nonsense. Maybe it's true, maybe it's not.”Very often small companies, especially early stage, do a relatively poor job of explaining the value of their product because everyone who works there knows the product intimately and they're too close to the problem. If you're going to explain what this does in a context where you have to work there and with that level of intensity on the problem space, you're really only pitching to the already converted as opposed to folks who have the expensive problem that gets in the way of them doing their actual job. And having those endless style engagements is great; they periodically then ask me, “Hey, do you want to build a bunch of custom content for us?” And the answer is, “No, because I'm bad at deadlines in that context.”And finding intelligent and fun and creative ways to tell stories takes up a tremendous amount of time and is something that I find just gets repetitive in a bunch of ways. So, I like doing the typical sponsorships that most people who listen to this are used to: “This episode is sponsored by our friends at Chex Mix.” And that's fine because I know how to handle that and I have that down to a set of study workflows. Every time I've done custom content, I find it's way more work than I anticipated, and honestly, I get myself in trouble with it.Jon: Well, when you come across it, you send them our way because guess what, we are actually taking those and we're diving deep with them. And yes, I used an Amazon term. But if you take their product—yeah [laugh]. I love the reaction I got from you. But we dive into the product. And you said it exactly: those people who are there at the facility, they understand it, they can say, “Yeah, it does this.”Well, that's not going to have somebody engaged. That's not going to get somebody excited. Let me give you an example. Yesterday, I had a call with an awesome company that I want to use their product. And I was like, “Listen, I want to know about your product a little bit more.”We demoed it for my current company, and I was like, “But how do you work for people like me: podcasters who do a lot of the work themselves? Or a social media expert?” You know, how do I get my content out there? How does that work? What's your pricing?And they're, like, “You know, we thought about getting it and see if there was a need in that space, and you're validating that there's a need.” I actually turned it around and I pitched them. I was like, “Listen, I'd love for you guys to be a sponsor on my show. I'd love for you to—let me do this. Let me do some demos. Let's get together.”And I pitched them this idea that I can be a spokesperson for their product because I actually believed in it that much just from two calls, 30 minutes. And I said, “This is going to be great for people like me out there and getting the voice, getting the volume out there, how to use it.” I said, “I can show some quick integration setups. You don't have to have the full-blown product that you sell out the businesses, us as individuals or small groupings, we're only going to use certain features because, one, is going to be overwhelming, and two, it's going to be costly. So, give us these features in a nice package and let's do this.” And they're like, “Let's set something up. I think we got to do this.”Corey: How do you avoid the problem where if you do a few pieces of content around a particular brand, you start to become indelibly linked to that brand? And I found that in my early days when I was doing a lot of advisory work and almost DevRel-for-hire as part of the sponsorship story thing that I was doing, and I found that that did not really benefit the larger thing I was trying to build, which is part of the reason that I got out of it. Because it makes sense for the first one; yeah, it's a slam dunk. And the second one, sure, but sooner or later, it feels like wow, I have five different sponsors in various ways that want me to be building stories and talking about their stuff as I travel the world. And now I feel like I'm not able to do any of them a decent service, while also confusing the living hell out of the audience of, “Who is it you work for again anyway?” It was the brand confusion, for lack of a better term.Jon: Okay, so you have two questions there. One of them is, how do you do this without being associated with the brand? I don't actually see a problem with that. Think of a race car; NASCAR drivers are walking around with all their stuff on their jackets, you know, sponsored by this person, this group, that group. Yeah, it's kind of overwhelming at times, but what's wrong with being tied to a couple of brands as long as the brands are trustworthy, like yourself? Or you believing those, right? So, there's nothing wrong with that.Second is the scalability that you're talking about where you're traveling all over the world and doing this and that. And that's where I'm looking for other leaders and trustworthy community members that are doing this type of thing to join a highly visible team, right? So, now you have a multitude and a diverse group of individuals who can get the same message out that's ultimately tied to—and I'm actually going to call it out here, I have it already as Myer Media, right? So, it's going to be under the Jon Myer Podcast; everything's going to be grouped in together under Myer Media, and then we're going to have a group of highly engaging individuals that enjoy doing this for a living, but also trust what they're talking about.Corey: If you can find a realistic way to scale that, that sounds like it's going to have some potential significant downstream consequences just as far as building almost a, I guess, a DevRel workshop, for lack of a better term. And I mean, that in the sense of an Andy Warhol workshop style approach, not just a training course. But you wind up with people in your orbit who become associated, affiliated with a variety of different brands. I mean, last time I did the numbers, I had something like 110 sponsors over the last five years. If I become deeply linked to those brands, no one knows what the hell I do because every company in the space, more or less, has at some level done a sponsorship with me at some point.Jon: I guess I'll cross that when it happens, or keep that in the top-of-mind as it moves forward. I mean, it's a good point of view, but I think if we keep our individualism, that's what's going to separate us as associated. So, think of advertising, you have a, you know, actor, actress that actually gets on there, and they're associated with a certain brand. Did they do it forever? I am looking at long-term relationships because that will help me understand the product in-depth and I'll be able to jump in there and provide them value in a expedited version.So, think about it. Like, they are launching a new version of their product or they're talking about something different. And they're, like, “Jon, we need to get this out ASAP.” I've had this long-term relationship with them that I'm able to actually turn it around rather quickly, but create highly engaging out of it. I guess, to really kind of signify that the question that you're asking is, I'm not worried about it yet.Corey: What stage or scale of company do you find is, I guess, the sweet spot for what you're trying to build out?Jon: I like the small to medium. And looking at it, the small to medium—Corey: Define your terms because to my mind, I'm still stuck in this ancient paradigm that I was in as an employee, where a big company is anything that has more than 200 people, which is basically everyone these days.Jon: So, think about startups. Startups, they are usually relatively 100 or less; medium, 200 or less. The reason I like that type of—is because we're able to move fast. As you get bigger, you're stuck in processes and you have to go through so many steps. If you want speed and you want scalability, you got to pay attention to some of the stuff that you're doing and the processes that are slowing it down.Granted, I will evaluate, you know, the enterprise companies, but the individuals who know the value of doing this will ultimately seek me and say, “Hey, listen, we need this because we're just kicking this off and we need highly visible content, and we want to engage with our current community, and we don't know how.”Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: I think that there's a fair bit of challenge somewhere in there. I'm not quite sure how to find it, that you're going to, I think, find folks that are both too small and too big, that are going to think that they're ready for this. I feel like this doesn't, for example, have a whole lot of value until a company has found product-market fit unless what you're proposing to do helps get them to that point. Conversely, at some point, you have some of the behemoth companies out there, it's, “Yeah, we can't hire DevRel people fast enough. We've hired 500 of them. Cool, can you come do some independent work for us?” At which point, it's… great, good luck standing out from the crowd in any meaningful way at that point.Jon: Well, even a high enterprise as hired X number of DevRels, the way you stand out is your personality and everything that you built behind your personal brand, and your value brand, and what you're trying to do, and the voice that you're trying to achieve out there. So, think about it—and this is very difficult for me to, kind of, boost and say, “Hey, listen, if I were to go to a DevRel of, like, say, 50 people, I will stand out. I might be one of the top five, or I might be two at the top five.” It doesn't matter. But for me why and what I do, the value that I am actually driving across is what will stand out, the engaging conversations.Every interview, every podcast that I do, at the end, everybody's like, “Oh, my God, you're, like, really good at it; you kind of keep us engaging, you know when to ask a question; you jump in there and you dive even deeper.” I literally have five bullet points on any conversation, and these are just, like, two or three sentences, maybe. And they're not exact questions. They're just topics that we need to talk about, just like we did going into this conversation. There is nothing that scripted. Everything that's coming across the questions that you're pulling out from me giving an answer to one of your questions and then you're diving deep on it.Corey: I think that that's probably a fair approach. And it's certainly going to lead to a better narrative than the organic storytelling that tends to arise internally. I mean, there's no better view to see a lot of these things than working on bills. One of my favorite aspects of what I do is I get to see the lies that clients tell to themselves, where it's—like, they believe these things, but it no longer matches the reality. Like developer environments being far too expensive as a proportion of the rest of their environment. It's miniscule just because production has scaled since you last really thought about it.Or the idea that a certain service is incredibly expensive. Well, sure. The way that it was originally configured and priced, it was and that has changed. Once people learn something, they tend to stop keeping current on that thing because now they know it. And that's a bit of a tricky thing.Jon: That's why we keep doing podcasts, you keep doing interviews, you keep talking with folks is because if you look at when you and I actually started doing these podcasts—and aka, like, webinars, and I hate to say webinars because it's always negative and—you know because they're not as highly engaging, but taking that story and that narrative and creating a conversation out of it and clicking record. There are so many times that when I go to a summit or an event, I will tell people, they're like, “So, what am I supposed to do for your podcast?” And we were talking for, like, ten minutes, I said, “You know, I would have clicked record and we would have ten minutes of conversation.” And they're like, “What?” I was like, “That's exactly what it is.”My podcast is all about the person that I'm interviewing, what they're doing, what they're trying to achieve, what's their message that they're trying to get across? Same thing, Corey. When you kick this off, you asked me a bunch of questions and then that's why we took it. And that's where this conversation went because it's—I mean, yeah, I'm spinning it around and making it about you, sometimes because obviously, it's fun to do that, and that's normally—I'm on the other side.Corey: No, it's always fun to wind up talking to people who have their own shows just because it's fun watching the narrative flow back and forth. It's kind of a blast.Jon: It's almost like commentators, though. You think about it at a sporting event. There's two in the booth.Corey: Do a team-up at some point, yeah.Jon: Yeah.Corey: In fact, doing the—what is it like the two old gentlemen in the Sesame Street box up in the corner? I forget their names… someone's going to yell at me for that one. But yeah, the idea of basically kibitzing back and forth. I feel like at some level, we should do a team up and start doing a play-by-play of the re:Invent keynotes.Jon: Oh… you know what, Corey, maybe we should talk about this offline. Having a huge event there, VIP receptions, a podcasting booth is set up at a villa that we have ready to go. We're going to be hosting social media influencers, live-tweeting happening for keynotes. Now, you don't have to go to the keynotes personally. You can come to this room, you can click record, we'll record a live session right there, totally unscripted, like everything else we do, right? We'll have a VIP reception, come in chat, do introductions. So, Corey, love to have you come into that and we can do a live one right there.Corey: Unfortunately, I'm going to be spending most of re:Invent this year dressed in my platypus costume, but you know how it works.Jon: [laugh]. Oh man, you definitely got to go for that because oh, I have a love to put that on the show. I'm actually doing something not similar, but in true style that I've been going to the last couple of re:Invents I will be doing something unique and standing out.Corey: I'm looking forward to it. It's always fun seeing how people continue to successfully exceed what they were able to do previously. That's the best part, on some level, is just watching it continually iterate until you're at a point where it just becomes, well frankly, either ridiculous or you flame out or it hits critical mass and suddenly you launch an entire TV network or something.Jon: Stay tuned. Maybe I will.Corey: You know, it's always interesting to see how that entire thing plays out. Last question before we call it a show. Talk to me about your process for building content, if you don't mind. What is your process when you sit down and stare at—at least from my perspective—that most accursed of all enemies, a blank screen? “All right time to create some content, Jackwagon, better be funny. And by the way, you're on a deadline.” That is the worst part of my job.Jon: All right, so the worst part of your job is the best part of my job. I have to tell you, I actually don't—and I'm going to have to knock on wood because I don't get content block. I don't sit at a screen when I'm doing it. I actually will go for a walk or, you know, I'll have my weirdest ideas at the weirdest time, like at the gym, I might have a quick idea of something like that and I'll have a backlog of these ideas that I write down. The thing that I do is I come down, I open up a document and I'll just drop this idea.And I'll write it out as almost as it seems like a script. And I'll never read it verbatim because I look at it and be like, “I know what I'm going to say right now.” An example, if you take a look at my intros that I do for my podcast, they are done after the recording because I recap what we do on a recording.So, let's take this back. Corey will talk about the one you and I just did. And you and I we hopped on, we did a recording. Afterwards, I put together the intro. And what I'm going to say the intro, I have no freaking clue until I actually get to it, and then all of a sudden, I think of something—not at my desk, but away from my desk—what I'm going to say about you or the guest.An example, there was a gentleman I did his name's called Mat Batterbee, and he's from the UK. And he's a Social Media Finalist. And he has this beard and he always wears, like, this hat or something. And I saw somebody on Twitter make a comment about, you know, following in his footsteps or looking like him. So, they spoofed him with a hat and everything—glasses.I actually bought a beard off of Amazon, put it on, glasses, hat, and I spoofed him for the intro. I had this idea, like, the day before. So, thank goodness for Prime delivery, that I was able to get this beard ASAP, put it on. One take; I only tried to do one take. I don't think I've ever recorded any more.Corey: I have a couple of times sometimes because the audio didn't capture—Jon: Yeah.Corey: —but that's neither here nor there. But yeah, I agree with you, I find that the back-and-forth with someone else is way easier from a content perspective for me. Because when you and I started talking, on this episode, for example, I had, like, three or four bullet points I wanted to cover and that's about it. The rest of it becomes this organic freewheeling conversation and that just tends to work when it's just me free-associating in front of the camera, it doesn't work super well. I need something that's a bit more structured in that sense. So apparently, my answer is just never be alone, ever.Jon: [laugh]. The content that I create, like how-to tutorials, demos, reviews, I'll take a lot more time on them and I'll put them together in the flow. And I record those in certain sections. I'll actually record the demo of walking through and clicking on everything and going through the process, and then I will actually put that in my recording software, and then I will record against it like a voiceover.But I don't record a script. I actually follow the flow that I did and in order to do that, I understand the product, so I'll dive deep on it, I'll figure out some of the things using keywords along the way to highlight the value of utilizing it. And I like to create these in, like, two to three minutes. So, my entire process of creating content—podcast—you know what we hop on, I give everybody the spiel, I click record and I say, “Welcome.” And I do the introduction. I cut that out later. We talk. I'll tell you what, I never edited anything throughout the entire length of it because whatever happens happens in his natural and comes across.And then I slap on an ending. And I try to make it as quick and as efficiently as possible because if I start doing cuts, people are going to be, like, “Oh, there's a cut there. What did he cut out?” Oh, there's this. It's a full-on free flow. And so, if I mess up and flub or whatever it is, I poke fun of myself and we move on.Corey: Oh, I have my own favorite punching bag. And I honestly think about that for a second. If I didn't mock myself the way that I do, I would be insufferable. The entire idea of being that kind of a blowhard just doesn't work. From my perspective, I am always willing to ask the quote-unquote dumb question.It just happens to turn out but I'm never the only person wondering about that thing and by asking it out loud, suddenly I'm giving a whole bunch of other folks air cover to say, “Yeah, I don't know the answer to that either.” I have no problem whatsoever doing that. I don't have any technical credibility to worry about burning.Jon: When you start off asking and say, “Hey, dumb question or dumb question,” you start being unsure of yourself. Start off and just ask the question. Never say it's a dumb question because I'll tell you what, like you said, there's probably 20 other people in that room that have the same question and they're afraid to ask it. You can be the one that just jumps up there and says it and then you're well-respected for it. I have no problem asking questions.Corey: Honestly, the problem I've got is I wish people would ask more questions. I think that it leads to such a better outcome. But people are always afraid to either admit ignorance. Or worse, when they do ask questions just for the joy they get from hearing themselves talk. We've all been conference talks where you there's someone who's just asking the question because they love the sound of their own voice. I say, they, but let's be serious; it's always a dude.Jon: That is very true.Corey: So, if people want to learn more about what you're up to, where's the best place to go?Jon: All right, so the best place to go is to follow me on LinkedIn. LinkedIn is my primary one, right? Jon Myer; can't miss me. At all. Twitter, I am active on Twitter. Not as well as Corey; I would love to get there one day, but my audience right now is LinkedIn.Else you can go to jonmyer.com. Yes, that's right, jonmyer.com. Because why not? I found I have to talk about this just a little bit. And the reason that I changed it—I actually do own the domain awsblogger, by the way and I still have it—is that when I was awsblogger, I had to chan—I didn't have to change anything' nobody required me to, but I changed it to, like, thedailytechshow. And that was pretty cool but then I just wanted to associated with me, and I felt that going with jonmyer, it allowed me not having to change the name ever again because, let's face it, I'm not changing my name. And I want to stick with it so I don't have to do a whole transition and when this thing takes off really huge, like it is doing right now, I don't have to change the name.Corey: Yeah. I would have named it slightly differently had I known was coming. But again, this far in—400 some-odd episodes in last I checked recorded—though I don't know what episode this will be when it airs—I really get the distinct impression that I am going to learn as I go and, you know, you can't change that this far in anymore.Jon: I am actually rounding so I'm not as far as you are with the episodes, but I'm happy to say that I did cross number 76—actually 77; I recorded yesterday, so it's pretty good. And 78 tomorrow, so I am very busy with all the episodes and I love it. I love everybody reaching out and enjoying the conversations that I have. And just the naturalness and the organicness of the podcast. It really puts people at ease and comfortable to start sharing more and more of their stories and what they want to talk about.Corey: I really want to thank you for being so generous with your time and speak with me today. Thanks. It's always a pleasure to talk with you and I look forward to seeing what you wind up building next.Jon: Thanks, Corey. I really appreciate you having me on. This is very entertaining, informative. I had a lot of fun just having a conversation with you. Thanks for having me on, man.Corey: Always a pleasure. Jon Myer, podcaster extraordinaire and content producer slash creator. The best folks really have no idea what to refer to themselves and I am no exception, so I made up my own job title. I am Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment telling me that I'm completely wrong and that you are a very interesting person. And then tell me what company you wrote a check to once upon a time.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About TimTim's tech career spans over 20 years through various sectors. Tim's initial journey into tech started as a US Marine. Later, he left government contracting for the private sector, working both in large corporate environments and in small startups. While working in the private sector, he honed his skills in systems administration and operations for large Unix-based datastores.Today, Tim leverages his years in operations, DevOps, and Site Reliability Engineering to advise and consult with clients in his current role. Tim is also a father of five children, as well as a competitive Brazilian Jiu-Jitsu practitioner. Currently, he is the reigning American National and 3-time Pan American Brazilian Jiu-Jitsu champion in his division.Links Referenced:Twitter: https://twitter.com/elchefe TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it's still somebody's problem. And a big part of that responsibility is app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you're actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That's S-N-Y-K.co/screamCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. A bit of a sad episode Today. I am joined by Duckbill Group principal cloud economist, Tim Banks, but by the time this publishes, he will have left the Duckbill nest, as it were. Tim, thank you for joining me, and can I just start by saying, this is sad?Tim: It is. I have really enjoyed being with Duckbill and I will never forget that message you sent me. It's like, “Hey, would you like to do this?” And I was like, “Boy would I.” It's been a fantastic ride and I have enjoyed working with a friend. And I'm glad that we remain friends to this day and always will be, so far as I can tell.Corey: Yes, yes. What you can't see while recording this, I'm actually sitting in the same room as Tim with a weapon pointed at him to make sure that he stays exactly on message. Yeah, I kid. There's been a lot that's happened over the last year. We only got to spend time together in person once at re:Invent. I think because re:Invent is such a blur for me, I don't remember who the hell I talk to.Someone can walk up and say, “Oh yeah, we met at re:Invent,” and I'll nod and say, “Oh yeah,” and I will have no recollection of that whatsoever. But you don't argue with people. But I do distinctly remember hanging out with you there. But since then, it's been a purely distributed company, purely distributed work.Tim: Yeah, that's the only time I've seen you since I've worked here. It's the only time I met Mike. But it's weird because it's like, someone you work with you see every day virtually and talk to, and then you actually get to, like, IRL them and like, “Oh, wow. I had all these, kind of, conceptions of, you know, what you are or who you are as a person, and then you get to, like, check yourself. Was I right? Was I wrong?” I was like, “Oh, you're taller than I thought; you're shorter than I thought,” you know, whatever it was.But I think the fun part about it was we all end up being so close by the nature of how we work that it was just like going back and seeing family after a while; you already know who they are and how they are and about them. So, it felt good, but it felt familiar. That's a great feeling to have. To me, that's a sign of a very successful distributed culture.Corey: Yeah, it's weird the kinds of friendships we've built during the pandemic. When I was in New York for the summit, I got to meet Linda Haviv at AWS for the first time, despite spending the past year or so talking to her repeatedly. As I referred to her the entire time I was in New York, this is Linda, my new old friend because that is exactly how it felt. It's the idea of meeting someone in person that you've had a long-term ongoing friendship with. It's just a really—it's a strange way Everything's new but it's not, all at the same time.It reminds me of the early days of the internet culture where I had more friends online than off, which in my case was not hard. And finally meeting them, some people were exactly like they were described and others were nothing at all like they presented. Now that we have Zoom and this constant level of Slack chatter and whatnot, it's become a lot easier to get a read on what someone is like, I think.Tim: I think so too, you know, we've gotten away—and I think largely because of the pandemic—of just talking about work at work, right? The idea of embracing, you know, almost a cliche of the whole person. But it's become a very necessary thing as people have dealt with pandemic, social upheaval, political climates, and whatever, while they're working from home. You can't compartmentalize that safely in perpetuity, right? So, you do end up getting to know people very well, especially in what their concerns are, what their anxieties are, what makes them happy, what makes them sad, things that go on in their lives.You bring all that to your distributed culture because it's not like you leave it at the door, when you walk out. You're not walking out anymore; you're walking to another room, and it's hard to walk away from those things in this day and age. And we shouldn't have to, right? I feel like for a successful and nurturing culture—whatever it is, whether it's tech culture, whether it's whatever kind of work culture—you can't say, “I only want your productivity and nothing else about you,” and expect people to sustain that. So, you see these companies are, like, you know, “We don't have political discussions. We don't have personal discussions. We're just about the work.” I'm like, “All right, well, that's not going to last.” A person cannot just be an automaton in perpetuity and expect them to grow and thrive.Corey: And this is why you're leaving. And I want to give that a little context because without, sounds absolutely freaking horrifying. You've been a strong advocate for an awful lot of bringing the human to work, on your philosophy around leadership, around management. And you've often been acting in that capacity throughout, I would say, the majority of your career. But here at The Duckbill Group, we don't have a scale of team where you being the director of the team or leader of the team is going to happen in anything approaching the near or mid-term.And so, much of your philosophy is great and all because it's easy to sit here at a small company and start talking about, “Oh, this is how you should be doing it.” You have the opportunity to wind up making a much deeper impact on a lot more people from a management perspective, but you do in fact, need a team to manage as opposed to sitting around there, “Oh, yeah. Who do you manage?” “This one person and I'm doing all of these things to make their life and job awesome.” It's like, “Yeah, how many hours a week are you spending in one-on-ones?” “20 to 25.”Okay, maybe you need a slightly larger team so you can diffuse that out a little bit. And we are definitely sad to be losing you; super excited to see where you wind up going next. This has been a long time coming where there are things that you have absolutely knocked out of the park here at The Duckbill Group, but you also have that growing—from what I picked up on anyway—need to set a good management example. And lord knows this industry needs more of those. So first, sad to lose you. Secondly, very excited for where you wind up next and what they're in for, even though it has a strong likelihood that they don't know the half of it yet.Tim: One of the things that I like about The Duckbill Group and how my time here has been is the first thing that I was asked in the interview was very sincere, like, “Well, what's your next job?” And I was very clear. It's like, “After this, I want to be a director or VP of engineering because I would like to be a force multiplier, right?” I would like to make engineering orgs better. I would like to make engineering practices better. I want to make the engineers better, right?And not by driving KPIs and not by management, right, not administrative functions. I want to do it via leadership. I want to do it by setting examples, making safe places for people, making people feel like they're important and invested in, nurturing them, right? I've said this before I—this analogy was getting me somewhere else and I love, it's like, if I plant a tree and I want it to grow apples, right, I'm not going to sit there and put a number down of apples it's expected to produce, and then put it on a performance plan if it doesn't get that number of apples, right? I need to nurture the tree, I need to fertilize it, I need to protect it, I need to keep it safe, I need to keep it safe from the elements, I need to make sure that it doesn't have parasites, I need to take care of that tree.And if that tree grows and it's healthy and it's thriving, it will produce, right? But I'm not—I can't just expect apples if I'm not taking care of the tree. Now, people are not trees, but you still have to take care of the people if you want them to do things. And if you can't take care of the people, if you can't manage the environment that they're in to make it safe, if you can't give them the things they need to be successful, then you're just going to be holding numbers over someone and expecting to hit them.And that doesn't work. That's not something that's sustainable. And it doesn't really—it's not even about how much you pay them. You must pay them well, right, but it has to be more than just that if you want people to succeed. And that doesn't necessarily mean—like, one thing is at the Duckbill Group I love, succeeding doesn't necessarily mean that I'm going to stay at—or your engineer is going to stay at one place in perpetuity. If you mentor and train and coach and give an engineer opportunity to grow and thrive and what they do is they go to another job for a title increase and a pay increase or something like that, you did your job.Corey: A lot of companies love to tell that lie and they almost convince themselves of it where I look at your resume, and great you have not generally crossed the two-year mark at companies for the last decade. I never did until I started at this place. But we magically always liked to pretend in job interviews that, “Oh yeah, this is my forever job—” like you're a rescue dog getting adopted or something, “—and I'm going to work here for 25 years and get a gold watch and a pension at the end of it.” It's lunacy. I have never seen the value in lying to ourselves like that, which is why we start our interviews with, “What's the job after this and how do we help you get there?”It's important that we ask those questions and acknowledge that reality. And the downside to it—if you can call it a downside—is you've got to live by it. It's not just words, you can slop onto an interview questionnaire; you actually have to mean it. People can see through insincerity.Tim: And it's one of the things, like, if you run an org and you grow your people and you don't have a place for them to grow into, you should expect and encourage them to find those opportunities elsewhere. It is not reasonable, I feel like, as a leader for you expect people to stay in a place where they have grown past or grown out of. You need to either need to give them a new pot to grow into or you need to let them move elsewhere and thrive and grow. And moving elsewhere—like, if you have a retention problem where you can't retain anybody, that's a problem, but if you have your junior engineers who become senior engineers at other places, right, and everyone leaves on good terms, and they got the role and you gave them a great recommendation and they give glowing recommendations to you, there's nothing wrong with that. That's not a failure; that's success.Corey: One bit of I would say pushback that I suspect you might get when talking to people about what's next is that, “Well, you are just a consultant, on some level, for a year.” You always know that someone is really arguing in good faith when they describe what you did with the word ‘just,' but we'll skip past that part. And it's, “You're just a consultant. What would you possibly know about team management and team dynamics?” And there is a little bit of truth to that insofar as the worst place in the world to get management advice is very clearly on Twitter.It turns out that most interpersonal scenarios are, one, far too personal to wind up tweeting about, and two, do not lend themselves to easy solutions that succinctly fit within 280 characters. Imagine that. The counter-argument though, is that you have—correctly from where I sit—identified a number of recurring dynamics on teams that you have encountered and worked with deeply as a large number of engagements. And these are recurring things, I want to be clear. So, I'm not talking about one particular client. If you're one of our clients and listen to this thinking that we're somehow subtweeting you with our voices—I don't know what that is; subwoofing, maybe?Tim: [crosstalk 00:12:05]—Corey: Is that what a subwoofer is? I'm not an audio person.Tim: Throwing shade, we'll just say—call it throwing shade.Corey: Yeah, we're not throwing shade at any one person, team, or group in particular; these are recurring things. Tim, what have you seen?Tim: And so, I think the biggest thing I see is folks that are on the precipice of a big technological change, right, and there is an extraordinary amount of anxiety, right? I've seen a number of customers through our engagements that, “We are moving away from this legacy platform,” or from this thing that we have been doing for X amount of time. And everyone has staked the other domains, staked out their areas of expertise and control and we're going to change that. And the solution to that is not a technical solution. You don't fix that by Helm charts, or Terraform, or CloudFormation. You fix that by conversations, and you fix that by listening. You fix that by finding ways to reassure folks and giving them confidence in their ability to adjust and thrive in a new environment.If you take somebody who's been, you know, an Oracle admin for 20 years, and you going to say, “Great. Now, you're going to learn, you're going to do this an RDS,” that's a whole new animal, and folks feel like, well, you know, I can't learn something new like that? Well, yeah you can. If you can learn Oracle, you can learn anything. I firmly believe that.But that's one of the conversations we have, it's never, almost never a technical problem folks have. We need to reassure people, right? And so, folks who reach out to us, it's typically folks who are trying to get their organizations in that direction. Another thing we see sometimes is that we find that there's a disconnect between leadership and the engineers. They have either different priorities or different understandings of what's going on. And we come in to solve a problem, which may be cost but that's not the problem we actually solve. The problem we actually solve is fixing this communication bridges between management and leadership.And that's almost an every time occurred. At some point or another, there's some disconnect there. And that's the best part of the job. Like, the reason I do this consulting gig is not because I want to bang away at code. If I've had to do that, that's an anomaly for sure because I want to have these conversations.And people want to have these conversations; they want to get these problems solved and sometimes they don't know how to. And that is the common thing, I think, through all of our customers. Like, we need some amount of expertise to help us find solutions to these things that aren't necessarily technical problems. And I think that's where we run into problems as an industry, right, where we think a lot of things are technical problems or have technical solutions, and they don't. There are people problems. They're—Corey: Here at The Duckbill Group, we're basically marriage counseling for engineering and finance in many cases.Tim: We really are.Corey: This is why were people not software.Tim: Yeah. And I will say this very firmly and you can quote me on this: like, you cannot replace us. You cannot replace the kind of engagements we do with software. You can't. Can't be done, right? Software is not empathetic.Corey: There are a whole series of questions we ask our clients at the start of an engagement and the answers to those questions change what we ask them going forward. In fact, even the level-setting in the conversation that we have at the start of that changes the nature of those. We're not reading from a list; we're trying to build an understanding. There is a process around what we do, but it's not process that can ever be scoped down to the point where it's just a list of questions or a questionnaire that isn't maddening for people to fill out because it's so deeply and clearly misses the mark around context of what they're actually doing.Tim: Mm-hm. Our engaged with their conversations. That's all they are. They're really in-depth conversations where we're going to start asking questions and we're going to ask questions about those answers. We're start pulling out strings and kicking over rocks and seeing what we find.And that's the kind of thing that, you know, you would expect anyone to do who's coming in and saying, “Okay, we have a problem. Now, let's figure it out.” Right? Well, you can't just look at something on the surface, and say, “Oh, I know what this is.” Right? You know, for someone to say, “Oh, I know how to fix this,” when they walk in is the surest way to know that someone doesn't know what they're talking about, right?Corey: Oh, easiest thing in the world is to walk in and say, “This is broken and wrong.” That can translate directly to, “Hi, I am very junior. Please feed my own ass to me.” Because no one shows up at work thinking they're going to do a crap job today on purpose. There's a reason things are the way that they are.Tim: Mm-hm. And that's the biggest piece of context we get from our customers is we can understand what the best practices are. You can go Google them right now and say, “This is the ten things you're supposed to do all the time,” right? And we would be really, really crappy consultants if we just read off that list, right? We need to have context: does this thing make sense? Is this the best practice? Maybe, but we want to know why you did it this way.And after you tell us that way, I'm like, “You know what? I would do it the exact same way for this use case.” And that's great. We can say like, “This is the best way to do that. Good job.” It's atypical; it's unusual, but it solves the problems that you need solving.And that's where I think a lot of people miss. Like, you know, you can go—and not to throw shade at AWS's Trusted Advisor, but we're going to throw shade at AWS's Trusted Advisor—and the fact that it will give you—Corey: It is Plausible Advisor at absolute best.Tim: [laugh]. It will give you suggestions that have no context. And a lot of the automated AI things that will recommend that you do this and this and this and this are pretty much all the same. And they have no context because they don't understand what you're trying to do. And that's what makes the difference between people. There's these people problems.And so, one of the things that I think is really interesting is that we have moved into doing a shorter engagement style that is very short. It's very quick, it's very kind of almost tactical, but we go in, we look at your bill, we ask you some questions, and we're going to give you a list of suggestions that are going to save you a significant amount of money right away, right? So, a lot of times, folks when they need quick wins, or they don't really need us to deep-dive into all their DynamoDB access patterns, right? They just want like, “Hey, what are the five things we can do to save us some money?” And we're like, “Well, here they are. And here's what we think they're going to save you.” And folks who really enjoyed that type of engagement. And it's one of my favorite ones to do.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: I can also predict that people are going to have questions for you—probably inane—of, well, you were a consultant, how are your actual technical chops? And I love answering these questions with data. So, I have here pulled up the last six months of The Duckbill Group's AWS bills. And for those who are unaware, every cloud economist has their own dedicated test account for testing out strange things that we come across. And again, can the correct answer in many consulting engagements is, “I don't know, but I'll find out.”Well, this is how we find out. We run tests and learn these things ourselves. I suppose we could extend this benefit, if you want to call it that, to people who aren't cloud economists but I'm not entirely sure what, I don't know, an audio engineer is going to do with an AWS account that isn't, you know, kind of horrifying. To the audio engineer that is editing this podcast, my condolences if you take that as a slight, and if there is something you would use an AWS account for, please let me know. We'll come talk about it here.But back to topic, looking at the last six months of your bill for your account—that's right, a ritualistic shaming of the AWS bill—in January you spent $16.06. In February, you spent 44 cents. And you realized that was too high, so back in March, you then spent 19 cents. And then $3.01 back in April. May wound up $10.02, and now you're $9.84 as of June. July has not yet finalized as of this recording.And what I want to highlight—and what that tells me when I look at these types of bills—and I assure you as the world's leading self-described expert in AWS billing, I'm right; listening to me is a best practice on these things—that shows the exact opposite of a steady-state workload. There's a lot of dynamism to those giant swings because we don't have cloud economists who are going to just run these things steady-state for the rest of our lives. Those are experiments of building and testing out new and exciting things in a whole bunch of very weird, very strange ways. Whenever I wind up talking to someone in one of the overarching AWS services at AWS and I pull up my account, a common refrain is, “Wow, you use an awful lot of services.” Right. I'm not just sitting here run and EC2 instances forever. Imagine that. And your account is a perfect microcosm of that entire philosophy.Tim: Well, I don't know all the answers, right? And I will never profess all the answers. And before I say, “You should do this—” or maybe I will say, “You might be able to do this. Let me go save as possible.” [laugh]. Right? And so, just let me just see, can you do this? Does this work? No, I guess it doesn't. Or AWS docu—especially, “The AWS documentation says this. Let me see if that's actually the case.”Corey: I don't believe that they intend to lie, but—Tim: No.Corey: —they also certainly don't get it correct all the time.Tim: And to be fair, they have, what, 728 services by this point, and that's a lot of documentation you're not going to get—Corey: Three more have launched since the start of this recording.Tim: I—yeah, actually—well, by the time this hits, they're probably going to have 22. But we'll [laugh] see. But yeah, no. And that's fine. And they're not going to have every use case, and every edge, kind of like, concern handled, and so that's why we need to kick the tires a little bit.And what I think more than anything else is, you know, sometimes we just do things out of convenience. Like, “Well, I don't want to run this on this; let me just fire it up because it's not my money.” [laugh]. But we also want to be fairly concerned about you know, how we do things. You don't want to run a fleet of z1ds, obviously.But there is a certain amount of tire-kicking and infrastructure spinning up that you have to do in order to maintain freshness, right? And it's not a thing where I'm going to say, “Oh, I know YAML off the top of my head, and I need to do—you know, I'm up to speed on every single possible API call that you can make.” No. My technical prowess has always been in architecture and operations. So, I think when we have these conversations, folks mostly tend to be impressed by not only business acumen and strategy, but also being able to get down to the weeds and talking with the developers and the engineers about the minutia. And you will have seen you know, the feedback that I've gotten about my technical prowess has always been good. You know, I can hang with anybody, I feel like.Corey: I would agree wholeheartedly. It's been really interesting watching you in conversations, internally and with our clients, where you will just idly bust out something fricking brilliant out of left field. And most of the time, I don't think you even realize it. It's just one of those things that makes intuitive and instinctive sense to you. And you basically just leave people stunned and their scribbling notes and trying to wrap their heads around what you just said.And it's adorable because sometimes you wind up almost, like, looking embarrassed, like, “Did I say something rude and not realize it? Like, I wasn't trying to be insulting.” It's like, “Nope, nope. You're just doing your thing, Tim. Just keep on doing it. That's fine.”Tim: Yeah, it's funny because, like you, one of the things that I've really enjoyed about it is, like, we'll just start bouncing ideas off of each other and come up with something brilliant. “Yeah, let's do that.” And then, “Okay, this is now a thing.” And it's like, you know, there's something to be said about being around smart people. So, it's not just me coming up with something brilliant; these are almost always fruits of a conversation and discussion being had, and then you formulate something great in your head.But again, this is why I love the aspect of talking and having conversations with people, so that way you can come up with something kind of brilliant. None of this is done in a silo. Like we're not really, really good at what we do because we don't rely or talk to or have conversations with other people.Corey: One thing that you did that I think is one of the most transformative things that has happened in company history in some respects has been when you started, and for the first half of your tenure here, we had two engagement types that we would wind up giving our consulting clients. There's contract negotiation, where we help companies negotiate their long-term commitment contracts with AWS—and we're effective at it and that's fun; that's basically what you would more or less expected to be—and the other is our cost optimization project engagements. And those tend to look six to eight weeks where we wind up going in deep-dives into the intricacies of an organization's AWS accounts, bills, strategy, growth plan, et cetera, et cetera, et cetera, to an exhaustive level of detail. And in an interest of being probably overly transparent here, I didn't like working on those engagements myself. I like coming in, finding the big things that will be transformative to reduce the bills—it's like solving a puzzle—and then the relatively in-depth analysis for things that are a relatively paltry portion of the AWS bill does not really lead me to enjoying the work very much.And I beat my head against that one for years. And you busted out one day with an idea that became our third type of engagement, which is the first pass, where we charge significantly less for the engagement and it essentially distills down into you get us to talk to your engineering teams for a day. Bring us any questions, give us access in advance to these things, and we will basically go on a whirlwind guided tour and lay waste to your AWS bill and highlight different opportunities that we see to optimize these things. And it has been an absolute smash success. People love the engagements.Very often, it leads to that second full-bore engagement that I was describing earlier, but it also aligns very well with the way that I like to think about these things. I'm a great consultant, specifically because once I've delivered the value, I like to leave. Whereas as an employee, I just sort of linger around, and then I go cause problems and other people's departments—ideally, not on purpose, but you know, I am me—and this really emphasizes that and keeps me moving quickly. I really, really like that engagement style and I have you to thank for coming up with the idea and finding a way to do it that didn't either not resonate with the market—in which case, we're not selling a damn thing—or wound up completely eviscerating the value of the longer-term deep-dive engagements, and you threaded that needle perfectly.Tim: I thank you; I appreciate that. There was this kind of vacuum that I saw where, both from a cost and from a resource point where six to eight weeks is a long time for an engineering org to dedicate to any one thing, especially if that one thing isn't directly making money. But engineering orgs are also very interested in saving money. But it's especially in smaller orgs where that velocity is very important, they don't have six to eight weeks for that. They can't dedicate the resources to those deep-dives all the time, and all the conversations we—and when we do a COP, it is exhaustive. We are exploring every avenue to almost an absurd level, right?And that's not the right engagement for a lot of orgs, right? So, coming in and saying, “Hey, you know, this is a quick one; these are the things that you can do. This is 90% of the savings you're going to realize. These things: bam, bam, bam, bam, bam.” Right?And then we give it to the folks and we let them work on it, and then they're like, “Hey, we need this because we want to negotiate EDP,” or, “We need this because, you know, we're just trying to make sure that our costs are in line so we can be more agile, so we can do this project, or whatever.” Right? And then there are a lot of other orgs that do need that exhaustive kind of thing, larger orgs especially, right? Larger, more complex orgs, orgs that are trying to maybe—like, if you're trying to make a play to get acquired, you want to get this very, very in-depth study so you know all your liabilities and all your assets, so that way you can fix those problems and make it very attractive for someone to buy you, right? Or orgs that just have, like, we are not having an impending EDP; we have a lot of time to be able to focus on these things, and we can build this into the roadmap, right?Then we can do a very exhaustive study of those things. But for a lot of times, people are just like, “Look, I just need to save X amount of money on my AWS bill and can you do that?” Well, sure. We can go in there and have those conversations and give you a lot of savings. And I'm very much in the camp of, you know, ‘perfect is the enemy of good.' I don't have to save down to the nth penny on your DynamoDB bill. But if I can, shave—cut it in half, that's great. Most people are very happy about those kinds of things. And that's a very routine finding for us.Corey: One other aspect that I really liked about it, too, is that it let us move down market a bit, away from companies that are spending millions of dollars a month. Because yeah, the ROI for those customers is a slam dunk on virtually any engagement that we could put together, but what about the smaller companies, the ones that are not spending that much money, yet? They've never felt great talk to them and say, “Oh, just go screw up your AWS bill some more. Then, then you will absolutely be able to generate some value. Maybe turn off MFA and post your credentials to GitHub or something. That'll speed up the process nicely.”That's terrible advice and we can't do it. But this enables us to move down to smaller companies that are earlier in their cloud estate build-out or are growing organically rather than trying to do a giant migration as sort of greenfield growth approach. I really, really like our ability to help companies that are a bit earlier in their cloud journey, as well as in smaller environments, just because I guess, on some level, for me, at least, when you see enormous multimillion-dollar levels of spend, the misconfigurations are generally less fun to find; they're less exciting. Because, yeah at a small scale, you can screw up and your Managed NAT Gateway bill is a third of your spend. When you're spending $80 million a year, you're not wasting that kind of money on Managed NAT Gateways because that misconfiguration becomes visible from frickin' orbit.So, someone has already found that stuff. And it's always then it's almost certainly EC2, RDS, and storage. Great. Then there's some weird data transfer stuff and it starts to look a lot more identical. Smaller accounts, at least from my perspective, tend to have a lot more of interesting things to learn hiding in the shadows.Tim: Oh, absolutely. And I think the impact that you make for the future for small companies much higher, right? You go in there and you have an engagement, you can say, “Okay, I understand the business reason why you did this here, but if you make these changes—bam, bam, bam—12 to 18 months and on, right, this is going to make a huge difference in your business. You're going to save a tremendous amount of money and you're going to be much more agile.”You did this thing because it worked for the POC, it worked for the MVP, right? That's great, but before it gets too big and becomes load-bearing technical debt, let's make some changes to put you in a better position, both for cost optimization and an architectural future that you don't have to then break a bone that's already set to try and fix it. So, getting in there before there's a tremendous load on their architecture—or rather on their infrastructure, it's super, super fun because you know that when you've done this, you have given that company more runway, or you've given them the things they need to actually be more successful, and so they can focus their time and efforts on growth and not on trying to stop the bleeding with their AWS bill.Corey: Tim, it's been an absolute pleasure to work with you. I'm going to miss working with you, but we are definitely going to remain in touch. Where can people find you to follow along with your continuing adventures?Tim: The best way to find me is on Twitter, I am @elchefe—E-L-C-H-E-F-E. And yeah, I will definitely keep in touch with you, Corey. Again, you have been a tremendous friend and I really appreciate you, your insights, and your honesty. Our partners are friends with each other and I do not think that they will let us ever drift too far apart. So.Corey: No, I think it is pretty clear that we are basically going to be both of their plus-ones forever.Tim: [laugh]. I think so.Corey: I'm just waiting for them when they pulled the prank of dressing us the exact same way because our styles are somewhat different, and I'm pretty sure that there's not a whole lot of convergence where we both wind up looking great. So, it's going to be hilarious regardless of what direction it goes in.Tim: Well, you do have velour tracksuits too, right?Corey: Not yet, but please don't tell that to Bethany.Tim: [laugh].Corey: Tim, it has been an absolute pleasure.Tim: The pleasure has been all mine, Corey. I really appreciate it.Corey: Tim Banks, for one last time, principal cloud economist at The Duckbill Group. I am Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and an insulting comment that says that we are completely wrong in our approach to management and the real answer is as follows, making sure to keep that answer less than 280 characters.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About AntonDr. Anton Chuvakin is now involved with security solution strategy at Google Cloud, where he arrived via Chronicle Security (an Alphabet company) acquisition in July 2019.Anton was, until recently, a Research Vice President and Distinguished Analyst at Gartner for Technical Professionals (GTP) Security and Risk Management Strategies team. (see chuvakin.org for more)Links Referenced: Google Cloud: https://cloud.google.com/ Cloud Security Podcast: https://cloud.withgoogle.com/cloudsecurity/podcast/ Twitter: https://twitter.com/anton_chuvakin Medium blog: https://medium.com/anton.chuvakin TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: Let's face it, on-call firefighting at 2am is stressful! So there's good news and there's bad news. The bad news is that you probably can't prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. My guest today is Anton Chuvakin, who is a Security Strategy Something at Google Cloud. And I absolutely love the title, given, honestly, how anti-corporate it is in so many different ways. Anton, first, thank you for joining me.Anton: Sure. Thanks for inviting me.Corey: So, you wound up working somewhere else—according to LinkedIn—for two months, which in LinkedIn time is about 20 minutes because their date math is always weird. And then you wound up going—according to LinkedIn, of course—leaving and going to Google. Now, that was an acquisition if I'm not mistaken, correct?Anton: That's correct, yes. And it kind of explains that timing in a little bit of a title story because my original title was Head of Security Solution Strategy, and it was for a startup called Chronicle. And within actually three weeks, if I recall correctly, I was acquired into Google. So, title really made little sense of Google, so I kind of go with, like, random titles that include the word security, and occasionally strategy if I feel generous.Corey: It's pretty clear, the fastest way to get hired at Google, given their famous interview process is to just get acquired. Like, “I'm going to start a company and raise it to, like, a little bit of providence, and then do an acquihire because that will be faster than going through the loop, and ideally, there will be less algorithm solving on whiteboards.” But I have to ask, did you have to solve algorithms on whiteboards for your role?Anton: Actually, no, but it did come close to that for some other people who were seen as non-technical and had to join technical roles. I think they were forced to solve coding questions and stuff, but I was somehow grandfathered into a technical role. I don't know exactly how it happened.Corey: Yeah, how you wound up in a technical role. Let's be clear, you are Doctor Anton Chuvakin, and you have written multiple books, you were a research VP at Gartner for many years, and once upon a time, that was sort of a punchline in the circles I hung out with, and then I figured out what Gartner actually does. And okay, that actually is something fairly impressive, let's be clear here. Even as someone who categorically defines himself as not an analyst, I find myself increasingly having a lot of respect for the folks who are actually analysts and the laborious amount of work that they do that remarkably few people understand.Anton: That's correct. And I don't want to boost my ego too much. It's kind of big enough already, obviously, but I actually made it all the way to Distinguished Analyst, which is the next rank after VP.Corey: Ah, my apologies. I did not realize it. This [challenges 00:02:53] the internal structure.Anton: [laugh]. Yeah.Corey: It's like, “Oh, I went from Senior to Staff,” or Staff to Senior because I'm external; I don't know the direction these things go in. It almost feels like a half-step away from oh, I went from [SDE3 to SDE4 00:03:02]. It's like, what do those things mean? Nobody knows. Great.Anton: And what's the top? Is it 17 or is it 113? [laugh].Corey: Exactly. It's like, oh okay, so you're Research VP—or various kinds of VPs—the real question is, how many people have to die before you're the president? And it turns out that that's not how companies think. Who knew?Anton: That's correct. And I think Gartner was a lot of hard work. And it's the type of work that a lot of people actually don't understand. Some people understand it wrong, and some people understand it wrong, kind of, for corrupt reasons. So, for example, a lot of Gartner machinery involves soaking insight from the outside world, organizing it, packaging it, writing it, and then giving it as advice to other people.So, there's nothing offensive about that because there is a lot of insight in the outside world, and somebody needs to be a sponge slash filter slash enrichment facility for that insight. And that, to me, is a good analyst firm, like Gartner.Corey: Yeah. It's a very interesting world. But you historically have been doing a lot of, well, let's I don't even know how to properly describe it because Gardner's clientele historically has not been startups because let's face it, Gartner is relatively expensive. And let's be clear, you're at Google Cloud now, which is a different kind of expensive, but in a way that works for startups, so good for you; gold star. But what was interesting there is that the majority of the Gartner clientele that I've spoken to tend to be big-E Enterprise, which runs legacy businesses, which is a condescending engineering term for ‘it makes money.'And they had the temerity to start their company before 15 years ago, so they built data centers and did things in a data center environment, and now they're moving in a cloudy direction. Your emphasis has always been on security, so my question for you to start with all this is where do you see security vendors fitting in? Because when I walk the RSA expo hall and find myself growing increasingly depressed, it seems like an awful lot of what vendors are selling looks very little removed from, “We took a box, now we shoved in a virtual machine and here you go; it's in your cloud environment. Please pay us money.” The end. And it feels, if I'm looking at this from a pure cloud-native, how I would build things in the cloud from scratch perspective, to be the wrong design. Where do you stand on it?Anton: So, this has been one of the agonizing questions. So, I'm going to kind of ignore some of the context. Of course, I'll come back to it later, but want to kind of frame it—Corey: I love ignoring context. My favorite thing; it's what makes me a decent engineer some days.Anton: So, the frame was this. One of the more agonizing questions for me as an analyst was, a client calls me and says, “We want to do X.” Deep in my heart, I know that X is absolutely wrong, however given their circumstances and how they got to decided to do X, X is perhaps the only thing they can logically do. So, do you tell them, “Don't do X; X is bad,” or you tell them, “Here's how you do X in a manner that aligns with your goals, that's possible, that's whatever.”So, cloud comes up a lot in this case. Somebody comes and says, I want to put my on-premise security information management tool or SIM in the cloud. And I say, deep in my heart, I say, “No, get cloud-native tool.” But I tell them, “Okay, actually, here's how you do it in a less painful manner.” So, this is always hard. Do you tell them they're on their own path, but you help them tread their own path with least pain? So, as an analyst, I agonized over that. This was almost like a moral decision. What do I tell them?Corey: It makes sense. It's a microcosm of the architect's dilemma, on some level, because if you ask a typical Google-style interview whiteboard question, one of my favorites in years past was ‘build a URL shortener.' Great. And you can scale it out and turn it into different things and design things on the whiteboard, and that's great. Most mid-level people can wind up building a passable designed for most things in a cloud sense, when you're starting from scratch.That's not hard. The problem is that the real world is messy and doesn't fit on a whiteboard. And when you're talking about taking a thing that exists in a certain state—for whatever reason, that's the state that it's in—and migrating it to a new environment or a new way of operating, there are so many assumptions that have to break, and in most cases, you don't get the luxury of just taking the thing down for 18 months so you can rework it. And even that it's never as easy as people think it is, so it's going to be 36. Great.You have to wind up meeting people where they are as they're contextualizing these things. And I always feel like the first step of the cloud migration has been to improve your data center environment at the cost of worsening your cloud environment. And that's okay. We don't all need to be the absolute vanguard of how everything should be built and pushing the bleeding edge. You're an insurance company, for God's sake. Maybe that's not where you want to spend your innovation energies.Anton: Yeah. And that's why I tend to lean towards helping them get out of this situation, or maybe build a five-step roadmap of how to become a little bit more cloud-native, rather than tell them, “You're wrong. You should just rewrite the app in a cloud-native way.” That advice almost never actually works in real world. So, I see a lot of the security people move their security stacks to the cloud.And if I see this, I deepen my heart and say, “Holy cow. What do you mean, you want to IDS every packet between Cloud instances? You want to capture every packet in cloud instances? Why? It's all encrypted anyway.” But I don't say that. I say, “Okay, I see how this is the first step for you. Let's describe the next seven steps.”Corey: The problem I keep smacking into is that very often folks who are pushing a lot of these solutions are, yes, they're meeting customers where they are, and that makes an awful lot of sense; I'm not saying that there's anything inherently wrong about that. The challenge is it also feels on the high end, when those customers start to evolve and transform, that those vendors act as a drag. Because if you wind up going in a full-on cloud-native approach, in the fullness of time, there's an entire swath of security vendors that do not have anything left to sell you.Anton: Yes, that is correct. And I think that—I had a fight with an EDR vendor, Endpoint Detection Response, vendor one day when they said, “Oh, we're going to be XDR and we'll do cloud.” And I told them, “You do realize that in a true cloud-native environment, there's no E? There is no endpoint the way you understand it? There is no OS. There is no server. And 99% of your IP isn't working on the clients and servers. How are you going to secure a cloud again?”And I get some kind of rambling answer from them, but the point is that you're right, I do see a lot of vendors that meet clients where they are during their first step in the cloud, and then they may become a drag, or the customer has to show switch to a cloud-native vendor, or to both sometimes, and pay into two mouths. Well, shove money into two pockets.Corey: Well, first, I just want to interject for a second here because when I was walking the RSA expo floor, there were something like 15 different vendors that were trying to sell me XDR. Not a single one of them bothered to expand the acronym—Anton: Just 15? You missed half of them.Corey: Well, yeah—Anton: Holy cow.Corey: As far as I know XDR cable. It's an audio thing right? I already have a bunch of those for my microphone. What's the deal here? Like, “I believe that's XLR.” It's like, “I believe you should expand your acronyms.” What is XDR?Anton: So, this is where I'm going to be very self-serving and point to a blog that I've written that says we don't know what's XDR. And I'm going to—Corey: Well, but rather than a spiritual meaning, I'm going to ask, what does the acronym stands for? I don't actually know the answer to that.Anton: Extended Detection and Response.Corey: Ah.Anton: Extended Detection and Response. But the word ‘extended' is extended by everybody in different directions. There are multiple camps of opinion. Gartner argues with Forrester. If they ever had a pillow fight, it would look really ugly because they just don't agree on what XDR is.Many vendors don't agree with many other vendors, so at this point, if you corner me and say, “Anton, commit to a definition of XDR,” I would not. I will just say, “TBD. Wait two years.” We don't have a consensus definition of XDR at this point. And RSA notwithstanding, 30 booths with XDRs on their big signs… still, sorry, I don't have it.Corey: The problem that I keep running into again and again and again, has been pretty consistently that there are vendors willing to help customers in a very certain position, and for those customers, those vendors are spot on the right thing to do.Anton: Mmm, yep.Corey: But then they tried to expand and instead of realizing that the market has moved on and the market that they're serving is inherently limited and long-term is going to be in decline, they instead start trying to fight the tide and saying, “Oh, no, no, no, no. Those new cloud things, can't trust them.” And they start out with the FU, the Fear, Uncertainty, and Doubt marketing model where, “You can't trust those newfangled cloud things. You should have everything on-prem,” ignoring entirely the fact that in their existing data centers, half the time the security team forgets to lock the door.Anton: Yeah, yeah.Corey: It just feels like there is so much conflict of interest about in the space. I mean, that's the reason I started my Thursday Last Week in AWS newsletter that does security round-ups, just because everything else I found was largely either community-driven where it understood that it was an InfoSec community thing—and InfoSec community is generally toxic—or it was vendor-captured. And I wanted a round-up of things that I had to care about running an infrastructure, but security is not in my job title, even if the word something is or is not there. It's—I have a job to do that isn't security full time; what do I need to know? And that felt like an underserved market, and I feel like there's no equivalent of that in the world of the emerging cloud security space.Anton: Yes, I think so. But it has a high chance of being also kind of captured by legacy vendors. So, when I was at Gartner, there was a lot of acronyms being made with that started with a C: Cloud. There was CSPM, there was CWBP, and after I left the coined SNAPP with double p at the end. Cloud-Native Application Protection Platform. And you know, in my time at Gartner, five-letter acronyms are definitely not very popular. Like, you shouldn't have done a five-letter acronym if you can help yourself.So, my point is that a lot of these vendors are more from legacy vendors. They are not born in the cloud. They are born in the 1990s. Some are born in the cloud, but it's a mix. So, the same acronym may apply to a vendor that's 2019, or—wait for it—1989.Corey: That is… well, I'd say on the one hand, it's terrifying, but on the other, it's not that far removed from the founding of Google.Anton: True, true. Well, '89, kind of, it's another ten years. I think that if you're from the '90s, maybe you're okay, but if you're from the '80s… you really need to have superpowers of adaptation. Again, it's possible. Funny aside: at Gartner, I met somebody who was an analyst for 32 years.So, he was I think, at Gartner for 32 years. And how do you keep your knowledge current if you are always in an ivory tower? The point is that this person did do that because he had a unique ability to absorb knowledge from the outside world. You can adapt; it's just hard.Corey: It always is. I'm going to pivot a bit and put you in a little bit of a hot seat here. Not intentionally so. But it is something that I've been really kicking around for a while. And I'm going to basically focus on Google because that's where you work.I yeah, I want you to go and mouth off about other cloud companies. Yeah, that's—Anton: [laugh]. No.Corey: Going to go super well and no one will have a problem with that. No, it's… we'll pick on Google for a minute because Google Cloud offers a whole bunch of services. I think it's directionally the right number of services because there are areas that you folks do not view as a core competency, and you actually—imagine that—partner with third parties to wind up delivering something great rather than building this shitty knockoff version that no one actually wants. Ehem, I might be some subtweeting someone here with this, only out loud.Anton: [laugh].Corey: The thing that resonates with me though, is that you do charge for a variety of security services. My perspective, by and large, is that the cloud vendors should not be viewing security as a profit center but rather is something that comes baked into the platform that winds up being amortized into the cost of everything else, just because otherwise you wind up with such a perverse set of incentives.Anton: Mm-hm.Corey: Does that sound ridiculous or is that something that aligns with your way of thinking. I'm willing to take criticism that I'm wrong on this, too.Anton: Yeah. It's not that. It's I almost start to see some kind of a magic quadrant in my mind that kind of categorizes some things—Corey: Careful, that's trademarked.Anton: Uhh, okay. So, some kind of vis—Corey: It's a mystical quadrilateral.Anton: Some kind of visual depiction, perhaps including four parts—not quadrants, mind you—that is focused on things that should be paid and aren't, things that should be paid and are paid, and whatever else. So, the point is that if you're charging for encryption, like basic encryption, you're probably making a mistake. And we don't, and other people, I think, don't as well. If you're charging for logging, then it's probably also wrong—because charging for log retention, keeping logs perhaps is okay because ultimately you're spending resources on this—charging for logging to me is kind of in the vile territory. But how about charging for a tool that helps you secure your on-premise environment? That's fair game, right?Corey: Right. If it's something you're taking to another provider, I think that's absolutely fair. But the idea—and again, I'm okay with the reality of, “Okay, here's our object storage costs for things, and by the way, when you wind up logging things, yeah, we'll charge you directionally what it costs to store that an object store,” that's great, but I don't have the Google Cloud price list shoved into my head, but I know over an AWS land that CloudWatch logs charge 50 cents per gigabyte, for ingress. And the defense is, “Well, that's a lot less expensive than most other logging vendors out there.” It's, yeah, but it's still horrifying, and at scale, it makes me want to do some terrifying things like I used to, which is build out a cluster of Rsyslog boxes and wind up having everything logged to those because I don't have an unbounded growth problem.This gets worse with audit logs because there's no alternative available for this. And when companies start charging for that, either on a data plane or a management plane level, that starts to get really, really murky because you can get visibility into what happened and reconstruct things after the fact, but only if you pay. And that bugs me.Anton: That would bug me as well. And I think these are things that I would very clearly push into the box of this is security that you should not charge for. But authentication is free. But, like, deeper analysis of authentication patterns, perhaps costs money. This to me is in the fair game territory because you may have logs, you may have reports, but what if you want some kind of fancy ML that analyzes the logs and gives you some insights? I don't think that's offensive to charge for that.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it's still somebody's problem. And a big part of that responsibility is app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you're actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That's S-N-Y-K.co/screamCorey: I think it comes down to what you're doing with it. Like, the baseline primitives, the things that no one else is going to be in a position to do because honestly, if I can get logging and audit data out of your control plane, you have a different kind of security problem, and—Anton: [laugh].Corey: That is a giant screaming fire in the building, as it should be. The other side of it, though, is that if we take a look at how much all of this stuff can cost, and if you start charging for things that are competitive to other log analytics tools, great because at that point, we're talking about options. I mean, I'd like to see, in an ideal world, that you don't charge massive amounts of money for egress but ingress is free. I'd like to see that normalized a bit.But yeah, okay, great. Here's the data; now I can run whatever analytics tools I want on it and then you're effectively competing on a level playing field, as opposed to, like, okay, this other analytics tool is better, but it'll cost me over ten times as much to migrate to it, so is it ten times better? Probably not; few things are, so I guess I'm sticking with the stuff that you're offering. It feels like the cloud provider security tools never quite hit the same sweet spot that third-party vendors tend to as far as usability, being able to display things in a way that aligns with various stakeholders at those companies. But it still feels like a cash grab and I have to imagine without having insight into internal costing structures, that the security services themselves are not a significant revenue driver for any of the cloud companies. And the rare times where they are is almost certainly some horrifying misconfiguration that should be fixed.Anton: That's fair, but so to me, it still fits into the bucket of some things you shouldn't charge for and most people don't. There is a bucket of things that you should not charge for, but some people do. And there's a bucket of things where it's absolutely fair to charge for I don't know the amount I'm not a pricing person, but I also seen things that are very clearly have cost to a provider, have value to a client, have margins, so it's very clear it's a product; it's not just a feature of the cloud to be more secure. But you're right if somebody positions as, “I got cloud. Hey, give me secure cloud. It costs double.” I'd be really offended because, like, what is your first cloud is, like, broken and insecure? Yeah. Replace insecure with broken. Why are you selling broken to me?Corey: Right. You tried to spin up a service in Google Cloud, it's like, “Great. Do you want the secure version or the shitty one?”Anton: Yeah, exactly.Corey: Guess which one of those costs more. It's… yeah, in the fullness of time, of course, the shitty one cost more because you find out about security breaches on the front page of The New York Times, and no one's happy, except maybe The Times. But the problem that you hit is that I don't know how to fix that. I think there's an opportunity there for some provider—any provider, please—to be a trendsetter, and, “Yeah, we don't charge for security services on our own stuff just because it'd be believed that should be something that is baked in.” Like, that becomes the narrative of the secure cloud.Anton: What about tiers? What about some kind of a good, better, best, or bronze, gold, platinum, where you have reasonable security, but if you want superior security, you pay money? How do you feel, what's your gut feel on this approach? Like, I can't think of example—log analysis. You're going to get some analytics and you're going to get fancy ML. Fancy ML costs money; yay, nay?Corey: You're bringing up an actually really interesting point because I think I'm conflating too many personas at once. Right now, just pulling up last months bill on Google Cloud, it fits in the free tier, but my Cloud Run bill was 13 cents for the month because that's what runs my snark.cloud URL shortener. And it's great. And I wound up with—I think my virtual machine costs dozen times that much. I don't care.Over in AWS-land, I was building out a serverless nonsense thing, my Last Tweet In AWS client, and that cost a few pennies a month all told, plus a whopping 50 cents for a DNS zone. Whatever. But because I was deploying it to all regions and the way that configural evaluations work, my config bill for that was 16 bucks. Now, I don't actually care about the dollar figures on this. I assure you, you could put zeros on the end of that for days and it doesn't really move the needle on my business until you get to a very certain number there, and then suddenly, I care a lot.Anton: [laugh]. Yeah.Corey: And large enterprises, this is expected because even the sheer cost of people's time to go through these things is valuable. What I'm thinking of is almost a hobby-level side project instead, where I'm a student, and I'm learning this in a dorm room or in a bootcamp or in my off hours, or I'm a career switcher and I'm doing this on my own dime out of hours. And I wind up getting smacked with the bill for security services that, for a company, don't even slightly matter. But for me, they matter, so I'm not going to enable them. And when I transition into the workforce and go somewhere, I'm going to continue to work the same way that I did when I was an independent learner, like, having a wildly generous free tier for small-scale accounts, like, even taking a perspective until you wind up costing, I don't know, five, ten—whatever it is—thousand dollars a month, none of the security stuff is going to be billable for you because it's it is not aimed at you and we want you comfortable with and using these things.This is a whole deep dive into the weeds of economics and price-driven behavior and all kinds of other nonsense, but every time I wind up seeing that, like, in my actual production account over at AWS land for The Duckbill Group, all things wrapped up, it's something like 1100 bucks a month. And over a third of it is monitoring, audit, and observability services, and a few security things as well. And on the one hand, I'm sitting here going, “I don't see that kind of value coming from it.” Now, the day there's an incident and I have to look into this, yeah, it's absolutely going to be worth having, but it's insurance. But it feels like a disproportionate percentage of it. And maybe I'm just sitting here whining and grousing and I sound like a freeloader who doesn't want to pay for things, but it's one of those areas where I would gladly pay more for a just having this be part of the cost and not complain at all about it.Anton: Well, if somebody sells me a thing that costs $1, and then they say, “Want to make it secure?” I say yes, but I'm already suspicious, and they say, “Then it's going to be 16 bucks.” I'd really freak out because, like, there are certain percentages, certain ratios of the actual thing plus security or a secure version of it; 16x is not the answer expect. 30%, probably still not the answer I expect, frankly. I don't know. This is, like, an ROI question [crosstalk 00:23:46]—Corey: Let's also be clear; my usage pattern is really weird. You take a look at most large companies at significant scale, their cloud environments from a billing perspective look an awful lot like a crap ton of instances—or possibly containers running—and smattering of other things. Yeah, you also database and storage being the other two tiers and because of… reasons data transfer loves to show up too, but by and large, everything else was more or less a rounding error. I have remarkably few of those things, just given the weird way that I use services inappropriately, but that is the nature of me, so don't necessarily take that as being gospel. Like, “Oh, you'll spend a third of your bill.”Like, I've talked to analyst types previously—not you, of course—who will hear a story like this and that suddenly winds up as a headline in some report somewhere. And it's, “Yeah, if your entire compute is based on Lambda functions and you get no traffic, yeah, you're going to see some weird distortions in your bill. Welcome to the conversation.” But it's a problem that I think is going to have to be addressed at some point, especially we talked about earlier, those vendors who are catering to customers who are not born in the cloud, and they start to see their business erode as the cloud-native way of doing things continues to accelerate, I feel like we're in for a time where they're going to be coming at the cloud providers and smacking them for this way harder than I am with my, “As a customer, wouldn't it be nice to have this?” They're going to turn this into something monstrous. And that's what it takes, that's what it takes. But… yeah.Anton: It will take more time than than we think, I think because again, back in the Gartner days, I loved to make predictions. And sometimes—I've learned that predictions end up coming true if you're good, but much later.Corey: I'm learning that myself. I'm about two years away from the end of it because three years ago, I said five years from now, nobody will care about Kubernetes. And I didn't mean it was going to go away, but I meant that it would slip below the surface level of awareness to point where most people didn't have to think about it in the same way. And I know it's going to happen because it's too complex now and it's going to be something that just gets handled in the same way that Linux kernels do today, but I think I was aggressive on the timeline. And to be clear, I've been misquoted as, “Oh, I don't think Kubernetes is going to be relevant.”It is, it's just going to not be something that you need to spend the quarter million bucks an engineer on to run in production safely.Anton: Yeah.Corey: So, we'll see. I'm curious. One other question I had for you while I've got you here is you run a podcast of your own: the Cloud Security Podcast if I'm not mistaken, which is—Anton: Sadly, you are not. [laugh].Corey: —the Cloud Se—yeah. Interesting name on that one, yeah. It's like what the Cloud Podcast was taken?Anton: Essentially, we had a really cool name [Weather Insecurity 00:26:14]. But the naming team here said, you must be descriptive as everybody else at Google, and we ended up with the name, Cloud Security Podcast. Very, very original.Corey: Naming is challenging. I still maintain that the company is renamed Alphabet, just so it could appear before Amazon in the yellow pages, but I don't know how accurate that one actually is. Yeah, to be clear, I'm not dunking on your personal fun podcast, for those without context. This is a corporate Google Cloud podcast and if you want to make the argument that I'm punching down by making fun of Google, please, I welcome that debate.Anton: [laugh]. Yes.Corey: I can't acquire companies as a shortcut to hire people. Yet. I'm sure it'll happen someday, but I can aspire to that level of budgetary control. So, what are you up to these days? You spent seven years at Gartner and now you're doing a lot of cloud security… I'll call it storytelling, and I want to be clear that I mean that as a compliment, not the, “Oh, you just tell stories rather than build things?”Anton: [laugh].Corey: Yeah, it turns out that you have to give people a reason to care about what you've built or you don't have your job for very long. What are you talking about these days? What narratives are you looking at going forward?Anton: So, one of the things that I've been obsessed with lately is a lot of people from more traditional companies come in in the cloud with their traditional on-premise knowledge, and they're trying to do cloud the on-premise way. On our podcast, we do dedicate quite some airtime to people who do cloud as if it were a rented data center, and sometimes we say, the opposite is called—we don't say cloud-native, I think; we say you're doing the cloud the cloudy way. So, if you do cloud, the cloudy way, you're probably doing it right. But if you're doing the cloud is rented data center, when you copy a security stack, you lift and shift your IDS, and your network capture devices, and your firewalls, and your SIM, you maybe are okay, as a first step. People here used to be a little bit more enraged about it, but to me, we meet customers where they are, but we need to journey with them.Because if all you do is copy your stack—security stack—from a data center to the cloud, you are losing effectiveness, you're spending money, and you're making other mistakes. I sometimes joke that you copy mistakes, not just practices. Why copy on-prem mistakes to the cloud? So, that's been bugging me quite a bit and I'm trying to tell stories to guide people out of a situation. Not away, but out.Corey: A lot of people don't go for the idea of the lift and shift migration and they say that it's a terrible pattern and it causes all kinds of problems. And they're right. The counterpoint is that it's basically the second-worst approach and everything else seems to tie itself for first place. I don't mean to sound like I'm trying to pick a fight on these things, but we're going to rebuild an application while we move it. Great.Then it doesn't work or worse works intermittently and you have no idea whether it's the rewrite, the cloud provider, or something else you haven't considered. It just sounds like a recipe for disaster.Anton: For sure. And so, imagine that you're moving the app, you're doing cut-and-paste to the cloud of the application, and then you cut-and-paste security, and then you end up with sizeable storage costs, possibly egress costs, possibly mistakes you used to make beyond five firewalls, now you make this mistake straight on the edge. Well, not on the edge edge, but on the edge of the public internet. So, some of the mistakes do become worse when you copy them from the data center to the cloud. So, we do need to, kind of, help people to get out of the situation but not by telling them don't do it because they will do it. We need to tell them what step B; what's step 1.5 out of this?Corey: And cost doesn't drive it and security doesn't drive it. Those are trailing functions. It has to be a capability story. It has to be about improving feature velocity or it does not get done. I have learned this the painful way.Anton: Whatever 10x cost if you do something in the data center-ish way in the cloud, and you're ten times more expensive, cost will drive it.Corey: To an extent, yes. However, the problem is that companies are looking at this from the perspective of okay, we can cut our costs by 90% if we make these changes. Okay, great. It cuts the cloud infrastructure cost that way. What is the engineering time, what is the opportunity cost that they gets baked into that, and what are the other strategic priorities that team has been tasked with this year? It has to go along for the ride with a redesign that unlocks additional capability because a pure cost savings play is something I have almost never found to be an argument that carries the day.There are always exceptions, to be clear, but the general case I found is that when companies get really focused on cost-cutting, rather than expanding into new markets, on some level, it feels like they are not in the best of health, corporately speaking. I mean, there's a reason I'm talking about cost optimization for what I do and not cost-cutting.It's not about lowering the bill to zero at all cost. “Cool. Turn everything off. Your bill drops to zero.” “Oh, you don't have a company anymore? Okay, so there's a constraint. Let's talk more about that.” Companies are optimized to increase revenue as opposed to reduce costs. And engineers are always more expensive than the cloud provider resources they're using, unless you've done something horrifying.Anton: And some people did, by replicating their mistakes for their inefficient data centers straight into the cloud, occasionally, yeah. But you're right, yeah. It costs the—we had the same pattern of Gartner. It's like, it's not about doing cheaper in the cloud.Corey: I really want to thank you for spending so much time talking to me. If people want to learn more about what you're up to, how you view the world, and what you're up to next, where's the best place for them to find you?Anton: At this point, it's probably easiest to find me on Twitter. I was about to say Podcast, I was about to say my Medium blog, but frankly, all of it kind of goes into Twitter at some point. And so, I think I am twitter.com/anton_chuvakin, if I recall correctly. Sorry, I haven't really—Corey: You are indeed. It's always great; it's one of those that you have a sizable audience, and you're like, “What is my Twitter handle, again? That's a good question. I don't know.” And it's your name. Great. Cool. “So, you're going to spell that for you, too, while you're at it?” We will, of course, put a link to that in the [show notes 00:32:09]. I really want to thank you for being so generous with your time. I appreciate it.Anton: Perfect. Thank you. It was fun.Corey: Anton Chuvakin, Security Strategy Something at Google Cloud. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment because people are doing it wrong, but also tell me which legacy vendor you work for.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About Anadelia Anadelia is a B2B marketing leader passionate about building tech brands and growing revenue. She is currently the Sr. Director of Demand Generation at Teleport. In her spare time she enjoys live music and craft beer.Links Referenced: Teleport: https://goteleport.com/ @anadeliafadeev: https://twitter.com/anadeliafadeev LinkedIn: https://www.linkedin.com/in/anadeliafadeev/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: Let's face it, on-call firefighting at 2am is stressful! So there's good news and there's bad news. The bad news is that you probably can't prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This may surprise some of you to realize, but every once in a while, I mention how these episodes are sponsored by different companies. Well, to peel back a little bit of the mystery behind that curtain, I should probably inform some of you that when I say that, that means that companies have paid me to talk about them. I know, shocking.This is a revelation that will topple the podcast industry if it gets out. That's why it's just between us. My guest today knows this better than most. Anadelia Fadeev is the Senior Director of Demand Generation at Teleport, who does in fact sponsor a number of different things that I do, but this is not a sponsored episode in that context. Anadelia, thank you for joining me today.Anadelia: Thank you for having me.Corey: It's interesting. I always have to double-check where it is that you happen to be working because when we first met you were a Senior Marketing Manager, also in Demand Gen, at InfluxData, then you were a Director of Demand Generation at LightStep, and then you became a Director of Demand Gen and Growth and then a Senior Director of Demand Gen, where you are now at Teleport. And the couple of things that I've noticed are, one, you seem to more or less be not only doing the same role, but advancing within it, and also—selfishly—it turns out that every time you wind up working somewhere, that company winds up sponsoring some of my nonsense. So first, thank you for your business. It's always appreciated. Now, what is demand gen exactly? Because I have to say, when I started podcasting and newslettering and shooting my mouth off on the internet, I had no clue.Anadelia: [laugh]. Well, to put it very simply, demand generation, our goal is to drive awareness and interest in your products or services. It's as simple as that. Now, how we do that, we could definitely dive into the specifics, but it's all about generating awareness and interest. Especially when you work for an early-stage startup, it's all about awareness, right? Just getting your name out there.Corey: Marketing is one of those things that I suspect in some ways is kind of like engineering, where you take a look at, “Oh, what do you do? I'm a software engineer.” Okay, great. For someone who is in that space, does that mean front-end? Does that mean back-end? Does that mean security? Oh, wait, you're crying and awake at weird hours and you're angry all the time. You're a DevOps, aren't you?And you start to realize that there are these breakdowns within engineering. And we realize this and we get offended when people in some cases miscategorize us as, “I am not that kind of engineer. How dare you?” Which I think is unwarranted and ridiculous, but it also sort of slips under our notice in the engineering space that marketing is every bit as divided into different functions, different roles, and the rest. For those of us who think of marketing in the naive approach, like I did when I started this place—“Oh, marketing. So basically, you do Super Bowl ads, right?” And it turns out, there might be more than one or two facets to marketing. What's your journey been like in the wide world of marketing? Where did you start? Where does it stop?Anadelia: Yeah. I have not gotten to the Super Bowl ads phase yet but on my way there. No, but when you think about the different core areas within marketing, right, you have your product marketing team, and this is the team that sets the positioning, the messaging, and the information about who your ideal audience is, what pain points are they having, and how is your product solving those pain points? Right, so they sort of set the direction for the rest of the team, you have another core function, which is the content team, right? So, with the direction from Product Marketing, now that we know what the pain points are and what our value prop for our product is, how do we tell that to the world in a compelling way, right? So, this is where content marketing really comes into play.And then you have your demand generation teams. And some companies might call it growth or revenue or… I guess those two are the ones that come to mind. But this team is taking the direction from Product Marketing, taking the content produced by the content team, and then just making sure that people actually see it, right? And across all those teams, you have a lot of support from operations making sure that there's processes and systems in place to support all of those marketing efforts, you have teams that help support web development and design, and brand.Corey: One of the challenges that I think people have when they don't really understand what marketing is they think back on what they know—maybe they've seen Mad Men, which to my understanding does not much resemble modern all workplaces, but then again, I've been on my own for five years, so one wonders—and they also see things in the context of companies that are targeting more mass-market, in some respects. If you're trying to advertise Coca-Cola, every person on the planet—give or take—knows what Coca-Cola is. And the job is just to resurface it, on some level, in people's awareness, so the correct marketing answer there apparently, is to slap the logo on a bunch of things, be it a stadium, be it a billboard, be it almost anything, whereas when we're talking about earlier stage companies—oh, I don't know Teleport, for example—if you were to slap the Teleport logo on a stadium somewhere for some sports game, I have the impression that most people looking at that, if they notice it at all, would instead respond to some level of confusion of, “Teleport, what is that exactly? Have scientists cracked the way of getting me to Miami from San Francisco in less than ten seconds? Because I feel like I would have heard about that.”There's a matter of targeting beyond just the general public or human beings walking around and starting to target people who might have a problem that you know how to solve. And then, of course, figuring out where those people are gathering and how to get in front of them in a way that resonates instead of being annoying. At least that has been my lived experience of watching the challenges that marketing people have talked to me about over the years. Is that directionally correct or are they all just shining me on and, like, “Oh, Corey, you're adorable, you almost understand how this stuff works. Now, go insult some more things on Twitter. It'll be fine.”Anadelia: [laugh]. The reality is that advertising is a big part of a demand generation program, but it's not all, right? So, good demand generation is meeting people where they are. So, the right channels, the right mediums, the right physical places. So, when you look at it from an inbound and outbound approach, inbound, you have a sign outside of your door inviting people to your house, right, and this is in the form of your website. And outbound is you go out to where people are and you knock on their door to introduce yourself.So, when we look at it from that approach, so on the inbound side, right, the goal is to get people to come to your website because that is where you are telling them what you do and giving them the option to start using your product. So, what reason are you giving people to come to you, right? How are you helping them become better at something or achieve certain results, right? So, understanding the motivations behind it is extremely important.And how are you driving people to you? Well, that's where SEO comes in, right? Search engine optimization.So, what content are you producing that is driving the right search results to get your website to show up and get people to come to you, right? There's also SEM or Search Engine Marketing. So, when people are searching for certain keywords that are relevant to you, are you showing up in those search results?And on the outbound side of things is, what do you do to contribute to existing communities, right? So, this is where things like advertising comes into play. So, I know you have a huge following and I want to be where you are. So, of course, I'm going to sponsor your podcast and your newsletters. And similarly, I'm looking for what events are out there where I know that our potential customers are spending their time and what can we do to join that conversation in a way that adds value?So, that can be in the form of supporting community events and meetups, giving community members a platform to share their experiences, and even supporting local businesses, right, it's all about adding value, and by doing so, you are building trust that will allow you to then talk about how your product can help these communities solve their problems.Corey: It's interesting because when we look at the places that you have been, you were at InfluxData, they are a time-series database company; you were at LightStep, which was effectively an observability company, and now you're at Teleport where you are an authentication and access company. And forgive me, none of these are your terms. These are my understandings of having talked to these folks. And on the one hand, from a product perspective, it sounds like you're hopping between this and that and doing all those other things, and yet, we had conversations about all three of those products and how the companies around them are structured and built, and you've advertised all three of those on this show and others and all three of those companies and products speak specifically to problems that I have dealt with personally in the way I go through my engineering existence as well. So, instead of specializing on a particular product or on a particular niche, it almost feels like you're specializing on a particular audience. Is that how you think about it, or is that just one of those happy accident, or in retrospect, we're just going to retcon everything, and, “Yeah, that's exactly why I did it.” And you're like, “Let we jot that down. That belongs on my resume somewhere.”Anadelia: [laugh]. No, so prior to me joining InfluxData, I was at other companies that were marketing to sales, HR, finance, different audiences, right? And the moment I joined Influx, it was really eye-opening for me to be part of a product that has an open-source community, and between that and marketing to a highly technical audience that probably very likely doesn't want to hear from marketers, I found that to be a really good challenge for myself because it challenged me to elevate my own technical knowledge. And also personally, I just want to be surrounded by people that are smarter than me, and so I know that by being part of a community that markets to a developer audience, I am putting myself in a position where I'm having to constantly continue to learn. So, it's a good challenge for a marketer in our industry. Just like in any others, there's always the latest buzzword or the latest trend, and so it's really easy to get caught up in those things. And I think that being a marketer whose audience is developers really forces you to kind of look at what you're doing and sort of remove the fluff. This happens everywhere.Corey: Well, I have to be careful about selling yourself too short on this because I've talked to a lot of different people who want to wind up promoting what it is that their companies do, and people come from all kinds of different places, and some of the less likely to be successful—in many cases, I turn the business down—are, “Well, this is our first real experience with marketing.” And the reason for that is people expect unrealistic things. I describe what I do as top-of-funnel where we get people's attention and we give them a glimpse and a hook of what it is the product does. And I do that by talking about the painful problem that the product solves. So, when people hear their pain reflected in what we talk about, then that gives them the little bit of a push to go and take a look and see if this solves it.And that's great, but there has to be a process on the other side, where oh, a prospect comes in and starts looking at what it is we do. Do we have a sales funnel that moves them from someone just idly browsing to someone who might sign up for a trial, or try this in their own time, or start to understand how the community views it and the rest because just dropping a bunch of traffic on someone's website doesn't, in isolation, achieve anything without a means to convert that traffic into something that's a bit more meaningful and material to the business? I've talked to other folks who are big on oh, well, we want to wind up just instrument in the living crap out of everything we put out there, so I want to know, when someone clicks on the ad, who they are, what they do for a living, what their signing authority is, et cetera, et cetera, et cetera. And my answer, that's super easy, “Cool. We don't do any of that.”Part of the reason that people like hearing from me, is because I generally tend to respect their time, I'm not supporting invasive tracking of what they do, they don't see my dumb face smiling with a open mouth grin as they travel across the internet on every property. Although one of these days I will see myself on the side of a bus; I'm just waiting for it. And it's really nice to be able to talk to people who get the nuances and the peculiarities of the audience that I tend to speak to the most. You've always had that unlocked, even since our first conversation.Anadelia: Yeah, well, first of all, thank you. And yeah, the reality is that, especially within my world, right—and demand generation, we are very metrics-driven because our goal [tends 00:13:00] to be pipeline, right? Pipeline for the sales team, so we want to generate sales opportunities, and in order to do that, we need to be able to measure what's working and what is not working. But the reality is that good marketing is all about building trust, right? So, that's why I stress the importance of providing something of value to your prospect so that you're not wasting their time, right? The message that you have for them is something that can help them in the future.And if building trust sometimes means I'm not able to measure the direct results of the activity that you're doing, then that is okay, right? Because when you're driving people to your website, there are things that you can measure, like, you have some web visits, and you know that percentage of those visitors might be interested in continue further, right? So, when you look at the journey across the buyer stages, you have to have a compelling offer for a person on each of the possible stages, right? So, if they are just learning about you today because this is the first time that heard your ad, it's probably not expected that they would immediately go to your website and fill out your form, right? They've just heard about you, and now you start building that recognition.Now, if all the stars align, and I actually have a need for a solution that's like yours today, then, of course, you can expect a conversion to happen in that time point. But the reality is that having offers that are aimed at every stage of the buyer's journey is important.Corey: I'm glad to hear you say this. And the reason is that I often feel like when I say it, it sounds incredibly self-serving. But if you imagine the ideal buyer and their journey, they have the exact problem that your product does and there's an ad on my podcast that mentions it. Well, I imagine—and maybe this isn't accurate, but it's how I engage with podcasts myself—I'm probably not sitting in front of a computer ready to type in whatever it is that gets talked about.I'm probably doing dishes or outside harassing a dog or something. And if it resonates is, “Oh, I should look into that.” In an ideal world. I'll remember the short URL that I can go to, but in practice, I might just Google the company name. And oh, this does solve the problem.If it's not just me and there's a team I have to have a buy-in on, I might very well mention it in our next group meeting. And, “Okay, we're going to go ahead and try it out with an open-source version or whatnot.” And, “Oh, this seems to be working. We'll have procurement reach out and see what it takes to wind up generating a longer-term deal.” And the original attribution of the engineer who heard it on a podcast, or the DevOps director who read it in my newsletter, or whatever it is, is long since lost. I've commiserated with marketing people over this, and the adage that I picked up that I love quoting is half your marketing budget is wasted, but you can spend an entire career trying to figure out which half and get nowhere by the end of it.Anadelia: And this sort of touches on the buyer's journey is not linear. On the other side of that ad, or that marketing offer is a human, right? So, of course, as marketers, we're going to try to build this path of once you landed on our website, we want to guide you through all the steps until you do the thing that we want you to do, but the reality is, that does not happen in your example, right? You see something, you come back to it later through another channel, there's no way for us to measure those. And that's okay because that's just the reality of how humans behave.And also, I think it's worth noting that it takes multiple touch points until a person is ready to even hear what you have to say, right? And it sort of goes back to that point of building trust, right? It takes many times until you've gained that person's trust enough for them to listen to what you have to say.Corey: Building trust is important.Anadelia: SIt is very important. And that's why I think that running brand awareness programs are an extremely important part of a marketing mix. And sometimes there's not going to be any direct attribution, and we just have to be okay with it.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it's still somebody's problem. And a big part of that responsibility is app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you're actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That's S-N-Y-K.co/screamCorey: I tend to take a perspective that trust is paramount, on some level, where we have our standard rules of, you know, don't break the law, et cetera, et cetera, that we do require our sponsors to conform to, but there are really two rules that I have that I care about. The first is you're not allowed to lie to the audience. Because if I wind up saying something is true in an ad or whatnot, and it's not, that damages my credibility. And I take this old world approach of, well, I believe trust is built over time, and you continually demonstrate a pattern of doing the right thing, and people eventually are willing to extend a little bit of credulousness when you say something that sounds that might be a little bit beyond their experience.The other is, and this is very nebulous, and difficult to define so I don't think we even have this in writing, but you have to be able to convince me if you're going to advertise something in one of my shows, that it will not, when used as directed, leave the user worse off than they were when they started. And that is a very strange thing. Like, a security product that has a bunch of typos on its page and is rolling its own crypto, for example—if you want an easy example—is one of those things that I will very gracefully decline not to wind up engaging with, just because I have the sneaking suspicion that if you trust that thing, you might very well live to regret it. In other cases, though—and this is almost never a problem because most companies that you have heard of and have established themselves as brands in this space already instinctively get that you're not able to build a lasting business by lying to people and then ripping them off.So, it's a relatively straightforward approach, but every once in a while, I see something that makes me raise an eyebrow. And it's not always bad. Sometimes I think that's a little odd. Teleport is a good example of this because, “Oh, really? You wound up doing access and authentication? That sounds exactly like the kind of thing I want something old and boring, not new and exciting, around, so let's dig into this and figure out whether this might be the one company you work at that doesn't get to sponsor stuff that I do.”But of course you do. You're absolutely focusing on an area that is relevant, useful, and having talked to people on your side of the world, you're doing the right thing. And okay, I would absolutely not be opposed to deploying this in the right production environment. But having that credulousness, having that exploratory conversation, makes it clear that I'm talking to people who know what they're doing and not effectively shilling for the highest bidder, which is not really a position I ever want to find myself in.Anadelia: And look, you have only one opportunity to make a first impression, right? So, being clear about what it is that you can do, and also being clear about what it is that you cannot do is extremely important, right? It kind of goes back to the point of just be a good human, don't waste people's time. You want to provide something of value to your audience. And so, setting those expectations early on is extremely important.And I don't know anyone that does this, but if your goal is only to drive people to your website, you can do that, probably very easily, but nothing will come out of it unless you have the right message.Corey: Oh, all you do is write something incendiary and offensive, and you'll have a lot of traffic. They won't buy anything and they'll hate you, but you'll get traffic, so maybe you want to be a little bit more intentional. It's the same reason that the companies that advertise on what I do pick me to advertise with as opposed to other things. It is more expensive than the mass-market podcasts and whatnot that speak to everyone. But you take a look at those podcasts and the things that they're advertising are things that actually apply to an awful lot more people, things like mattresses, and click-and-design website services, and the baseline stuff that a lot of people would be interested in, whereas the things that advertise on what I do tend to look a lot more like B2B SaaS companies where they're talking to folks who spend a lot of time working in cloud computing.And one of the weird things to think about from that perspective, at least for me, is if one person is listening to a show that I'm putting out and they go through the journey and become a customer, well, at the size of some of these B2B contracts between large companies, that one customer has basically paid for everything I can sell for advertising for the next decade and change, just because the long-term value of some of these customers is enormous. But it's why, for example—and I kept expecting it to happen, but it didn't—I've never been subjected to outreach from the mattress companies of, “Hey, you want to go talk about that to your guests?” No, because for those folks, it is pure raw numbers: how many millions of subscribers do you have? Here, it's—the newsletter is the easy one to get numbers on because lies, damned lies, and podcast statistics. I have 31,000 people that receive emails. Great, that's not the biggest newsletter in the world by a longshot, but the people who are the type of person to sign up for cloud computing-style newsletters, that alone says something very specific about them and it doesn't require anyone do anything creepy to wind up reaching out from that perspective.It doesn't require spying on customers to intuit that, hmm, maybe people who care about what AWS is up to and have big AWS-sized problems might sign up to a newsletter called Last Week in AWS. That's the sort of easy thinking about advertising that I tend to go for, which yeah, admittedly sounds a lot like something out of that Mad Men era. But I think that we got a lot right back then, and everything's new all the time.Anadelia: [laugh]. And actually, that's exactly what demand generation is, right? We want to find the right channels to reach our audience. And so, for a consumer company that sells mattresses, right, anyone might be on the market for a mattress, right? You want to go as broad as possible. But for something that's more specific, you want to find what are the right channels to reach that audience where you know that there's—it might be a smaller audience size, but it's the right people.And we've talked about the other core areas of marketing. So, with demand generation, it's all about finding people where they are, right, and providing them their message to you and attracting them to come to you, right? It kind of goes back to that inbound and outbound motion that I mentioned earlier. But at the end of the day also, if you don't have the right messaging to keep them engaged, once you got them to your website, then that's a different problem, right? So, demand gen alone cannot be successful without really strong product marketing and without really strong content, and everything else that's needed to support that, right? I mentioned the—if your website is not loading fast enough, then you're losing people if your form is not working. So, there's so many, so many different factors that come into play.Corey: Oh, God, the forms. Don't get me started on the forms. Hey, we have a great report that's super useful. Okay, cool. I'll click the link and I'll follow that. I talk to sponsors about this all the time. And it's, you have 30 mandatory fields on that website that I need to fill out. I am never going to do that.What is the absolute bare minimum that you need in an ideal world? Don't put any sort of gateway in front of it and just make it that good that I will reach out to thank you for it or something, but just make it an email address or something and that's it. You don't need to know the size of my company, the industry we're in, the level of my signing authority, et cetera, et cetera, et cetera. Because if this is good, I might very well be in touch. And if it's not, all you're going to do is harass me forever with pointless calls and emails and whatnot, and I don't want to deal with that. There's something to be said for adding value early in the conversation and letting other people sometimes make the first move. But this is also, to be clear, a very inbound type of approach.Anadelia: It's a never-ending debate, to gate or not to gate. And I don't know if there is a right answer. My approach is that if your content is good, people will come back to you. They'll keep coming back, and they'll want to take the next step with you. And so, I have some gated assets, and I have some that are not, and—but—Corey: But your gates have also never been annoying of the type that I'm talking about where it's the, “Oh, great. You need to, like, put in, like, how big is your company? What's the budget?” It feels like I'm answering a survey at some point. AWS is notorious for this.I counted once; there are 19 mandatory fields I had to fill out in order to watch a webinar that AWS was putting on.Anadelia: [laugh].Corey: And the worst part is they asked me the same questions every time I want to watch a different webinar. It's like, for a company that says the data is so valuable, you'd really think they'd be better at managing it.Anadelia: You know, like, some of the questions keep getting stranger. Like, I would not be surprised if people start asking what's your favorite color, or what's the answer to your—Corey: The one they always ask now for, like, big data seminars and whatnot, is where this really gets me, is this in relation to your professional interests or your personal interests? It's… “What do you think my hobbies are over there? Oh, yeah, I like big enterprise software. That's my hobby.” “Okay, I guess.” But I really do wonder what happens if someone checks the personal interest [vibe 00:25:33]. Do they wind up just with various AWS employees showing up want to hang out on the weekends and go surfing or something? I don't know.Anadelia: As somebody who has been on the receiving end of lists like this—for example, we sponsor a conference and we get people stop by to talk to us, and now we get the list of those people. And there's 25 columns. Like, honestly, that data does not come in helpful because at the end of the day, whatever you've marked on the required question is not going to change how I am going to communicate to you after, right, because we just had a conversation in person at this event.Corey: My budget is not material to the reason I let you scan my badge. The reason I let you scan my badge because I really wanted one of those fun plastic toy things, so I waited in line for 45 minutes to get it. But that doesn't mean that I'm going to be a buyer; it just means that now I'm in your funnel, although I could not possibly care less about what you do. One thing I do at re:Invent and a couple other conferences, for example, is I will have swag at a booth—because I don't tend to get booths myself, I don't have the staff to man it and I'm bad at that type of thing. But when people come up to get a sticker for Last Week in AWS or when of our data transfer diagram things or whatnot, the rule that we've always put in place is, you're not going to mandate a badge scan for that.And the kind of company I like doing that with gets it because the people who walk by and are interested will say, “Hey, can you scan my badge as well?” But they don't want to pollute their own lead lists with a bunch of people who are only there to get a sticker featuring a sarcastic platypus, as opposed to getting them confused with people actually care about what it is that they're solving for. And that's a delicate balance to strike sometimes, but the nice thing about being me is I have customers who come back again and again and again. Although I will argue that I probably got better at being a service provider when I started also being a customer at the same time, where I hired out a marketing department here because it turns out that fixing the AWS bill is something that does a fair bit of marketing work. It's not something people talk about at large scale in public, so you have to be noisy enough so that inbound finds its path to you a bunch of times. That's always tricky.And learning about how no matter what it is you do, in the case of my consulting work, we are quite honestly selling money, bring us in for an engagement, you will turn a profit on that engagement and we don't come back with a whole bunch of extra add-ons after the fact to basically claw back more things. It's one of the easiest sales in the world. And it's still nuanced, and challenging, and finding the right way to talk about it to the right people at the right time explains why marketing is the industry that it is. It's hard. None of this is easy.Anadelia: It is. And you know, in your example, you're not scanning that badge, but giving the person the sticker, right? Like, it's all about making a good first impression, and if the person's not ready to talk to you, that is okay. But there are ways that you can stay top-of-mind so that the moment that they have a need, they'll come to you. It kind of goes back again to my earlier points of adding value in supporting existing communities, right? So, what are you doing to stay top-of-mind with that person that wasn't quite ready back then, but the moment they have a need, they'll think of you first because you made a good first impression.Corey: And that's really what it comes down to. It's nice to talk to people who actually work in marketing because a lot of what I do in the marketing space, I've got to be honest, is terrible. Because I've done the old engineering thing of, well, I'm no marketer, but I know how to write code, so how hard could marketing really be and I invent this theory of marketing from first principles, which not only is mostly wrong, but also has a way of being incredibly insulting to people who have actually made this their profession and excel at it. But it's an evolutionary process and trying to figure out the right way to do things and how to think about things from particular point of view has been transformative. Really easy example of this: when I first started selling sponsorships, I was constantly worried that a sponsor was going to reach out and say, “Well, hang on a second. We didn't get the number of clicks that we expected to on this campaign. What do you have to say about that?”Because I'm a consultant. I am used to clients not getting results that they expected having some harsh words for me. In practice, I don't believe I've ever had a deep conversation about that with a marketing person. I've talked to them and they've said, “Well, some of these things worked. Some of these things didn't. Here's what works; here's what didn't, and for our next round, here's what we want to try instead.” Those are the great constructive conversations.The ones that I was fearing somehow would assume that I held this iron grip of control over exactly how many people would be clicking on a thing in a newsletter, and I'm not. We barely provide click-tracking at this point in the aggregate, let alone anything more specific, just because it's so hard to actually tell and get value out of it. You talk as well, about there being brand awareness. Even if someone doesn't click an ad, they're potentially reading it, they're starting to associate your company with the problem space. That's one of those things that are effectively impossible to track, but it does pay dividends.When you suddenly have a problem in a particular area. And there's one or two companies off the top of your mind that you know work in that space. Well, what do you think marketing is? There has been huge money put into making that association in your mind. It's not just about click the link; it's not just about buy the thing; it's about shaping the way that we think about different things.Anadelia: And I spend a lot of time thinking about how people think we talk about what are the things that motivate you. When you have a problem, where do you go to look for a solution, or who do you go to, right? So, just understanding what the thought process is when someone is trying to solve a problem or making a purchasing decision, I think that a lot of demand generation is what are the different ways by which someone is trying to solve a problem that they're having? And I had an interest in psychology growing up; both my parents are psychologists, and I think that marketing tends to bring some aspects of that in business and creativity, which is what led me to a career in marketing.And you ended up being sort of a connector, right? Like your job was to connect to people who would benefit from meeting each other. Just one of them happens to be a product, or you know, it depends on your company, right, but you're just introducing people and making sure they know about each other because there's going to be a mutually beneficial relationship between them.Corey: That seems to be what so many jobs ultimately distilled down to in the final analysis of things. I really want to thank you for being so generous with your time and talking about how you view the world slash industry in which we live. If people want to learn more about what you're up to and how you think about these things, where's the best place to find you?Anadelia: You can follow me on Twitter at @anadeliafadeev, or connect with me on LinkedIn.Corey: Oh, you're one of the LinkedIn peoples. I used to do that a bit, and then I just started getting deluged with all kinds of nonsense, and let me adjust my notification settings, and there are 600 of them. And no, no, no, no, no. And I basically have quit the field, by and large, on LinkedIn. But power to you for not having done that. Links to that will of course be in the [show notes 00:32:38]. Thank you so much for being so generous with your time.Anadelia: Thank you for having me. I appreciate it.Corey: Anadelia Fadeev, Senior Director of Demand Generation at Teleport. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry ranting comment about how we got it completely wrong and that marketing does not work on you in the least. And by the way, when you close out that ranting comment, tell me what kind of brand of shoes you're wearing today.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About JeffJeff Smith has been in the technology industry for over 20 years, oscillating between management and individual contributor. Jeff currently serves as the Director of Production Operations for Basis Technologies (formerly Centro), an advertising software company headquartered in Chicago, Illinois. Before that he served as the Manager of Site Reliability Engineering at Grubhub.Jeff is passionate about DevOps transformations in organizations large and small, with a particular interest in the psychological aspects of problems in companies. He lives in Chicago with his wife Stephanie and their two kids Ella and Xander.Jeff is also the author of Operations Anti-Patterns, DevOps Solutions with Manning publishing. (https://www.manning.com/books/operations-anti-patterns-devops-solutions) Links Referenced: Basis Technologies: https://basis.net/ Operations Anti-Patterns: https://attainabledevops.com/book Personal Site: https://attainabledevops.com LinkedIn: https://www.linkedin.com/in/jeffery-smith-devops/ Twitter: https://twitter.com/DarkAndNerdy Medium: https://medium.com/@jefferysmith duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by our friends at Fortinet. Fortinet's partnership with AWS is a better-together combination that ensures your workloads on AWS are protected by best-in-class security solutions powered by comprehensive threat intelligence and more than 20 years of cybersecurity experience. Integrations with key AWS services simplify security management, ensure full visibility across environments, and provide broad protection across your workloads and applications. Visit them at AWS re:Inforce to see the latest trends in cybersecurity on July 25-26 at the Boston Convention Center. Just go over to the Fortinet booth and tell them Corey Quinn sent you and watch for the flinch. My thanks again to my friends at Fortinet.Corey: Let's face it, on-call firefighting at 2am is stressful! So there's good news and there's bad news. The bad news is that you probably can't prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the fun things about doing this show for long enough is that you eventually get to catch up with people and follow up on previous conversations that you've had. Many years ago—which sounds like I'm being sarcastic, but is increasingly actually true—Jeff Smith was on the show talking about a book that was about to release. Well, time has passed and things have changed. And Jeff Smith is back once again. He's the Director of Product Operations at Basis Technologies, and the author of DevOps Anti-Patterns? Or what was the actual title of the book it was—Jeff: Operations Anti-Patterns.Corey: I got hung up in the anti-patterns part because it's amazing. I love the title.Jeff: Yeah, Operations Anti-Patterns, DevOps Solutions.Corey: Got you. Usually in my experience, alway been operations anti-patterns, and here I am to make them worse, probably by doing something like using DNS as a database or some godforsaken thing. But you were talking about the book aspirationally a few years ago, and now it's published and it has been sent out to the world. And it went well enough that they translated it to Japanese, I believe, and it has seen significant uptick. What was your experience of it? How did it go?Jeff: You know, it was a great experience. This is definitely the first book that I've written. And the Manning process was extremely smooth. You know, they sort of hold your hand through the entire process. But even after launch, just getting feedback from readers and hearing how it resonated with folks was extremely powerful.I was surprised to find out that they turned it into an audiobook as well. So, everyone reaches out and says, “Did you read the audiobook? I was going to buy it, but I wasn't sure.” I was like, “No, unfortunately, I don't read it.” But you know, still cool to have it out there.Corey: My theory has been for a while now that no one wants to actually write a book; they want to have written a book. Now that you're on the other side, how accurate is that? Are you in a position of, “Wow, sure glad that's done?” Or are you, “That was fun. Let's do it again because I like being sad all the time.” I mean, you do work Kubernetes for God's sake. I mean, there's a bit of masochism inherent to all of us in this space.Jeff: Yeah. Kubernetes makes me cry a little bit more than the writing process. But it's one of the things when you look back on it, you're like, “Wow, that was fun,” but not in the heat of the moment, right? So, I totally agree with the sentiment that people want to have written a book but not actually gone through the process. And that's evident by the fact that how many people try to start a book on their own without a publisher behind them, and they end up writing it for 15 years. The process is pretty grueling. The feedback is intense at first, but you start to get into a groove and you—I could see, you know, in a little while wanting to write another book. So, I can see the appeal.Corey: And the last time you were on the show, I didn't really bother to go in a particular topical direction because, what's the point? It didn't really seem like it was a top-of-mind issue to really bring up because what's it matter; it's a small percentage of the workforce. Now I feel like talking about remote work is suddenly taking on a bit of a different sheen than it was before the dark times arrived. Where do you land on the broad spectrum of opinions around the idea of remote work, given that you have specialized in anti-patterns, and well, as sarcastic as I am, I tend to look at almost every place I've ever worked is expressing different anti-patterns from time to time. So, where do you land on the topic?Jeff: So, it's funny, I started as a staunch office supporter, right? I like being in the office. I like collaborating in person; I thought we were way more productive. Since the pandemic, all of us are forced into remote work, I've hired almost half of my team now as remote. And I am somewhat of a convert, but I'm not on the bandwagon of remote work is just as good or is better as in person work.I've firmly landed in the camp of remote work is good. It's got its shortcomings, but it's worth the trade off. And I think acknowledging what those trade-offs are important to keeping the team afloat. We just recently had a conversation with the team where we were discussing, like, you know, there's definitely been a drop in productivity over the past six months to a year. And in that conversation, a lot of the things that came up were things that are different remote that were better in person, right, Slack etiquette—which is something, you know, I could talk a little bit about as well—but, you know, Slack etiquette in terms of getting feedback quickly, just the sort of camaraderie and the lack of building that camaraderie with new team members as they come on board and not having those rituals to replace the in-person rituals. But through all that, oddly enough, no one suggested going back into the office. [laugh].Corey: For some strange reason, yeah. I need to be careful what I say here, I want to disclaim the position that I'm in. There is a power imbalance and nothing I say is going to be able to necessarily address that because I own the company and if my team members are listening to this, they're going to read a lot into what I say that I might not necessarily intend. But The Duckbill Group, since its founding, has been a fully distributed company. My business partner lives in a different state than I do so there's never been the crappy version of remote, which is, well, we're all going to be in the same city, except for Theodore. Theodore is going to be timezones away and then wonder why he doesn't get to participate in some of the conversations where the real decisions get made.Like that's crappy. I don't like that striated approach to things. We don't have many people who are co-located in any real sense, nor have we for the majority of the company's life. But there are times when I am able to work on a project in a room with one of my colleagues, and things go a lot more smoothly. As much as we want to pretend that video is the same, it quite simply isn't.It is a somewhat poor substitute for the very high bandwidth of a face-to-face interaction. And yes, I understand this is also a somewhat neurotypical perspective, let's be clear with that as well, and it's not for everyone. But I think that for the base case, a lot of the remote work advocates are not being fully, I guess, honest with themselves about some of the shortcomings remote has. That is where I've mostly landed on this. Does that generally land with where you are?Jeff: Yeah, that's exactly where I'm at. I completely agree. And when we take work out of the equation, I think the shortcomings lay themselves bare, right? Like I was having a conversation with a friend and we were like, well, if you had a major breakup, right, I would never be like, “Oh, man. Grab a beer and hop on Zoom,” right? [laugh]. “Let's talk it out.”No, you're like, hey, let's get in person and let's talk, right? We can do all of that conversation over Zoom, but the magic of being in person and having that personal connection, you know, can't be replaced. So, you know, if it's not going to work, commiserating over beers, right? I can't imagine it's going to work, diagramming some complex workflows and trying to come to an answer or a solution on that. So again, not to say that, you know, remote work is not valuable, it's just different.And I think organizations are really going to have to figure out, like, okay, if I want to entice people back into the office, what are the things that I need to do to make this realistic? We've opened the floodgates on remote hiring, right, so now it's like, okay, everyone's janky office setup needs to get fixed, right? So, I can't have a scenario where it's like, “Oh, just point your laptop at the whiteboard, right?” [laugh]. Like that can't exist, we have to have office spaces that are first-class citizens for our remote counterparts as well.Corey: Right because otherwise, the alternative is, “Great, I expect you to take the home that you pay for and turn it into an area fit for office use. Of course, we're not going to compensate you for that, despite the fact that, let's be realistic, rent is often larger than the AWS bill.” Which I know, gasp, I'm as shocked as anyone affected by that, but it's true. “But oh, you want to work from home? Great. That just means you can work more hours.”I am not of the school of thought where I consider time in the office to be an indicator of anything meaningful. I care if the work gets done and at small-scale, this works. Let me also be clear, we're an 11-person company. A lot of what I'm talking about simply will not scale to companies that are orders of magnitude larger than this. And from where I sit, that's okay. It doesn't need to.Jeff: Right. And I think a lot of the things that you talk about will scale, right? Because in most scenarios, you're not scaling it organizationally so much as you are with a handful of teams, right? Because when I think about all the different teams I interact with, I never really interact with the organization as a whole, I interact with my little neighborhood in the organization. So, it is definitely something that scales.But again, when it comes to companies, like, enticing people back into the office, now that I'm talking about working from home five days a week, I've invested in my home setup. I've got the monitor I want, I've got the chair that I want, I've got the mouse and keyboard that I want. So, you're going to bring me back to the office so I can have some standard Dell keyboard and mouse with some janky, you know—maybe—21-inch monitor or something like that, right? Like, you really have to decide, like, okay, we're going to make the office a destination, we're going to make it where people want to go there where it's not just even about the collaboration aspect, but people can still work and be effective.And on top of that, I think how we look at what the office delivers is going to change, right? Because now when I go to the office now, I do very little work. It's connections, right? It's like, you know, “Oh, I haven't seen you in forever. Let's catch up.” And a lot of that stuff is valuable. You know, there's these hallway conversations that exist that just weren't happening previously because how do I accidentally bump into you on Slack? [laugh]. Right, it has to be much more it of a—Corey: Right. It takes some contrivance to wind up making that happen. I remember back in the days of working in offices, I remember here in San Francisco where we had unlimited sick time and unlimited PTO, I would often fake a sick day, but just stay home and get work done. Because I knew if I was in the office, I'd be constantly subjected to drive-bys the entire time of just drive-by requests, people stopping by to ask, “Oh, can you just help me with this one thing,” that completely derails my train of thought. Then at the end of the day, they'd tell me, “You seem distractible and you didn't get a lot of work done.”It's, “Well, no kidding. Of course not. Are you surprised?” And one of the nice things about starting your own company—because there are a lot of downsides, let me be very clear—one of the nice things is you get to decide how you want to work. And that was a study in, first, amazement, and then frustration.It was, “All right, I just landed a big customer. I'm off to the races and going to take this seriously for a good six to twelve months. Great sky's the limit, I'm going to do up my home office.” And then you see how little money it takes to have a nice chair, a good standing desk, a monitor that makes sense and you remember fighting tooth-and-nail for nothing that even approached this quality at companies and they acted like it was going to cost them 20-grand. And here, it's two grand at most, when I decorated this place the first time.And it was… “What the hell?” Like, it feels like the scales fall away from your eyes, and you start seeing things that you didn't realize were a thing. Now I worry that five years in, there's no way in the world I'm ever fit to be an employee again, so this is probably the last job I'll ever have. Just because I've basically made myself completely unemployable across six different axes.Jeff: [laugh]. And I think one of the things when it comes to, like, furniture, keyboard, stuff like that, I feel like part of it was just, like, this sort of enforced conformity, right, that the office provided us the ability to do. We can make sure everyone's got the same monitor, the same keyboard that way, when it breaks, we can replace it easily. In a lot of organizations that I've been in, you know, that sort of like, you know, even if it was the same amount or ordering a custom keyboard was a big exception process, right? Like, “Oh, we've got to do a whole thing.” And it's just like, “Well, it doesn't have to be that complicated.”And like you said, it doesn't cost much to allow someone to get the tools that they want and prefer and they're going to be more productive with. But to your point really quickly about work in the office, until the pandemic, I personally didn't recognize how difficult it actually was to get work done in the office. I don't think I appreciated it. And now that I'm remote, I'm like, wow, it is so much easier for me to close this door, put my headphones on, mute Slack and go heads down. You know, the only drive-by I've got is my wife wondering if I want to go for a walk, and that's usually a text message that I can ignore and come back to later.Corey: The thing that just continues to be strange for me and breaks in some of the weirdest ways has just been the growing awareness of how much of office life is unnecessary and ridiculous. When you're in the office every day, you have to find a way to make it work and be productive and you have this passive-aggressive story of this open office, it's for collaboration purposes. Yeah, I can definitively say that is not true. I had a boss who once told me that there was such benefits to working in an open plan office that if magically it were less expensive to give people individual offices, he would spare the extra expense for open plan. That was the day I learned he would lie to me while looking me in the eye. Because of course you wouldn't.And it's for collaboration. Yeah, it means two loud people—often me—are collaborating and everyone else wears noise-canceling headphones trying desperately to get work done, coming in early, hours before everyone else to get things done before people show up and distracted me. What the hell kind of day-to-day work environment is that?Jeff: What's interesting about that, though, is those same distractions are the things that get cited as being missed from the perspective of the person doing the distracting. So, everyone universally hates that sort of drive-by distractions, but everyone sort of universally misses the ability to say like, “Hey, can I just pull on your ear for a second and get your feedback on this?” Or, “Can we just walk through this really quickly?” That's the thing that people miss, and I don't think that they ever connect it to the idea that if you're not the interruptee, you're the interruptor, [laugh] and what that might do to someone else's productivity. So, you would think something like Slack would help with that, but in reality, what ends up happening is if you don't have proper Slack etiquette, there's a lot of signals that go out that get misconstrued, misinterpreted, internalized, and then it ends up impacting morale.Corey: And that's the most painful part of a lot of that too. Is that yeah, I want to go ahead and spend some time doing some nonsense—as one does; imagine that—and I know that if I'm going to go into an office or meet up with my colleagues, okay, that afternoon or that day, yeah, I'm planning that I'm probably not going to get a whole lot of deep coding done. Okay, great. But when that becomes 40 hours a week, well, that's a challenge. I feel like being full remote doesn't work out, but also being in the office 40 hours a week also feels a little sadistic, more than almost anything else.I don't know what the future looks like and I am privileged enough that I don't have to because we have been full remote the entire time. But what we don't spend on office space we spend on plane tickets back and forth so people can have meetings. In the before times, we were very good about that. Now it's, we're hesitant to do it just because it's we don't want people traveling before the feel that it's safe to do so. We've also learned, for example, when dealing with our clients, that we can get an awful lot done without being on site with them and be extraordinarily effective.It was always weird have traveled to some faraway city to meet with the client, and then you're on a Zoom call from their office with the rest of the team. It's… I could have done this from my living room.Jeff: Yeah. I find those sorts of hybrid meetings are often worse than if we were all just remote, right? It's just so much easier because now it's like, all right, three of us are going to crowd around one person's laptop, and then all of the things that we want to do to take advantage of being in person are excluding the people that are remote, so you got to do this careful dance. The way we've been sort of tackling it so far—and we're still experimenting—is we're not requiring anyone to come back into the office, but some people find it useful to go to the office as a change of scenery, to sort of, like break things up from their typical routine, and they like the break and the change. But it's something that they do sort of ad hoc.So, we've got a small group that meets, like, every Thursday, just as a day to sort of go into the office and switch things up. I think the idea of saying everyone has to come into the office two or three days a week is probably broken when there's no purpose behind it. So, my wife technically should go into the office twice a week, but her entire team is in Europe. [laugh]. So, what point does that make other than I am a body in a chair? So, I think companies are going to have to get flexible with this sort of hybrid environment.But then it makes you wonder, like, is it worth the office space and how many people are actually taking advantage of it when it's not mandated? We find that our office time centers around some event, right? And that event might be someone in town that's typically remote. That might be a particular project that we're working on where we want to get ideas and collaborate and have a workshop. But the idea of just, like, you know, we're going to systematically require people to be in the office x many days, I don't see that in our future.Corey: No, and I hope you're right. But it also feels like a lot of folks are also doing some weird things around the idea of remote such as, “Oh, we're full remote but we're going to pay you based upon where you happen to be sitting geographically.” And we find that the way that we've done this—and again, I'm not saying there's a right answer for everyone—but we wind up paying what the value of the work is for us. In many cases, that means that we would be hard-pressed to hire someone in the Bay Area, for example. On the other hand, it means that when we hire people who are in places with relatively low cost of living, they feel like they've just hit the lottery, on some level.And yeah, some of them, I guess it does sort of cause a weird imbalance if you're a large Amazon-scale company where you want to start not disrupting local economies. We're not hiring that many people, I promise. So, there's this idea of figuring out how that works out. And then where does the headquarters live? And well, what state laws do we wind up following on what we're doing? Just seems odd.Jeff: Yeah. So, you know, one thing I wanted to comment on that you'd mentioned earlier, too, was the weird things that people are doing, and organizations are doing with this, sort of, remote work thing, especially the geographic base pay. And you know, a lot of it is, how can we manipulate the situation to better us in a way that sounds good on paper, right? So, it sounds perfectly reasonable. Like, oh, you live in New York, I'm going to pay you in New York rates, right?But, like, you live in Des Moines, so I'm going to pay you Des Moines rates. And on the surface, when you just go you're like, oh, yeah, that makes sense, but then you think about it, you're like, “Wait, why does that matter?” Right? And then, like, how do I, as a manager, you know, level that across my employees, right? It's like, “Oh, so and so is getting paid 30 grand less. Oh, but they live in a cheaper area, right?” I don't know what your personal situation is, and how much that actually resonates or matters.Corey: Does the value that they provide to your company materially change based upon where they happen to be sitting that week?Jeff: Right, exactly. But it's a good story that you can tell, it sounds fair at first examination. But then when you start to scratch the surface, you're like, “Wait a second, this is BS.” So, that's one thing.Corey: It's like tipping on some level. If you can't afford the tip, you can't afford to eat out. Same story here. If you can't afford to compensate people the value that they're worth, you can't afford to employ people. And figure that out before you wind up disappointing people and possibly becoming today's Twitter main character.Jeff: Right. And then the state law thing is interesting. You know, when you see states like California adopting laws similar to, like, GDPR. And it's like, do you have to start planning for the most stringent possibility across every hire just to be safe and to avoid having to have this sort of patchwork of rules and policies based on where someone lives? You might say like, “Okay, Delaware has the most stringent employer law, so we're going to apply Delaware's laws across the board.” So, it'll be interesting to see how that sort of plays out in the long run. Luckily, that's not a problem I have to solve, but it'll be interesting to see how it shakes out.Corey: It is something we had to solve. We have an HR consultancy that helps out with a lot of these things, but the short answer is that we make sure that we obey with local laws, but the way that we operate is as if everyone were a San Francisco employee because that is—so far—the locale that, one, I live here, but also of every jurisdiction we've looked at in the United States, it tends to have the most advantageous to the employee restrictions and requirements. Like one thing we do is kind of ridiculous—and we have to do for me and one other person, but almost no one else, but we do it for everyone—is we have to provide stipends every month for electricity, for cellphone usage, for internet. They have to be broken out for each one of those categories, so we do 20 bucks a month for each of those. It adds up to 100 bucks, as I recall, and we call it good. And employees say, “Okay. Do we just send you receipts? Please don't.”I don't want to look at your cell phone bill. It's not my business. I don't want to know. We're doing this to comply with the law. I mean, if it were up to me, it would be this is ridiculous. Can we just give everyone $100 a month raise and call it good? Nope. The forms must be obeyed. So, all right.We do the same thing with PTO accrual. If you've acquired time off and you leave the company, we pay it out. Not every state requires that. But paying for cell phone access and internet access as well, is something Amazon is currently facing a class action about because they didn't do that for a number of their California employees. And even talking to Amazonians, like, “Well, they did, but you had to jump through a bunch of hoops.”We have the apparatus administratively to handle that in a way that employees don't. Why on earth would we make them do it unless we didn't want to pay them? Oh, I think I figured out this sneaky, sneaky plan. I'm not here to build a business by exploiting people. If that's the only way to succeed, and the business doesn't deserve to exist. That's my hot take of the day on that topic.Jeff: No, I totally agree. And what's interesting is these insidious costs that sneak up that employees tend to discount, like, one thing I always talk about with my team is all that time you're thinking about a problem at work, right, like when you're in the shower, when you're at dinner, when you're talking it over with your spouse, right? That's work. That's work. And it's work that you're doing on your time.But we don't account for it that way because we're not typing; we're not writing code. But, like, think about how much more effective as people, as employees, we would be if we had time dedicated to just sit and think, right? If I could just sit and think about a problem without needing to type but just critically think about it. But then it's like, well, what does that look like in the office, right? If I'm just sitting there in my chair like this, it doesn't look like I'm doing anything.But that's so important to be able to, like, break down and digest some of the complex problems that we're dealing with. And we just sort of write it off, right? So, I'm like, you know, you got to think about how that bleeds into your personal time and take that into account. So yeah, maybe you leave three hours early today, but I guarantee you, you're going to spend three hours throughout the week thinking about work. It's the same thing with these cellphone costs that you're talking about, right? “Oh, I've got a cell phone anyways; I've got internet anyways.” But still, that's something that you're contributing to the business that they're not on the hook for, so it seems fair that you get compensated for that.Corey: I just think about that stuff all the time from that perspective, and now that I you know, own the place, it's one of those which pocket of mine does it come out of? But I hold myself to a far higher standard about that stuff than I do the staff, where it's, for example, I could theoretically justify paying my internet bill here because we have business-class internet and an insane WiFi system because of all of the ridiculous video production I do. Now. It's like, like, if anyone else on the team was doing this, yes, I will insist we pay it, but for me, it just it feels a little close to the edge. So, it's one of those areas where I'm very conservative around things like that.The thing that also continues to just vex me, on some level, is this idea that time in a seat is somehow considered work. I'll never forget one of the last jobs I had before I started this place. My boss walked past me and saw that I was on Reddit. And, “Is that really the best use of your time right now?” May I use the bathroom when I'm done with this, sir?Yeah, of course it is. It sounds ridiculous, but one of the most valuable things I can do for The Duckbill Group now is go on the internet and start shit posting on Twitter, which sounds ridiculous, but it's also true. There's a brand awareness story there, on some level. And that's just wild to me. It's weird, we start treating people like adults, they start behaving that way. And if you start micromanaging them, they live up or down to the expectations you tend to hold. I'm a big believer in if I have to micromanage someone, I should just do the job myself.Jeff: Yeah. The Reddit story makes me think of, like, how few organizations have systematic ways of getting vital information. So, the first thing I think about is, like, security and security vulnerabilities, right? So, how does Basis Technologies, as an organization, know about these things? Right now, it's like, well, my team knows because we're plugged into Reddit and Twitter, right, but if we were gone Basis, right, may not necessarily get that information.So, that's something we're trying to correct, but it just sort of highlights the importance of freedom for these employees, right? Because yeah, I'm on Reddit, but I'm on /r/sysadmin. I'm on /r/AWS, right, I'm on /r/Atlassian. Now I'm finding out about this zero-day vulnerability and it's like, “Oh, guys, we got to act. I just heard about this thing.” And people are like, “Oh, where did this come from?” And it's like it came from my network, right? And my network—Corey: Mm-hm.Jeff: Is on Twitter, LinkedIn, Reddit. So, the idea that someone browsing the internet on any site, really, is somehow not a productive use of their time, you better be ready to itemize exactly what that means and what that looks like. “Oh, you can do this on Reddit but you can't do that on Reddit.”Corey: I have no boss now, I have no oversight, but somehow I still show up with a work ethic and get things done.Jeff: Right. [laugh].Corey: Wow, I guess I didn't need someone over my shoulder the whole time. Who knew?Jeff: Right. That's all that matters, right? And if you do it in 30 hours or 40 hours, that doesn't really matter to me, you know? You want to do it at night because you're more productive there, right, like, let's figure out a way to make that happen. And remote work is actually empowering us ways to really retain people that wasn't possible before I had an employee that was like, you know, I really want to travel. I'm like, “Dude, go to Europe. Work from Europe. Just do it. Work from Europe,” right? We've got senior leaders on the C-suite that are doing it. One of the chief—Corey: I'm told they have the internet, even there. Imagine that?Jeff: Yeah. [laugh]. So, our chief program officer, she was in Greece for four weeks. And it worked. It worked great. They had a process. You know, she would spent one week on and then one week off on vacation. But you know, she was able to have this incredible, long experience, and still deliver. And it's like, you know, we can use that as a model to say, like—Corey: And somehow the work got done. Wow, she must be amazing. No, that's the baseline expectation that people can be self-managing in that respect.Jeff: Right.Corey: They aren't toddlers.Jeff: So, if she can do that, I'm sure you can figure out how to code in China or wherever you want to visit. So, it's a great way to stay ahead of some of these companies that have a bit more lethargic policies around that stuff, where it's like, you know, all right, I'm not getting that insane salary, but guess what, I'm going to spend three weeks in New Zealand hanging out and not using any time off or anything like that, and you know, being able to enjoy life. I wish this pandemic had happened pre-kids because—Corey: Yeah. [laugh].Jeff: —you know, we would really take advantage of this.Corey: You and me both. It would have very different experience.Jeff: Yeah. [laugh]. Absolutely, right? But with kids in school, and all that stuff, we've been tethered down. But man, I you know, I want to encourage the young people or the single people on my team to just, like, hey, really, really embrace this time and take advantage of it.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it's still somebody's problem. And a big part of that responsibility is app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you're actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That's S-N-Y-K.co/screamCorey: One last topic I want to get into before we call it an episode is, I admit, I read an awful lot of books, it's a guilty pleasure. And it's easy to fall into the trap, especially when you know the author, of assuming that snapshot of their state of mind at a very fixed point in time is somehow who they are, like a fly frozen in amber, and it's never true. So, my question for you is, quite simply, what have you learned since your book came out?Jeff: Oh, man, great question. So, when I was writing the book, I was really nervous about if my audience was as big as I thought it was, the people that I was targeting with the book.Corey: Okay, that keeps me up at night, too. I have no argument there.Jeff: Yeah. You know what I mean?Corey: Please, continue.Jeff: I'm surrounded, you know, by—Corey: Is anyone actually listening to this? Yeah.Jeff: Right. [laugh]. So, after the book got finished and it got published, I would get tons of feedback from people that so thoroughly enjoyed the book, they would say things like, you know, “It feels like you were in our office like a fly on the wall.” And that was exciting, one, because I felt like these were experiences that sort of resonated, but, two, it sort of proved this thesis that sometimes you don't have to do something revolutionary to be a positive contribution to other people, right? So, like, when I lay out the tips and things that I do in the book, it's nothing earth-shattering that I expect Google to adopt. Like, oh, my God, this is the most unique view ever.But being able to talk to an audience in a way that resonates with them, that connects with them, that shows that I understand their problem and have been there, it was really humbling and enlightening to just see that there are people out there that they're not on the bleeding edge, but they just need someone to talk to them in a language that they understand and resonate with. So, I think the biggest thing that I learned was this idea that your voice is important, your voice matters, and how you tell your story may be the difference between someone understanding a concept and someone not understanding a concept. So, there's always an audience for you out there as you're writing, whether it be your blog post, the videos that you produce, the podcasts that you make, somewhere there's someone that needs to hear what you have to say, and the unique way that you can say it. So, that was extremely powerful.Corey: Part of the challenge that I found is when I start talking to other people, back in the before times, trying to push them into conference talks and these days, write blog posts, the biggest objection I get sometimes is, “Well, I don't have anything worth saying.” That is provably not true. One of my favorite parts about writing Last Week in AWS is as I troll the internet looking for topics about AWS that I find interesting, I keep coming across people who are very involved in one area or another of this ecosystem and have stories they want to tell. And I love, “Hey, would you like to write a guest post for Last Week in AWS?” It's always invite only and every single one of them has been paid because people die of exposure and I'm not about that exploitation lifestyle.A couple have said, “Oh, I can't accept payment for a variety of reasons.” Great. Pick a charity that you would like it to go to instead because we do not accept volunteer work, we are a for-profit entity. That is the way it works here. And that has been just one of the absolute favorite parts about what I do just because you get to sort of discover new voices.And what I find really neat is that for a lot of these folks, this is their start to writing and telling the story, but they don't stop there, they start telling their story in other areas, too. It leads to interesting career opportunities for them, it leads to interesting exposure that they wouldn't have necessarily had—again, not that they're getting paid in exposure, but the fact that they are able to be exposed to different methodologies, different ways of thinking—I love that. It's one of my favorite parts about doing what I do. And it seems to scale a hell of a lot better than me sitting down with someone for two hours to help them build a CFP that they wind up not getting accepted or whatnot.Jeff: Right. It's a great opportunity that you provide folks, too, because of, like, an instant audience, I think that's one of the things that has made Medium so successful as, like, a blogging platform is, you know, everyone wants to go out and build their own WordPress site and launch it, but then it like, you write your blog post and it's crickets. So, the ability for you to, you know, use your platform to also expose those voices is great and extremely powerful. But you're right, once they do it, it lights a fire in a way that is admirable to watch. I have a person that I'm mentoring and that was my biggest piece of advice I can give. It was like, you know, write. Just write.It's the one thing that you can do without anyone else. And you can reinforce your own knowledge of a thing. If you just say, you know, I'm going to teach this thing that I just learned, just the writing process helps you solidify, like, okay, I know this stuff. I'm demonstrating that I know it and then four years from now, when you're applying for a job, someone's like, “Oh, I found your blog post and I see that you actually do know how to set up a Kubernetes cluster,” or whatever. It's just extremely great and it—Corey: It's always fun. You're googling for how to do something and you find something you wrote five years ago.Jeff: Right, yeah. [laugh]. And it's like code where you're like, “Oh, man, I would do that so much differently now.”Corey: Since we last spoke, one of the things I've been doing is I have been on the hook to write between a one to two-thousand-word blog post every week, and I've done that like clockwork, for about a year-and-a-half now. And I was no slouch at storytelling before I started doing that. I've given a few hundred conference talks in the before times. And I do obviously long Twitter threads in the past and I write reports a lot. But forcing me to go through that process every week and then sit with an editor and go ahead and get it improved, has made me a far better writer, it's made me a better storyteller, I am far better at articulating my point of view.It is absolutely just unlocking a host of benefits that I would have thought I was, oh, I passed all this. I'm already good at these things. And I was, but I'm better now. I think that writing is one of those things that people need to do a lot more of.Jeff: Absolutely. And it's funny that you mentioned that because I just recently, back in April, started to do the same thing I said, I'm going to write a blog post every week, right? I'm going to get three or four in the can, so that if life comes up and I miss a beat, right, I'm not actually missing the production schedule, so I have a steady—and you're right. Even after writing a book, I'm still learning stuff through the writing process, articulating my point of view.It's just something that carries over, and it carries over into the workforce, too. Like, if you've ever read a bad piece of documentation, right, that comes from—Corey: No.Jeff: Right? [laugh]. That comes from an inability to write. Like, you know, you end up asking these questions like who's the audience for this? What is ‘it' in this sentence? [laugh].Corey: Part of it too, is that people writing these things are so close to the problem themselves that the fact that, “Well, I'm not an expert in this.” That's why you should write about it. Talk about your experience. You're afraid everyone's going to say, “Oh, you're a fool. You didn't understand how this works.”Yeah, my lived experiences instead—and admittedly, I have the winds of privilege of my back on this—but it's also yeah, I didn't understand that either. It turns out that you're never the only person who has trouble with a concept. And by calling it out, you're normalizing it and doing a tremendous service for others in your shoes.Jeff: Especially when you're not an expert because I wrote some documentation about the SSL process and it didn't occur to me that these people don't use the AWS command line, right? Like, you know, in our organization, we sort of mask that from them through a bunch of in-house automation. Now we're starting to expose it to them and simple things like oh, you need to preface the AWS command with a profile name. So, then when we're going through the setup, we're like, “Oh. What if they already have an existing profile, right?” Like, we don't want to clobber that.SSo, it just changed the way you write the documentation. But like, that's not something that initially came to mind for me. It wasn't until someone went through the docs, and they're like, “Uh, this is blowing up in a weird way.” And I was like, “Oh, right. You know, like, I need to also teach you about profile management.”Corey: Also, everyone has a slightly different workflow for the way they interact with AWS accounts, and their shell prompts, and the way they set up local dev environments.Jeff: Yeah, absolutely. So, not being an expert on a thing is key because you're coming to it with virgin eyes, right, and you're able to look at it from a fresh perspective.Corey: So, much documentation out there is always coming from the perspective of someone who is intimately familiar with the problem space. Some of the more interesting episodes that I have, from a challenge perspective, are people who are deep technologists in a particular area and they love they fallen in love with the thing that they are building. Great. Can you explain it to the rest of us mere mortals so that we can actually we can share your excitement on this? And it's very hard to get them to come down to a level where it's coherent to folks who haven't spent years thinking deeply about that particular problem space.Jeff: Man, the number one culprit for that is, like, the AWS blogs where they have, like, a how-to article. You follow that thing and you're like, “None of this is working.” [laugh]. Right? And then you realize, oh, they made an assumption that I knew this, but I didn't right?So, it's like, you know, I didn't realize this was supposed to be, like, a handwritten JSON document just jammed into the value field. Because I didn't know that, I'm not pulling those values out as JSON. I'm expecting that just to be, like, a straight string value. And that has happened more and more times on the AWS blog than I can count. [laugh].Corey: Oh, yeah, very often. And then there's other problems, too. “Oh, yeah. Set up your IAM permissions properly.” That's left as an exercise for the reader. And then you wonder why everything's full of stars. Okay.Jeff: Right. Yep, exactly, exactly.Corey: Ugh. It's so great to catch up with you and see what you've been working on. If people want to learn more, where's the best place to find you?Jeff: So, the best place is probably my website, attainabledevops.com. That's a place where you can find me on all the other places. I don't really update that site much, but you can find me on LinkedIn, Twitter, from that jumping off point, links to the book are there if anyone's interested in that. Perfect stocking stuffers. Mom would love it, grandma would love it, so definitely, definitely buy multiple copies of that.Corey: Yeah, it's going to be one of my two-year-old's learning to read books, it'd be great.Jeff: Yeah, it's perfect. You know, you just throw it in the crib and walk away, right? They're asleep at no time. Like I said, I've also been taking to, you know, blogging on Medium, so you can catch me there, the links will be there on Attainable DevOps as well.Corey: Excellent. And that link will of course, be in the show notes. Thank you so much for being so generous with your time. I really do appreciate it. And it's great to talk to you again.Jeff: It was great to catch up.Corey: Really was. Jeff Smith, Director of Product Operations at Basis Technologies. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice or smash the like and subscribe buttons on the YouTubes, whereas if you've hated this podcast, do the exact same thing—five-star review, smash the buttons—but also leave an angry, incoherent comment that you're then going to have edited and every week you're going to come back and write another incoherent comment that you get edited. And in the fullness of time, you'll get much better at writing angry, incoherent comments.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About BenjaminBenjamin Anderson is CTO, Cloud at EDB, where he is responsible for developing and driving strategy for the company's Postgres-based cloud offerings. Ben brings over ten years' experience building and running distributed database systems in the cloud for multiple startups and large enterprises. Prior to EDB, he served as chief architect of IBM's Cloud Databases organization, built an SRE practice at database startup Cloudant, and founded a Y Combinator-funded hardware startup.Links Referenced: EDB: https://www.enterprisedb.com/ BigAnimal: biganimal.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it's still somebody's problem. And a big part of that responsibility is app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you're actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That's S-N-Y-K.co/screamCorey: This episode is sponsored by our friends at Fortinet. Fortinet's partnership with AWS is a better-together combination that ensures your workloads on AWS are protected by best-in-class security solutions powered by comprehensive threat intelligence and more than 20 years of cybersecurity experience. Integrations with key AWS services simplify security management, ensure full visibility across environments, and provide broad protection across your workloads and applications. Visit them at AWS re:Inforce to see the latest trends in cybersecurity on July 25-26 at the Boston Convention Center. Just go over to the Fortinet booth and tell them Corey Quinn sent you and watch for the flinch. My thanks again to my friends at Fortinet.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted guest episode is brought to us by our friends at EDB. And not only do they bring us this promoted episode, they bring me their CTO for Cloud, Benjamin Anderson. Benjamin, thank you so much for agreeing to suffer the slings and arrows that I will no doubt throw at you in a professional context, because EDB is a database company, and I suck at those things.Benjamin: [laugh]. Thanks, Corey. Nice to be here.Corey: Of course. So, databases are an interesting and varied space. I think we can all agree—or agree to disagree—that the best database is, of course, Route 53, when you misuse TXT records as a database. Everything else is generally vying for number two. EDB was—back in the days that I was your customer—was EnterpriseDB, now rebranded as EDB, which is way faster to say, and I approve of that.But you were always the escalation point of last resort. When you're stuck with a really weird and interesting Postgres problem, EDB was where you went because if you folks couldn't solve the problem, it was likely not going to get solved. I always contextualized you folks as a consulting shop. That's not really what you do. You are the CTO for Cloud.And, ah, interesting. Do databases behave differently in cloud environments? Well, they do when you host them as a managed service, which is an area you folks have somewhat recently branched into. How'd you get there?Benjamin: Ah, that's interesting. So, there's a bunch of stuff to unpack there. I think EDB has been around for a long time. It's something like 13, 14, 15 years, something like that, and really it's just been kind of slowly growing, right? We did start very much as a product company. We built some technology to help customers get from Oracle database on to Postgres, way back in 2007, 2008.That business has just slowly been growing. It's been going quite well. Frankly, I only joined about 18 months ago, and it's really cool tech, right? We natively understand some things that Oracle is doing. Customers don't have to change their schemas to migrate from Oracle to Postgres. There's some cool technology in there.But as you point out, I think a lot of our position in the market has not been that product focused. There's been a lot of people seeing us as the Postgres experts, and as people who can solve Postgres problems, in general. We have, for a long time, employed a lot of really sharp Postgres people. We still employ a lot of really sharp Postgres people. That's very much, in a lot of ways, our bread and butter. That we're going to fix Postgres problems as they come up.Now, over the past few years, we've definitely tried to shift quite a bit into being more of a product company. We've brought on a bunch of people who've been doing more enterprise software product type development over the past few years, and really focusing ourselves more and more on building products and investing in ourselves as a product company. We're not a services company. We're not a consulting company. We do, I think, provide the best Postgres support in the market. But it's been a journey. The cloud has been a significant part of that as well, right? You can't get away.Corey: Oh, yeah. These days, when someone's spinning up a new workload, it's unlikely—in most cases—they're going to wind up spinning up a new data center, if they don't already have one. Yes, there's still a whole bunch of on-prem workloads. But increasingly, the default has become cloud. Instead of, “Why cloud?” The question's become, “Why not?”Benjamin: Right, exactly. Then, as people are more and more accepting of managed services, you have to be a product company. You have to be building products in order to support your database customers because what they want his managed services. I was working in managed databases and service, something like, ten years ago, and it was like pulling teeth. This is after RDS launched. This was still pulling teeth trying to get people to think about, oh, I'm going to let you run my database. Whereas, now obviously, it's just completely different. We have to build great products in order to succeed in the database business, in general.Corey: One thing that jumped out at me when you first announced this was the URL is enterprisedb.com. That doesn't exactly speak to, you know, non-large companies, and EDB is what you do. You have a very corporate logo, but your managed service is called BigAnimal, which I absolutely love. It actually expresses a sense of whimsy and personality that I can no doubt guess that a whole bunch of people argued against, but BigAnimal, it is. It won through. I love that. Was that as contentious as I'm painting it to be, or people actually have a sense of humor sometimes?Benjamin: [laugh]. Both, it was extremely contentious. I, frankly, was one of the people who was not in favor of it at first. I was in favor of something that was whimsical, but maybe not quite that whimsical.Corey: Well, I call it Postgres-squeal, so let's be very clear here that we're probably not going to see eye-to-eye on most anything in pronunciation things. But we can set those differences aside and have a conversation.Benjamin: Absolutely, no consider that. It was deliberate, though, to try to step away a little bit from the blue-suit-and-tie, enterprise, DB-type branding. Obviously, a lot of our customers are big enterprises. We're good at that. We're not trying to be the hip, young startup targeting business in a lot of ways. We have a wide range of customers, but we want to branch out a little bit.Corey: One of the challenges right now is if I spin up an environment inside of AWS, as one does, and I decide I certainly don't want to take the traditional approach of running a database on top of an EC2 instance—the way that we did in the olden days—because RDS was crappy. Now that it's slightly less crappy, that becomes a not ideal path. I start looking at their managed database offerings, and there are something like 15 distinct managed databases that they offer, and they never turn anything off. And they continue to launch things into the far future. And it really feels, on some level, like 20 years from now—what we call a DBA today—their primary role is going to look a lot more like helping a company figure out which of Amazon's 40 managed databases is the appropriate fit for this given workload. Yet, when I look around at what the industry has done, it seems that when we're talking about relational databases. Postgres has emerged back when I was, more or less, abusing servers in person in my data center days, it was always MySQL. These days, Postgres is the de facto standard, full stop. I admit that I was mostly keeping my aura away from any data that was irreplaceable at that time. What happened? What did I miss?Benjamin: It's a really good question. And I certainly am not a hundred percent on all the trends that went on there. I know there's a lot of folks that are not happy about the MySQL acquisition by Oracle. I think there's a lot of energy that was adopted by the NoSQL movement, as well. You have people who didn't really care about transactional semantics that were using MySQL because they needed a place to store their data. And then, things like MongoDB and that type of system comes along where it's significantly easier than MySQL, and that subset of the population just sort of drifts away from MySQL.Corey: And in turn, those NoSQL projects eventually turn into something where, okay, now we're trying to build a banking system on top of it, and it's, you know, I guess you can use a torque wrench as a hammer if you're really creative about it, but it seems like there's a better approach.Benjamin: Yeah, exactly. And those folks are coming back around to the relational databases, exactly. At the same time, the advancements in Postgres from the early eight series to today are significant, right? We shouldn't underestimate how much Postgres has really moved forward. It wasn't that long ago that replication was hardly a thing and Postgres, right? It's been a journey.Corey: One thing that your website talks about is that you accelerate your open-sourced database transformation. And this is a bit of a hobby horse I get on from time to time. I think that there are a lot of misunderstandings when people talk about this. You have the open-source purists—of which I shamefully admit I used to be one—saying that, “Oh, it's about the idea of purity and open and free as in software.” Great. Okay, awesome. But when I find that corporate customers are talking about when they say open-source database, they don't particularly care if they have access to the source code because they're not going to go in and patch a database engine, we hope. But what they do care about is regardless of where they are today—even if they're perfectly happy there—they don't want to wind up beholden to a commercial database provider, and/or they don't want to wind up beholden to the environment that is running within. There's a strategic Exodus that's available in theory, which on some level serves to make people feel better about not actually Exodus-ing, but it also means if they're doing a migration at some point, they don't also have to completely redo their entire data plan.Benjamin: Yeah, I think that's a really good point. I mean, I like to talk—there's a big rat's nest of questions and problems in here—but I generally like talk to about open APIs, talk about standards, talk about how much is going to have to change if you eliminate this vendor. We're definitely not open-source purists. Well, we employ a lot of open-source purists. I also used to be an open—Corey: Don't let them hear you say that, then. Fair enough. Fair enough.Benjamin: [laugh] we have proprietary software at EDB, as well. There's a kind of wide range of businesses that we participate in. Glad to hear you also mention this where-it's-hosted angle, as well. I think there's some degree to which people are—they figured out that having at least open APIs or an open-source-ish database is a good idea rather than being beholden to proprietary database. But then, immediately forget that when they're picking a cloud vendor, right? And realizing that putting their data in Cloud Vendor A versus Cloud Vendor B is also putting them in a similar difficult situation. They need to be really wary of when they're doing that. Now, obviously, I work at an independent software company, and I have some incentive to say this, but I do think it's true. And you know, there's meaningful data gravity risk.Corey: I assure you, I have no incentive. I don't care what cloud provider you're on. My guidance has been, for years, to—as a general rule—pick a provider, I care about which one, and go all in until there's a significant reason to switch. Trying to build an optionality, “Oh, everything we do should be fully portable at an instance notice.” Great. Unless you're actually doing it, you're more or less, giving up a whole bunch of shortcuts and feature velocity you could otherwise have, in the hopes of one day you'll do a thing, but all the assumptions you're surrounded by baked themselves in regardless. So, you're more or less just creating extra work for yourself for no defined benefit. This is not popular in some circles, where people try to sell something that requires someone to go multi-cloud, but here we are.Benjamin: No, I think you're right. I think people underestimate the degree to which the abstractions are just not very good, right, and the degree to which those cloud-specific details are going to leak in if you're going to try to get anything done, you end up in kind of a difficult place. What I see more frequently is situations where we have a big enterprise—not even big, even medium-sized companies where maybe they've done an acquisition or two, they've got business units that are trying to do things on their own. And they end up in two or three clouds, sort of by happenstance. It's not like they're trying to do replication live between two clouds, but they've got one business unit in AWS and one business unit and Azure, and somebody in the corporate—say enterprise architect or something like that—really would like to make things consistent between the two so they get a consistent security posture and things like that. So, there are situations where the multi-cloud is a reality at a certain level, but maybe not at a concrete technical level. But I think it's still really useful for a lot of customers.Corey: You position your cloud offering in two different ways. One of them is the idea of BigAnimal, and the other—well, it sort of harkens back to when I was in sixth grade going through the American public school system. They had a cop come in and talk to us and paint to this imaginary story of people trying to push drugs. “Hey, kid. You want to try some of this?” And I'm reading this and it says EDB, Postgres for Kubernetes. And I'm sent back there, where it's like, “Hey, kid. You want to run your stateful databases on top of Kubernetes?” And my default answer to that is good lord, no. What am I missing?Benjamin: That's a good question. Kubernetes has come a long way—I think is part of that.Corey: Oh, truly. I used to think of containers as a pure story for stateless things. And then, of course, I put state into them, and then, everything exploded everywhere because it turns out, I'm bad at computers. Great. And it has come a long way. I have been tracking a lot of that. But it still feels like the idea being that you'd want to have your database endpoints somewhere a lot less, I guess I'll call it fickle, if that makes sense.Benjamin: It's an interesting problem because we are seeing a lot of people who are interested in our Kubernetes-based products. It's actually based on—we recently open-sourced the core of it under a project called cloud-native PG. It's a cool piece of technology. If you think about sort of two by two. In one corner, you've got self-managed on-premise databases. So, you're very, very slow-moving, big-iron type, old-school database deployments. And on the opposite corner, you've got fully-managed, in the cloud, BigAnimal, Amazon RDS, that type of thing. There's a place on that map where you've got customers that want a self-service type experience. Whether that's for production, or maybe it's even for dev tests, something like that. But you don't want to be giving the management capability off to a third party.For folks that want that type of experience, trying to build that themselves by, like, wiring up EC2 instances, or doing something in their own data center with VMware, or something like that, can be extremely difficult. Whereas if you've go to a Kubernetes-based product, you can get that type of self-service experience really easily, right? And customers can get a lot more flexibility out of how they run their databases and operate their databases. And what sort of control they give to, say application developers who want to spin up a new database for a test or for some sort of small microservice, that type of thing. Those types of workloads tend to work really well with this first-party Kubernetes-based offering. I've been doing databases on Kubernetes in managed services for a long time as well. And I don't, frankly, have any concerns about doing it. There are definitely some sharp edges. And if you wanted to do to-scale, you need to really know what you're doing with Kubernetes because the naive thing will shoot you in the foot.Corey: Oh, yes. So, some it feels almost like people want to cosplay working for Google, but they don't want to pass the technical interview along the way. It's a bit of a weird moment for it.Benjamin: Yeah, I would agree.Corey: I have to go back to my own experiences with using RDS back at my last real job before I went down this path. We were migrating from EC2-Classic to VPC. So, you could imagine what dates me reasonably effectively. And the big problem was the database. And the joy that we had was, “Okay, we have to quiesce the application.” So, the database is now quiet, stop writes, take a snapshot, restore that snapshot into the environment. And whenever we talk to AWS folks, it's like, “So, how long is this going to take?” And the answer was, “Guess.” And that was not exactly reassuring. It went off without a hitch because every migration has one problem. We were sideswiped in an Uber on the way home. But that's neither here nor there. This was two o'clock in the morning, and we finished in half the maintenance time we had allotted. But it was the fact that, well, guess we're going to have to take the database down for many hours with no real visibility, and we hope it'll be up by morning. That wasn't great. But that was the big one going on, on an ongoing basis, there were maintenance windows with a database. We just stopped databasing for a period of time during a fairly broad maintenance window. And that led to a whole lot of unfortunate associations in my mind with using relational databases for an awful lot of stuff. How do you handle maintenance windows and upgrading and not tearing down someone's application? Because I have to assume, “Oh, we just never patch anything. It turns out that's way easier,” is in fact, the wrong answer.Benjamin: Yeah, definitely. As you point out, there's a bunch of fundamental limitations here, if we start to talk about how Postgres actually fits together, right? Pretty much everybody in RDS is a little bit weird. The older RDS offerings are a little bit weird in terms of how they do replication. But most folks are using Postgres streaming replication, to do high availability, Postgres in managed services. And honestly, of course—Corey: That winds up failing over, or the application's aware of both endpoints and switches to the other one?Benjamin: Yeah—Corey: Sort of a database pooling connection or some sort of proxy?Benjamin: Right. There's a bunch of subtleties that get into their way. You say, well, did the [vit 00:16:16] failover too early, did the application try to connect and start making requests before the secondaries available? That sort of thing.Corey: Or you misconfigure it and point to the secondary, suddenly, when there's a switchover of some database, suddenly, nothing can write, it can only read, then you cause a massive outage on the weekend?Benjamin: Yeah. Yeah.Corey: That may have been of an actual story I made up.Benjamin: [laugh] yeah, you should use a managed service.Corey: Yeah.Benjamin: So, it's complicated, but even with managed services, you end up in situations where you have downtime, you have maintenance windows. And with Postgres, especially—and other databases as well—especially with Postgres, one of the biggest concerns you have is major version upgrades, right? So, if I want to go from Postgres 12 to 13, 13 to 14, I can't do that live. I can't have a single cluster that is streaming one Postgres version to another Postgres version, right?So, every year, people want to put things off for two years, three years sometimes—which is obviously not to their benefit—you have this maintenance, you have some sort of downtime, where you perform a Postgres upgrade. At EDB, we've got—so this is a big problem, this is a problem for us. We're involved in the Postgres community. We know this is challenging. That's just a well-known thing. Some of the folks that are working EDB are folks who worked on the Postgres logical replication tech, which arrived in Postgres 10. Logical replication is really a nice tool for doing things like change data capture, you can do Walter JSON, all these types of things are based on logical replication tech.It's not really a thing, at least, the code that's in Postgres itself doesn't really support high availability, though. It's not really something that you can use to build a leader-follower type cluster on top of. We have some techs, some proprietary tech within EDB that used to be called bi-directional replication. There used to be an open-source project called bi-directional replication. This is a kind of a descendant of that. It's now called Postgres Distributed, or EDB Postgres Distributed is the product name. And that tech actually allows us—because it's based on logical replication—allows us to do multiple major versions at the same time, right? So, we can upgrade one node in a cluster to Postgres 14, while the other nodes in the clusters are at Postgres 13. We can then upgrade the next node. We can support these types of operations in a kind of wide range of maintenance operations without taking a cluster down from maintenance.So, there's a lot of interesting opportunities here when we start to say, well, let's step back from what your typical assumptions are for Postgres streaming replication. Give ourselves a little bit more freedom by using logical replication instead of physical streaming replication. And then, what type of services, and what type of patterns can we build on top of that, that ultimately help customers build, whether it's faster databases, more highly available databases, so on and so forth.Corey: Let's face it, on-call firefighting at 2am is stressful! So there's good news and there's bad news. The bad news is that you probably can't prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: One approach that I took for, I guess you could call it backup sort of, was intentionally staggering replication between the primary and the replica about 15 minutes or so. So, if I drop a production table or something like that, I have 15 short minutes to realize what has happened and sever the replication before it is now committed to the replica and now I'm living in hell. It felt like this was not, like, option A, B, or C, or the right way to do things. But given that meeting customers where they are as important, is that the sort of thing that you support with BigAnimal, or do you try to talk customers into not being ridiculous?Benjamin: That's not something we support now. It's not actually something that I hear that many asks for these days. It's kind of interesting, that's a pattern that I've run into a lot in the past.Corey: I was an ancient, grumpy sysadmin. Again, I'm dating myself here. These days, I just store everything at DNS text records, and it's way easier. But I digress.Benjamin: [laugh] yeah, it's something that we see a lot for and we had support for a point-in-time restore, like pretty much anybody else in the business at this point. And that's usually the, “I fat-fingered something,” type response. Honestly, I think there's room to be a bit more flexible and room to do some more interesting things. I think RDS is setting a bar and a lot of database services out there and kind of just meeting that bar. And we all kind of need to be pushing a little bit more into more interesting spaces and figuring out how to get customers more value, get customers to get more out of their money for the database, honestly.Corey: One of the problems we tend to see, in the database ecosystem at large, without naming names or companies or anything like that, is that it's a pretty thin and blurry line between database advocate, database evangelist, and database zealot. Where it feels like instead, we're arguing about religion more than actual technical constraints and concerns. So, here's a fun question that hopefully isn't too much of a gotcha. But what sort of workloads would you actively advise someone not to use BigAnimal for in the database world? But yes, again, if you try to run a DNS server, it's probably not fit for purpose without at least a shim in the way there. But what sort of workloads are you not targeting that a customer is likely to have a relatively unfortunate time with?Benjamin: Large-scale analytical workloads is the easy answer to that, right? If you've got a problem where you're choosing between Postgres and Snowflake, you're seriously considering—you actually have as much data that you seriously be considering Snowflake? You probably don't want to be using Postgres, right? You want to be using something that's column, or you want to be using a query planner that really understands a columnar layout that's going to get you the sorts of performance that you need for those analytical workloads. We don't try to touch that space.Corey: Yeah, we're doing some of that right now with just the sheer volume of client AWS bills we have. We don't really need a relational model for a lot of it. And Athena is basically fallen down on the job in some cases, and, “Oh, do you want to use Redshift, that's basically Postgres.” It's like, “Yeah, it's Postgres, if it decided to run on bars of gold.” No, thank you. It just becomes this ridiculously overwrought solution for what feels like it should be a lot similar. So, it's weird, six months ago or so I wouldn't have had much of an idea what you're talking about. I see it a lot better now. Generally, by virtue of trying to do something the precise wrong way that someone should.Benjamin: Right. Yeah, exactly. I think there's interesting room for Postgres to expand here. It's not something that we're actively working on. I'm not aware of a lot happening in the community that Postgres is, for better or worse, extremely extensible, right? And if you see the JSON-supported Postgres, it didn't exist, I don't know, five, six years ago. And now it's incredibly powerful. It's incredibly flexible. And you can do a lot of quote-unquote, schemaless stuff straight in Postgres. Or you look at PostGIS, right, for doing GIS geographical data, right? That's really a fantastic integration directly in the database.Corey: Yeah, before that people start doing ridiculous things almost looks similar to a graph database or a columnar store somehow, and yeah.Benjamin: Yeah, exactly. I think sometimes somebody will do a good column store that's an open-source deeply integrated into Postgres, rather than—Corey: I've seen someone build one on top of S3 bucket with that head, a quarter of a trillion objects in it. Professional advice, don't do that.Benjamin: [laugh]. Unless you're Snowflake. So, I mean, it's something that I'd like to see Postgres expand into. I think that's an interesting space, but not something that, at least especially for BigAnimal, and frankly, for a lot of EDB customers. It's not something we're trying to push people toward.Corey: One thing that I think we are seeing a schism around is the idea that some vendors are one side of it, some are on the other, where on the one side, you have, oh, every workload should have a bespoke, purpose-built database that is exactly for this type of workload. And the other school of thought is you should generally buy us for a general-purpose database until you have a workload that is scaled and significant to a point where running that on its own purpose-built database begins to make sense. I don't necessarily think that is a binary choice, where do you tend to fall on that spectrum?Benjamin: I think everybody should use Postgres. And I say not just because I work in a Postgres company.Corey: Well, let's be clear. Before this, you were at IBM for five years working on a whole bunch of database stuff over there, not just Postgres. And you, so far, have not struck me as the kind of person who's like, “Oh, so what's your favorite database?” “The one that pays me.” We've met people like that, let's be very clear. But you seem very even-handed in those conversations.Benjamin: Yeah, I got my start in databases, actually, with Apache CouchDB. I am a committer on CouchDB. I worked on a managed at CouchDB service ten years ago. At IBM, I worked on something in nine different open-source databases and managed services. But I love having conversations about, like, well, I've got this workload, should I use Postgres, rr should I use Mongo, should I use Cassandra, all of those types of discussions. Frankly, though, I think in a lot of cases people are—they don't understand how much power they're missing out on if they don't choose a relational database. If they don't understand the relational model well enough to understand that they really actually want that. In a lot of cases, people are also just over-optimizing too early, right? It's just going to be much faster for them to get off the ground, get product in customers hands, if they start with something that they don't have to think twice about. And they don't end up with this architecture with 45 different databases, and there's only one guy in the company that knows how to manage the whole thing.Corey: Oh, the same story of picking a cloud provider. It's, “Okay, you hire a team, you're going to build a thing. Which cloud provider do you pick?” Every cloud provider has a whole matrix and sales deck, and the rest. The right answer, of course, is the one your team's already familiar with because learning a new cloud provider while trying not to run out of money at your startup, can't really doesn't work super well.Benjamin: Exactly. Yeah.Corey: One thing that I think has been sort of interesting, and when I saw it, it was one of those, “Oh, I sort of like them.” Because I had that instinctive reaction and I don't think I'm alone in this. As of this recording a couple of weeks ago, you folks received a sizable investment from private equity. And default reaction to that is, “Oh, well, I guess I put a fork in the company, they're done.” Because the narrative is that once private equity takes an investment, well, that company's best days are probably not in front of it. Now, the counterpoint is that this is not the first time private equity has invested in EDB, and you folks from what I can tell are significantly better than you were when I was your customer a decade ago. So clearly, there is something wrong with that mental model. What am I missing?Benjamin: Yeah. Frankly, I don't know. I'm no expert in funding models and all of those sorts of things. I will say that my experience has been what I've seen at EDB, has definitely been that maybe there's private equity, and then there's private equity. We're in this to build better products and become a better product company. We were previously owned by a private equity firm for the past four years or so. And during the course of those four years, we brought on a bunch of folks who were very product-focused, new leadership. We made a significant acquisition of a company called 2ndQuadrant, which they employed a lot of the European best Postgres company. Now, they're part of EDB and most of them have stayed with us. And we built the managed cloud service, right? So, this is a pretty significant—private equity company buying us to invest in the company. I'm optimistic that that's what we're looking at going forward.Corey: I want to be clear as well, I'm not worried about what I normally would be in a private equity story about this, where they're there to save money and cut costs, and, “Do we really need all these database replicas floating around,” and, “These backups, seems like that's something we don't need.” You have, at last count, 32 Postgres contributors, 7 Postgres committers, and 3 core members. All of whom would run away screaming loudly and publicly, in the event that such a thing were taking place. Of all the challenges and concerns I might have about someone running a cloud service in the modern day. I do not have any fear that you folks are not doing what will very clearly be shown to be the right thing by your customers for the technology that you're building on top of. That is not a concern. There are companies I do not have that confidence in, to be clear.Benjamin: Yeah, I'm glad to hear that. I'm a hundred percent on board as well. I work here, but I think we're doing the right thing, and we're going to be doing great stuff going forward.Corey: One last topic I do want to get into a little bit is, on some level, launching in this decade, a cloud-hosted database offering at a time when Amazon—whose product strategy of yes is in full display—it seems like something ridiculous, that is not necessarily well thought out that why would you ever try to do this? Now, I will temper that by the fact that you are clearly succeeding in this direction. You have customers who say nice things about you, and the reviews have been almost universally positive anywhere I can see things. The negative ones are largely complaining about databases, which I admit might be coming from me.Benjamin: Right, it is a crowded space. There's a lot of things happening. Obviously, Amazon, Microsoft, Google are doing great things, both—Corey: Terrible things, but great, yes. Yes.Benjamin: [laugh] right, there's good products coming in. I think AlloyDB is not necessarily a great product. I haven't used it myself yet, but it's an interesting step in the direction. I'm excited to see development happening. But at the end of the day, we're a database company. Our focus is on building great databases and supporting great databases. We're not entering this business to try to take on Amazon from an infrastructure point of view. In fact, the way that we're structuring the product is really to try to get the strengths of both worlds. We want to give customers the ability to get the most out of the AWS or Azure infrastructure that they can, but come to us for their database.Frankly, we know Postgres better than anybody else. We have a greater ability to get bugs fixed in Postgres than anybody else. We've got folks working on the database in the open. We got folks working on the database proprietary for us. So, we give customers things like break/fix support on that database. If there is a bug in Postgres, there's a bug in the tech that sits around Postgres. Because obviously, Postgres is not a batteries-included system, really. We're going to fix that for you. That's part of the contract that we're giving to our customers. And I know a lot of smaller companies maybe haven't been burned by this sort of thing very much. We start to talk about enterprise customers and medium, larger-scale customers, this starts to get really valuable. The ability to have assurance on top of your open-source product. So, I think there's a lot of interesting things there, a lot of value that we can provide there.I think also that I talked a little bit about this earlier, but like the box, this sort of RDS-shaped box, I think is a bit too small. There's an opportunity for smaller players to come in and try to push the boundaries of that. For example, giving customers more support by default to do a good job using their database. We have folks on board that can help consult with customers to say, “No, you shouldn't be designing your schemas that way. You should be designing your schemas this way. You should be using indexes here,” that sort of stuff. That's been part of our business for a long time. Now, with a managed service, we can bake that right into the managed service. And that gives us the ability to kind of make that—you talk about shared responsibility between the service writer and the customer—we can change the boundaries of that shared responsibility a little bit, so that customers can get more value out of the managed database service than they might expect otherwise.Corey: There aren't these harsh separations and clearly defined lines across which nothing shall pass, when it makes sense to do that in a controlled responsible way.Benjamin: Right, exactly. Some of that is because we're a database company, and some of that is because, frankly, we're much smaller.Corey: I'll take it a step further beyond that, as well, that I have seen this pattern evolve a number of times where you have a customer running databases on EC2, and their AWS account managers suggests move to RDS. So, they do. Then, move to Aurora. So, they do. Then, I move this to DynamoDB. At which point, it's like, what do you think your job is here, exactly? Because it seems like every time we move databases, you show up in a nicer car. So, what exactly is the story here, and what are the incentives? Where it just feels like there is a, “Whatever you're doing is not the way that it should be done. So, it's time to do, yet, another migration.”There's something to be said for companies who are focused around a specific aspect of things. Then once that is up and working and running, great. Keep on going. This is fine. As opposed to trying to chase the latest shiny, on some level. I have a big sense of, I guess, affinity for companies that wind up knowing where they start, and most notably, where they stop.Benjamin: Yeah, I think that's a really good point. I don't think that we will be building an application platform anytime soon.Corey: “We're going to run Lambda functions on top of a database.” It's like, “Congratulations. That is the weirdest stored procedure I can imagine this week, but I'm sure we can come up with a worse one soon.”Benjamin: Exactly.Corey: I really want to thank you for taking the time to speak with me so much about how you're thinking about this, and what you've been building over there. If people want to learn more, where's the best place to go to find you?Benjamin: biganimal.com.Corey: Excellent. We will throw a link to that in the show notes and it only just occurred to me that the Postgres mascot is an elephant, and now I understand why it's called BigAnimal. Yeah, that's right. He who laughs last, thinks slowest, and today, that's me. I really want to thank you for being so generous with your time. I appreciate it.Benjamin: Thank you. I really appreciate it.Corey: Benjamin Anderson, CTO for Cloud at EDB. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you then wind up stuffing into a SQLite database, converting to Base64, and somehow stuffing into the comment field.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About BrandonBrandon West was raised in part by video games and BBSes and has been working on web applications since 1999. He entered the world of Developer Relations in 2011 as an evangelist for a small startup called SendGrid and has since held leadership roles at companies like AWS. At Datadog, Brandon is focused on helping developers improve the performance and developer experience of the things they build. He lives in Seattle where enjoys paddle-boarding, fishing, and playing music.Links Referenced: Datadog: https://www.datadoghq.com/ Twitter: https://twitter.com/bwest TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: This episode is sponsored in part by our friends at Fortinet. Fortinet's partnership with AWS is a better-together combination that ensures your workloads on AWS are protected by best-in-class security solutions powered by comprehensive threat intelligence and more than 20 years of cybersecurity experience. Integrations with key AWS services simplify security management, ensure full visibility across environments, and provide broad protection across your workloads and applications. Visit them at AWS re:Inforce to see the latest trends in cybersecurity on July 25-26 at the Boston Convention Center. Just go over to the Fortinet booth and tell them Corey Quinn sent you and watch for the flinch. My thanks again to my friends at Fortinet.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is someone I've been trying to get on the show for years, but I'm very bad at, you know, following up and sending the messages and all the rest because we all struggle with our internal demons. My guest instead struggles with external demons. He is the team lead for developer experience and tools advocacy at what I can only assume is a Tinder for Pets style company, Date-A-Dog. Brendon West, thank you for joining me today.Brandon: Hey, Corey, thanks for having me. I'm excited to be here. Finally, like you said, it's been a couple of years. But glad that it's happening. And yeah, I'm on the DevRel team at Datadog.Corey: Yes, I'm getting a note here in the headset of breaking news coming in. Yes, you're not apparently a dog dating company, you are a monitoring slash observability slash whatever the cool kids are calling it today telemetry outputer dingus nonsense. Anyone who has ever been to a community or corporate event has no doubt been tackled by one of the badge scanners that you folks have orbiting your booth, but what is it that you folks do?Brandon: Well, the observability, the monitoring, the distributed tracing, all that stuff that you mentioned. And then a lot of other interesting things that are happening. Security is a big focus—InfoSec—so we're adding some products around that, automated security monitoring, very cool. And then the sort of stuff that I'm representing is stuff that helps developers provide a better experience to their end-users. So, things like front-end monitoring, real-time user monitoring, synthetic testing of your APIs, whatever it might be.Corey: Your path has been somewhat interesting because you—well, everyone's path has been somewhat interesting; yours has been really interesting because back in 2011, you entered the world of developer relations, or being a DevReloper as I insist on calling it. And you were in a—you call it a small startup called SendGrid. Which is, on some level, hilarious from my point of view. I've been working with you folks—you folks being SendGrid—for many years now. I cared a lot about email once upon a time.And now I send an email newsletter every week, that deep under the hood, through a couple of vendor abstraction layers is still SendGrid, and I don't care about email because that's something that I can pay someone else to worry about. You went on as well to build out DevRel teams at AWS. You decided okay, you're going to take some time off after that. You went to a small scrappy startup and ah, nice. You could really do things right and you have a glorious half of the year and then surprise, you got acquired by Datadog. Congratu-dolances on that because now you're right back in the thick of things at big company-style approaches. Have I generally nailed the trajectory of the past decade for you?Brandon: Yeah, I think the broad strokes are all correct there. SendGrid was a small company when I joined, you know? There were 30 of us or so. So, got to see that grow into what it is today, which was super, super awesome. But other than that, yeah, I think that's the correct path.Corey: It's interesting to me, in that you were more or less doing developer relations before that was really a thing in the ecosystem. And I understand the challenge that you would have in a place like SendGrid because that is large-scale email sending, transactional or otherwise, and that is something that by and large, has slipped below the surface level of awareness for an awful lot of folks in your target market. It's, “Oh, okay, and then we'll just have the thing send an email,” they say, hand-waving over what is an incredibly deep and murky pool. And understanding that is a hard thing requires a certain level of technical sophistication. So, you started doing developer relations for something that very clearly needed some storytelling chops. How did you fall into it originally?Brandon: Well, I wanted to do something that let me use those storytelling chops, honestly. I had been writing code at an agency for coal mines and gold mines and really actively inserting evil into the world, power plants, and that sort of thing. And, you know, I went to school for English literature. I loved writing. I played in thrash metal bands when I was a kid, so I've been up on stage being cussed at and told that I suck. So I—Corey: Oh, I get that conference talks all the time.Brandon: Yeah, right? So, that's why when people ask me to speak, I'm like, “Absolutely.” There's no way I can bomb harder than I've bombed before. No fear, right? So yeah, I wanted to use those skills. I wanted to do something different.And one of my buddies had a company that he had co-founded that was going through TechStars in Boulder. SendGrid was the first accelerator-backed company to IPO which is pretty cool. But they had gone through TechStars in 2009. They were looking for a developer evangelist. So, SendGrid was looking for developer evangelist and my friend introduced me said, “I think you'd be good at this. You should have a conversation.” My immediate thought was what the hell is a developer evangelist?Corey: And what might a SendGrid be? And all the rest. Yes, it's that whole, “Oh, how do I learn to swim?” Someone throws you off the end of the dock and then retrospect, it's, “I don't think they were trying to teach me how to swim.” Yeah. Hindsight.Brandon: Yeah. It worked out great. I will say, though, that I think DevRel has been around for a long time, you know? The title has been around since the original Macintosh at Apple in 1980-ish. There's a whole large part of the tech world that would like you to think that it's new because of all the terrible things that their DevRel team did at Microsoft in the late-90s.And you can go read all about this. There were trials about it. These documents were released to the public, James Plamondon is the lead architect of all of this nastiness. But I think there was then a concerted effort to memory-hole that and say, “No, DevRel is new and shiny.” And then Google came along and said, “Well, it's not evangelism anymore. It's advocacy.”Corey: It's not sysadmin work anymore. It's SRE. It's not on-prem, it's Sparkling Kubernetes, et cetera, et cetera.Brandon: Yeah, so there's this sense in a lot of places that DevRel is new, but it's actually been around a long time. And you can learn a lot from reading about the history and understanding it, something I've given a talk on and written a bit about. So.Corey: My philosophy around developer relations for a while has been that in many cases, its biggest obstacle is the way that it is great at telling stories about fantastically complex, deeply technical things; it can tell stories about almost anything except itself. And I keep seeing similar expressions of the same problem again, and again, and again. I mean, AWS, where you worked, as an example: they love to talk about their developer advocates, and you read the job descriptions and these are high-level roles with sweeping responsibilities, broad basis of experience being able to handle things at a borderline executive level. And then they almost neuter the entire thing by slapping a developer advocate title on top of those people, which means that some of the people that would be most effectively served by talking to them will dismiss them as, “Well, I'm a director”—or a VP—“What am I going to do talking to a developer advocate?” It feels like there's a swing and a miss as far as encapsulating the value that the function provides.I want to be clear, I am not sitting here shitting on DevRel or its practitioners, I see a problem with how it [laugh] is being expressed. Now, feel free to argue with me and just scream at me for the next 20 minutes, and this becomes a real short show. But—Brandon: [laugh].Corey: —It'll be great. Hit me.Brandon: No, you're correct in many ways, which makes me sad because these are the same conversations that I've been having for the 11, 12 years that I've been in DevRel now. And I thought we would have moved past this at some point, but the problem is that we are bad at advocating for advocacy. We do a bad job of relating to people about DevRel because we spend so much time worried about stuff that doesn't really matter. And we get very loud voices in the echo chamber screaming about titles and evangelism versus advocate versus community manager, and which department you should report up to, and all of these things that ultimately don't matter. And it just seems like bickering from the outside. I think that the core of what we do is super awesome. And I don't think it's very hard to articulate. It's just that we don't spend the time to do that.Corey: It's always odd to me when I talk to someone like, “Oh, you're in DevRel. What does that mean?” And their immediate response is, “Well, it's not marketing, I'll tell you that.” It's feels like there might be some trauma that is being expressed in some strange ways. I do view it as marketing, personally, and people who take umbrage at that don't generally tend to understand what marketing is.Yeah, you can look at any area of business or any function and judge it by some of the worst examples that we've all seen, but when someone tells me they work in sales, I don't automatically assume that they are sending me horrifyingly passive-aggressive drip campaigns, or trying to hassle me in a car lot. It's no, there's a broad spectrum of people. Just like I don't assume that you're an engineer. And I immediately think, oh, you can't solve FizzBuzz on a whiteboard. No, there's always going to be a broad spectrum of experience.Marketing is one of those awesome areas of business that's dramatically misunderstood a lot. Similarly to the fact that, you know, DevRel can't tell stories, you think marketing could tell stories about itself, but it's still struggles, too, in a bunch of ways. But I do believe that even if they're not one of the same, developer relations and marketing are aligned around an awful lot of things like being able to articulate value that is hard to quantify.Brandon: I completely agree with that. And if I meet someone in DevRel that starts off the conversation by saying that they're not in marketing, then I know they're probably not that great at their job. I mean, I think there's a place of tech hubris, where we want to disrespect anything that's not a hard skill where it's not putting zeros and ones into a chip—Corey: And spoiler, they're all very hard skills.Brandon: [laugh]. Yeah. And so, first off, like, stop disrespecting marketing. It's important; your business probably wouldn't survive if you didn't have it. And second of all, you're not immune to it, right?Like, Heartbleed had a logo and a name for vulnerability because tech people are so susceptible to it, right? People don't just wake up and wait in line for three days for a new iPhone because tech marketing doesn't work, right?Corey: “Oh, tech marketing doesn't work on me,” says someone who's devoted last five years of their life to working on Kubernetes. Yeah, sure it doesn't.Brandon: Yeah exactly. So, that whole perspective is silly. I think part of the problem is that they don't want to invest in learning how to communicate what they do to a marketing org. They don't want to spend the time to say, “Here's how the marketing world thinks, and here's how we can fit into that perspective.” They want to come in and say, “Well, you don't understand DevRel. Let me define DevRel for you and tell you what we do.” And all those sorts of things. It's too prescriptive and less collaborative.Corey: Anytime you start getting into the idea of metrics around how do you measure someone in a developer advocacy role, the answer is, “Well, your metrics that you're using are wrong, and any metrics you use are wrong, and there's no good way to do it.” And I am sympathetic to that. When I started this place, I knew that if I went to a bunch of events and did my thing, good things would happen for the business. And how did I articulate that? Gut feel, but when you own the place, you can do that.Whereas when you are a function inside of another org, inside of another org, and you start looking at from the executive leadership position at these things, it's, “Okay, so let me get this straight. You cost as much as an engineer, you cost as much as that again, in your expenses because you're traveling all the time, you write zero production code, whenever people ask you what it is you do here, you have a very strange answer, and from what we can tell, it looks like you hang out with your friends in exotic locations, give a 15-minute talk from time to time that mentions our name at the beginning, and nothing else relevant to our business, and then you go around and the entire story is ‘just trust me, I'm adding value.'” Yeah, when it's time to tighten belts and start cutting back, is it any wonder that the developer advocacy is often one of the first departments hit from that perspective?Brandon: It doesn't surprise me. I mean, I've been a part of DevRel teams where we had some large number of events that we had attended for the year—I think 450-something—and the director of the team was very excited to show that off, right, you should have seen the CFOs face when he heard that, right, because all he sees is outgoing dollar signs. Like, how much expense? What's the ROI on 450 events?Corey: Yeah, “450 events? That's more than one a day. Okay, great. That's a big number and I already know what we're spending. Great. How much business came out of that?”And that's when the hemming and hawing starts. Like, well, sort of, and yadda—and yeah, it doesn't present well in the language that they are prepared to speak. But marketing can tell those stories because they have for ages. Like, “Okay, how much business came from our Superbowl ad?” “I dunno. The point is, is that there's a brand awareness play, there's the chance to remain top of the mental stack when people think about this space. And over the next few months, we can definitely see there's been a dramatic uptick in our business. Now, how do we attribute that back? Well, I don't know.”There's a saying in marketing, that half of your marketing budget is wasted. Now, figuring out which half will spend the rest of your career, you'll never get even close. Because people don't know the journey that customers go through, not really. Even customers don't often see it.Take this podcast, for example. I have sponsors that I do love and appreciate who say things from time to time on this show. And people will hear it and occasionally will become customers of those sponsors. But very often, it's, “Oh, I heard about that on the podcast. I'll Google it when I get to work and then I'll have a conversation with my team and we'll agree to investigate that.”And any UTM tracking has long since fallen by the wayside. You might get to that from discussions with users in their interview process, but very often, they won't remember where it came up. And it's one of those impossible to quantify things. Now, I sound like one of those folks where I'm trying to say, “Oh, buy sponsorships that you can never prove add value.” But that is functionally how advertising tends to work, back in the days before it spied on you.Brandon: Yeah, absolutely. And we've added a bunch of instrumentation to allow us to try and put that multi-touch attribution model together after the fact, but I'm still not sure that that's worth the squeeze, right? You don't get much juice out. One of the problems with metrics in DevRel is that the things that you can measure are very production-focused. It's how many talks did you give? How many audience members did you reach?Some developer relations folks do actually write production code, so it might be how many of the official SDK that you support got downloaded? That can be more directly attributed to business impact, those sorts of things are fantastic. But a lot of it is kind of fuzzy and because it's production-focused, it can lead to burnout because it's disconnected from business impact. “It's how many widgets did your line produce today?” “Well, we gave all these talks and we had 150,000 engaged developer hours.” “Well, cool, what was the business outcome?” And if you can't answer that for your own team and for your own self in your role, that leads pretty quickly to burnout.Corey: Anytime you start measuring something and grading people based on it, they're going to optimize for what you measure. For example, I send an email newsletter out, at time of this recording, to 31,000 people every week and that's awesome. I also periodically do webinars about the joys of AWS bill optimization, and you know, 50 people might show up to one of those things. Okay, well, from a broad numbers perspective, yeah, I'd much rather go and send something out to those 31,000, folks until you realize that the kind of person that's going to devote half an hour, forty-five minutes to having a discussion with you about AWS bill optimization is far likelier to care about this to the point where they become a customer than someone who just happens to be in an audience for something that is orthogonally-related. And that is the trick because otherwise, we would just all be optimizing for the single biggest platforms out there if oh, I'm going to go talk at this conference and that conference, not because they're not germane to what we do, but because they have more people showing up.And that doesn't work. When you see that even on the podcast world, you have Joe Rogan, as the largest podcast in the world—let's not make too many comparisons in different ways because I don't want to be associated with that kind of tomfoolery—but there's a reason that his advertisers, by and large, are targeting a mass-market audience, whereas mine are targeting B2B SaaS, by and large. I'm not here shilling for various mattress companies. I'm instead talking much more about things that solve the kind of problem that listeners to this show are likely to have. It's the old-school of thought of advertising, where this is a problem that is germane to a certain type of audience, and that certain type of audience listens to shows like this. That was my whole school of thought.Brandon: Absolutely. I mean, the core value that you need to do DevRel, in my opinion is empathy. It's all about what Maya Angelou said, right? “People may not remember what you said, but they'll definitely remember how you made them feel.” And I found that to be incredibly true.Like, the moments that I regret the most in DevRel are the times when someone that I've met and spent time with before comes up to have a conversation and I don't remember them because I met 200 people that night. And then I feel terrible, right? So, those are the metrics that I use internally. It's hearts and minds. It's how do people feel? Am I making them feel empowered and better at their craft through the work that I do?That's why I love DevRel. If I didn't get that fulfillment, I'd go write code again. But I don't get that sense of satisfaction, and wow, I made an impact on this person's trajectory through their career that I do from DevRel. So.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it's still somebody's problem. And a big part of that responsibility is app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you're actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That's S-N-Y-K.co/screamCorey: The way that I tend to see it, too, is that there's almost a bit of a broadening of DevRel. And let's be clear, it's a varied field with a lot of different ways to handle that approach. I'm have a terrible public speaker, so I'm not going to ever succeed in DevRel. Well, that's certainly not true. People need to write blog posts; people need to wind up writing some of the sample code, in some cases; people need to talk to customers in a small group environment, as opposed to in front of 3000 people and talk about the things that they're seeing, and the rest.There's a broad field and different ways that it applies. But I also see that there are different breeds of developer advocate as well. There are folks, like you for example. You and I have roughly the same amount of time in the industry working on different things, whereas there's also folks who it seems like they graduate from a boot camp, and a year later, they're working in a developer advocacy role. Does that mean that they're bad developer advocates?I don't think so, but I think that if they try and present things the same way that you were I do from years spent in the trenches working on these things, they don't have that basis of experience to fall back on, so they need to take a different narrative path. And the successful ones absolutely do.Brandon: Yeah.Corey: I think it's a nuanced and broad field. I wish that there was more acceptance and awareness of that.Brandon: That's absolutely true. And part of the reason people criticize DevRel and don't take it seriously, as they say, “Well, it's inconsistent. This org, it reports to product; or, this org, it reports up to marketing; this other place, it's part of engineering.” You know, it's poorly defined. But I think that's true of a lot of roles in tech.Like, engineering is usually done a different way, very differently at some orgs compared to others. Product teams can have completely different methodologies for how they track and manage and estimate their time and all of those things. So, I would like to see people stop using that as a cudgel against the whole profession. It just doesn't make any sense. At the same time, two of the best evangelist I ever hired were right out of university, so you're completely correct.The key thing to keep in mind there is, like, who's the audience, right, because ultimately, it's about building trust with the audience. There's a lot of rooms where if you and I walk into the room; if it's like a college hackathon, we're going to have a—[laugh], we're going to struggle.Corey: Yeah, we have some real, “Hello fellow kids,” energy going on when we do that.Brandon: Yeah. Which is also why I think it's incredibly important for developer relations teams to be aware of the makeup of their team. Like, how diverse is your team, and how diverse are the audiences you're speaking to? And if you don't have someone who can connect, whether it's because of age or lived experience or background, then you're going to fail because like I said that the number one thing you need to be successful in this role is empathy, in my opinion.Corey: I think that a lot of the efforts around a lot of this—trying to clarify what it is—some cases gone in well, I guess I'm going to call it the wrong direction. And I know that sounds judgy and I'm going to have to live with that, I suppose, but talk to me a bit about the, I guess, rebranding that we've seen in some recent years around developer advocates. Specifically, like, I like calling folks DevRelopers because it's cutesy, it's a bit of a portmanteau. Great. But it's also not something I seriously suggest most people put on business cards.But there are people who are starting to, I think, take a similar joke and actually identify with it where they call themselves developer avocados, which I don't fully understand. I have opinions on it, but again, having opinions that are not based in data is something I try not to start shouting from the rooftops wherever I can. You live in that world a lot more posted than I do, where do you stand?Brandon: So, I think it was well-intentioned and it was an attempt to do some of the awareness and brand building for DevRel, broadly, that we had lacked. But I see lots of problems with it. One, we already struggle to be taken seriously in many instances, as we've been discussing, and I don't think we do ourselves any favors by giving ourselves cutesy nicknames that sort of infantilize the role like I can't think of any other job that has a pet name for the work that they do.Corey: Yeah. The “ooh-woo accounting”. Yeah, I sort of don't see that happening very often in most business orgs.Brandon: Yeah. It's strange to me at the same time, a lot of the people who came up with it and popularized it are people that I consider friends and good colleagues. So hopefully, they won't be too offended, but I really think that it kind of set us back in many ways. I don't want to represent the work that I do with an emoji.Corey: Funny, you bring that up. As we record this through the first recording, I have on my new ridiculous desktop computer thing from Apple, which I have named after a—you know, the same naming convention that you would expect from an AWS region—it's us-shitpost-one. Instead of the word shit, it has the poop emoji. And you'd be amazed at the number of things that just melt when you start trying to incorporate that. GitHub has a problem with that being the name of an SSH key, for example.I don't know if I'll keep it or I'll just fall back to just spelling words out, but right now, at least, it really is causing all kinds of strange computer problems. Similarly, it causes strange cultural problems when you start having that dissonance and seeing something new and different like that in a business context. Because in some cases, yeah, it helps you interact with your audience and build rapport; in many others, it erodes trust and confidence that you know what you're talking about because people expect things to be cast a certain way. I'm not saying they're right. There's a shitload of bias that bakes into that, but at the same time, I'd like to at least bias for choosing when and where I'm going to break those expectations.There's a reason that increasingly, my Duckbillgroup.com website speaks in business terms, rather than in platypus metaphors, whereas lastweekinaws.com, very much leans into the platypus. And that is the way that the branding is breaking down, just because people expect different things in different places.Brandon: Yeah and, you know, this framing matters. And I've gone through two exercises now where I've helped rename an evangelism team to an advocacy team, not because I think it's important to me—it's a bunch of bikeshedding—but it has external implications, right? Especially evangelism, in certain parts of the world, has connotations. It's just easier to avoid those. And how we present ourselves, the titles that we choose are important.I wish we would spend way less time arguing about them, you know, advocacy has won evangelism, don't use it. DevRel, if you don't want to pick one, great. DevRel is broader umbrella. If you've got community managers, people who can't write code that do things involving your events or whatever, program managers, if they're on your team, DevRel, great description. I wish we could just settle that. Lots of wasted air discussing that one.Corey: Constantly. It feels like this is a giant distraction that detracts from the value of DevRel. Because I don't know about you, but when I pick what I want to do next in my career, the things I want to explain to people and spend that energy on are never, I want to explain what it is that I do. Like I've never liked those approaches where you have to first educate someone before they're going to be in a position where they want to become your customer.I think, honestly, that's one of the things that Datadog has gotten very right. One of the early criticisms lobbed against Datadog when it first came out was, “Oh, this is basically monitoring by Fisher-Price.” Like, “This isn't the deep-dive stuff.” Well yeah, but it turns out a lot of your buying audience are fundamentally toddlers with no visibility into what's going on. For an awful lot of what I do, I want it to be click, click, done.I am a Datadog customer for a reason. It's not because I don't have loud and angry opinions about observability; it's because I just want there to be a dashboard that I can look at and see what's working, what's not, and do I need to care about things today? And it solves that job admirably because if I have those kinds of opinions about every aspect, I'm never going to be your customer anyway, or anyone's customer. I'm going to go build my own and either launch a competitor or realize this is my what I truly love doing and go work at a company in this space, possibly yours. There's something to be said for understanding the customer journey that those customers do not look like you.And I think that's what's going on with a lot of the articulation around what developer relations is or isn't. The people on stage who go to watch someone in DevRel give a talk, do not care, by and large, what DevRel is. They care about the content that they're about to hear about, and when the first half of it is explaining what the person's job is or isn't, people lose interest. I don't even like intros at the beginning of a talk. Give me a hook. Talk for 45 seconds. Give me a story about why I should care before you tell me who you are, what your credentials are, what your job title is, who you work for. Hit me with something big upfront and then we'll figure it out from there.Brandon: Yeah, I agree with you. I give this speaking advice to people constantly. Do not get up on stage and introduce yourself. You're not a carnival hawker. You're not trying to get people to roll up and see the show.They're already sitting in the seat. You've established your credibility. If they had questions about it, they read your abstract, and then they went and checked you out on LinkedIn, right? So, get to the point; make it engaging and entertaining.Corey: I have a pet theory about what's going on in some cases where, I think, on some level, it's an outgrowth of an impostor-syndrome-like behavior, where people don't believe that they deserve to be onstage talking about things, so they start backing up their bona fides to almost reassure themselves because they don't believe that they should be up there and if they don't believe it, why would anyone else. It's the wrong approach. By holding the microphone, you inherently deserve to hold the microphone. And go ahead and tell your story. If people care enough to dig into you and who you are and well, “What is this person's background, really?” Rest assured the internet is pretty easy to use these days, people will find out. So, let them do that research if they care. If they don't, then there's an entire line of people in this world who are going to dislike you or say you're not qualified for what it is you're doing or you don't deserve it. Don't be in that line, let alone at the front of it.Brandon: So, you mentioned imposter syndrome and it got me thinking a little bit. And hopefully this doesn't offend anyone, but I kind of starting to think that imposter syndrome is in many ways invented by people to put the blame on you for something that's their fault. It's like a carbon footprint to the oil and gas industry, right? These companies can't provide you psychological safety and now they've gone and convinced you that it's your fault and that you're suffering from this syndrome, rather than the fact that they're not actually making you feel prepared and confident and ready to get up on that stage, even if it's your first time giving a talk, right?Corey: I hadn't considered it like that before. And again, I do tend to avoid straying into mental health territory on this show because I'm not an—Brandon: Yes.Corey: Expert. I'm a loud, confident white guy in tech. My failure mode is a board seat and a book deal, but I am not board-certified, let's be clear. But I think you're onto something here because early on in my career, I was very often faced with a whole lot of nebulous job description-style stuff and I was never sure if I was working on the right thing. Now that I'm at this stage of my career, and as you become more senior, you inherently find yourselves in roles, most of the time, that are themselves mired in uncertainty. That is, on some level, what seniority leads to.And that's fine, but early on in your career, not knowing if you're succeeding or failing, I got surprise-fired a number of times when I thought I was doing great. There are also times that I thought I was about to be fired on the spot and, “Come on in; shut the door.” And yeah, “Here's a raise because you're just killing it.” And it took me a few years after that point to realize, wait a minute. They were underpaying me. That's what that was, and they hope they didn't know.But it's that whole approach of just trying to understand your place in the world. Do I rock? Do I suck? And it's that constant uncertainty and unknowing. And I think companies do a terrible job, by and large, of letting people know that they're okay, they're safe, and they belong.Brandon: I completely agree. And this is why I would strongly encourage people—if you have the privilege—please do not work at a company that does not want you to bring your whole self to work, or that bans politics, or however they want to describe it. Because that's just a code word for we won't provide you psychological safety. Or if they're going to, it ends at a very hard border somewhere between work and life. And I just don't think anyone can be successful in those environments.Corey: I'm sure it's possible, but it does bias for folks who, frankly, have a tremendous amount of privilege in many respects where I mentioned about, like, I'm a white dude in tech—you are too—and when we say things, we are presumed competent and people don't argue with us by default. And that is a very easy to forget thing. Not everyone who looks like us is going to have very similar experiences. I have gotten it hilariously wrong before when I gave talks on how to wind up negotiating for salaries, for example, because well, it worked for me, what's the problem? Yeah, I basically burned that talk with fire, redid the entire thing and wound up giving it with a friend of mine who was basically everything that I am not.She was an attorney, she was a woman of color, et cetera, et cetera. And suddenly, it was a much stronger talk because it wasn't just, “How to Succeed for White Guys.” There's value in that, but you also have to be open to hearing that and acknowledging that you were born on third; you didn't hit a triple. There's a difference. And please forgive the sports metaphor. They do not sound natural coming from me.Brandon: [laugh]. I don't think I have anything more interesting to add on that topic.Corey: [laugh]. So, I really want to thank you for taking the time to speak with me today. If people want to learn more about what you're up to and how you view the world, what's the best place to find you.Brandon: So, I'm most active on Twitter at @bwest, but you know, it's a mix of things so you may or may not just get tech. Most recently, I've been posting about a—Corey: Oh, heaven forbid you bring your whole self to school.Brandon: Right? I think most recently, I've been posting about a drill press that I'm restoring. So, all kinds of fun stuff on there.Corey: I don't know it sounds kind of—wait for it—boring to me. Bud-dum-tiss.Brandon: [laugh]. [sigh]. I can't believe I missed that one.Corey: You're welcome.Brandon: Well, done. Well, done. And then I also will be hiring for a couple of developer relations folks at Datadogs soon, so if that's interesting and you like the words I say about how to do DevRel, then reach out.Corey: And you can find all of that in the show notes, of course. I want to thank you for being so generous with your time. I really appreciate it.Brandon: Hey, thank you, Corey. I'm glad that we got to catch up after all this time. And hopefully get to chat with you again sometime soon.Corey: Brandon West, team lead for developer experience and tools advocacy at Datadog. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry and insulting comment that is talking about how I completely misunderstand the role of developer advocacy. And somehow that rebuttal features no fewer than 400 emoji shoved into it.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About ChrisChris Short has been a proponent of open source solutions throughout his over two decades in various IT disciplines, including systems, security, networks, DevOps management, and cloud native advocacy across the public and private sectors. He currently works on the Kubernetes team at Amazon Web Services and is an active Kubernetes contributor and Co-chair of OpenGitOps. Chris is a disabled US Air Force veteran living with his wife and son in Greater Metro Detroit. Chris writes about Cloud Native, DevOps, and other topics at ChrisShort.net. He also runs the Cloud Native, DevOps, GitOps, Open Source, industry news, and culture focused newsletter DevOps'ish.Links Referenced: DevOps'ish: https://devopsish.com/ EKS News: https://eks.news/ Containers from the Couch: https://containersfromthecouch.com opengitops.dev: https://opengitops.dev ChrisShort.net: https://chrisshort.net Twitter: https://twitter.com/ChrisShort TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Coming back to us since episode two—it's always nice to go back and see the where are they now type of approach—I am joined by Senior Developer Advocate at AWS Chris Short. Chris, been a few years. How has it been?Chris: Ha. Corey, we have talked outside of the podcast. But it's been good. For those that have been listening, I think when we recorded I wasn't even—like, when was season two, what year was that? [laugh].Corey: Episode two was first pre-pandemic and the rest. I believe—Chris: Oh. So, yeah. I was at Red Hat, maybe, when I—yeah.Corey: Yeah. You were doing Red Hat stuff, back when you got to work on open-source stuff, as opposed to now, where you're not within 1000 miles of that stuff, right?Chris: Actually well, no. So, to be clear, I'm on the EKS team, the Kubernetes team here at AWS. So, when I joined AWS in October, they were like, “Hey, you do open-source stuff. We like that. Do more.” And I was like, “Oh, wait, do more?” And they were like, “Yes, do more.” “Okay.”So, since joining AWS, I've probably done more open-source work than the three years at Red Hat that I did. So, that's kind of—you know, like, it's an interesting point when I talk to people about it because the first couple months are, like—you know, my friends are like, “So, are you liking it? Are you enjoying it? What's going on?” And—Corey: Do they beat you with reeds? Like, all the questions people have about companies? Because—Chris: Right. Like, I get a lot of random questions about Amazon and AWS that I don't know the answer to.Corey: Oh, when I started telling people, I fixed Amazon bills, I had to quickly pivot that to AWS bills because people started asking me, “Well, can you save me money on underpants?” It's I—Chris: Yeah.Corey: How do you—fine. Get the prime credit card. It docks 5% off the bill, so there you go. But other than that, no, I can't.Chris: No.Corey: It's—Chris: Like, I had to call my bank this morning about a transaction that I didn't recognize, and it was from Amazon. And I was like, that's weird. Why would that—Corey: Money just flows one direction, and that's the wrong direction from my employer.Chris: Yeah. Like, what is going on here? It shouldn't have been on that card kind of thing. And I had to explain to the person on the phone that I do work at Amazon but under the Web Services team. And he was like, “Oh, so you're in IT?”And I'm like, “No.” [laugh]. “It's actually this big company. That—it's a cloud company.” And they're like, “Oh, okay, okay. Yeah. The cloud. Got it.” [laugh]. So, it's interesting talking to people about, “I work at Amazon.” “Oh, my son works at Amazon distribution center,” blah, blah, blah. It's like, cool. “I know about that, but very little. I do this.”Corey: Your son works in Amazon distribution center. Is he a robot? Is normally my next question on that? Yeah. That's neither here nor there.So, you and I started talking a while back. We both write newsletters that go to a somewhat similar audience. You write DevOps'ish. I write Last Week in AWS. And recently, you also have started EKS News because, yeah, the one thing I look at when I'm doing these newsletters every week is, you know what I want to do? That's right. Write more newsletters.Chris: [laugh].Corey: So, you are just a glutton for punishment? And, yeah, welcome to the addiction, I suppose. How's it been going for you?Chris: It's actually been pretty interesting, right? Like, we haven't pushed it very hard. We're now starting to include it in things. Like we did Container Day; we made sure that EKS news was on the landing page for Container Day at KubeCon EU. And you know, it's kind of just grown organically since then.But it was one of those things where it's like, internally—this happened at Red Hat, right—when I started live streaming at Red Hat, the ultimate goal was to do our product management—like, here's what's new in the next version thing—do those live so anybody can see that at any point in time anywhere on Earth, the second it's available. Similar situation to here. This newsletter actually is generated as part of a report my boss puts together to brief our other DAs—or developer advocates—you know, our solutions architects, the whole nine yards about new EKS features. So, I was like, why can't we just flip that into a weekly newsletter, you know? Like, I can pull from the same sources you can.And what's interesting is, he only does the meeting bi-weekly. So, there's some weeks where it's just all me doing it and he ends up just kind of copying and pasting the newsletter into his document, [laugh] and then adds on for the week. But that report meeting for that team is now getting disseminated to essentially anyone that subscribes to eks.news. Just go to the site, there's a subscribe thing right there. And we've gotten 20 issues in and it's gotten rave reviews, right?Corey: I have been a subscriber for a while. I will say that it has less Chris Short personality—Chris: Mm-hm.Corey: —to it than DevOps'ish does, which I have to assume is by design. A lot of The Duckbill Group's marketing these days is no longer in my voice, rather intentionally, because it turns out that being a sarcastic jackass and doing half-billion dollar AWS contracts can not to be the most congruent thing in the world. So okay, we're slowly ameliorating that. It's professional voice versus snarky voice.Chris: Well, and here's the thing, right? Like, I realized this year with DevOps'ish that, like, if I want to take a week off, I have to do, like, what you did when your child was born. You hired folks to like, do the newsletter for you, or I actually don't do the newsletter, right? It's binary: hire someone else to do it, or don't do it. So, the way I structured this newsletter was that any developer advocate on my team could jump in and take over the newsletter so that, you know, if I'm off that week, or whatever may be happening, I, Chris Short, am not the voice. It is now the entire developer advocate team.Corey: I will challenge you on that a bit. Because it's not Chris Short voice, that's for sure, but it's also not official AWS brand voice either.Chris: No.Corey: It is clearly written by a human being who is used to communicating with the audience for whom it is written. And that is no small thing. Normally, when oh, there's a corporate newsletter; that's just a lot of words to say it's bad. This one is good. I want to be very clear on that.Chris: Yeah, I mean, we have just, like, DevOps'ish, we have sections, just like your newsletter, there's certain sections, so any new, what's new announcements, those go in automatically. So, like, that can get delivered to your inbox every Friday. Same thing with new blog posts about anything containers related to EKS, those will be in there, then Containers from the Couch, our streaming platform, essentially, for all things Kubernetes. Those videos go in.And then there's some ecosystem news as well that I collect and put in the newsletter to give people a broader sense of what's going on out there in Kubernetes-land because let's face it, there's upstream and then there's downstream, and sometimes those aren't in sync, and that's normal. That's how Kubernetes kind of works sometimes. If you're running upstream Kubernetes, you are awesome. I appreciate you, but I feel like that would cause more problems and it's worse sometimes.Corey: Thank you for being the trailblazers. The rest of us can learn from your misfortune.Chris: [laugh]. Yeah, exactly. Right? Like, please file your bugs accordingly. [laugh].Corey: EKS is interesting to me because I don't see a lot of it, which is, probably, going to get a whole lot of, “Wait, what?” Moments because wait, don't you deal with very large AWS bills? And I do. But what I mean by that is that EKS, until you're using its Fargate expression, charges for the control plane, which rounds to no money, and the rest is running on EC2 instances running in a company's account. From the billing perspective, there is no difference between, “We're running massive fleets of EKS nodes.” And, “We're managing a whole bunch of EC2 instances by hand.”And that feels like an interesting allegory for how Kubernetes winds up expressing itself to cloud providers. Because from a billing perspective, it just looks like one big single-tenant application that has some really strange behaviors internally. It gets very chatty across AZs when there's no reason to, and whatnot. And it becomes a very interesting study in how to expose aspects of what's going on inside of those containers and inside of the Kubernetes environment to the cloud provider in a way that becomes actionable. There are no good answers for this yet, but it's something I've been seeing a lot of. Like, “Oh, I thought you'd be running Kubernetes. Oh, wait, you are and I just keep forgetting what I'm looking at sometimes.”Chris: So, that's an interesting point. The billing is kind of like, yeah, it's just compute, right? So—Corey: And my insight into AWS and the way I start thinking about it is always from a billing perspective. That's great. It's because that means the more expensive the services, the more I know about it. It's like, “IAM. What is that?” Like, “Oh, I have no idea. It's free. How important could it be?” Professional advice: do not take that philosophy, ever.Chris: [laugh]. No. Ever. No.Corey: Security: it matters. Oh, my God. It's like you're all stars. Your IAM policy should not be. I digress.Chris: Right. Yeah. Anyways, so two points I want to make real quick on that is, one, we've recently released an open-source project called Carpenter, which is really cool in my purview because it looks at your Kubernetes file and says, “Oh, you want this to run on ARM instance.” And you can even go so far as to say, right, here's my limits, and it'll find an instance that fits those limits and add that to your cluster automatically. Run your pod on that compute as long as it needs to run and then if it's done, it'll downsize—eventually, kind of thing—your cluster.So, you can basically just throw a bunch of workloads at it, and it'll auto-detect what kind of compute you will need and then provision it for you, run it, and then be done. So, that is one-way folks are probably starting to save money running EKS is to adopt Carpenter as your autoscaler as opposed to the inbuilt Kubernetes autoscaler. Because this is instance-aware, essentially, so it can say, like, “Oh, your massive ARM application can run here,” because you know, thank you, Graviton. We have those processors in-house. And you know, you can run your ARM64 instances, you can run all the Intel workloads you want, and it'll right size the compute for your workloads.And I'll look at one container or all your containers, however you want to configure it. Secondly, the good folks over at Kubecost have opencost, which is the open-source version of Kubecost, basically. So, they have a service that you can run in your clusters that will help you say, “Hey, maybe this one notes too heavy; maybe this one notes too light,” and you know, give you some insights into Kubernetes spend that are a little bit more granular as far as usage and things like that go. So, those two projects right there, I feel like, will give folks an optimal savings experience when it comes to Kubernetes. But to your point, it's just compute, right? And that's really how we treat it, kind of, here internally is that it's a way to run… compute, Kubernetes, or ECS, or any of those tools.Corey: A fairly expensive one because ignoring entirely for a second the actual raw cost of compute, you also have the other side of it, which is in every environment, unless you are doing something very strange or pre-funding as a one-person startup in your spare time, your payroll costs will it—should—exceed your AWS bill by a fairly healthy amount. And engineering time is always more expensive than services time. So, for example, looking at EKS, I would absolutely recommend people use that rather than rolling their own because—Chris: Rolling their own? Yeah.Corey: —get out of that engineering space where your time is free. I assure you from a business context, it is not. So, there's always that question of what you can do to make things easier for people and do more of the heavy lifting.Chris: Yeah, and to your rather cheeky point that there's 17 ways to run a container on AWS, it is answering that question, right? Like those 17 ways, like, how much of this do you want to run yourself, you could run EKS distro on EC2 instances if you want full control over your environment.Corey: And then run IoT Greengrass core on top within that cluster—Chris: Right.Corey: So, I can run my own Lambda function runtime, so I'm not locked in. Also, DynamoDB local so I'm not locked into AWS. At which point I have gone so far around the bend, no one can help me.Chris: Well—Corey: Pro tip, don't do that. Just don't do that.Chris: But to your point, we have all these options for compute, and specifically containers because there's a lot of people that want to granularly say, “This is where my engineering team gets involved. Everything else you handle.” If I want EKS on Spot Instances only, you can do that. If you want EKS to use Carpenter and say only run ARM workloads, you can do that. If you want to say Fargate and not have anything to manage other than the container file, you can do that.It's how much does your team want to manage? That's the customer obsession part of AWS coming through when it comes to containers is because there's so many different ways to run those workloads, but there's so many different ways to make sure that your team is right-sized, based off the services you're using.Corey: I do want to change gears a bit here because you are mostly known for a couple of things: the DevOps'ish newsletter because that is the oldest and longest thing you've been doing the time that I've known you; EKS, obviously. But when prepping for this show, I discovered you are now co-chair of the OpenGitOps project.Chris: Yes.Corey: So, I have heard of GitOps in the context of, “Oh, it's just basically your CI/CD stuff is triggered by Git events and whatnot.” And I'm sitting here going, “Okay, so from where you're sitting, the two best user interfaces in the world that you have discovered are YAML and Git.” And I just have to start with the question, “Who hurt you?”Chris: [laugh]. Yeah, I share your sentiment when it comes to Git. Not so much with YAML, but I think it's because I'm so used to it. Maybe it's Stockholm Syndrome, maybe the whole YAML thing. I don't know.Corey: Well, it's no XML. We'll put it that way.Chris: Thankfully, yes because if it was, I would have way more, like, just template files laying around to build things. But the—Corey: And rage. Don't forget rage.Chris: And rage, yeah. So, GitOps is a little bit more than just Git in IaC—infrastructure as Code. It's more like Justin Garrison, who's also on my team, he calls it infrastructure software because there's four main principles to GitOps, and if you go to opengitops.dev, you can see them. It's version one.So, we put them on the website, right there on the page. You have to have a declared state and that state has to live somewhere. Now, it's called GitOps because Git is probably the most full-featured thing to put your state in, but you could use an S3 bucket and just version it, for example. And make it private so no one else can get to it.Corey: Or you could use local files: copy-of-copy-of-this-thing-restored-parentheses-use-this-one-dot-final-dot-doc-dot-zip. You know, my preferred naming convention.Chris: Ah, yeah. Wow. Okay. [laugh]. Yeah.Corey: Everything I touch is terrifying.Chris: Yes. Geez, I'm sorry. So first, it's declarative. You declare your state. You store it somewhere. It's versioned and immutable, like I said. And then pulled automatically—don't focus so much on pull—but basically, software agents are applying the desired state from source. So, what does that mean? When it's—you know, the fourth principle is implemented, continuously reconciled. That means those software agents that are checking your desired state are actually putting it back into the desired state if it's out of whack, right? So—Corey: You're talking about agents running it persistently on instances, validating—Chris: Yes.Corey: —a checkpoint on a cron. How is this meaningfully different than a Puppet agent running in years past? Having spent I learned to speak publicly by being a traveling trainer for Puppet; same type of model, and in fact, when I was at Pinterest, we wound up having a fair bit—like, that was their entire model, where they would have—the Puppet's code would live in an S3 bucket that was then copied down, I believe, via Git, and then applied to the instance on a schedule. Like, that sounds like this was sort of a early days GitOps.Chris: Yeah, exactly. Right? Like so it's, I like to think of that as a component of GitOps, right? DevOps, when you talk about DevOps in general, there's a lot of stuff out there. There's a lot of things labeled DevOps that maybe are, or maybe aren't sticking to some of those DevOps core things that make you great.Like the stuff that Nicole Forsgren writes about in books, you know? Accelerate is on my desk for a reason because there's things that good, well-managed DevOps practices do. I see GitOps as an actual implementation of DevOps in an open-source manner because all the tooling for GitOps these days is open-source and it all started as open-source. Now, you can get, like, Flux or Argo—Argo, specifically—there's managed services out there for it, you can have Flux and not maintain it, through an add-on, on EKS for example, and it will reconcile that state for you automatically. And the other thing I like to say about GitOps, specifically, is that it moves at the speed of the Kubernetes Audit Log.If you've ever looked at a Kubernetes audit log, you know it's rather noisy with all these groups and versions and kinds getting thrown out there. So, GitOps will say, “Oh, there's an event for said thing that I'm supposed to be watching. Do I need to change anything? Yes or no? Yes? Okay, go.”And the change gets applied, or, “Hey, there's a new Git thing. Pull it in. A change has happened inGit I need to update it.” You can set it to reconcile on events on time. It's like a cron or it's like an event-driven architecture, but it's combined.Corey: How does it survive the stake through the heart of configuration management? Because before I was doing all this, I wasn't even a T-shaped engineer: you're broad across a bunch of things, but deep in one or two areas, and one of mine was configuration management. I wrote part of SaltStack, once upon a time—Chris: Oh.Corey: —due to a bunch of very strange coincidences all hitting it once, like, I taught people how to use Puppet. But containers ultimately arose and the idea of immutable infrastructure became a thing. And these days when we were doing full-on serverless, well, great, I just wind up deploying a new code bundle to the Lambdas function that I wind up caring about, and that is a immutable version replacement. There is no drift because there is no way to log in and change those things other than through a clear deployment of this as the new version that goes out there. Where does GitOps fit into that imagined pattern?Chris: So, configuration management becomes part of your approval process, right? So, you now are generating an audit log, essentially, of all changes to your system through the approval process that you set up as part of your, how you get things into source and then promote that out to production. That's kind of the beauty of it, right? Like, that's why we suggest using Git because it has functions, like, requests and issues and things like that you can say, “Hey, yes, I approve this,” or, “Hey, no, I don't approve that. We need changes.” So, that's kind of natively happening with Git and, you know, GitLab, GitHub, whatever implementation of Git. There's always, kind of—Corey: Uh, JIF-ub is, I believe, the pronunciation.Chris: JIF-ub? Oh.Corey: Yeah. That's what I'm—Chris: Today, I learned. Okay.Corey: Exactly. And that's one of the things that I do for my lasttweetinaws.com Twitter client that I build—because I needed it, and if other people want to use it, that's great—that is now deployed to 20 different AWS commercial regions, simultaneously. And that is done via—because it turns out that that's a very long to execute for loop if you start down that path—Chris: Well, yeah.Corey: I wound up building out a GitHub Actions matrix—sorry a JIF-ub—actions matrix job that winds up instantiating 20 parallel builds of the CDK deploy that goes out to each region as expected. And because that gets really expensive with native GitHub Actions runners for, like, 36 cents per deploy, and I don't know how to test my own code, so every time I have a typo, that's another quarter in the jar. Cool, but that was annoying for me so I built my own custom runner system that uses Lambda functions as runners running containers pulled from ECR that, oh, it just runs in parallel, less than three minutes. Every time I commit something between I press the push button and it is out and running in the wild across all regions. Which is awesome and also terrifying because, as previously mentioned, I don't know how to test my code.Chris: Yeah. So, you don't know what you're deploying to 20 regions sometime, right?Corey: But it also means I have a pristine, re-composable build environment because I can—Chris: Right.Corey: Just automatically have that go out and the fact that I am making a—either merging a pull request or doing a direct push because I consider main to be my feature branch as whenever something hits that, all the automation kicks off. That was something that I found to be transformative as far as a way of thinking about this because I was very tired of having to tweak my local laptop environment to, “Oh, you didn't assume the proper role and everything failed again and you broke it. Good job.” It wound up being something where I could start developing on more and more disparate platforms. And it finally is what got me away from my old development model of everything I build is on an EC2 instance, and that means that my editor of choice was Vim. I use the VS Code now for these things, and I'm pretty happy with it.Chris: Yeah. So, you know, I'm glad you brought up CDK. CDK gives you a lot of the capabilities to implement GitOps in a way that you could say, like, “Hey, use CDK to declare I need four Amazon EKS clusters with this size, shape, and configuration. Go.” Or even further, connect to these EKS clusters to RDS instances and load balancers and everything else.But you put that state into Git and then you have something that deploys that automatically upon changes. That is infrastructure as code. Now, when you say, “Okay, main is your feature branch,” you know, things happen on main, if this were running in Kubernetes across a fleet of clusters or the globe-wide in 20 regions, something like Flux or Argo would kick in and say, “There's been a change to source, main, and we need to roll this out.” And it'll start applying those changes. Now, what do you get with GitOps that you don't get with your configuration?I mean, can you rollback if you ever have, like, a bad commit that's just awful? I mean, that's really part of the process with GitOps is to make sure that you can, A, roll back to the previous good state, B, roll forward to a known good state, or C, promote that state up through various environments. And then having that all done declaratively, automatically, and immutably, and versioned with an audit log, that I think is the real power of GitOps in the sense that, like, oh, so-and-so approve this change to security policy XYZ on this date at this time. And that to an auditor, you just hand them a log file on, like, “Here's everything we've ever done to our system. Done.” Right?Like, you could get to that state, if you want to, which I think is kind of the idea of DevOps, which says, “Take all these disparate tools and processes and procedures and culture changes”—culture being the hardest part to adopt in DevOps; GitOps kind of forces a culture change where, like, you can't do a CAB with GitOps. Like, those two things don't fly. You don't have a configuration management database unless you absolutely—Corey: Oh, you CAB now but they're all the comments of the pull request.Chris: Right. Exactly. Like, don't push this change out until Thursday after this other thing has happened, kind of thing. Yeah, like, that all happens in GitHub. But it's very democratizing in the sense that people don't have to waste time in an hour-long meeting to get their five minutes in, right?Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: So, would it be overwhelmingly cynical to suggest that GitOps is the means to implement what we've all been pretending to have implemented for the last decade when giving talks at conferences?Chris: Ehh, I wouldn't go that far. I would say that GitOps is an excellent way to implement the things you've been talking about at all these conferences for all these years. But keep in mind, the technology has changed a lot in the, what 11, 12 years of the existence of DevOps, now. I mean, we've gone from, let's try to manage whole servers immutably to, “Oh, now we just need to maintain an orchestration platform and run containers.” That whole compute interface, you go from SSH to a Docker file, that's a big leap, right?Like, you don't have bespoke sysadmins; you have, like, a platform team. You don't have DevOps engineers; they're part of that platform team, or DevOps teams, right? Like, which was kind of antithetical to the whole idea of DevOps to have a DevOps team. You know, everybody's kind of in the same boat now, where we see skill sets kind of changing. And GitOps and Kubernetes-land is, like, a platform team that manages the cluster, and its state, and health and, you know, production essentially.And then you have your developers deploying what they want to deploy in when whatever namespace they've been given access to and whatever rights they have. So, now you have the potential for one set of people—the platform team—to use one set of GitOps tooling, and your applications teams might not like that, and that's fine. They can have their own namespaces with their own tooling in it. Like, Argo, for example, is preferred by a lot of developers because it has a nice UI with green and red dots and they can show people and it looks nice, Flux, it's command line based. And there are some projects out there that kind of take the UI of Argo and try to run Flux underneath that, and those are cool kind of projects, I think, in my mind, but in general, right, I think GitOps gives you the choice that we missed somewhat in DevOps implementations of the past because it was, “Oh, we need to go get cloud.” “Well, you can only use this cloud.” “Oh, we need to go get this thing.” “Well, you can only use this thing in-house.”And you know, there's a lot of restrictions sometimes placed on what you can use in your environment. Well, if your environment is Kubernetes, how do you restrict what you can run, right? Like you can't have an easily configured say, no open-source policy if you're running Kubernetes. [laugh] so it becomes, you know—Corey: Well, that doesn't stop some companies from trying.Chris: Yeah, that's true. But the idea of, like, enabling your developers to deploy at will and then promote their changes as they see fit is really the dream of DevOps, right? Like, same with production and platform teams, right? I want to push my changes out to a larger system that is across the globe. How do I do that? How do I manage that? How do I make sure everything's consistent?GitOps gives you those ways, with Kubernetes native things like customizations, to make consistent environments that are robust and actually going to be reconciled automatically if someone breaks the glass and says, “Oh, I need to run this container immediately.” Well, that's going to create problems because it's deviated from state and it's just that one region, so we'll put it back into state.Corey: It'll be dueling banjos, at some point. You'll try and doing something manually, it gets reverted automatically. I love that pattern. You'll get bored before the computer does, always.Chris: Yeah. And GitOps is very new, right? When you think about the lifetime of GitOps, I think it was coined in, like, 2018. So, it's only four years old, right? When—Corey: I prefer it to ChatOps, at least, as far as—Chris: Well, I mean—Corey: —implementation and expression of the thing.Chris: —ChatOps was a way to do DevOps. I think GitOps—Corey: Well, ChatOps is also a way to wind up giving whoever gets access to your Slack workspace root in production.Chris: Mmm.Corey: But that's neither here nor there.Chris: Mm-hm.Corey: It's yeah, we all like to pretend that's not a giant security issue in our industry, but that's a topic for another time.Chris: Yeah. And that's why, like, GitOps also depends upon you having good security, you know, and good authorization and approval processes. It enforces that upon—Corey: Yeah, who doesn't have one of those?Chris: Yeah. If it's a sole operation kind of deal, like in your setup, your case, I think you kind of got it doing right, right? Like, as far as GitOps goes—Corey: Oh, to be clear, we are 11 people and we do have dueling pull requests and all the rest.Chris: Right, right, right.Corey: But most of the stuff I talk about publicly is not our production stuff, so it really is just me. Just as a point of clarity there. I've n—the 11 people here do not all—the rest of you don't just sit there and clap as I do all the work.Chris: Right.Corey: Most days.Chris: No, I'm sure they don't. I'm almost certain they don't clap… for you. I mean, they would—Corey: No. No, they try and talk me out of it in almost every case.Chris: Yeah, exactly. So, the setup that you, Corey Quinn, have implemented to deploy these 20 regions is kind of very GitOps-y, in the sense that when main changes, it gets updated. Where it's not GitOps-y is what if the endpoint changes? Does it get reconciled? That's the piece you're probably missing is that continuous reconciliation component, where it's constantly checking and saying, “This thing out there is deployed in the way I want it. You know, the way I declared it to be in my source of truth.”Corey: Yeah, when you start having other people getting involved, there can—yeah, that's where regressions enter. And it's like, “Well, I know where things are so why would I change the endpoint?” Yeah, it turns out, not everyone has the state of the entire application in their head. Ideally it should live in—Chris: Yeah. Right. And, you know—Corey: —you know, Git or S3.Chris: —when I—yeah, exactly. When I think about interactions of the past coming out as a new DevOps engineer to work with developers, it's always been, will developers have access to prod or they don't? And if you're in that environment with—you're trying to run a multi-billion dollar operation, and your devs have direct—or one Dev has direct access to prod because prod is in his brain, that's where it's like, well, now wait a minute. Prod doesn't have to be only in your brain. You can put that in the codebase and now we know what is in your brain, right?Like, you can almost do—if you document your code, well, you can have your full lifecycle right there in one place, including documentation, which I think is the best part, too. So, you know, it encourages approval processes and automation over this one person has an entire state of the system in their head; they have to go in and fix it. And what if they're not on call, or in Jamaica, or on a cruise ship somewhere kind of thing? Things get difficult. Like, for example, I just got back from vacation. We were so far off the grid, we had satellite internet. And let me tell you, it was hard to write an email newsletter where I usually open 50 to 100 tabs.Corey: There's a little bit of internet out Californ-ie way.Chris: [laugh].Corey: Yeah it's… it's always weird going from, like, especially after pandemic; I have gigabit symmetric here and going even to re:Invent where I'm trying to upload a bunch of video and whatnot.Chris: Yeah. Oh wow.Corey: And the conference WiFi was doing its thing, and well, Verizon 5G was there but spotty. And well, yeah. Usual stuff.Chris: Yeah. It's amazing to me how connectivity has become so ubiquitous.Corey: To the point where when it's not there anymore, it's what do I do with myself? Same story about people pushing back against remote development of, “Oh, I'm just going to do it all on my laptop because what happens if I'm on a plane?” It's, yeah, the year before the pandemic, I flew 140,000 miles domestically and I was almost never hamstrung by my ability to do work. And my only local computer is an iPad for those things. So, it turns out that is less of a real world concern for most folks.Chris: Yeah I actually ordered the components to upgrade an old Nook that I have here and turn it into my, like, this is my remote code server, that's going to be all attached to GitHub and everything else. That's where I want to be: have Tailscale and just VPN into this box.Corey: Tailscale is transformative.Chris: Yes. Tailscale will change your life. That's just my personal opinion.Corey: Yep.Chris: That's not an AWS opinion or anything. But yeah, when you start thinking about your network as it could be anywhere, that's where Tailscale, like, really shines. So—Corey: Tailscale makes the internet work like we all wanted to believe that it worked.Chris: Yeah. And Wireguard is an excellent open-source project. And Tailscale consumes that and puts an amazingly easy-to-use UI, and troubleshooting tools, and routing, and all kinds of forwarding capabilities, and makes it kind of easy, which is really, really, really kind of awesome. And Tailscale and Kubernetes—Corey: Yeah, ‘network' and ‘easy' don't belong in the same sentence, but in this case, they do.Chris: Yeah. And trust me, the Kubernetes story in Tailscale, there is a lot of there. I understand you might want to not open ports in your VPC, maybe, but if you use Tailscale, that node is just another thing on your network. You can connect to that and see what's going on. Your management cluster is just another thing on the network where you can watch the state.But it's all—you're connected to it continuously through Tailscale. Or, you know, it's a much lighter weight, kind of meshy VPN, I would say, if I had to sum it up in one sentence. That was not on our agenda to talk about at all. Anyways. [laugh]Corey: No, no. I love how many different topics we talk about on these things. We'll have to have you back soon to talk again. I really want to thank you for being so generous with your time. If people want to learn more about what you're up to and how you view these things, where can they find you?Chris: Go to ChrisShort.net. So, Chris Short—I'm six-four so remember, it's Short—dot net, and you will find all the places that I write, you can go to devopsish.com to subscribe to my newsletter, which goes out every week. This year. Next year, there'll be breaks. And then finally, if you want to follow me on Twitter, Chris Short: at @ChrisShort on Twitter. All one word so you see two s's. Like, it's okay, there's two s's there.Corey: Links to all of that will of course be in the show notes. It's easier for people to do the clicky-clicky thing as a general rule.Chris: Clicky things are easier than the wordy things, yes.Corey: Says the Kubernetes guy.Chris: Yeah. Says the Kubernetes guy. Yeah, you like that, huh? Like I said, Argo gives you a UI. [laugh].Corey: Thank you [laugh] so much for your time. I really do appreciate it.Chris: Thank you. This has been fun. If folks have questions, feel free to reach out. Like, I am not one of those people that hides behind a screen all day and doesn't respond. I will respond to you eventually.Corey: I'm right here, Chris. Come on, come on. You're calling me out in front of myself. My God.Chris: Egh. It might take a day or two, but I will respond. I promise.Corey: Thanks again for your time. This has been Chris Short, senior developer advocate at AWS. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice and if it's YouTube, click the thumbs-up button. Whereas if you've hated this podcast, same thing, smash the buttons five-star review and leave an insulting comment that is written in syntactically correct YAML because it's just so easy to do.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.