Podcasts about elastic cloud

In this episode of Making Markets, Ash Kulkarni, CEO of Elastic, talks with host Daniel Newman about the company's recent earnings and growth of Elastic Cloud, the increasing demand for its solutions in the areas of search, observability, and security, and how he expects generative AI to help Elastic and other companies get more out of information systems in a much more human, intuitive way. Disclaimer: The Making Markets podcast is for information and entertainment purposes only. Over the course of this podcast, we may talk about companies that are publicly traded and we may even reference that fact and their equity share price, but please do not take anything that we say as a recommendation about what you should do with your investment dollars. We are not investment advisors and we do not ask that you treat us as such. Quick LinksGet Embed PlayerShare on SocialDownload Audio Fi

About ThomasThomas Hazel is Founder, CTO, and Chief Scientist of ChaosSearch. He is a serial entrepreneur at the forefront of communication, virtualization, and database technology and the inventor of ChaosSearch's patented IP. Thomas has also patented several other technologies in the areas of distributed algorithms, virtualization and database science. He holds a Bachelor of Science in Computer Science from University of New Hampshire, Hall of Fame Alumni Inductee, and founded both student & professional chapters of the Association for Computing Machinery (ACM).Links Referenced: ChaosSearch: https://www.chaossearch.io/ Twitter: https://twitter.com/ChaosSearch Facebook: https://www.facebook.com/CHAOSSEARCH/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at AWS AppConfig. Engineers love to solve, and occasionally create, problems. But not when it's an on-call fire-drill at 4 in the morning. Software problems should drive innovation and collaboration, NOT stress, and sleeplessness, and threats of violence. That's why so many developers are realizing the value of AWS AppConfig Feature Flags. Feature Flags let developers push code to production, but hide that that feature from customers so that the developers can release their feature when it's ready. This practice allows for safe, fast, and convenient software development. You can seamlessly incorporate AppConfig Feature Flags into your AWS or cloud environment and ship your Features with excitement, not trepidation and fear. To get started, go to snark.cloud/appconfig. That's snark.cloud/appconfig.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted episode is brought to us by our returning sponsor and friend, ChaosSearch. And once again, the fine folks at ChaosSearch has seen fit to basically subject their CTO and Founder, Thomas Hazel, to my slings and arrows. Thomas, thank you for joining me. It feels like it's been a hot minute since we last caught up.Thomas: Yeah, Corey. Great to be on the program again, then. I think it's been almost a year. So, I look forward to these. They're fun, they're interesting, and you know, always a good time.Corey: It's always fun to just take a look at companies' web pages in the Wayback Machine, archive.org, where you can see snapshots of them at various points in time. Usually, it feels like this is either used for long-gone things and people want to remember the internet of yesteryear, or alternately to deliver sick burns with retorting a “This you,” when someone winds up making an unpopular statement. One of the approaches I like to use it for, which is significantly less nefarious—usually—is looking back in time at companies' websites, just to see how the positioning of the product evolves over time.And ChaosSearch has had an interesting evolution in that direction. But before we get into that, assuming that there might actually be people listening who do not know the intimate details of exactly what it is you folks do, what is ChaosSearch, and what might you folks do?Thomas: Yeah, well said, and I look forward to [laugh] doing the Wayback Time because some of our ideas, way back when, seemed crazy, but now they make a lot of sense. So, what ChaosSearch is all about is transforming customers' cloud object stores like Amazon S3 into an analytical database that supports search and SQL-type use cases. Now, where's that apply? In log analytics, observability, security, security data lakes, operational data, particularly at scale, where you just stream your data into your data lake, connect our service, our SaaS service, to that lake and automagically we index it and provide well-known APIs like Elasticsearch and integrate with Kibana or Grafana, and SQL APIs, something like, say, a Superset or Tableau or Looker into your data. So, you stream it in and you get analytics out. And the key thing is the time-cost complexity that we all know that operational data, particularly at scale, like terabytes and a day and up causes challenges, and we all know how much it costs.Corey: They certainly do. One of the things that I found interesting is that, as I've mentioned before, when I do consulting work at The Duckbill Group, we have absolutely no partners in the entire space. That includes AWS, incidentally. But it was easy in the beginning because I was well aware of what you folks were up to, and it was great when there was a use case that matched of you're spending an awful lot of money on Elasticsearch; consider perhaps migrating some of that—if it makes sense—to ChaosSearch. Ironically, when you started sponsoring some of my nonsense, that conversation got slightly trickier where I had to disclose, yeah our media arm is does have sponsorships going on with them, but that has no bearing on what I'm saying.And if they take their sponsorships away—please don't—then we would still be recommending them because it's the right answer, and it's what we would use if we were in your position. We receive no kickbacks or partner deal or any sort of reseller arrangement because it just clouds the whole conflict of interest perception. But you folks have been fantastic for a long time in a bunch of different ways.Thomas: Well, you know, I would say that what you thought made a lot of sense made a lot of sense to us as well. So, the ChaosSearch idea just makes sense. Now, you had to crack some code, solve some problems, invent some technology, and create some new architecture, but the idea that Elasticsearch is a useful solution with all the tooling, the visualization, the wonderful community around that, was a good place to start, but here's the problem: setting it up, scaling it out, keep it up, when things are happening, things go bump in the night. All those are real challenges, and one of them was just the storaging of the data. Well, what if you could make S3 the back-end store? One hundred percent; no SSDs or HDDs. Makes a lot of sense.And then support the APIs that your tooling uses. So, it just made a lot of sense on what we were trying to do, just no one thought of it. Now, if you think about the Northstar you were talking about, you know, five, six years ago, when I said, transforming cloud storage into an analytical database for search and SQL, people thought that was crazy and mad. Well, now everyone's using Cloud Storage, everyone's using S3 as a data lake. That's not in question anymore.But it was a question five, six, you know, years ago. So, when we met up, you're like, “Well, that makes sense.” It always made sense, but people either didn't think was possible, or were worried, you know, I'll just try to set up an Elastic cluster and deal with it. Because that's what happens when you particularly deal with large-scale implementations. So, you know, to us, we would love the Elastic API, the tooling around it, but what we all know is the cost, the time the complexity, to manage it, to scale it out, just almost want to pull your hair out. And so, that's where we come in is, don't change what you do, just change how you do it.Corey: Every once in a while, I'll talk to a client who's running an Amazon Elasticsearch cluster, and they have nothing but good things to say about it. Which, awesome. On the one hand, part of me wishes that I had some of their secrets, but often what's happened is that they have this down to a science, they have a data lifecycle that's clearly defined and implemented, the cluster is relatively static, so resizes aren't really a thing, and it just works for their use cases. And in those scenarios, like, “Do you care about the bill?” “Not overly. We don't have to think about it.”Great. Then why change? If there's no pain, you're not going to sell someone something, especially when we're talking, this tends to be relatively smaller-scale as well. It's okay, great, they're spending $5,000 a month on it. It doesn't necessarily justify the engineering effort to move off.Now, when you start looking at this, and, “Huh, that's a quarter million bucks a month we're spending on this nonsense, and it goes down all the time,” yeah, that's when it starts to be one of those logical areas to start picking apart and diving into. What's also muddied the waters since the last time we really went in-depth on any of this was it used to be we would be talking about it exactly like we are right now, about how it's Elasticsearch-compatible. Technically, these days, we probably shouldn't be saying it is OpenSearch compatible because of the trademark issues between Elastic and AWS and the Schism of the OpenSearch fork of the Elasticsearch project. And now it feels like when you start putting random words in front of the word search, ChaosSearch fits right in. It feels like your star is rising.Thomas: Yeah, no, well said. I appreciate that. You know, it's funny when Elastic changed our license, we all didn't know what was going to happen. We knew something was going to happen, but we didn't know what was going to happen. And Amazon, I say ironically, or, more importantly, decided they'll take up the open mantle of keeping an open, free solution.Now, obviously, they recommend running that in their cloud. Fair enough. But I would say we don't hear as much Elastic replacement, as much as OpenSearch replacement with our solution because of all the benefits that we talked about. Because the trigger points for when folks have an issue with the OpenSearch or Elastic stack is got too expensive, or it was changing so much and it was falling over, or the complexity of the schema changing, or all the above. The pipelines were complex, particularly at scale.That's both for Elasticsearch, as well as OpenSearch. And so, to us, we want either to win, but we want to be the replacement because, you know, at scale is where we shine. But we have seen a real trend where we see less Elasticsearch and more OpenSearch because the community is worried about the rules that were changed, right? You see it day in, day out, where you have a community that was built around open and fair and free, and because of business models not working or the big bad so-and-so is taking advantage of it better, there's a license change. And that's a trust change.And to us, we're following the OpenSearch path because it's still open. The 600-pound gorilla or 900-pound gorilla of Amazon. But they really held the mantle, saying, “We're going to stay open, we assume for as long as we know, and we'll follow that path. But again, at that scale, the time, the costs, we're here to help solve those problems.” Again, whether it's on Amazon or, you know, Google et cetera.Corey: I want to go back to what I mentioned at the start of this with the Wayback Machine and looking at how things wound up unfolding in the fullness of time. The first time that it snapshotted your site was way back in the year 2018, which—Thomas: Nice. [laugh].Corey: Some of us may remember, and at that point, like, I wasn't doing any work with you, and later in time I would make fun of you folks for this, but back then your brand name was in all caps, so I would periodically say things like this episode is sponsored by our friends at [loudly] CHAOSSEARCH.Thomas: [laugh].Corey: And once you stopped capitalizing it and that had faded from the common awareness, it just started to look like I had the inability to control the volume of my own voice. Which, fair, but generally not mid-sentence. So, I remember those early days, but the positioning of it was, “The future of log management and analytics,” back in 2018. Skipping forward a year later, you changed this because apparently in 2019, the future was already here. And you were talking about, “Log search analytics, purpose-built for Amazon S3. Store everything, ask anything all on your Amazon S3.”Which is awesome. You were still—unfortunately—going by the all caps thing, but by 2020, that wound up changing somewhat significantly. You were at that point, talking for it as, “The data platform for scalable log analytics.” Okay, it's clearly heading in a log direction, and that made a whole bunch of sense. And now today, you are, “The data lake platform for analytics at scale.” So, good for you, first off. You found a voice?Thomas: [laugh]. Well, you know, it's funny, as a product mining person—I'll take my marketing hat off—we've been building the same solution with the same value points and benefits as we mentioned earlier, but the market resonates with different terminology. When we said something like, “Transforming your Cloud Object Storage like S3 into an analytical database,” people were just were like, blown away. Is that even possible? Right? And so, that got some eyes.Corey: Oh, anything is a database if you hold that wrong. Absolutely.Thomas: [laugh]. Yeah, yeah. And then you're saying log analytics really resonated for a few years. Data platform, you know, is more broader because we do more broader things. And now we see over the last few years, observability, right? How do you fit in the observability viewpoint, the stack where log analytics is one aspect to it?Some of our customers use Grafana on us for that lens, and then for the analysis, alerting, dashboarding. You can say that Kibana in the hunting aspect, the log aspects. So, you know, to us, we're going to put a message out there that resonates with what we're hearing from our customers. For instance, we hear things like, “I need a security data lake. I need that. I need to stream all my data. I need to have all the data because what happens today that now, I need to know a week, two weeks, 90 days.”We constantly hear, “I need at least 90 days forensics on that data.” And it happens time and time again. We hear in the observability stack where, “Hey, I love Datadog, but I can't afford it more than a week or two.” Well, that's where we come in. And we either replace Datadog for the use cases that we support, or we're auxiliary to it.Sometimes we have an existing Grafana implementation, and then they store data in us for the long tail. That could be the scenario. So, to us, the message is around what resonates with our customers, but in the end, it's operational data, whether you want to call it observability, log analytics, security analytics, like the data lake, to us, it's just access to your data, all your data, all the time, and supporting the APIs and the tooling that you're using. And so, to me, it's the same product, but the market changes with messaging and requirements. And this is why we always felt that having a search and SQL platform is so key because what you'll see in Elastic or OpenSearch is, “Well, I only support the Elastic API. I can't do correlations. I can't do this. I can't do that. I'm going to move it over to say, maybe Athena but not so much. Maybe a Snowflake or something else.”Corey: “Well, Thomas, it's very simple. Once you learn our own purpose-built, domain-specific language, specifically for our product, well, why are you still sitting here, go learn that thing.” People aren't going to do that.Thomas: And that's what we hear. It was funny, I won't say what the company was, a big banking company that we're talking to, and we hear time and time again, “I only want to do it via the Elastic tooling,” or, “I only want to do it via the BI tooling.” I hear it time and time again. Both of these people are in the same company.Corey: And that's legitimate as well because there's a bunch of pre-existing processes pointing at things and we're not going to change 200 different applications in their data model just because you want to replace a back-end system. I also want to correct myself. I was one tab behind. This year's branding is slightly different: “Search and analyze unlimited log data in your cloud object storage.” Which is, I really like the evolution on this.Thomas: Yeah, yeah. And I love it. And what was interesting is the moving, the setting up, the doubling of your costs, let's say you have—I mean, we deal with some big customers that have petabytes of data; doubling your petabytes, that means, if your Elastic environment is costing you tens of millions and then you put into Snowflake, that's also going to be tens of millions. And with a solution like ours, you have really cost-effective storage, right? Your cloud storage, it's secure, it's reliable, it's Elastic, and you attach Chaos to get the well-known APIs that your well-known tooling can analyze.So, to us, our evolution has been really being the end viewpoint where we started early, where the search and SQL isn't here today—and you know, in the future, we'll be coming out with more ML type tooling—but we have two sides: we have the operational, security, observability. And a lot of the business side wants access to that data as well. Maybe it's app data that they need to do analysis on their shopping cart website, for instance.Corey: The thing that I find curious is, the entire space has been iterating forward on trying to define observability, generally, as whatever people are already trying to sell in many cases. And that has seemed to be a bit of a stumbling block for a lot of folks. I figured this out somewhat recently because I've built the—free for everyone to use—the lasttweetinaws.com, Twitter threading client.That's deployed to 20 different AWS regions because it's go—the idea is that should be snappy for people, no matter where they happen to be on the planet, and I use it for conferences when I travel, so great, let's get ahead of it. But that also means I've got 20 different sources of logs. And given that it's an omnibus Lambda function, it's very hard to correlate that to users, or user sessions, or even figure out where it's going. The problem I've had is, “Oh, well, this seems like something I could instrument to spray logs somewhere pretty easily, but I don't want to instrument it for 15 different observability vendors. Why don't I just use otel—or Open Telemetry—and then tell that to throw whatever I care about to various vendors and do a bit of a bake-off?” The problem, of course, is that open telemetry and Lambda seem to be in just the absolute wrong directions. A lot.Thomas: So, we see the same trend of otel coming out, and you know, this is another API that I'm sure we're going to go all-in on because it's getting more and more talked about. I won't say it's the standard that I think is trending to all your points about I need to normalize a process. But as you mentioned, we also need to correlate across the data. And this is where, you know, there are times where search and hunting and alerting is awesome and wonderful and solves all your needs, and sometimes correlation. Imagine trying to denormalize all those logs, set up a pipeline, put it into some database, or just do a SELECT *, you know, join this to that to that, and get your answers.And so, I think both OpenTelemetry and SQL and search all need to be played into one solution, or at least one capability because if you're not doing that, you're creating some hodgepodge pipeline to move it around and ultimately get your questions answered. And if it takes weeks—maybe even months, depending on the scale—you may sometimes not choose to do it.Corey: One other aspect that has always annoyed me about more or less every analytics company out there—and you folks are no exception to this—is the idea of charging per gigabyte ingested because that inherently sets up a weird dichotomy of, well, this is costing a lot, so I should strive to log less. And that is sort of the exact opposite, not just of the direction you folks want customers to go in, but also where customers themselves should be going in. Where you diverge from an awful lot of those other companies because of the nature of how you work, is that you don't charge them again for retention. And the idea that, yeah, the fact that anything stored in ChaosSearch lives in your own S3 buckets, you can set your own lifecycle policies and do whatever you want to do with that is a phenomenal benefit, just because I've always had a dim view of short-lived retention periods around logs, especially around things like audit logs. And these days, I would consider getting rid of audit logging data and application logging data—especially if there's a correlation story—any sooner than three years feels like borderline malpractice.Thomas: [laugh]. We—how many times—I mean, we've heard it time and time again is, “I don't have access to that data because it was too costly.” No one says they don't want the data. They just can't afford the data. And one of the key premises that if you don't have all the data, you're at risk, particularly in security—I mean, even audits. I mean, so many times our customers ask us, you know, “Hey, what was this going on? What was that go on?” And because we can so cost-effectively monitor our own service, we can provide that information for them. And we hear this time and time again.And retention is not a very sexy aspect, but it's so crucial. Anytime you look in problems with X solution or Y solution, it's the cost of the data. And this is something that we wanted to address, officially. And why do we make it so cost-effective and free after you ingest it was because we were using cloud storage. And it was just a great place to land the data cost-effective, securely.Now, with that said, there are two types of companies I've seen. Everybody needs at least 90 days. I see time and time again. Sure, maybe daily, in a weeks, they do a lot of their operation, but 90 days is where it lands. But there's also a bunch of companies that need it for years, for compliance, for audit reasons.And imagine trying to rehydrate, trying to rebuild—we have one customer—again I won't say who—has two petabytes of data that they rehydrate when they need it. And they say it's a nightmare. And it's growing. What if you just had it always alive, always accessible? Now, as we move from search to SQL, there are use cases where in the log world, they just want to pay upfront, fixed fee, this many dollars per terabyte, but as we get into the more ad hoc side of it, more and more folks are asking for, “Can I pay per query?”And so, you'll see coming out soon, about scenarios where we have a different pricing model. For logs, typically, you want to pay very consistent, you know, predetermined cost structure, but in the case of more security data lakes, where you want to go in the past and not really pay for something until you use it, that's going to be an option as well coming out soon. So, I would say you need both in the pricing models, but you need the data to have either side, right?Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: You'd like to hope. I mean, you could always theoretically wind up just pulling what Ubiquiti apparently did—where this came out in an indictment that was unsealed against an insider—but apparently one of their employees wound up attempting to extort them—which again, that's not their fault, to be clear—but what came out was that this person then wound up setting the CloudTrail audit log retention to one day, so there were no logs available. And then as a customer, I got an email from them saying there was no evidence that any customer data had been accessed. I mean, yeah, if you want, like, the world's most horrifyingly devilish best practice, go ahead and set your log retention to nothing, and then you too can confidently state that you have no evidence of anything untoward happening.Contrast this with what AWS did when there was a vulnerability reported in AWS Glue. Their analysis of it stated explicitly, “We have looked at our audit logs going back to the launch of the service and have conclusively proven that the only time this has ever happened was in the security researcher who reported the vulnerability to us, in their own account.” Yeah, one of those statements breeds an awful lot of confidence. The other one makes me think that you're basically being run by clowns.Thomas: You know what? CloudTrail is such a crucial—particularly Amazon, right—crucial service because of that, we see time and time again. And the challenge of CloudTrail is that storing a long period of time is costly and the messiness the JSON complexity, every company struggles with it. And this is how uniquely—how we represent information, we can model it in all its permutations—but the key thing is we can store it forever, or you can store forever. And time and time again, CloudTrail is a key aspect to correlate—to your question—correlate this happened to that. Or do an audit on two years ago, this happened.And I got to tell you, to all our listeners out there, please store your CloudTrail data—ideally in ChaosSearch—because you're going to need it. Everyone always needs that. And I know it's hard. CloudTrail data is messy, nested JSON data that can explode; I get it. You know, there's tricks to do it manually, although quite painful. But CloudTrail, every one of our customers is indexing with us in CloudTrail because of stories like that, as well as the correlation across what maybe their application log data is saying.Corey: I really have never regretted having extra logs lying around, especially with, to be very direct, the almost ridiculously inexpensive storage classes that S3 offers, especially since you can wind up having some of the offline retrieval stuff as part of a lifecycle policy now with intelligent tiering. I'm a big believer in just—again—the Glacier Deep Archive I've at the cost of $1,000 a month per petabyte, with admittedly up to 12 hours of calling that as a latency. But that's still, for audit logs and stuff like that, why would I ever want to delete things ever again?Thomas: You're exactly right. And we have a bunch of customers that do exactly that. And we automate the entire process with you. Obviously, it's your S3 account, but we can manage across those tiers. And it's just to a point where, why wouldn't you? It's so cost-effective.And the moments where you don't have that information, you're at risk, whether it's internal audits, or you're providing a service for somebody, it's critical data. With CloudTrail, it's critical data. And if you're not storing it and if you're not making it accessible through some tool like an Elastic API or Chaos, it's not worth it. I think, to your point about your story, it's epically not worth it.Corey: It's really not. It's one of those areas where that is not a place to overly cost optimize. This is—I mean we talked earlier about my business and perceptions of conflict of interest. There's a reason that I only ever charge fixed-fee and not percentage of savings or whatnot because, at some point, I'll be placed in a position of having to say nonsense, like, “Do you really need all of these backups?” That doesn't make sense at that point.I do point out things like you have hourly disk snapshots of your entire web fleet, which has no irreplaceable data on them dating back five years. Maybe cleaning some of that up might be the right answer. The happy answer is somewhere in between those two, and it's a business decision around exactly where that line lies. But I'm a believer in never regretting having kept logs almost into perpetuity. Until and unless I start getting more or less pillaged by some particularly rapacious vendor that's oh, yeah, we're going to charge you not just for ingest, but also for retention. And for how long you want to keep it, we're going to treat it like we're carving it into platinum tablets. No. Stop that.Thomas: [laugh]. Well, you know, it's funny, when we first came out, we were hearing stories that vendors were telling customers why they didn't need their data, to your point, like, “Oh, you don't need that,” or, “Don't worry about that.” And time and time again, they said, “Well, turns out we didn't need that.” You know, “Oh, don't index all your data because you just know what you know.” And the problem is that life doesn't work out that way business doesn't work out that way.And now what I see in the market is everyone's got tiering scenarios, but the accessibility of that data takes some time to get access to. And these are all workarounds and bandaids to what fundamentally is if you design an architecture and a solution is such a way, maybe it's just always hot; maybe it's just always available. Now, we talked about tiering off to something very, very cheap, then it's like virtually free. But you know, our solution was, whether it's ultra warm, or this tiering that takes hours to rehydrate—hours—no one wants to live in that world, right? They just want to say, “Hey, on this date on this year, what was happening? And let me go look, and I want to do it now.”And it has to be part of the exact same system that I was using already. I didn't have to call up IT to say, “Hey, can you rehydrate this?” Or, “Can I go back to the archive and look at it?” Although I guess we're talking about archiving with your website, viewing from days of old, I think that's kind of funny. I should do that more often myself.Corey: I really wish that more companies would put themselves in the customers' shoes. And for what it's worth, periodically, I've spoken to a number of very happy ChaosSearch customers. I haven't spoken to any angry ones yet, which tells me you're either terrific at crisis comms, or the product itself functions as intended. So, either way, excellent job. Now, which team of yours is doing that excellent job, of course, is going to depend on which one of those outcomes it is. But I'm pretty good at ferreting out stories on those things.Thomas: Well, you know, it's funny, being a company that's driven by customer ask, it's so easy build what the customer wants. And so, we really take every input of what the customer needs and wants—now, there are cases where we relace Splunk. They're the Cadillac, they have all the bells and whistles, and there's times where we'll say, “Listen, that's not what we're going to do. We're going to solve these problems in this vector.” But they always keep on asking, right? You know, “I want this, I want that.”But most of the feedback we get is exactly what we should be building. People need their answers and how they get it. It's really helped us grow as a company, grow as a product. And I will say ever since we went live now many, many years ago, all our roadmap—other than our Northstar of transforming cloud storage into a search SQL big data analytics database has been customer-driven, market customer-driven, like what our customer is asking for, whether it's observability and integrating with Grafana and Kibana or, you know, security data lakes. It's just a huge theme that we're going to make sure that we provide a solution that meets those needs.So, I love when customers ask for stuff because the product just gets better. I mean, yeah, sometimes you have to have a thick skin, like, “Why don't you have this?” Or, “Why don't you have that?” Or we have customers—and not to complain about customers; I love our customers—but they sometimes do crazy things that we have to help them on crazy-ify. [laugh]. I'll leave it at that. But customers do silly things and you have to help them out. I hope they remember that, so when they ask for a feature that maybe takes a month to make available, they're patient with us.Corey: We sure can hope. I really want to thank you for taking so much time to once again suffer all of my criticisms, slings and arrows, blithe market observations, et cetera, et cetera. If people want to learn more, where's the best place to find you?Thomas: Well, of course, chaossearch.io. There's tons of material about what we do, use cases, case studies; we just published a big case study with Equifax recently. We're in Gartner and a whole bunch of Hype Cycles that you can pull down to see how we fit in the market.Reach out to us. You can set up a trial, kick the tires, again, on your cloud storage like S3. And ChaosSearch on Twitter, we have a Facebook, we have all this classic social medias. But our website is really where all the good content and whether you want to learn about the architecture and how we've done it, and use cases; people who want to say, “Hey, I have a problem. How do you solve it? How do I learn more?”Corey: And we will, of course, put links to that in the show notes. For my own purposes, you could also just search for the term ChaosSearch in your email inbox and find one of their sponsored ads in my newsletter and click that link, but that's a little self-serving as we do it. I'm kidding. I'm kidding. There's no need to do that. That is not how we ever evaluate these things. But it is funny to tell that story. Thomas, thank you so much for your time. As always, it's appreciated.Thomas: Corey Quinn, I truly enjoyed this time. And I look forward to upcoming re:Invent. I'm assuming it's going to be live like last year, and this is where we have a lot of fun with the community.Corey: Oh, I have no doubt that we're about to go through that particular path very soon. Thank you. It's been an absolute pleasure.Thomas: Thank you.Corey: Thomas Hazel, CTO and Founder of ChaosSearch. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry, insulting comment that I will then set to have a retention period of one day, and then go on to claim that I have received no negative feedback.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Wie würde heutzutage ein moderner Logging, Metriken, Monitoring, Alerting und Tracing-Stack aussehen?Im Infrastruktur-Bereich gibt es zu jedem Bereich etliche Tools. Cloud-Native ist das Buzzword der Stunde. In dieser Episode erzählt Andy, wie er einen modernen Stack für ein Side-Projekt für die Bereiche Logging, Metriken, Monitoring, Alerting und Tracing aufsetzen würde. Unter anderem geht es dabei um Fragen wie: Was sollte man eigentlich alles loggen? Wie kann man von einem Alert angerufen werden? Wie visualisiert man Daten in schönen Graphen? Brauchen wir Tracing? Und was ist Observability?Bonus: Engineering Porn und Buzzword-Bingo.Feedback (gerne auch als Voice Message)Email: stehtisch@engineeringkiosk.devTwitter: https://twitter.com/EngKioskWhatsApp +49 15678 136776Gerne behandeln wir auch euer Audio Feedback in einer der nächsten Episoden, einfach Audiodatei per Email oder WhatsApp Voice Message an +49 15678 136776LinksEpisode #37 Mit IT-Büchern Geld verdienen? Wer liest überhaupt noch Bücher?: https://engineeringkiosk.dev/podcast/episode/37-mit-it-b%C3%BCchern-geld-verdienen-wer-liest-%C3%BCberhaupt-noch-b%C3%BCcher/?pkn=shownotes Episode #17 Was können wir beim Incident Management von der Feuerwehr lernen?: https://engineeringkiosk.dev/podcast/episode/17-was-k%C3%B6nnen-wir-beim-incident-management-von-der-feuerwehr-lernen/?pkn=shownotes Sentry: https://sentry.io/Datadog: https://www.datadoghq.com/Splunk: https://www.splunk.com/Elasticsearch: https://www.elastic.co/de/enterprise-search/Logstash: https://github.com/elastic/logstashKibana: https://github.com/elastic/kibanaOpenSearch: https://opensearch.org/Elastic Cloud: https://www.elastic.co/de/cloud/Aiven: https://aiven.io/Fluentd: https://www.fluentd.org/Amazon S3 und S3 Glacier: https://aws.amazon.com/de/s3/Amazon Athena: https://aws.amazon.com/de/athena/Prometheus: https://prometheus.io/VictoriaMetrics: https://github.com/VictoriaMetrics/VictoriaMetricsInfluxDB: https://www.influxdata.com/M3 Metrics Engine: https://m3db.io/Prometheus Node Exporter: https://github.com/prometheus/node_exporterGrafana: https://github.com/grafana/grafanaPromQL: https://prometheus.io/docs/prometheus/latest/querying/basics/OpsGenie: https://www.atlassian.com/de/software/opsgenieJaeger: https://www.jaegertracing.io/Zipkin: https://zipkin.io/OpenTracing: https://opentracing.io/OpenTelemetry: https://opentelemetry.io/yak shaving: https://seths.blog/2005/03/dont_shave_that/Cloud Native Computing Foundation: https://www.cncf.io/Sprungmarken(00:00:00) Intro(00:00:50) Wolfgangs MySQL-Buch(00:02:11) Heutiges Thema: Wie würde Andy die Themen Monitoring, Alerting, Metriken und Logging bei einem Side Projekt angehen?(00:04:49) Warum brauchst du Logging, Monitoring, Metriken und Tracing?(00:07:29) Logging von Exceptions, Warnings und anderen Fehler, Logging und der ELK-Stack(00:16:06) Was sollte man eigentlich alles loggen?(00:19:22) Log-Rotation und Log-Retention auf Object-Storage(00:27:30) Metriken mit Prometheus(00:31:46) Visualisierung von Metriken mit Grafana(00:34:25) Intelligente Alerting Systeme und die richtigen Schwellenwerte finden(00:38:47) Alerts senden und anrufen lassen(00:43:22) Tracing: Was ist das und brauchen wir das?(00:48:49) Was ist Observability?(00:51:42) Iterativer Aufbau seiner Plattform und Alternativen(00:54:49) Keine bezahlte Werbung(00:55:14) Outro und FeedbackHostsWolfgang Gassler (https://twitter.com/schafele)Andy Grunwald (https://twitter.com/andygrunwald)Feedback (gerne auch als Voice Message)Email: stehtisch@engineeringkiosk.devTwitter: https://twitter.com/EngKioskWhatsApp +49 15678 136776

Robby has a chat with Courtney Wilburn (She/Her/Hers), the Sr. Engineering Manager at Elastic Cloud, the leading platform for search-powered solutions. She is an experienced DevOps Engineer, speaker, and writer. With solutions in enterprise search, observability, and security, Elastic helps enhance customer and employee search experiences, keep mission-critical applications running smoothly, and protect against cyber threats. For Courtney, well-maintained software is all about software having a good community around it that is enthusiastic about its long-term success. She shares her expertise on the traits of excellent documentation and talks about how engineers should go about joining a software team. Courtney uses the metaphor technical debt and she will graciously break down how her team discusses, prioritizes, and documents what and when they focus on it. She also talks about the challenges that come with process debt, how to go about hiring junior-level engineers, and what we can do to foster mentorship in our teams. It's going to be a very interesting conversation so don't miss out.Book Recommendations:Emergent Strategy: Shaping Change, Changing Worlds By Adrienne Maree BrownHelpful LinksCourtney on TwitterCourney's WebsiteCourtney on LinkedInSubscribe to Maintainable on:Apple PodcastsOvercastSpotifyOr search "Maintainable" wherever you stream your podcasts.Join the discussion in the Maintainable Discord Community

We do love AWS, but sometimes we have to admit that it's not always a silver bullet. There are definitely use cases where it might be worth considering alternatives to AWS. In this episode we will discuss some of these use cases and try to highlight what are the advantages that other platforms or services can have over AWS in very specific circumstances. First of all we clarify why we like AWS and why (and when) it's worth sticking with it. Then, we discuss what are some of the reasons why it might be worth considering alternatives to AWS. At this point we go into the specifics and talk about authentication services (Auth0), search services (ElasticSearch, Algolia), CDN Services (GitHub Pages, Netlify, Vercel, CloudFlare, Fastly, Akamai), Databases (MongoDB Atlas, Digital Ocean managed databases, IBM Compose, CloudFlare D1, Upstash, Confluent Kafka), Headless CMS services (ContentFul, Storyful, AirTable, Google Spreadsheet), Virtual Machine services (Digital Ocean, Linode). In this episode, we mentioned the following resources: - Episode 3. "​​How do you deploy a static website on AWS?”: https://awsbites.com/3-how-do-you-deploy-a-static-website-on-aws/ - Auth0: https://auth0.com/ - Amazon OpenSearch: https://aws.amazon.com/opensearch-service/the-elk-stack/what-is-opensearch/ - Elastic Cloud: https://www.elastic.co/cloud/ - Algolia: https://www.algolia.com/ - Vercel: https://vercel.com/ - Netlify: https://www.netlify.com/ - MongoDB Atlas: https://www.mongodb.com/atlas/database - Digital Ocean managed database: https://try.digitalocean.com/managed-databases/ - Compose (now IBM Cloud Databases): https://www.compose.com/ - Upstash: https://upstash.com/ - Confluent: https://www.confluent.io/ - AirTable: https://airtable.com/ - Linode: https://www.linode.com/ This episode is also available on YouTube: https://www.youtube.com/AWSBites You can listen to AWS Bites wherever you get your podcasts: - Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-bites/id1585489017 - Spotify: https://open.spotify.com/show/3Lh7PzqBFV6yt5WsTAmO5q - Google: https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy82YTMzMTJhMC9wb2RjYXN0L3Jzcw== - Breaker: https://www.breaker.audio/aws-bites - RSS: ​​https://anchor.fm/s/6a3312a0/podcast/rss Do you have any AWS questions you would like us to address? Connect with us on Twitter: - https://twitter.com/eoins - https://twitter.com/loige

Siguiendo con nuestra serie de capítulos orientada a servicio Cloud, llega el turno de Elastic Cloud, con la inestimable colaboración de Oscar Cabanillas, Solution Architect en Elastic. Música: Aliaksei Yuknhevich - Background Ukulele Makesound - Ambient Motivational Inspirational

About TomaszTomasz is a Frontend Engineer at Stedi, Co-Founder/Head of React at Cloudash, egghead.io instructor with over 200 lessons published, a tech speaker, an AWS Community Hero and a lifelong learner.Links Referenced: Cloudash: https://cloudash.dev/ Twitter: https://twitter.com/tlakomy TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It's always a pleasure to talk to people who ask the bold questions. One of those great bold questions is, what if CloudWatch's web page didn't suck? It's a good question. It's one I ask myself all the time.And then I stumbled across a product that wound up solving this for me, and I'm a happy customer. To be clear, they're not sponsoring anything that I do, nor should they. It's one of those bootstrapped, exciting software projects called Cloudash. Today, I'm joined by the Head of React at Cloudash, Tomasz Łakomy. Tomasz, thank you for joining me.Tomasz: It's a pleasure to be here.Corey: So, where did this entire idea come from? Because I sit and I get upset every time I have to go into the CloudWatch dashboard because first, something's broken. In an ideal scenario, I don't have to care about monitoring or observability or anything like that. But then it's quickly overshadowed by the fact that this interface is terrible. And the reason I know it's terrible is that every time I'm in there, I feel dumb.My belief is—for the longest time, I thought that was a problem with me. But no, invariably, when you wind up working with something and consistently finding it a bad—you don't know enough to solve for it, it's not you. It is, in fact, the signs of a poorly designed experience, start to finish. “You should be smarter to use this tool,” is very rarely correct. And there are a bunch of observability tools and monitoring tools for serverless things that have made sense over the years and made this easier, but one of the most—and please don't take this the wrong way—stripped down, bare essentials of just the facts, style of presentation is Cloudash. It's why I continue to pay for it every month with a smile on my face. How did you get here from there?Tomasz: Yeah that's a good question. I would say that. Cloudash was born out of desire for simple things to be simple. So, as you mentioned, Cloudash is basically the monitoring and troubleshooting tool for serverless applications, made for serverless developers because I am very much into serverless space, as is Maciej Winnicki, who is the another half of Cloudash team. And, you know, the whole premise of serverless was things are going to be simpler, right?So, you know, you have a bunch of code, you're going to dump it into a Lambda function, and that's it. You don't have to care about servers, you don't have to care about, you know, provisioning stuff, you don't have to care about maintenance, and so on. And that is not exactly true because why PagerDuty still continues to be [unintelligible 00:02:56] business even in serverless spaces. So, you will get paged every now and then. The problem is—what we kind of found is once you have an incident—you know, PagerDuty always tends to call it in the middle of the night; it's never, like, 11 a.m. during the workday; it's always the middle of the night.Corey: And no one's ever happy when it calls them either. It's, “Ah, hell.” Whatever it rings, it's yeah, the original Call of Duty. PagerDuty hooked up to Nagios. I am old enough to remember those days.Tomasz: [unintelligible 00:03:24] then business, like, imagine paying for something that's going to wake you up in the middle of the night. It doesn't make sense. In any case—Corey: “So, why do you pay for that product? Because it's really going to piss me off.” “Okay, well… does that sound like a good business to you? Well, AWS seems to think so. No one's happy working with that stuff.” “Fair. Fair enough.”Tomasz: So, in any case, like we've established an [unintelligible 00:03:43]. So you wake up, you go to AWS console because you saw a notification that this-and-this API has, you know, this threshold was above it, something was above the threshold. And then you go to the CloudWatch console. And then you see, okay, those are the logs, those are the metrics. I'm going to copy this request ID. I'm going to go over here. I'm going to go to X-Ray.And again, it's 3 a.m. so you don't exactly remember what do you investigate; you have, like, ten minutes. And this is a problem. Like, we've kind of identified that it's not simple to do these kinds of things, too—it's not simple to open something and have an understanding, okay, what exactly is happening in my serverless app at this very moment? Like, what's going on?So, we've built that. So, Cloudash is a desktop app; it lives on your machine, which is a single pane of glass. It's a single pane of glass view into your serverless system. So, if you are using CloudFormation in order to provision something, when you open Cloudash, you're going to see, you know, all of the metrics, all the Lambda functions, all of the API Gateways that you have provisioned. As of yesterday, API Gateway is no longer cool because they did launch the direct integration, so you have—you can call Lambda functions with [crosstalk 00:04:57]—Corey: Yeah, it's the one they released, and then rolled back and somehow never said a word—because that's an AWS messaging story, and then some—right around re:Invent last year. And another quarter goes by and out it goes.Tomasz: It's out yesterday.Corey: Yeah, it's terrific. I love that thing. The only downside to it is, ah, you have to use one of their—you have to use their domain; no custom domain support. Really? Well, you can hook up CloudFront to it, but the pricing model that way makes it more expensive than API Gateway.Okay, so I could use Cloudflare in front of it, and then it becomes free, so I bought a domain just for that purpose. That's right, my serverl—my direct Lambda URLs now live behind the glorious domain of cheapass.cloud because of course. They are. It's a day-one product from AWS, so of course, it's not feature-complete.But one of the things I like about the serverless model, and it's also a challenge when it comes to troubleshooting stuff is that it's very much set it and forget it style because serverless in many cases, at least the way that I tend to use it, is back-office stuff, its back-end things, it's processing on things that are not necessarily always direct front and center. So, these things can run on their own for years until finally, you find a strange bug in a new use case, or you want to go and change something. And then it's how the hell did this ever work? And it's still working, kind of, but what fool built this? Of course, it was me; it's always me.But what happened here? You're basically excavating your own legacy code, trying to understand what's going on. And so, you're already upset then. Cloudash makes this easier to find the things, to navigate through a whole bunch of different accounts. And there are a bunch of decisions that you made while building the app that are so clearly correct, that I get actively annoyed when others don't because oh, it looks at your AWS configuration file in your user home directory. Great, awesome. It's a desktop app, but it still consults that file. Yay, integration between ClickOps and the terminal. Wonderful.But ah, use SSO for a lot of stuff, so that's going to fix your little red wagon. I click on that app, and suddenly, bam, a browser opens asking me to log in and authenticate, allow the request. It works, and then suddenly, it goes back to doing exactly what you'd expect it to. It's really nice. The affordances behind this are glorious.Tomasz: Like I said, one of our kind of design goals when building Cloudash was to make simple things simple again. The whole purpose is to make sure that you can get into the root cause of an issue within, like, five minutes, if not less. And this is kind of the app that you're going to tend to open whenever that—as I said, because some of the systems can be around for, like, ages, literally without any incident whatsoever, then the data is going to change because somebody [unintelligible 00:07:30] got that the year is 2020 and off you go, we have an incident.But what's important about Cloudash is that we don't send logs anywhere. And that's kind of important because you don't pay for [PUT 00:07:42] metric API because we are not sending those logs anywhere. If you install Cloudash on your machine, we are not going to get your logs from the last ten years, put them in into a system, charge you for that, just so you are able to, you know, find out what happened in this particular hour, like, two weeks ago. We genuinely don't care about your logs; we have enough of our own logs at work to, you know, to analyze, to investigate, and so on; we are not storing them anywhere.In fact, you know, whatever happens on your machine stays on the machine. And that is partially why this is a desktop app. Because we don't want to handle your credentials. We don't—absolutely, we don't want you to give us any of your credentials or access keys, you know, whatever. We don't want that.So, that is why you install Cloudash, it's going to run on your machine, it's going to use your local credentials. So, it's… effectively, you could say that this is a much more streamlined and much more laser-focused browser or like, an eye into AWS systems, which live on the serverless side of things.Corey: I got to deal with it in a bit of an interesting way, recently. I have a detector in my company's production AWS org, to detect when ClickOps is afoot. Now, I'm a big proponent of ClickOps, but I also want to know what's going on, so I have a whole thing that [runs detects 00:09:04] when people are doing things in the console versus via API. And it alerts on certain subsets of them. I had to build a special case for the user agent string coming out of Cloudash because no, no, this is an app, this is not technically ClickOps—it is also read-only, which is neither here nor there, to my understanding.But it was, “Oh yeah, this is effectively an Electron app.” It just wraps, effectively, a browser and presents that as an application. And cool. From my perspective, that's an implementation detail. It feels like a native app—because it is—and I can suddenly see the things I care about in a way that is much more straightforward without having to have four different browser tabs open where, okay, here's the CloudTrail log for this thing, here's the metrics next to it. Oh, those are two separate windows already, and so on and so forth. It just makes hunting down to the obnoxious problems so much nicer.It's also, you're one of those rare products where if I don't use it for a month, I don't get the bill at the end of the month and think, “Ooh, that's going to—did I waste the money?” It's no, nice. I had a whole month where I didn't have to mess with this. It's great.Tomasz: Exactly. I feel like, you know, it's one of those systems where, as you said, we send you an email at the end of every month that we're going to charge you X dollars for the month—by the way, we have fixed pricing and then you can cancel anytime—and it's like one of those things that, you know, I didn't have to open this up for a month. This is awesome because I didn't have any incidents. But I know whenever again, PagerDuty is going to decide, “Hey, dude, wake up. You know, if slept for three hours. That is definitely long enough,” then you know that; you know, this app is there and you can use that.We very much care about, you know, building this stuff, not only for our customers, but we also use that on a daily basis. In fact, I… every single time that I have to—I want to investigate something in, like, our serverless systems at Stedi because everything that we do at work, at Stedi, since this incident serverless paradigm. So, I tend to open Cloudash, like, 95% of the time whenever I want to investigate something. And whenever I am not able to do something in Cloudash, this goes, like, straight to the top of our, you know, issue lists or backlog or whatever you want to call it. Because we want to make this product, not only awesome, you know, for customers to buy a [unintelligible 00:11:22] or whatever, but we also want to be able to use that on a daily basis.And so far, I think we've kind of succeeded. But then again, we have quite a long way to go because we have more ideas, than we have the time, definitely, so we have to kind of prioritize what exactly we're going to build. So, [unintelligible 00:11:39] integrations with alarms. So, for instance, we want to be able to see the alarms directly in the Cloudash UI. Secondly, integration with logs insights, and many other ideas. I could probably talk for hours about what we want to build.Corey: I also want to point out that this is still your side gig. You are by day a front-end engineer over at Stedi, which has a borderline disturbing number of engineers with side gigs, generally in the serverless space, doing interesting things like this. Dynobase is another example, a DynamoDB desktop client; very similar in some respects. I pay for that too. Honestly, for a company in Stedi's space, which is designed as basically a giant API for deep, large enterprise business stuff, there's an awful lot of stuff for small-scale coming out of that.Like, I wind up throwing a disturbing amount of money in the general direction of Stedi for not being their customer. But there's something about the culture that you folks have built over there that's just phenomenal.Tomasz: Yeah. For the record, you know, having a side gig is another part of interview process at Stedi. You don't have to have [laugh] a side project, but yeah, you're absolutely right, you know, the amount of kind of side projects, and you know, some of those are monetized, as you mentioned, you know, Cloudash and Dynobase and others. Some of those—because for instance, you talked to Aidan, I think a couple of weeks ago about his shenanigans, whenever you know, AWS is going to announce something he gets in and try to [unintelligible 00:13:06] this in the most amusing ways possible. Yeah, I mean, I could probably talk for ages about why Stedi is by far the best company I've ever worked at, but I'm going to say this: that this is the most talented group of people I've ever met, and myself, honestly.And, you know, the fact that I think we are the second largest, kind of, group of AWS experts outside of AWS because the density of AWS Heroes, or ex-AWS employees, or people who have been doing cloud stuff for years, is frankly, massive, I tend to learn something new about cloud every single day. And not only because of the Last Week in AWS but also from our Slack.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: There's something to be said for having colleagues that you learn from. I have never enjoyed environments where I did not actively feel like the dumbest person in the room. That's why I love what I do now. I inherently am. I have to talk about so many different things, that whenever I talk to a subject matter expert, it is a certainty that they know more about the thing than I do, with the admitted and depressing exception of course of the AWS bill because it turns out the reason I had to start becoming the expert in that was because there weren't any. And here we are now.I want to talk as well about some of—your interaction outside of work with AWS. For example, you've been an Egghead instructor for a while with over 200 lessons that you published. You're an AWS Community Hero, which means you have the notable distinction of volunteering for a for-profit company—good work—no, the community is very important. It's helping each other make sense of the nonsense coming out of there. You've been involved within the ecosystem for a very long time. What is it about, I guess—the thing I'm wondering about myself sometimes—what is it about the AWS universe that drew you in, and what keeps you here?Tomasz: So, give you some context, I've started, you know, learning about the cloud and AWS back in early-2019. So, fun fact: Maciej Winnicki—again, the co-founder of Cloudash—was my manager at the time. So, we were—I mean, the company I used to work for at the time, OLX Group, we are in the middle of cloud transformation, so to speak. So, going from, you know, on-premises to AWS. And I was, you know, hired as a senior front-end engineer doing, you know, all kinds of front-end stuff, but I wanted to grow, I wanted to learn more.So, the idea was, okay, maybe you can get AWS Certified because, you know, it's one of those corporate goals that you have to have something to put that checkbox next to it. So, you know, getting certified, there you go, you have a checkbox. And off you go. So, I started, you know, diving in, and I saw this whole ocean of things that, you know, I was not entirely aware of. To be fair, at the time I knew about this S3, I knew that you can put a file in an S3 bucket and then you can access it from the internet. This is, like, the [unintelligible 00:16:02] idea of my AWS experiences.Corey: Ideally, intentionally, but one wonders sometimes.Tomasz: Yeah, exactly. That is why you always put stuff as public, right? Because you didn't have to worry about who [unintelligible 00:16:12] [laugh] public [unintelligible 00:16:15]. No, I'm kidding, of course. But still, I think what's [unintelligible 00:16:20] to AWS is what—because it is this endless ocean of things to learn and things to play with, and, you know, things to teach.I do enjoy teaching. As you said, I have quite a lot of, you know, content, videos, blog posts, conference talks, and a bunch of other stuff, and I do that for two reasons. You know, first of all, I tend to learn the best by teaching, so it helps me very much, kind of like, solidify my own knowledge. Whenever I record—like, I have two courses about CDK, you know, when I was recording those, I definitely—that kind of solidify my, you know, ideas about CDK, I get to play with all those technologies.And secondly, you know, it's helpful for others. And, you know, people have opinions about certificates, and so on and so forth, but I think that for somebody who's trying to get into either the tech industry or, you know, cloud stuff in general, being certified helps massively. And I've heard stories about people who are basically managed to double or triple their salaries by going into tech, you know, with some of those certificates. That is why I strongly believe, by the way, that those certificates should be free. Like, if you can pass the exam, you shouldn't have to worry about this $150 of the fee.Corey: I wrote a blog post a while back, “The Dumbest Dollars a Cloud Provider Can Make,” and it's charging for training and certification because if someone's going to invest that kind of time in learning your platform, you're going to try and make $150 bucks off them? Which in some cases, is going to put people off from even beginning that process. “What cloud provider I'm not going to build a project on?” Obviously, the one I know how to work with and have a familiarity with, in almost every case. And the things you learn in your spare time as an independent learner when you get a job, you tend to think about your work the same way. It matters. It's an early on-ramp that pays off down the road and the term of years.I used to be very anti-cert personally because it felt like I was jumping through hoops, and paying, in some cases, for the privilege. I had a CCNA for a while from Cisco. There were a couple of smaller companies, SaltStack, for example, that I got various certifications from at different times. And that was sort of cheating because I helped write the software, but that's neither here nor there. It's the—and I do have a standing AWS cert that I get a different one every time—mine is about to expire—because it gets me access to lounges at physical events, which is the dumbest of all reasons to get certs, but here you go. I view it as the $150 lounge pass with a really weird entrance questionnaire.But in my case it certs don't add anything to what I do. I am not the common case. I am not early in my career. Because as you progress through your career, things—there needs to be a piece of paper that says you know things, and early on degree or certifications are great at that. In the time it becomes your own list of experience on your resume or CV or LinkedIn or God knows what. Polywork if you're doing it the right way these days.And it shows a history of projects that are similar in scope and scale and impact to the kinds of problems that your prospective employer is going to have to solve themselves. Because the best answer to hear—especially in the ops world—when there's a problem is, “Oh, I've seen this before. Here's how you fix it.” As opposed to, “Well, I don't know. Let me do some research.”There's value to that. And I don't begrudge anyone getting certs… to a point. At least that's where I sit on it. At some point when you have 25 certs, it's when you actually do any work? Because it's taking the tests and learning all of these things, which in many ways does boil down to trivia, it stands in counterbalance to a lot of these things.Tomasz: Yeah. I mean, I definitely, totally agree. I remember, you know, going from zero to—maybe not Hero; I'm not talking about AWS Hero—but going from zero to be certified, there was the Solutions Architect Associate. I think it took me, like, 200 hours. I am not the, you know, the brightest, you know, the sharpest tool in the shed, so it probably took me, kind of, somewhat more.I think it's doable in, like, 100 hours, but I tend to over-prepare for stuff, so I didn't actually take the actual exam until I was able to pass the sample exams with, like, 90% pass, just to be extra sure that I'm actually going to pass it. But still, I think that, you know, at some point, you probably should focus on, you know, getting into the actual stuff because I hold two certificates, you know, one of those is going to expire, and I'm not entirely sure if I want to go through the process again. But still, if AWS were to introduce, like, a serverless specialty exam, I would be more than happy to have that. I genuinely enjoy, kind of, serverless, and you know, the fact that I would be able to solidify my knowledge, I have this kind of established path of the things that I should learn about in order to get this particular certificate, I think this could be interesting. But I am not probably going to chase all the 12 certificates.Maybe if AWS IQ was available in Poland, maybe that would change because I do know that with IQ, those certs do matter. But as of [unintelligible 00:21:26] now, I'm quite happy with my certs that I have right now.Corey: Part of the problem, too, is the more you work with these things, the harder it becomes to pass the exams, which sounds weird and counterintuitive, but let me use myself as an example. When I got the cloud practitioner cert, which I believe has lapsed since then, and I got one of the new associate-level betas—I'll keep moving up the stack until I start failing exams. But I got a question wrong on the cloud practitioner because it was, “How long does it take to restore an RDS database from a snapshot backup?” And I gave the honest answer of what I've seen rather than what it says in the book, and that honest answer can be measured in days or hours. Yeah.And no, that's not the correct answer. Yeah, but it is the real one. Similarly, a lot of the questions get around trivia, syntax of which of these is the correct argument, and which ones did we make up? It's, I can explain in some level of detail, virtually every one of AWS has 300 some-odd services to you. Ask me about any of them, I could tell you what it is, how it works, how it's supposed to work and make a dumb joke about it. Fine, whatever.You'll forgive me if I went down that path, instead of memorizing what is the actual syntax of this YAML construct inside of a CloudFormation template? Yeah, I can get the answer to that question in the real world, with about ten seconds of Googling and we move on. That's the way most of us learn. It's not cramming trivia into our heads. There's something broken about the way that we do certifications, and tech interviews in many cases as well.I look back at some of the questions I used to ask people for Linux sysadmin-style jobs, and I don't remember the answer to a lot of these things. I could definitely get back into it, but if I went through one of these interviews now, I wouldn't get the job. One would argue I shouldn't because of my personality, but that's neither here nor there.Tomasz: [laugh]. I mean, that's why you use CDK, so you'd have to remember random YAML comments. And if you [unintelligible 00:23:26] you don't have YAML anymore. [unintelligible 00:23:27].Corey: Yes, you're quite the CDK fanboy, apparently.Tomasz: I do like CDK, yes. I don't like, you know, mental overhead, I don't like context switching, and the way we kind of work at Stedi is everything is written in TypeScript. So, I am a front-end engineer, so I do stuff in the front-end line in TypeScript, all of our Lambda functions are written in TypeScript, and our [unintelligible 00:23:48] is written in TypeScript. So, I can, you know, open up my Visual Studio Code and jump between all of those files, and the language stays the same, the syntax stays the same, the tools stay the same. And I think this is one of the benefits of CDK that is kind of hard to replicate otherwise.And, you know, people have many opinions about the best to deploy infrastructure in the cloud, you know? The best infrastructure-as-code tool is the one that you use at work or in your private projects, right? Because some people enjoy ClickOps like you do; people—Corey: Oh yeah.Tomasz: Enjoy CloudFormation by hand, which I don't; people are very much into Terraform or Serverless Framework. I'm very much into CDK.Corey: Or the SAM CLI, like, three or four more, and I use—Tomasz: Oh, yeah. [unintelligible 00:24:33]—Corey: —all of these things in various ways in some of my [monstrous 00:24:35] projects to keep up on all these things. I did an exploration with the CDK. Incidentally, I think you just answered why I don't like it.Tomasz: Because?Corey: Because it is very clear that TypeScript is a first-class citizen with the CDK. My language of choice is shitty bash because, grumpy old sysadmin; it happens. And increasingly, that is switching over to terrible Python because I'm very bad at that. And the problem that I run into as I was experimenting with this is, it feels like the Python support is not fully baked, most people who are using the CDK are using a flavor of JavaScript and, let's be very clear here, the every time I have tried to explore front-end, I have come away more confused than I was when I started, part of me really thinks I should be learning some JavaScript just because of its versatility and utility to a whole bunch of different problems. But it does not work the way I think, on some level, that it should because of my own biases and experiences. So, if you're not a JavaScript person, I think that you have a much rockier road with the CDK.Tomasz: I agree. Like I said, I tend to talk about my own experiences and my kind of thoughts about stuff. I'm not going to say that, you know, this tool or that tool is the best tool ever because nothing like that exists. Apart from jQuery, which is the best thing that ever happened to the web since, you know, baked bread, honestly. But you are right about CDK, to the best of my knowledge, kind of, all the other languages that are supported by CDK are effectively transpiled down from TypeScript. So it's, like, first of all, it is written in TypeScript, and then kind of the Python, all of the other languages… kind of come second.You know, and afterwards, I tend to enjoy CDK because as I said, I use TypeScript on a daily basis. And you know, with regards to front-end, you mentioned that you are, every single time you is that you end up being more confused. It never goes away. I've been doing front-end stuff for years, and it's, you know, kind of exactly the same. Fun story, I actually joined Cloudash because, well, Maciej started working on Cloudash alone, and after quite some time, he was so frustrated with the modern front-end landscape that he asked me, “Dude, you need to help me. Like, I genuinely need some help. I am tired of React. I am tired of React hooks. This is way too complex. I want to go back to doing back-end stuff. I want to go back, you know, thinking about how we're going to integrate with all those APIs. I don't want to do UI stuff anymore.”Which was kind of like an interesting shift because I remember at the very beginning of my career, where people were talking about front-end—you know, “Front-end is not real programming. Front-end is, you know, it's easy, it's simple. I can learn CSS in an hour.” And the amount of people who say that CSS is easy, and are good at CSS is exactly zero. Literally, nobody who's actually good at CSS says that, you know, CSS, or front-end, or anything like that is easy because it's not. It's incredibly complex. It's getting probably more and more complex because the expectations of our front-end UIs [unintelligible 00:27:44].Corey: It's challenging, it is difficult, and one of the things I find most admirable about you is not even your technical achievements, it's the fact that you're teaching other people to do this. In fact, this gets to the last point I want to cover on our conversation today. When I was bouncing topic ideas off of you, one of the points you brought up that I'm like, “Oh, we're keeping that and saving that for the end,” is why—to your words—why speaking at tech events gets easier, but never easy. Let's dive into that. Tell me more about it.Tomasz: Basically, I've accidentally kickstarted my career by speaking at meetups which later turned into conferences, which later turned into me publishing courses online, which later turned into me becoming an AWS Hero, and here we are, you know, talking to each other. I do enjoy, you know, going out in public and speaking and being on stage. I think, you know, if somebody has, kind of, the heart, the ability to do that, I do strongly recommend, you know, giving it a shot, not only to give, like, an honestly life-changing experience because the first time you go in front of hundreds of people, this is definitely, you know, something that's going to shake you, while at the same time acknowledging that this is absolutely, definitely not for everyone. But if you are able to do that, I think this is definitely worth your time. But as you said—by quoting me—that it gets easier, so every single time you go on stage, talk at a meetup or at a conference or online conferences—which I'm not exactly a fan of, for the record—it's—Corey: It's too much like work, too much like meetings. There's nothing different about it.Tomasz: Yeah, exactly. Like, there's no journey. There's no adventure in online conferences. I know that, of course, you know, given all of that, you know, we had to kind of switch to online conferences for quite some time where I think we are pretending that Covid is not a thing anymore, so we, you know, we're effectively going back, but kind of the point I wanted to make is that I am a somewhat experienced public speaker—I'd like to say that because I've been doing that for years—but I've been, you know, talking to people who actually get paid to speak at the conferences, to actually kind of do that for a living, and they all say the same thing. It gets simpler, it gets easier, but it's never freaking easy, you know, to go out there, and you know, to share whatever you've learned.Corey: I'm one of those people. I am a paid public speaker fairly often, even ignoring the podcast side, and I've spoken on conference stages a couple hundred times at least. And it does get easier but never easy. That's a great way of framing it. You… I get nervous before every talk I give.There are I think two talks I've given that I did not have an adrenaline hit and nervous energy before I went onstage, and both of those were duds. Because I think that it's part of the process, at least for me. And it's like, “Oh, how do you wind up not being scared for before you go on stage?” You don't. You really don't.But if that appeals to you and you enjoy the adrenaline rush of the rest, do it. If you're one of those people who've used public speaking as, “I would prefer death over that,” people are more scared of public speaking their death, in some cases, great. There are so many ways to build audiences and to reach people that fine, if you don't like doing it on stage, don't force yourself to. I'd say try it once; see how it feels meetups are great for this.Tomasz: Yeah. Meetups are basically the best way to get started. I'm yet to meet a meetup, either, you know, offline or online, who is not looking for speakers. It's always quite the opposite, you know? I was, you know, co-organizing a meetup in my city here in Poznań, Poland, and the story always goes like this: “Okay, we have a date. We have a venue. Where are the speakers?” And then you know, the tumbleweed is going to roll across the road and, “Oh, crap, we don't have any speakers.” So, we're going to try to find some, reach out to people. “Hey, I know that you did this fantastic project at your workplace. Come to us, talk about this.” “No, I don't want to. You know, I'm not an expert. I am, you know, I have on the 50 years of experience as an engineer. This is not enough.” Like I said, I do strongly recommend it, but as you said, if you're more scared of public speaking than, like, literally dying, maybe this is not for you.Corey: Yeah. It comes down to stretching your limits, finding yourself interesting. I find that there are lots of great engineers out there. The ones that I find myself drawn to are the ones who aren't just great at building something, but at storytelling around the thing that they are built of, yes, you build something awesome, but you have to convince me to care about it. You have to show me the thing that got you excited about this.And if you can't inspire that excitement in other people, okay. Are you really excited about it? Or what is the story here? And again, it's a different skill set. It is not for everyone, but it is absolutely a significant career accelerator if it's leveraged right.Tomasz: [crosstalk 00:32:45].Corey: [crosstalk 00:32:46] on it.Tomasz: Yeah, absolutely. I think that we don't talk enough about, kind of, the overlap between engineering and marketing. In the good sense of marketing, not the shady kind of marketing. The kind of marketing that you do for yourself in order to elevate yourself, your projects, your successes to others. Because, you know, try as you might, but if you are kind of like sitting in the corner of an office, you know, just jamming on your keyboard 40 hours per week, you're not exactly likely to be promoted because nobody's going to actively reach out to you to find out about your, you know, recent successes and so on.Which at the same time, I'm not saying that you should go @channel in Slack every single time you push a commit to the main branch, but there's definitely, you know, a way of being, kind of, kind to yourself by letting others know that, “Okay, I'm here. I do exist, I have, you know, those particular skills that you may be interested about. And I'm able to tell a story which is, you know, convincing.” So it's, you know, you can tell a story on stage, but you can also tell your story to your customers by building a future that they're going to use. [unintelligible 00:33:50].Corey: I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place to find you?Tomasz: So, the best place to find me is on Twitter. So, my Twitter handle is @tlakomy. So, it's T-L-A-K-O-M-Y. I'm assuming this is going to be in the [show notes 00:34:06] as well.Corey: Oh, it absolutely is. You beat me to it.Tomasz: [laugh]. So, you can find Cloudash at cloudash.dev. You can probably also find my email, but don't email me because I'm terrible, absolutely terrible at email, so the best way to kind of reach out to me is via my Twitter DMs. I'm slightly less bad at those.Corey: Excellent. And we will, of course, put links to that in the [show notes 00:34:29]. Thank you so much for being so generous with your time. I appreciate it.Tomasz: Thank you. Thank you for having me.Corey: Tomasz Łakomy, Head of React at Cloudash. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, and if you're on the YouTubes, smash the like and subscribe button, as the kids say. Whereas if you've hated this episode, please do the exact same thing—five-star reviews smash the buttons—but this time also leave an insulting and angry comment written in the form of a CloudWatch log entry that no one is ever able to find in the native interface.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About MichaelMichael is the creator of IT automation platforms Cobbler and Ansible, the latter allegedly used by ~60% of the Fortune 500, and at one time one of the top 10 contributed to projects on GitHub.Links Referenced: Speaking Tech: https://michaeldehaan.substack.com/ michaeldehaan.net: https://michaeldehaan.net Twitter: https://twitter.com/laserllama TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by our friends at Revelo. Revelo is the Spanish word of the day, and its spelled R-E-V-E-L-O. It means “I reveal.” Now, have you tried to hire an engineer lately? I assure you it is significantly harder than it sounds. One of the things that Revelo has recognized is something I've been talking about for a while, specifically that while talent is evenly distributed, opportunity is absolutely not. They're exposing a new talent pool to, basically, those of us without a presence in Latin America via their platform. It's the largest tech talent marketplace in Latin America with over a million engineers in their network, which includes—but isn't limited to—talent in Mexico, Costa Rica, Brazil, and Argentina. Now, not only do they wind up spreading all of their talent on English ability, as well as you know, their engineering skills, but they go significantly beyond that. Some of the folks on their platform are hands down the most talented engineers that I've ever spoken to. Let's also not forget that Latin America has high time zone overlap with what we have here in the United States, so you can hire full-time remote engineers who share most of the workday as your team. It's an end-to-end talent service, so you can find and hire engineers in Central and South America without having to worry about, frankly, the colossal pain of cross-border payroll and benefits and compliance because Revelo handles all of it. If you're hiring engineers, check out revelo.io/screaming to get 20% off your first three months. That's R-E-V-E-L-O dot I-O slash screaming.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: Once upon a time, Docker came out and change an entire industry forever. But believe it or not, for many of you, this predates your involvement in the space. There was a time where we had to manage computer systems ourselves with our hands—kind of—like in the prehistoric days, chiseling bits onto disk and whatnot. It was an area crying out for automation, as we started using more and more computers to run various websites. “Oh, that's a big website. It needs three servers now.” Et cetera.The times have changed rather significantly. One of the formative voices in that era was Michael DeHaan, who's joining me today, originally one of the—or if not the creator of Cobbler, and later—for which you became better known—Ansible. First, thanks for joining me.Michael: Thank you for having me. You're also making me feel very, very old there. So, uh, yes.Corey: I hear you. I keep telling people, I'm in my mid-30s, and my wife gets incensed because I'm turning 40 in July. But still. I go for the idea of yeah, the middle is expanding all the time, but it's always disturbing talking to people who are in our sector, who are younger than some of the code that we're using, which is just bizarre to me. We're all standing on the backs of giants. Like it or not, one of them's you.Michael: Oh, well, thank you. Thank you very much. Yeah, I was, like, talking to some undergrads, I was doing a little bit of stuff helping out my alma mater for a little bit, and teaching somebody the REST lecture. I was like, “In another year, REST is going to be older than everybody in the room.” And then I was just kind of… scared.Corey: Yeah. It's been a wild ride for basically everyone who's been around long enough if you don't fall off the teeter-totter and wind up breaking a limb somewhere. So, back in the bad old days, before cloud, when everything was no longer things back then were constrained by how much room you had on your credit card like they are today with cloud, but instead by things like how much space you had in the data center, what kind of purchase order you could ram through your various accounting departments. And one of the big problems you have is, great. So, finally—never on time—Dell has shipped out a whole bunch of servers—or HP or Supermicro or whoever—and the remote hands—which is always distinct from smart hands, which says something very insulting, but they seem to be good about it—would put them into racks for you.And great, so you'd walk in and see all of these brand new servers with nothing on them. How do we go ahead and configure these things? And by hand was how most of us started, and that means, oh, great, we're going to screw things up and not do them all quite the same, and it's just a treasure and a joy. Cobbler was something that you came up with that revolutionized how provisioning of bare-metal systems worked. Tell me about it.Michael: Yeah, um, so it's basically just glue. So, the story of how I came up with that is I was working for the Emerging Technologies Group at Red Hat, and I just joined. And they were like, “We have to have a solution to install Xen and KVM virtual machines.” So obviously, everybody's familiar with, like, EC2 and things now, but this was about people running non-VMware virtualization themselves. So, that was part of the problem, but in order to make that interesting, we really needed to have some automation around bare-metal installs.And that's PXE boot. So, it's TFTP and DHCP protocol and all that kind of boring stuff. And there was glue that existed, but it was usually humans would have to click on buttons to—like Red Hat had system-config-netboot, but what really happened was sysadmins all wrote their own automation at, like, every single company. And the idea that I had, and it was sort of cemented by the fact that, like, my boss, a really good guy left for another company and I didn't have a boss for, like, a couple years, was like, I'm just going to make IRC my boss, and let's get all these admins together and build a tool we can share, right?So, that was a really good experience, and it's just basically gluing all that stuff together to fully automate an install over a network so that when a system comes on, you can either pick it out from a menu; or maybe you've already got the MAC address and you can just say, “When you see this MAC address, go install this operating system.” And there's a kickstart file, or a preseed in the case of Debian, that says, “When you're booting up through the installer, basically, here's just the answers and go do these things.” And that install processes a lot slower than what we're used to, but for a bare-metal machine, that's a pretty good way to do it.Corey: Yeah, it got to a point where you could walk through and just turn on all the servers in a rack and go out to lunch, come back, they would all be configured and ready to go. And it sounds relatively basic the way we're talking about it now, but there were some gnarly cases. Like, “When I've rebooted the database server, why did it wipe itself and reprovision?” And it's, “Oh, dear.” And you have to make sure that things are—that there's a safety built into these things.And you also don't want to have to wind up plugging in a keyboard and monitor to all of these individual machines one-by-one to hit yes and acknowledge the thing. And it was a colossal pain in the ass. That's one of the things that cloud has freed us from.Michael: Yeah, definitely. And one of the nice things about the whole cloud environment is like, if you want to experiment with those ideas, like, I want to set up some DHCP or DNS, I don't have to have this massive lab and all the electricity and costs. But like, if I want to play with a load balancer, I can just get one. That kind of gives the experience of playing with all these data center technologies to everybody, which is pretty cool.Corey: On some level, you can almost view the history of all these things as speeding things up. With a well-tuned Cobbler install, it still took multiple minutes, in some cases, tens of minutes to go from machine you're powering on to getting it provisioned and ready to go. Virtual machines dropped that down to minutes. And cloud, of course, accelerated that a bit. But then you wind up with things like Docker and it gets down to less than a second. It's the meantime to dopamine.But in between the world of containers and bare-metal, there was another project—again, the one you're best known for—Ansible. Tell me about that because I have opinions on this whole space.Michael: [laugh]. Yeah. So, how Ansible got started—well, I guess configuration management is pretty old, so the people writing their own scripts, CFEngine came out, Puppet was a much better CFEngine. I was working at a company and I kind of wanted another open-source project because I enjoyed the Cobbler experience. So, I started Ansible on the side, kind of based on some frustrations around Puppet but also the desire to unify Capistrano kind of logic, which was like, “How do I push out my apps onto these servers that are already running,” with Puppet-style logic was like, “Is this computer's firewall configured directly? And is the time set correctly?”And you can obviously use that to install apps, but there's some places where that blurred together where a lot of people are using two different tools. And there's some prior art that I worked on called Funk, which I wrote with Seth Vidal and Adrian Likins at Red Hat, which was, like, 50% of the Ansible idea, and we just never built the config management layer on top. So, the idea was make something really, really simple that just uses SSH, which was controversial at the time because people thought it, like, wouldn't scale, because I was having trouble with setting up Puppet security because, like, it had DNS or timing issues or whatever.Corey: Yeah. Let's dive in a bit to what config management is first because it turns out that not everyone was living in the trenches in quite the same way that we were. I was a traveling trainer for Puppet for a summer once, and the best descriptor I found to explain it to people who are not in this space was, “All right, let's say that you go and you buy a new computer. What do you do? Well, you're going to install the applications you'd like to use, you're going to set up your own user account, you're going to set your password correctly, you're going to set up preferences, copy some files over so you have the stuff you care about. Great. Now, imagine you need to do that to a thousand computers and they all need to be the same. How do you do that?” Well, that is the world of configuration management.And there was sort of a bifurcation there, where there was the idea of, first, we're going to have configuration management that just describes what the system should look like, and that's going to run on a schedule or whatnot, and then you're going to have the other side of it, which is the idea of remote execution, of I want to run an arbitrary command on this server, or this set of servers, or all the servers, depending upon what it is. And depending on where you started on the side of that world, you wound up wanting things from the other side of that space. With Puppet, for example, is very oriented configuration management and the question became, well, can you use this for remote execution with arbitrary commands? And they wound up doing some work with Mcollective, which was a very complicated and expensive way to say, “No, not really.” There was a need for things that needed to hang out in that space.The two that really stuck out from that era were Ansible, which had its wild runaway success, and the one that I was smacking around for a bit, SaltStack, which never saw anywhere approaching that level of popularity.Michael: Yeah, sure. I mean, I think that you hit it pretty much exactly right. And it's hard to say what makes certain things take off, but I think, like, the just SSH approach was interesting because, well for one, everybody's running it. But there was this belief that this would not scale. And I tried to optimize the heck out of that because I liked performance, but it turns out that wasn't really a business problem because if you can imagine you just wrote this little bit of automation, and you're going to run it against your entire infrastructure and you've got 30,000 machines, do you want that to—if you were to, like, run an update command on 30,000 machines at once, you're going to DDoS something. Definitely, right?Corey: Yeah. Suddenly you have 30,000 machines all talk to the same things at the same times. And you want to do them in batches or smear it across.Michael: Right, so because that was there, like, you just add batch support in Ansible and things are fine, right? People want to target little small groups of things. So, like, that whole story wasn't true, and I think it was just a matter of testing this belief that everybody thought that we needed to have this whole network of things. And honestly, Salt's idea of using a message bus is great, but we took a little bit different approach with YAML because we have YAML variables in it, but they had something that compiled down to YAML. And I think those are some differences in the dialect and some things other people preferred, but—Corey: And they use Jinja, at one point to wind up making it effectively Turing complete; you could wind up having this ridicu—like, loop flow control and loops and the rest. And it was an interesting exposure to things, but yikes, at some l—at the same time.Michael: If you use all the language features in anything you can make something complicated, and too complicated. And I was like, I wanted automation to look like grocery lists. And when I started out, I said, “Hey, if anybody is doing this all day, for a day job, I will have failed.” And it clearly shows you that I have because there are people that are doing that all day. And the goal was, let me concentrate on dev and ops and my other things and keep this really, really simple.And some people just, like, get really, really into that automation technology, which is—in my opinion—why some of the earlier stuff was really popular because sysadmin were bored, so they see something new and it's kind of like a Java developer finding Perl for the first time. They're like, “I'm going to use all these things.” And they have all their little widgets, and it gets, like, really complicated.Corey: The thing that I always found interesting and terrifying at the same time about Ansible was the fact that you did ride on top of SSH, which is great because every company already had a way of controlling access by SSH to IT systems; everyone uses it, so it has an awful lot of eyes on the security protocol on the rest. The thing that I found terrifying in the early days was that more or less every ops person would wind up checking this out onto their laptop or whatnot, so whenever they wanted to run something, they would just run it from their laptop over a VPN or whatnot from wherever they happen to be, and you wind up with a dueling banjos type of circumstance where people were often not doing it from a centralized place. And in time, best practices emerged where, okay, that is going to be the command and control server where that runs at, and you log into it. And then you start guarding that with CI/CD flows and the rest. And like anything else, it wound up building some operational approaches to it.Michael: Yeah. Like, I kind of think that created a problem that allowed us to sell a product, right, which was good. If you knew what you were doing, you could use Jenkins completely and you'd be fine, right, if you had some level of discipline and access control, and you wanted to wire that up. And if you think about cloud, this whole, like, shadow IT idea of, “I just want to do this thing, therefore I'm going to get an Amazon account,” it's kind of the same thing. It's like, “I want to use this config management, but it's not approved. Who can stop me?” Right?And that kind of probably got us in the door in few accounts that way. But yeah, it did definitely create the problem where multiple people could be running things at the same time. So yeah, I mean, that's true.Corey: And the idea of, “Hey, maybe I should be controlling these things in Git,” or some other form of version control was sort of one of those evolutionary ideas that, oh, we could treat this like code. And the early days of DevOps, that was a controversial thing. These days, you say you're not doing it and people look at you very strangely. And things were going reasonably well in that direction for a while. Then this whole Docker thing showed up, where, well, what if instead of having these long-lived servers where you have to install updates and run patches and maintain a whole user list on them, instead you had this immutable infrastructure that every time there was a change, you would just go ahead and deploy a brand new set of servers?And you could do this in the olden days with virtual machines and whatnot; it just took a long time to push things out, so do I really want to roll the entire fleet for a two-line config change? Probably not, so we're going to batch it up, or maybe do this hybrid model. With Docker, it takes less than a second to wind up provisioning the—switching over to the new container series and you're done; you can keep going with that. That really solved a lot of these problems.But there were companies that, like, the entire configuration management space, who suddenly found themselves in a really weird position. Some of them tried to fight the tide forever and say, “Oh, this is terrible because it means we don't have a business model anymore.” But you can only fight the future for so long. And I think today, we'd be hard-pressed to say that Docker hasn't won, on some level.Michael: I mean, I think it has, like, the technology has won. But I guess the interesting thing is, config management now seems to be trying to pivot towards networking where I think the tool hasn't ever been designed for networking, so it's kind of a round peg, square hole. But it's all people have that unless they're buying something. Or, like, deploying the undercloud because, like, people are still running essentially clouds on top of clouds to get their Kubernetes deployments going and those are monstrous. Or maybe to deploy a data layer; like, I know Kafka has gotten off of ZooKeeper, but the Kafka-ZooKeeper thing—and I don't remember ZooKeeper [unintelligible 00:14:37] require [unintelligible 00:14:38] or not, but managing those sort of long, persistent implications, it still has a little bit of a place where it exists.But I mean, I think the whole immutable systems idea is theoretically completely great. I never was really happy with the whole Docker development workflow, and I think it does create a problem where people don't know what they're deploying and you kind of encourage that to where they could be deploying different versions of libraries, or—and that's kind of just a problem of the whole microservices thing in general where, “Did somebody change this?” And then I was working very briefly at one company where we essentially built a whole dashboard to detect service versions and what version of the base image everybody was on, and all these other things, and it can get out of hand, too. So, it's kind of like trading some problems for other problems, I think to me. But in general, containerization is good. I just wished the management glue around it was easy, right?Corey: I wound up giving a talk at a conference a while back, 2015 or so, called, “Heresy in the Church of Docker,” and it was a throwaway five-minute lightning talk, and someone approached me afterwards with, “Hey, can you give the full version of that at ContainerCon?” “There's a full version? Yes. Yes, I can.” And it talked about a number of problems with the management layer and the rest.Now, Kubernetes absolutely solves virtually every problem that I identified with it, but when you look at the other side of it, getting Kubernetes rolled out is effectively you get to cosplay being a cloud provider yourself. It is incredibly complicated, and of course, we're right back to managing it all with YAML.Michael: Right. And I think that's an interesting point, too, is I don't know who's exactly responsible for, like, the YAML explosion. And I like it as a data format; it's really good for humans. Cobbler originally used it more of an internal storage, which I think was a mistake because, like, even—I was trying to avoid setting up a database at the time, so—because I knew if I had to require setting up a database in 2007 or 2008, I'd get way less users, so it used flat files.A lot of the YAML dialects people are developing now are very, very nested and they requires, like, loading a webpage, for the Docks, like, all the time and reading what's valid here, what's valid there. I think people learn the wrong lesson from Ansible's YAML usage, right? It was supposed to be, like, YAML's good for things that are grocery lists. And there's a lot of places where I didn't do a good job. But when you see methods taking 15 parameters and you have to constantly have the reference up, maybe that's a sign that you should do something else.Corey: At least you saved us, on some level, from having to do this all in XML. But still, there are wrong ways and more wrong ways to do it. I don't think anyone could ever agree on the right way to approach these things.Michael: Yeah. I mean, and YAML, at the time was a good answer because I knew I didn't want to write and maintain a parser as, like, a guy that was running a project. We had a lot of awesome contributors, but if I had to also maintain a DSL, not only does that mean that I have to write the code for this thing—which I, you know, observed slowing down some other projects—but also that I'd have to explain it to people. Looking kind of like Bash was not a bad thing. Not having to know and learn something, so you can kind of feel really effective in about 15 minutes or something like that.Corey: One of the things that I find really interesting about you personally is that you were starting off in a bare-metal world; Ansible was sort of wherever you wanted to run it. Great, as long as there are systems that can receive these things, we're great. And now the world has changed, and for better or worse, configuration management slash remote execution is not the problem it once was and it is not a best practice way of solving a lot of those problems either. But you aren't spending your time basically remembering the glory years. You're actively moving forward doing some fairly interesting stuff. How would you describe what you're into these days?Michael: I tried to create a few projects to, like, kind of build other systems management things for the same audience for a while. I was building a build server and a new—trying to do some next-gen config stuff. And I saw people weren't interested. But I like having conversations with people, right, and I think one of the lessons from Ansible was how to explain highly technical things to technical audiences and cut out a lot of the marketing goo and all that; how to get people excited about an idea and make a community be really authentic. So, I've been writing about that for really, it's—rebooted blog is only a couple of weeks old. But also kind of trying to do some—helping out companies with some, like, basic marketing kind of stuff, right?There's just this pattern that everybody has where every website starts with this little basic slogan and two buttons and then there's a bunch of adjectives, but it doesn't say anything. So, how can you have really good documentation, and how can you explain an idea? Because, like, really, the reason you're in it is not just to sell stuff, but it's to help people and to see them get excited about your ideas. And there's just, like, we're not doing a good job in this, like, world where there's thousands upon thousands of applications, all competing at once to, like—how do you rise above that?Corey: And that's always the hard part is at some point, this does become your identity and you become known for a thing. And when you start branching out from that thing, you bring the expertise from that area that you were in, but you start applying it to new things. I feel like so many companies get focused—and people get focused—on assuming that their audience is just like them, where they're coming in with the exact same biases, the exact same experiences. And given that basically no one was as deep in the weeds as you were when it came to configuration management, that meant that you were spending time in that side of the world, not in other pursuits which aligned in some ways more directly with people developing other things. So, I suspect this might be one of the weird things we have in common when we show up and see something new.And a company is really excited. It's like, it's basically a few people talking [unintelligible 00:20:12] that both founders are technical. And they're super excited about something they can't quite articulate. And it's this, “Slow down. Tell me exactly what it is your product does.” And that's a hard thing to do because my default response was always the if I don't understand that is clearly the way in which I am deficient somehow. But marketing is really about clear communication and there's not that much of it in our space, at least not for early-stage companies.Michael: Yeah, I don't know why that is. I mean, I think there's this belief in that there's, like, this buyer audience where there's some vice president that's going to buy your stuff if you drop the right buzzwords. And 15 years ago, like, you had to say ‘synergy,' and now you say ‘time to value' or ‘total cost of ownership' or something. And I don't think that's true. I mean, I think people use products that they like and that they need to be shown them to try them out.So like, why can't your webpage have a diagram and a screenshot instead of this, like, picture of a couple of people drinking coffee around a computer, right? It's basic stuff. But I agree with you, I kind of feel dumb when I'm looking at all these tech products that I should be excited about, and, like, the way that we get there, as we ask questions. And the way that I've actually figured out what some of these things do is usually having to ask questions from someone who uses them that I randomly find on my diminishing circle of friends, right? And that's kind of busted.So, Ansible definitely had a lot of privilege in the way that it was launched in the sense that I launched it off Cobbler list and Cobbler list started off of [ET Management Tools 00:21:34] which was a company list. But people can do things like meetup groups really easily, they can give talks, they can get their blogs reblogged, and, you know, hope for some Hacker News or Reddit juice or whatever. But in order to get that to happen, you have to be able to talk to engineers that really want to know what you're doing, and they should be excited about it. So, learn to talk to them.Corey: You have to speak their language but without going so deep in the weeds that the only people that understand it are the folks who are never going to use your product because they want to build it themselves. It's a delicate balance to strike.Michael: And it's a difficult thing to do, too, when you know about it. So, when I was, like, developing all the Ansible docs, I've told people many times—and I hope it's true—that I, like, spent, like, 40% of my time just on the website and the docs, and whenever I heard somebody complain, I tried to fix it. But the idea was like, you can lose somebody really fast, but you kind of have to forget what you know about the product. So, the worst person to sometimes look at that as the person that built it. So, you have to forget what you know, and try to see, like, what questions they're asking, what do they need to find out? How do they want to learn something?And for me, I want to see a lot of pictures. A lot of people write a bunch of giant walls of text, or worse for me is when there's just these little pithy expressions and I don't know what they mean, right? And everybody's, like, kind of doing that these days.Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.48]Corey: One thing that I've really found myself enjoying recently has been your substack-based newsletter, Speaking Techis what you call it. And I didn't quite know what to expect when I signed up for it, but it's been a few weeks now, and you are more or less hitting across the board on a bunch of different things, ranging from engineering design patterns, to a teardown of random company's entire website from a marketing and messaging perspective—which I just adore personally; like that is very aligned with how I see the world—Michael: There's more of that coming.Corey: Yeah, [unintelligible 00:23:17] a bunch of other stuff. Let's talk about, for example, the idea of those teardowns. I always found that I have to be somewhat careful in how I talk about it when I'm doing a tweet thread or something like that because you are talking about people's work, let's be clear here, and I tend to be a lot kinder to small, early-stage companies than I am to, you know, $1.6 trillion companies who really should have solved for this by now, on some level. But so much of it misses the mark of great, here's the way that I think about these things. Here's the way that I don't understand what the hell you're telling me.An easy example of this for me, at least I'm curious to get your thoughts on it, I tend to almost always just skim what they're saying, great. Let's look at the pricing page because I find that speaks to people in a way that very often companies forget that they're speaking to customers.Michael: Yeah, for sure. I always tried to find the product page lately, and then, like, the product page now is, like, a regurgitation of the homepage. But it's what you said earlier. I think I try to stay nice to everybody, but it's good to show people how to understand things by counterexample, to some extent, right? Like, oh, I've got some stuff coming out—I don't know when this is actually going to get published—but next week, where I was like just taking random snippets of home pages, and like, “What's everybody doing with the header these days?”And there's just, like, ridiculous amounts of copying going on. But it's not just for, like, people's companies because everybody listening here isn't going to have a company. If you have a project and you wanted to get it noticed, right, I think, like, in the early days, the projects that I paid attention to and got excited about were often the ones that spend time on their website and their messaging and their experience. So, everybody kind of understands you have to write a good readme now but some of, like, the early Ruby crowd, for instance, did awesome, awesome web pages. They know how to pick out fonts, and I still don't know how to pick out fonts. But—Corey: I ask someone good at those things. That's how I pick ‘em.Michael: Yeah, yeah. That's not my job; get somebody that's good at that. But all that matters, right? So, if you do invest a little bit in not promoting yourself, not promoting your company, but trying to help people and communicate to them, you can build that audience around your thing and it makes it a lot more interesting.Corey: There's so many great tools out there that I find on GitHub that other people have to either point me to or I find it when I'm looking at it from a code-first perspective, just trying to find a particular example of the library being used, where they do such a terrible job of describing the problem that they solve, and it doesn't take much; it takes a paragraph or two at most. Or the idea that, “Oh, yeah, here's a way to do this thing. Just go ahead and get your credential file somewhere else.” Great. Could you maybe link to an example of how to do that?It's the basic stuff; assume that someone who isn't you might possibly want to use this. And I'm not even slightly suggesting that you wind up talking your way through how to do all of that. Just link to somewhere that has a good write-up of it and call it good. Just don't get in the way of people's first-time user experiences.Michael: Yeah, for sure. And—Corey: For some reason, that's a radical thought.Michael: Yeah, I think one of the things the industry has—well, not the industry; it's not their problem to solve, but, like, we don't really have a way for people to find what's cool and interesting anymore. So, various people have their own little lists on GitHub or whatever, but there's just so many people posting on the one or two forums people read and it goes by in a day. So, it's really, really hard to get attention. Even your own circle of followers isn't really logging into Twitter or anything, or LinkedIn. Or there's all the congratulations for your five years of Acme Corp kind of posts, and it's really, really hard to get attention.And I feel for everybody, so like, if somebody like GitHub or Microsoft is listening, and you wanted to build, like, a dashboard of here's the cool 15 projects for the week, kind of thing where everybody would see it, and start spotlight some of these really cool new things, that would be awesome, right?Corey: Whenever you see those roundups, that was things like Kubernetes and Docker. And great, I don't think those projects need the help in the same way.Michael: No, no, they don't. It's like maybe somebody's cool data thing, or a cool visualization, or the other thing that's—it's completely random, but I used to write fun graphics programs for fun or games and libraries. And I don't see that anymore, right? Maybe if you find it, you can look for it, but the things that get people excited about programming. Maybe they have no commercial value at all, but the way that people discover stuff is getting so consolidated is about Docker and Kubernetes. And everyone's talking about these three things, and if you're not Google or you're not Facebook, it's really—or Amazon, obviously—it's hard to get attention.Corey: Open-source on some level has changed from a community perspective. And part of it is because once upon a time, you could start with the very low-level stuff and build something, get it up and working. And that's where things like [Cobbler 00:27:44] and Ansible came out of. Now it's, “Click the button and use the thing everyone else is using. And if you're not doing that, what are you doing over there?”So, the idea of getting started tinkering with computers are built on top of so many frameworks and other things. And that's always been the case, but now it's much more apparent in some ways. “Okay, I'm going to go ahead and build out my first HTML file and serve it out using something in Node.” “Great, what is those NPM stuff that's scrolling past?” It's like, “The devil. That is the devil's own language you are seeing scroll past. And you don't need to worry about that; just pretend it's not there.”But back when I was learning all this stuff, we're paying attention to things scrolling past, like, you know, compilation messages and the Linux boot story as it wound up scrolling past. Terrible story; the protagonist was unreliable, but all right. And you start learning how these things work when you start scratching at the things that you're just sort of hand-waving and glossing over. These days, it feels like every time I use a modern project, that's everything.Michael: I mean, it is. And like what, React has, like, 2000 dependencies, right? So, how do you ever feel like you understand it? Or when recruiters are asking for ten years at Amazon. And then—or we find a library that it can only explain itself by being like this other library and requiring these other five.And you read one of those, and it becomes, like, this… tree of knowledge that you have no way of possibly understanding. So, we've just built these stacks upon stacks upon stacks of things. And I tend to think I kind of believe in minimalism. And like, wouldn't it be cool if we just burned this all and start—you know, we burn the forest and let something new regrow. But we tend to not do that. We just—now running a cloud on top of a cloud, and our JavaScript is thousands of miles high.Corey: I really wish that there were better paths for getting started. Like, I used to think that the right way to wind up learning how all this stuff work is to do what I did: Start off as, you know, the grumpy sysadmin type, and then—or help desk—and then work your way up and the rest. Those jobs aren't there anymore, and it doesn't leave people in a productive environment. “Oh, you want to build a computer game. Great. For an iPhone? Terrific.” Where do you go to get started on that? It's a hard thing to do.And people don't care at that scale, nor should they necessarily, on how to run your own servers. Back in the day when you wanted to have a blog on the internet, you were either reduced to using LiveJournal or MySpace, or you were running your own web server and had to learn how to make sure that it didn't become an attack platform. There was a learning curve that was fairly steep. Now, there are so many different paths to go down, you don't really need to know how any of these things even work.Michael: Yeah, I think, like, one of the—I don't know whether DevOps means anything as a topic or not, but one of the original pieces around that movement was systems administrators learning to code things and really starting to enjoy it, whether that was Python or Ruby, and so on. And now it feels like we're gluing all the things together, but that's happening in App Dev as well, right? The number of people that can build a really, really good library from the ground up, like, something that has C bindings, that's a really, really small crowd. And most of it, what we're doing is gluing together other people's libraries and compensating for the flaws and bugs in them, and duct tape and error handling or whatever. And it feels like programming has changed a lot because of this—and it's good if you want to get an idea up quickly, no doubt. But it's a different experience.Corey: The problem I always ran into was the similar problems I had with doing Debian packaging. It was always the, oh, great, there's going to be four or five different guides on how to do it—same story with RPM—and they're all going to be assuming different things, and you can crossover between them without realizing it. And then you just do something monstrous that kind of works until an actual Debian developer shoves you aside like you were a hazard to everyone around you. Let me do it for you. And there we go.It's basically, get people to do work for you by being really bad at it. And I don't love that pattern, but I'm still reminded of that because there are so many different ways to achieve any outcome that, okay, I want to run a ridiculous Hotdog or Not Hotdog style website out there. Great. I can upload things. Well, Docker or serverless? What provider do I want to put it on? And oh, by the way, a lot of those decisions very early on are one-way doors that you don't realize you're crossing through, as well as not knowing what the nuances of all of those things are. And that's dangerous.Michael: I think people are also learning the vendor as well, right? Some people get really engrossed in whether it's Amazon, or Google, or HashiCorp, or somebody's API, and you spend so much of your brain cells just learning how these people's systems work versus, like, general programming practices or whatever.Corey: I make it a point to build something on other cloud providers that aren't Amazon every now and then, just because I don't want to wind up effectively embracing a monoculture.Michael: Yeah, for sure. I mean, I think that's kind of the trend I see with people looking just at the Kubernetes stuff, or whatever, it's that I don't think it necessarily existed in web dev; there seems to be a lot of—still a lot of creativity and different frameworks there, but people are kind of… what's popular? What gets me my next job, and that kind of thing. Whereas before it was… I wasn't necessarily a sysadmin; I kind of stumbled into building admin tools. I kind of made hammers not houses or whatever, but basically, everybody was kind of building their own tools and deciding what they wanted. Now, like, people that are wanting to make money or deciding what people want for them. And it's kind of not always the simplest, easiest thing.Corey: So, many open-source projects now are—for example, one that I was dealing with recently was the AWS CLI. Great, like, I'm thrilled to throw in issues and challenges here, but I'm not going to spend significant time writing code against it because, one, it's basically impossible to get these things accepted when all the maintainers work at Amazon, and two, is it really an open-source project in the way that you and I think about community and the rest, but it's basically sole purpose is to funnel money to Amazon faster. Like, that isn't really a community ethos I feel comfortable getting behind to be perfectly honest. They're a big company; they can afford to pay people to build these things out, full time.Michael: Yeah. And GitHub, I mean, we all mostly, I think, appreciate the fact that we can host the Git repo and it's performant and everything, and we don't have blazing unicorns quite as often or whatever they used to have, but it kind of changed the whole open-source culture because we used to talk about things on mailing lists, like, what should this be, and there was like an upfront conversation, or it might happen on IRC. And now people are used to just saying, “I've got a problem. Fix it for me.” Or they're throwing code over the wall and it might not be the code or feature that you wanted because they're not really part of your thing.So before, people would get really engrossed with, like, just a couple of projects, and if they were working on them as kind of like a collective of people working against different organizations, we'd talk about things, and they kind of know what was going on. And now it's very easy to get a patch that you don't want and you're, like, “Oh, can you change all of these things?” And then somebody's, like, now they're offended because now they have to do all this extra work, whereas that conversation didn't happen. And GitHub could absolutely remodel themselves to encourage those kinds of conversations and communities, but part of the death of open-source and the fact that now it's, “Give me free code,” is because of that kind of absence of the—because we're looking at that is, like, the front of a community versus, like, a conversation.Corey: I really want to appreciate your taking so much time out of your day to basically reminisce about some of these things. But on a forward-looking basis, if people want to learn more about how you see things, where's the best place to find you?Michael: Yeah. So, if you're interested in my blog, it's pretty random, but it's michaeldehaan.substack.com. I run a small emerging consultancy thing off of michaeldehaan.net. And that's basically it. My Twitter is laserllama if you want to follow that. Yeah, thank you very much for having me. Great conversation. Definitely making all this technology feel old and busted, but maybe there's still some merit in going back—Corey: Old and busted because it wasn't built this year? Great—Michael: Yes.Corey: —yes, its legacy, which is a condescending engineering term for ‘it makes money.' Yeah, there's an entire universe of stuff out there. There are companies that are still toying with virtualization: “Is this something we get on board with?” There's nothing inherently wrong with that. I find that judging what a bunch of startups are doing or ‘company started today' is a poor frame of reference to look at what you should do with your 200-year-old insurance company.Michael: Yeah, like, [unintelligible 00:35:53] software engineering is just ridiculously new. Like, if you compare it to, like, bridge-building, or even electrical engineering, right? The industry doesn't know what it's doing and it's kind of stumbling around trying to escape local maxima and things like that.Corey: I will, of course, put links to where to find you into the [show notes 00:36:09]. Thanks again for being so generous with your time. It's appreciated.Michael: Yeah, thank you very much.Corey: Michael DeHaan, founder of Cobbler, Ansible, and oh, so much more than that. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice—and/or smash the like and subscribe buttons on the YouTubes—whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, smash the buttons as mentioned, and leave a loud, angry comment explaining what you hated about it that I will then summarily reject because it wasn't properly formatted YAML.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About KateKate Holterhoff, an industry analyst with RedMonk, has a background in frontend engineering, academic research, and technical communication. Kate comes to RedMonk from the digital marketing sector and brings with her expertise in frontend engineering, QA, accessibility, and scrum best practices.Before pursuing a career in the tech industry Kate taught writing and communication courses at several East Coast universities. She earned a PhD from Carnegie Mellon in 2016 and was awarded a postdoctoral fellowship (2016-2018) at Georgia Tech, where she is currently an affiliated researcher.Links: RedMonk: https://redmonk.com/ Visual Haggard: https://visualhaggard.org Twitter: https://twitter.com/kateholterhoff TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured, and fully managed with built-in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price-performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: Make your data sing.Corey: This episode is sponsored by our friends at Revelo. Revelo is the Spanish word of the day, and its spelled R-E-V-E-L-O. It means, “I reveal.” Now, have you tried to hire an engineer lately? I assure you it is significantly harder than it sounds. One of the things that Revelo has recognized is something I've been talking about for a while, specifically that while talent is evenly distributed, opportunity is absolutely not. They're exposing a new talent pool to, basically, those of us without a presence in Latin America via their platform. It's the largest tech talent marketplace in Latin America with over a million engineers in their network, which includes—but isn't limited to—talent in Mexico, Costa Rica, Brazil, and Argentina. Now, not only do they wind up spreading all of their talent on English ability, as well as you know, their engineering skills, but they go significantly beyond that. Some of the folks on their platform are hands down the most talented engineers that I've ever spoken to. Let's also not forget that Latin America has high time zone overlap with what we have here in the United States, so you can hire full-time remote engineers who share most of the workday as your team. It's an end-to-end talent service, so you can find and hire engineers in Central and South America without having to worry about, frankly, the colossal pain of cross-border payroll and benefits and compliance because Revelo handles all of it. If you're hiring engineers, check out revelo.io/screaming to get 20% off your first three months. That's R-E-V-E-L-O dot I-O slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while on the Twitters, I see a glorious notification. Now, doesn't happen often, but when it does, I have all well, atwitter, if you'll pardon the term. They have brought someone new in over at RedMonk.RedMonk has been a longtime friend of the show. They're one of the only companies that can say that about and not immediately get a cease-and-desist for having said that. And their most recent hire is joining me today. Kate Holterhoff is a newly minted analyst over at RedMonk. Kate, thank you for joining me.Kate: It's great to be here.Corey: One of the things that's always interesting about RedMonk is how many different directions you folks seem to go in all at once. It seems that I keep crossing paths with you folks almost constantly: When I'm talking to clients, when I'm talking to folks in the industry. And it could easily be assumed that you folks are 20, 30, 40 people, but to my understanding, there are not quite that many of you there.Kate: That is very true. Yes. I am the fifth analyst on a team of seven. And yeah, brought on the first of the year, and I'm thrilled to be here. I actually, I would say, recruited by one of my friends at Georgia Tech, Kelly Fitzpatrick, who I taught technical communication with when we were both postdocs in their Brittain Fellowship program.Corey: So, you obviously came out of an academic background. Is this your first excursion to industry?Kate: No, actually. After getting my PhD in literary and cultural studies at Carnegie Mellon in 2016, I moved to Atlanta and took a postdoc at Georgia Tech. And after that was kind of winding down, I decided to make the jump to industry. So, my first position out of that was at a digital marketing agency in Atlanta. And I was a frontend engineer for several years.Towards the end of my tenure there, I moved into doing more of their production engineering and QA work. Although it was deeply tied to my frontend work, so we spent a lot of time looking at how the web sites look at different media queries, making sure that there were no odd break points. So, it certainly was an organic move there as their team expanded.Corey: You spent significant amounts of time in the academic landscape. When you start talking about, “Well, I took on a postdoc position,” that's usually the sign of not your first year on a college campus in most cases. I mean, again, with an eighth grade education, I'm not really the person to ask, but I sit here in awe as people who are steeped in academia wind up going about the magic that, from where I sit, they tend to do. What was it that made you decide that I really enjoy the field that I've gotten a doctorate in. You just recently published a book in that is—or at least tangentially related to this space.But you decide, “You know what I really want to do now? That's right, frontend engineering. I want to spend, more or less, 40-some-odd hours a week slowly going mad because CSS, and I can't quite get that thing to line up the way that I want it to.” Now, at least that's my experience with it, for folks who are, you know, competent at it, I presume that's a bit of a different story.Kate: Yes. I considered naming my blog at RedMonk, “How to Center a Div.” So yes, that is certainly an ongoing issue, I think, for anyone in [unintelligible 00:06:15] any, you know, practitioners. So, I guess my story probably began in 2013, the real move into technology. So, getting a PhD, of course, takes a very, very long time.So, I started at Carnegie Mellon in 2009, and in 2013, I started a digital archive called Visual Haggard. And it's a Ruby on Rails site; you can visit it at visualhaggard.org. And it is a digital archive of illustrations that were created to accompany a 19th century writer, H. Rider Haggard.And I became very interested in all the illustrations that had been created to accompany both the serialization of his fictions, but also the later novelizations. And it's kind of like how we have all these different movie adaptations of, like, Spider Man that come out every couple of years. These illustrations were just very iterative. And generally, this fellowship that I saw really only focused on, you know, the first illustrations that, you know, came out. So, this was a sort of response to that: How can we use technology to showcase all the different types of illustrations and how maybe different artists would interpret that literature differently?And so, that drove me into a discipline called the digital humanities, which really sort of, you know, focuses on that question, which is, you know, how to computers help us to understand the humanities better? And so, that incorporates not only the arts, but also literature, philosophy, you know, new media. But it's an extremely broad subject, and it's evolving, as you can imagine, as the things that technology can do expands. So, I became interested in this subject and really was drawn to the sort of archival aspects of this. Which wasn't really my training; I think that's something that, you know, you think of librarians as being more focused on, but I became acquainted with all these, you know, very obscure editions.But in any event, it also taught me how to [laugh] use technology, I really—I was involved in the [RDF 00:08:08] export for [laugh] incorporating the site on Nines, which is sort of a larger agglomeration of 19th century archives. And I was just really drawn to a lot of the new things that we could do. So, I began to use it more in my teaching. So, not only did I—and of course as I taught communication courses at Carnegie Mellon, and then I moved to teaching them at Georgia Tech, you can imagine I had many students who were engineers, and they were very interested in these sorts of questions as well. So, the move felt very organic to me, but I think any academic that you speak to, their identity is very tied up in their sort of, you know, academic standing.And so, the idea of jumping ship, of not being labeled an academic anymore is kind of terrifying. But I, you know, ultimately opted to do it. It certainly was, yeah, but you know, what [laugh] what I learned is that there's the status called an affiliated researcher. So, I didn't necessarily have to be a professor or someone on the tenure track in order to continue doing research.Corey: Was it hard for you?Kate: So, the book project, which is titled Illustration in Fin-de-Siècle Transatlantic Romance Fiction, and has a chapter devoted to H. Rider Haggard, I wrote it, while really not even being an instructor or sort of traditional academic. I had access to the library through this affiliated researcher status, which I maintained by keeping a relationship with the folks at Georgia Tech, and was able to do all my research while you know, having a job in industry. And I think what a lot of academics need to do is think about what it is about academia that they really value. Is it the teaching?Because in industry, we spend a lot of time teaching [laugh]. Sharing our knowledge is something that's extremely important. Is that the research? As an analyst, I get to do research all the time, which is really fun for me. And then, you know, is it really just kind of focusing on historical aspects? And that was also important to me.So, you know, this status allowed me to keep all the best parts of being an academic while kind of sloughing off the [laugh] parts that weren't so good, which is, um, say the fact that 80% of courses in the university are taught now by adjuncts or folks who are not on the tenure track line. Which is, you know, pretty shocking, you know. The academy is going through some… troubles right now, and hiring issues are—they need to be acknowledged, and I think folks who are considering getting a PhD need to look for other career paths beyond just through modeling it on their advisors, or, you know, in order to become, ostensibly, a professor themselves.Corey: I don't know if I've told the story before in public, but I briefly explored the possibility of getting a PhD myself, which is interesting given that I'd have to… there's some prerequisites I'd probably have to nail first, like, get a formal GED might be, like, step one, before proceeding on. And strangely enough for me, it was not the higher level, I guess, contribution to a body of knowledge in a particular direction. I mean, cloud economics being sort of an easy direction for me [laugh] to go in, given that I eat, sleep, live, and breathe it, but rather the academic rigor around so much of it. And the incentives feel very different, which to be clear, is a good thing. My entire career path has always been focused on not starving to death, and how do we turn this problem into money, whereas academia has always seemed to be focused on knowledge for the sake of knowledge without much, if any, thought toward the practical application slash monetization thereof? Is that a fair characterization from where you sit? I'm trying not to actively be insulting, but it's possible I may be unintentionally so.Kate: No, I think you're right on. And so yeah, like, the book that I published, I probably won't see any remuneration for that. There is very little—I'm actually [laugh] not even sure what the contract says, but I don't intend to make any money with this. Professors, even those who have reached the height of their career, unless they're, you know, on specific paths, don't make a lot of money, those in the humanities, especially. You don't do this to become wealthy.And the Visual Haggard archive, I don't—you know, everything is under a Creative Commons license. I don't make money from people, you know, finding images that they're looking for to reproduce, say, on a t-shirt or something. So yeah, I suspect you do it for the love. I always explained it as having a sort of existential anxiety of, like, trying to, you know, cheat death. I think it was Umberto Eco who said that in order to live forever, you have to have a child and a book.And at this point, I have two children and a book now, so I can just, you know, die and my, you know, [laugh] my legacy lives on. But I do feel like the reasons that folks go into upper higher education vary, and so I wouldn't want to speak for everyone. But for me, yeah, it is not a place to make money, it's a place to establish sort of more intangible benefits.Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: I guess one of the weird things from where I sit is looking at the broad sweep of industry and what I know of RedMonks perspective, you mentioned that as a postdoc, you taught technical communication. Then you went to go to frontend engineering, which in many respects is about effectively, technically—highly technical and communicating with the end-user. And now you are an analyst at RedMonk. And seeing what I have seen of your organization in the larger ecosystem, teaching technical communication is a terrific descriptor of what it is you folks actually do. So, from a certain point of view, I would argue that you're still pursuing the path that you are on in some respects. Is that even slightly close to the way that you view things, or am I just more or less ineffectively grasping at straws, as I am wont to do?Kate: No, I feel like there is a continuous thread. So, even before I got my PhD, I got a—one of my bachelor's degrees was in art. So, I used to paint murals; I was very interested in public art. And so, it you know, it feels to me that there is this thread that goes from an interest in the arts and how the public can access them to, you know, doing web development that's focused on the visual aspects, you know, how are these things responsive? What is it that actually makes the DOM communicate in this visual way? You know, how are cascading style sheets,allowing us to do these sorts of marvelous things?You know, I could talk about my favorite, you know, selectors and things. [laugh]. Because I will defend CSS. I actually don't hate it, although we use SASS if it matters. But you know, that I think there's a lot to be said for the way that the web looks today rather than, you know, 20 years ago.So there, it feels very natural to me to have moved from an interest in illustration to trying to, you know, work in a more frontend way, but then ultimately [laugh] move from that into doing, sort of, QA, which is, like, well, let's take a look at how we're communicating visually and see if we can improve that to, you know, look for things that maybe aren't coming across as well as they could. Which really forced me to work in the interactive team more with the UI/UX folks who are, you know, obviously telling the designers where to put the buttons and, you know, how to structure the, you know, the text blocks in relationship to the images and things like that. So, it feels natural to me, although it might not seem so on the outside. You know, in the process, I really I guess, acquired a love of that entire area.And I think what's great about working at RedMonk now is that I get to see how these technologies are evolving. So, you know, I actually just spun up a site on [unintelligible 00:16:27] not long ago. And, I mean, it is so cool. I mean, you know, coming from a background where we were working with, you know, jQuery, [laugh] things have really evolved. You know, it's exciting. And I think we're seeing the, [like, as 00:16:39] the full stack approach to this.Corey: I used to volunteer for the jQuery infrastructure team and help run jquery.net, once upon a time.Kate: Ohh.Corey: I assume that is probably why it is no longer in vogue. Like, oh, Corey was too close to it got his stink all over the thing. Let's find something better immediately, which honestly, not the worst approach in the world to take.Kate: I'm so impressed. I had no idea.Corey: It was mostly—because again, I was bad at frontend; always have been, but I know how to make computers run—kind of—and on the backend side of things and the infrastructure piece of it. It's like I tend to—at least at the time—break the world into more or less three sets: You had the ops types, think of database admins and the rest; you had the backend engineers, people who wrote code that made things talk to each other from an API perspective, and you had frontend folks who took all of the nonsense and had this innovative idea that, “Huh, maybe a green screen glowing text terminal isn't the pinnacle of user experience that we might possibly think about, and start turning it into something that a human being can use.”And whatever I hear folks from one of those constituencies start talking disparagingly about the others, it's… yeah, go walk a mile in their shoes and then tell me how you feel. A couple years ago, I took a two week break to, all right, it's time for me to learn JavaScript. And by the end of the two weeks period, I was more confused than I was when I began. And it's just a very different way of thinking than I have become accustomed to working with. So, from where I sit, people who work on that stuff successfully are effectively just this side of wizards.I think that there's—I feel the same way about database types. That's an area I never go into either because I'm terrible at that, and the stakes over their company-killing proportions in a way that I took down a web server usually doesn't.Kate: Yeah, I think that's often the motto, well, at least at my last company, which was like, “It's just a website. No one will die.” [laugh].Corey: Honestly, I find that the people who have really have the best attitude about that tend to be, strangely enough, military veterans because it's, “The site is down. How are you so calm?” It's, “Well, no one's shooting at me and no one's going to die? It's fine.” Like, “We're all going to go home to our families tonight. It'll work out.” It having perspective is important.Kate: Yeah. It is interesting how the impetus—I mean, going back to your question about, you know, making money at this field, you know, how that kind of factors in, I guess, frontend does tend to have a more relaxed attitude than say, yeah, if you drop a table or something. But at the same time, you know, compared to academia, it did feel a little bit more [laugh] like, “Okay, well, this—you know, we've got the project manager that is breathing down our neck. They got to send them something, you know, what's going on here?” So, yeah, it does become a little bit more, I don't know, these things ramped up a little bit, and the importance, you know, varies by, you know, whatever part of it you're working on.It's interesting, as an analyst, I don't hear the terms backend and frontend as much, and that was really how my team was divided, you know? It was really, kind of, opaque when you walked in. Started the job, I was like, “Okay, well, is this something that the frontend should be dealing with or the backend? You know, what's going on?” And then, you know, ultimately, I was like, “Oh, no, I know exactly what this is.”And then anyone who came on later, I was like, “No, no, no. We talk to the backend folks for this sort of problem.” So, I don't know if that's also something that's falling out of vogue, but that was, you know, the backend handled all the DevOps aspects as well, and so, you know, anything with our virtual boxes and, you know, trying to get things running and, you know, access to our… yeah, the servers, you know, all of that was kind of handled by backend. But yeah, I worked with some really fantastic frontend, folks. They were just—I feel like they we could bet had been better categorized as full stack. And many of them have CS degrees and they chose to go into frontend. So, you know, it's a—I have no patience for, you know—Corey: Oh God, you mean you chose this instead of it being something that happened to you in a horrible accident one of these days?Kate: [laugh]. Exactly.Corey: And that's not restricted to frontend; that's working with computers, in my experience.Kate: [laugh].Corey: Like, oh, God, it's hard to remember I chose this at one point. Now, it feels almost like I'm not suited for anything else. You have a clear ability to effectively communicate technical concepts. If not, you more or less wasted most of your academic career, let's be very clear. Then you decided that you're going to go and be an engineer for a while, and you did that.Why RedMonk? Why was that the next step because with that combination of skills, the world is very much your oyster. What made you look at RedMonk and say, “Yes, this is where I should work?” And let me be very clear. There are days I have strongly considered, like, if I weren't doing this, where would I be? And yeah, I would probably annoy RedMonk into actively blocking me on all social media or hiring me. There's no third option there. So, I agree wholeheartedly with the decision. What was it that made it for you?Kate: I mean, it was certainly not just one thing. One of the parts of academia that I really enjoyed was the ability to go to conferences and just travel and really get to meet people. And so, that was something that seemed to be a big part of it [unintelligible 00:21:27] so that's kind of the part that maybe doesn't get mentioned so much. And then especially in the Covid era, you know, we're not doing as much traveling, as you're well aware.Corey: We're spending all of our time having these conversations via screen.Kate: You know, I do enjoy that.Corey: Yeah. Like in the before times, probably one out of every eight episodes or so of this show was recorded in person.Kate: Wow.Corey: Now, it's, “I don't know. I don't really know if I want to go across town.” It's a—honestly, I've become a bit of a shut-in here. But you get it down to a science. But you lose something by doing it.Kate: That's true.Corey: There's a lack of high bandwidth communication.Kate: And many of my academic friends, when they would go to conferences, they would just kind of hide in their hotel room until they had to present. And I was the kind of person that was down in the bar hanging out. So, to me, it [laugh] felt very natural. But in terms of the intellectual parts, in all seriousness, I think the ability to pull apart arguments is something that I just truly enjoy. So, when I was teaching, which of course was how—was why they paid me to be an academic, you know, I loved when I could sit in a classroom and I would ask a question. You know, I kind of come up with these questions ahead of time.And the students would say something totally unexpected, and then I'd have another one, say something totally out of the blue as well. And I get to take them and say, “You're both right. Here's how we combine them, and here's how we're going to move forward.” Sort of, the ability to take an argument and sort of mold it into something constructive, I think can be very useful, both in, you know, meeting with clients who maybe are, you know, coming at things a little bit differently than then maybe we would recommend in order to, you know, help them to reach developers, the practitioners, but also, you know, moderating panels is something that a lot of my colleagues do. I mean, that's a big part of the job, too, is, you know, speaking and… well, not only doing sort of keynote talks, which my colleague Rachel is doing that at, I think, a [GlueCon 00:23:14] this year.And then—but also, you know, just in video format, you know, to having multiple presenters and, kind of, taking their ideas and making something out of that sort of forwards the argument. I think that's a lot of fun. I like to think I do an okay job at it. And I certainly have a lot of experience with it. And then just finally, you know, listening to argument [unintelligible 00:23:30] a big part of the job is going to briefings where clients explain what their product does, and we listen and try to give them feedback about how to reach the developer audience, and, you know, just trying to work on that communication aspect.And I think what I would like to push is more of the visual part of this. So, I think a lot of times, people don't always think through the icons that they include, or the illustrations, or the just the stock photos. And I find those so fascinating. [laugh]. I know, that's not always the most—the part that everyone wants to focus on, but to me, the visuals of these pitches are truly interesting. They really, kind of, maybe say things that they don't intend always, and that also can really make concrete ideas that are, especially with some of this really complex technology, it can really help potential buyers to understand what it accomplishes better.Corey: Some of the endless engagements I've been on that I enjoy the most have been around talking to vendors who are making things. And it starts off invariably as, “Yeah, we want to go ahead and tell the world about this thing that we've done.” And my perspective has always been just a subtle frame shift. It's like, “Yeah, let me save some time. No one cares. Absolutely no one cares. You're in love with the technical thing that you built, and the only people who are going to love it as much as you do are either wanting to work where you, or they're going to go build their own and they're not going to be your customer. So, don't talk about you. No one cares about you. Talk about the pain that you solve. Talk about the painful thing that you're target customer is struggling with that you make disappear.”And I didn't think that would be, A, as revelatory as it turned out to be, and B, a lesson that I had to learn myself. When I was starting o—when I was doing some product development here where I once again fell into the easy trap of assuming if I know something, everyone must know it, therefore, it's easy, whereas if I don't know something, it's very hard, and no one could possibly wrap their head around it. And we all come from different places, and meeting people wherever they are in their journey, it's a delicate lesson to learn. I never understood what analysts did until I started being an analyst myself, and I've got to level with you, I spent six months of doing those types of engagements feeling like a giant fraud. I'm just a loudmouth with an opinion, what is what does that mean?Well, in many ways, it means analyst. Because it's having an opinion is in so many ways, what customers are really after. Raw data, you can find that a thousand different ways, but finding someone who could talk on what something means, that's harder. And I think that we don't teach anything approaching that in most of our STEM curriculum.Kate: Yeah, I think that's really on point. Yeah, I mean, especially when some of these briefings are so mired in acronyms, and sort of assumed specialization. I know I spend a lot of time just thinking about what it is that confuses me about their pitch, more so than what, you know, is actually coming through. So, I think actually, one of the tools that we use—writing instructors; my past life—was thinking like someone with an eighth grade education. So, I actually think that your reference to having [laugh] you know, that's sort of chestnut, that can actually be useful because you say, “If I, you know, took my slide deck and showed it to a bunch of eighth graders, would they understand what it is that I'm saying?”You know, maybe you don't want them to get the technical details, but what problem does it solve? If they don't understand that, you're not doing a good enough job. And so that, to me, is [laugh] actually something that a lot of folks need to hear. That yeah, these vendors because they're just so deep in it, they're so in the weeds, that they can't maybe see how someone who's just looking for a database, or a platform, or whatever, they actually need this sort of simplified and yet broad enough explanation for what it is that they're actually trying to do what service they actually provide.Corey: From where I sit, one of the hardest things is just reaching people in the right way. And I'm putting out a one to two-thousand word blog post every week because I apparently hate myself. And that was a constant struggle for me when I started doing that a year or two ago. And what has worked for me that really get me moving down that is, instead of trying to teach everyone all the things, I pick an individual—and it varies from week to week—that I think about and I want to explain something to that person. And then I wind up directing what it is I'm about to tell—what it is I'm writing—to that person.Sometimes they're a complete layperson. Other times they are fairly advanced in a particular area of technology. And the responses to these things differ, but it's always—I always learn something from the feedback that I get. And if nothing else, is one of those ways to become a better writer. While I would start by writing. Just do it, don't whine—don't worry about getting it perfect; just go out there and power through things.At least, that's my approach. And I'm talking about the burden of writing a thousand words a week. You wrote an actual book. My belief is that, the more people I've talked to who've done that, no one actually wants to write a book; people want to have written a book, and that definitely resonates with me. I am tempted to just slap a bunch of these—Kate: Yeah.Corey: —blogs posts together and call it a book one of these days as an anthology. But it feels like it's cheating. If I ever decide to go down that path, I want to do it right.Kate: I guess, I come at it from the perspective of I don't know what I think until I write it down. So, it helps me to formulate ideas better. I also feel like my strength is in rereading things and trying to edit them down to really get to the kernel of what it is I think. And a lot of times how I begin a chapter or a blog post or whatever is not where it should begin, that maybe I'm somewhere in the middle, maybe this is a conclusion. There's something magical, in my view, that [laugh] happens when you write, that you are able to pause and take a little bit more time and maybe come up with a better word for what it is that you're trying to communicate.I also am—I benefit from readers. So, for instance, in my book, I have one chapter that really focuses on Harper's Weekly, which is an American newspaper. I'm not an Americanist; I don't have a deep knowledge of that, so what I did is I revise that chapter and send it to American periodicals and got feedback from their readers. Super useful. In terms of my blog at RedMonk, anytime I publish something, you can bet that at least one founder and probably at least one other analyst has read it through and giving me some extremely incisive feedback. It never is just from my mind. It's something that is collaboration.And I am grateful to anyone who takes the time to read my writing because, you know, all of us have so much time, of course. It really helps me to understand what it is that I'm trying to dig into. So, for instance, I've been writing a series for RedMonk on certifications, which makes a lot of sense; I've come from an academic background, here it is, you know, I'm seeing all these tech certifications. And so, it's interesting to me to see similarities and differences and what sort of issues that we're seeing come up with them. So, for instance, I just wrote about the vendor-specific versus vendor-neutral certifications. What are the advantages of getting a certification from the CN/CF versus from say, VMware and—Corey: Oh, I have opinions, on all of [those 00:30:44]—Kate: I—Corey: —and most of them are terrible.Kate: —I'm sure you do. [laugh]. It came naturally out of the job, you know, sitting through briefings and, kind of, seeing these things evolve, and the questions that I have from a long history of teaching, but. I think it also suggests the collaborative aspect of this, of coming to my colleagues—you know, I've been here before, for what, four months?—and saying, you know, “Is this normal? Like, what are we seeing here? Let me write a little bit about what I think is going on with certifications, and then you tell me, you know, what it is that you've seen with your years and years of expertise,” right?So, Stephen O'Grady's been doing this for longer than he really likes to admit, right? So, this is grateful to have such well-established colleagues that can help me on that journey. But, you know, to kind of spiral back to your original question, I think that writing to me is an exploration, it's something that helps me to get to something a little more, I guess, meaningful than just where I began. You know, just the questions that I have, I can kind of dig down and find some substance there. I would encourage you to take any one of your blog posts and think about maybe where they—or using the jumping off points for your eventual book, which I will be looking for on newsstands any day now.Corey: I am looking forward to seeing how you continue to evolve your coverage area, as well as reading more of your writings around these things. I am—they always say that the cobblers children have no shoes, and I am having an ongoing war with the RedMonk RSS feed because I've been subscribed to it three times now, and I'm still not seeing everything that comes through, such as your posts. Time for me to go and yell at some people over on your end about how these things work because it is such good content. And every time RedMonk puts something out, it doesn't matter who over there has written it, I wind up reading it with this sense of envy, in that I wish I had written something like this. It is always an experience, and your writing is absolutely no exception to that. You fit in well over there.Kate: It means a lot to me. Thank you. [laugh].Corey: No, thank you. I want to thank you for spending so much time talking to me about things that I feel like I'm still not quite smart enough to wrap my head around, but that's all right. If people want to learn more, where's the best place to find you?Kate: Certainly Twitter. So, my Twitter handle is just my name, @kateholterhoff. And I don't post as often as maybe I should, but I try to maintain an ongoing presence there.Corey: And we will of course, put a link to that in the [show notes 00:33:04].Kate: Thank you.Corey: Thank you so much for your time. I appreciate it. Kate Holterhoff, analyst at RedMonk. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice—or if you're on YouTube, smash that like and subscribe button—whereas if you've hated this podcast, please do the exact same thing—five-star review, smashed buttons—but then leave an angry, incoherent comment, and it's going to be extremely incoherent because you never learned to properly, technically communicate.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble. 

About TomTom enjoys being a bridge between people and technology. When he's not thinking about ways to make enterprise demos less boring, Tom enjoys spending time with his wife and dogs, reading, and gaming with friends.Links Referenced: LaunchDarkly: https://launchdarkly.com Heidi Waterhouse Twitter: https://twitter.com/wiredferret TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: This episode is sponsored by our friends at Revelo. Revelo is the Spanish word of the day, and its spelled R-E-V-E-L-O. It means “I reveal.” Now, have you tried to hire an engineer lately? I assure you it is significantly harder than it sounds. One of the things that Revelo has recognized is something I've been talking about for a while, specifically that while talent is evenly distributed, opportunity is absolutely not. They're exposing a new talent pool to, basically, those of us without a presence in Latin America via their platform. It's the largest tech talent marketplace in Latin America with over a million engineers in their network, which includes—but isn't limited to—talent in Mexico, Costa Rica, Brazil, and Argentina. Now, not only do they wind up spreading all of their talent on English ability, as well as you know, their engineering skills, but they go significantly beyond that. Some of the folks on their platform are hands down the most talented engineers that I've ever spoken to. Let's also not forget that Latin America has high time zone overlap with what we have here in the United States, so you can hire full-time remote engineers who share most of the workday as your team. It's an end-to-end talent service, so you can find and hire engineers in Central and South America without having to worry about, frankly, the colossal pain of cross-border payroll and benefits and compliance because Revelo handles all of it. If you're hiring engineers, check out revelo.io/screaming to get 20% off your first three months. That's R-E-V-E-L-O dot I-O slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today's promoted episode is brought to us by our friends at LaunchDarkly. And it's always interesting when there's a promoted guest episode because they generally tend to send someone who has a story to tell in different ways.Sometimes they send me customers of theirs. Other times they send me executives. And for this episode, they have sent me Tom Totenberg, who's a senior solutions engineer at LaunchDarkly. Tom, thank you for drawing the short straw. It's appreciated.Tom: [laugh]. Anytime. Thank you so much for having me, Corey.Corey: So, you're a senior solutions engineer, which in many different companies is interpreted differently, but one of the recurring themes tends to pop up is often that is a different way of saying sales engineer because if you say sales, everyone hisses and recoils when you enter the conversation. Is that your experience or do you see your role radically differently?Tom: Well, I used to be one of those people who did recoil when I heard the word sales. I was raised in a family where you didn't talk about finances, you know? That's considered to be faux pas, and when you hear the word sales, you immediately think of a car lot. But what I came to realize is that, especially when we talk about cloud software or any sort of community where you start to run into the same people at conferences over and over and over again, turns out the good salespeople are the ones who actually try to form relationships and try to solve problems. And I realized that oh, I like to work with those people. It's pretty exciting. It's nice to be aspirational about what people can do and bring in the technical chops to see if you can actually make it happen. So, that's where I fit in.Corey: The way that I've always approached it has been rather different. Because before I got into tech, I worked in sales a bunch of times and coming up from the—I guess, clawing your way up doing telesales was a polite way of describing—back in the days before there were strong regulations against it, calling people at dinner to sell them credit cards. And what's worse is I was surprisingly effective at it for a kid who, like, you grew up in a family where we didn't talk about money. And it's easy to judge an industry by its worst examples. Another one of these would be recruiting, for example.When everyone talks about how terrible third-party recruiters are because they're referring to the ridiculous spray-and-pray model of just blasting out emails to everything that hold still long enough that meets a keyword. And yeah, I've also met some recruiters that are transformative as far as the conversations you have with them go. But some of that with sales. It's, “Oh, well, you can't be any fun to talk to because I had a really bad experience buying a used car once and my credit was in the toilet.”Tom: Yeah, exactly. And you know, I have a similar experience with recruiters coming to LaunchDarkly. So, not even talking about the product; I was a skeptic, I was happy where I was, but then as I started talking to more and more people here, I'm assuming you've read the book Accelerate; you probably had a hand in influencing part of it.Corey: I can neither confirm nor deny because stealing glory is something I only do very intentionally.Tom: Oh okay, excellent. Well, I will intentionally let you have some of that glory for you then. But as I was reading that book, it reminded me again of part of why I joined LaunchDarkly. I was a skeptic, and they convinced me through everyone that I talked to just what a nice place it is, and the great culture, it's safe to fail, it's safe to try stuff and build stuff. And then if it fails, that's okay. This is the place where that can happen, and we want to be able to continue to grow and try something new.That's again, getting back to the solutions engineer, sales engineer part of it, how can we effectively convey this message and teach people about what it is that we do—LaunchDarkly or not—in a way that makes them excited to see the possibilities of it? So yeah, it's really great when you get to work with those type of people, and it absolutely shouldn't be influenced by the worst of them. Sometimes you need to find the right ones to give you a chance and get in the door to start having those conversations so you can make good decisions on your own, not just try to buy whatever someone's—whatever their initiative is or whatever their priority is, right?Corey: Once upon a time when I first discovered LaunchDarkly, it was pretty easy to describe what you folks did. Feature flags. For longtime listeners of the show, and I mean very longtime listeners of the show, your colleague Heidi Waterhouse was guest number one. So, I've been talking to you folks about a variety of different things in a variety of different ways. But yeah, “LaunchDarkly. Oh, you do feature flags.”And over time that message has changed somewhat into something I have a little bit of difficulty to be perfectly honest with you in pinning down. At the moment we're recording this, if I pull up launchdarkly.com, it says, “Fundamentally change how you deliver software. Innovate faster, deploy fearlessly, and make each release a masterpiece.”And I look at the last release I pushed out, which wound up basically fixing a couple of typos there, and it's like, “Well, shit. Is it going to make me sign my work because I'm kind of embarrassed by a lot of it.” So, it's aspirational, I get it, but it also somehow [occludes 00:05:32] a little bit of meaning. What is it you'd say it is you do here.Tom: Oh, Office Space. Wonderful. Good reference. And also, to take about 30 seconds back, Heidi Waterhouse, what a wonderful human. wiredferret on Twitter. Please, please go look her up. She's got just always such wonderful things to say. So—Corey: If you don't like Heidi Waterhouse, it is a near certainty it is because you've not yet met her. She's wonderful.Tom: Exactly. Yes, she is. So, what is it we'd say we do here? Well, when people think about feature flags—or at this point now, ‘feature management,' which is a broader scope—that's the term that we're using now, it's really talking about that last bit of software delivery, the last mile, the last leg, whatever your—you know, when you're pushing the button, and it's going to production. So, you know, a feature flag, if you ask someone five or ten years ago, they might say, oh, it's a fancy if statement controlled by a config file or controlled by a database.But with a sort of modern architecture, with global delivery, instant response time or fraction of a second response time, it's a lot more fundamental than that. That's why the word fundamental is there: Because it comes down to psychological safety. It comes down to feeling good about your life every day. So, whether it is that you're fixing a couple typos, or if you're radically changing some backend functionality, and trying out some new sort of search algorithm, a new API route that you're not sure if it's going to work at scale, honestly, you shouldn't have to stay up at night, you shouldn't have to think about deploying on a weekend because you should be able to deploy half-baked code to production safely, you should be able to do all of that. And that's honestly what we're all about.Now, there's some extra elements to it: Feedback loops, experimentation, metrics to make sure that your releases are doing well and doing what you anticipated that they would do, but really, that's what it comes down to is just feeling good about your work and making sure that if there is a fire, it's a small fire, and the entire audience isn't going to get part of the splash zone, right? We're making it just a little safer. Does that answer your question? Is that what you're getting at? Or am I still just speaking in the lingo?Corey: That gets it a lot closer. One of the breakthrough moments—of course I picked it up from one of Heidi's talks—is feature flag seems like a front end developer thing, yadda, yadda, yadda. And she said historically, yeah, in some ways, in some cases, that's how it started. But think about it this way. Think about separating out configuration from your deploy process. And what would that mean? What would that entail?And I look at my current things that I have put out there, and there is no staging environment, my feature branches main, and what would that change? In my case, basically nothing. But that's okay. Because I'm an irresponsible lunatic who should not be allowed near anything expensive, which is why I'm better at stateless things because I know better than to take my aura near things like databases.Tom: Yeah. So, I don't know how old you are Corey. But back—Corey: I'm in my mid-30s, which—Tom: Hey—Corey: —enrages my spouse who's slightly older. Because I'm turning 40 in July, but it's like, during the pandemic, as it has for many of us, the middle has expanded.Tom: There you go. Right. Exa—[laugh] exactly. Can neither confirm nor deny. You can only see me from about the mid-torso up, so, you know, you're not going to see whether I've expanded.But when we were in school doing group projects, we didn't have Google Docs. We couldn't see what other people were working on. You'd say, “Hey, we've got to write this paper. Corey, you take the first section, I'll take the second section, and we'll go and write and we'll try to squish it back together afterward.” And it's always a huge pain in the ass, right? It's terrible. Nobody likes group projects.And so the old method of Gitflow, where we're creating these feature branches and trying to squish them back later, and you work on that, and you work on this thing, and we can't see what each other are doing, it all comes down to context switching. It is time away from work that you care about, time away from exciting or productive work that you actually get to see what you're doing and put it into production, try it out. Nobody wants to deal with all the extra administrative overhead. And so yeah, for you, when you've got your own trunk-based development—you know, it's all just main—that's okay. When we're talking about teams of 40, 50, 100, 1000 suddenly becomes a really big deal if you were to start to split off and get away from trunk-based development because there's so much extra work that goes into trying to squish all that work back together, right? So, nobody wants to do all the extra stuff that surrounds getting software out there.Corey: It's toil. It feels consistently like it is never standardized so you always have to wind up rolling your own CI/CD thing for whatever it is. And forget between jobs; between different repositories and building things out, it's, “Oh, great. I get to reinvent the wheel some more.” It's frustrating.Tom: [laugh]. It's either that or find somebody else's wheel that they put together and see if you can figure out where all those spokes lead off to. “Is this secure? I don't know.”Corey: How much stuff do you have running in your personal stuff that has more or less been copied around for a decade or so? During the pandemic, I finally decided, all right, you know what I'm doing? That's right, being productive. We should fix that. I'm going to go ahead and redo my shell config—my zshrc—from scratch because, you know, 15 years of technical debt later, a lot of the things I used to really need it to do don't really apply anymore.Let's make it prettier, and let's make it faster. And that was great and all, but just looking through it, it was almost like going back in time for weird shell aliases that I don't need anymore. It's, well, that was super handy when I ran a Ruby production environment, but I haven't done that in seven years, and I haven't been in this specific scenario that one existed for since 2011. So maybe, maybe I can turn that one off.Tom: Yeah, maybe. Maybe we can get rid of that one. I mean, when's the last time you ran npm install on something you were going to try out here and paid attention to the warnings that came up afterward? “Hey, this one's deprecated. That one's deprecated.” Well, let's see if it works first, and then we'll worry about that later.Corey: Exactly. Security problems? Whatever. It's a Lambda function. What do I care?Tom: Yeah, it's fine. [laugh]. Exactly. Yeah. So, a lot of this is hypothetical for someone in my position, too, because I didn't ever get formal training as a software developer. I can copy and paste from Stack Overflow with the best of them and there's all sorts of resources out there, but really the people that we're talking to are the ones who actually live that day in, day out.And so I try to step into their shoes and try to feel that pain. But it's tough. Like, you have to be able to speak both languages and try to relate to people to see what are they actually running into, and is that something that we can help with? I don't know.Corey: The way that I tend to think about these things—and maybe it's accurate, and maybe it's not—it's just, no one shows up hoping to do a terrible job at work today, but we are constrained by a whole bunch of things that are imposed upon us. In some of the more mature environments, some of that is processes there for damn good reasons. “Well, why can't I just push everything I come up with to production?” “It's because we're a bank, genius. How about you think a little bit before you open your mouth?”Other times, it's because well, I have to go and fight with the CI/CD system, and I'm just going to go ahead and patch this one-line change into production. Better processes, better structure have made that a lot more… they've made it a lot easier to be able to do things the right way. But I would say we're nowhere near its final form, yet. There's so much yak-shaving that has to go into building out anything that it's frustrating, on some level, just all of the stuff you have to do, just to get the scaffolding in place to write nonsense. I mean, back when they announced Lambda functions it was, “In the future, the only code you'll write is business logic.”Yeah, well, I use a crap-ton of Lambda here and it feels like most of the code I write is gluing all of the weird formats and interchanges together in different APIs. Not a lot of business logic in that; and awful lot of JSON finickiness.Tom: Yeah, I'm with you. And especially at scale, I still have a hard time wrapping my mind around how all of that extra translation is possibly going to give the same sort of performance and same sort of long-term usability, as opposed to something that just natively speaks the same language end-to-end. So yeah, I agree, there's still some evolution, some standardization that still needs to happen because otherwise we're going to end up with a lot of cruft at various points in the code to, just like you said, translate and make sure we're speaking the same language.Getting back to process though, I spent a good chunk of my career working with companies that are, I would say, a little more conservative, and talking to things like automotive companies, or medical device manufacturers. Very security-conscious, compliant places. And so agile is a four-letter word for them, right, [laugh] where we're going faster automatically means we're being dangerous because what would the change control board say? And so there's absolutely a mental shift that needs to happen on the business side. And developers are fighting this cultural battle, just to try to say, hey, it's better if we can make small iterative changes, there is less risk if we can make small, more iterative changes, and convincing people who have never been exposed to software or know the ins and outs of what it takes to get something from my laptop to the cloud or production or you know, wherever, then that's a battle that needs to be fought before you can even start thinking about the tooling. Living in the Midwest, there's still a lot of people having that conversation.Corey: So, you are clearly deep in the weeds of building and deploying things into production. You're clearly deep into the world of explaining various solutions to different folks, and clearly you have the obvious background for this. You majored in music. Specifically, you got a master's in it. So, other than the obvious parallel of you continue to sing for your supper, how do you get from there to here?Tom: Luck and [laugh]. Natural curiosity. Corey, right now you are sitting on the desk that is also housing my PC gaming computer, right? I've been building computers just to play video games since I was a teenager. And that natural curiosity really came in handy because when I—like many people—realize that oh, no, the career choice that I made when I was 18 ended up being not the career choice that I wanted to pursue for the rest of my life, you have to be able to make a pivot, right, and start to apply some of the knowledge that you got towards some other industries.So, like many folks who are now solutions engineers, there's no degree for solutions engineering, you can't go to school for it; everyone comes from somewhere else. And so in my case, that just happened to be music theory, which was all pedagogy and teaching and breaking down big complex pieces of music into one node at a time, doing analysis, figuring out what's going on underneath the hood. And all of those are transferable skills that go over to software, right? You open up some giant wall of spaghetti code and you have to start following the path and breaking it down because every piece is easy one note at a time, every bit of code—in theory—is easy one line at a time, or one function at a time, one variable at a time. You can continue to break it down further and further, right?So, it's all just taking the transferable skills that you may not see how they get transferred, but then bringing them over to share your unique perspective, because of your background, to wherever it is you're going. In my case, it was tech support, then training, and then solutions engineering.Corey: There's a lot to be said for blending different disciplines. I think that there was, uh, the naughts at least, and possibly into the teens, there was a bias for hiring people who look alike. And no, I'm not referring to the folks who are the white dudes you and I clearly present as but the people with a similar background of, “Oh, you went to these specific schools”—as long as they're Stanford—“And you majored in a narrow list of things”—as long as they're all computer science. And then you wind up going into the following type of role because this is the pedigree we expect and everything, soup to nuts, is aligned around that background and experience. Where you would find people who would be working in the industry for ten years, and they would bomb the interview because it turns out that most of us don't spend our days implementing quicksort on whiteboards or doing other algorithmic-based problems.We're mostly pushing pixels around a screen hoping to make ourselves slightly happier than we were. Here we are. And that becomes a strange world; it becomes a really, really weird moment, and I don't know what the answer is for fixing any of that.Tom: Yeah, well, if you're not already familiar with a quote, you should be, which is that—and I'm going to paraphrase here—but, “Diverse backgrounds lead to diversity in thought,” right? And that presents additional opportunities, additional angles to solve whatever problems you're encountering. And so you're right, you know, we shouldn't be looking for people who have the specific background that we are looking for. How it's described in Accelerate? Can you tell that I read it recently?Which we should be looking for capabilities, right? Are you capable? Do you have the capacity to do the problem-solving, the logic? And of course, some education or experience to prove that, but are you the sort of person who will be able to tackle this challenge? It doesn't matter, right, if you've handled that specific thing before because if you've handled that specific thing before, you're probably going to implement it the same way, again, even if that's not the appropriate solution, this time.So, scrap that and say, let's find the right people, let's find people who can come up with creative solutions to the problems that we're facing. Think about ways to approach it that haven't been done before. Of course don't throw out everything with the—you know, the bathwater out with a baby or whatever that is, but come in with some fresh perspectives and get it done.Corey: I really wish that there was more of an acceptance for that. I think we're getting there. I really do, but it takes time. And it does pay dividends. I mean, that's something I want to talk to you about.I love the sound of my own voice. I wouldn't have two podcasts if I didn't. The counterargument, though, is that there's an awful lot of things that get, you know, challenging, especially when, unlike in a conference setting, it's most people consider it rude to get up and walk out halfway through. When we're talking and presenting information to people during a pandemic situation, well, that changes a lot. What do you do to retain people's interest?Tom: Sure. So, Covid really did a number on anyone who needs to present information or teach. I mean, just ask the millions of elementary, middle school, and high schoolers out there, even the college kids. Everyone who's still getting their education suddenly had to switch to remote learning.Same thing in the professional world. If you are doing trainings, if you're doing implementation, if you're doing demos, if you're trying to convey information to a new audience, it is so easy to get distracted at the computer. I know this firsthand. I'm one of those people where if I'm sitting in an airport lobby and there's a TV on my eyes are glued to that screen. That's me. I have a hard time looking away.And the same thing happens to anyone who's on the receiving end of any sort of information sharing, right? You got Slack blowing you up, you've got email that's pinging you, and that's bound to be more interesting than whatever the person on the screen is saying. And so I felt that very acutely in my job. And there's a couple of good strategies around it, right, which is, we need to be able to make things interactive. We shouldn't be monologuing like I am doing to you right now, Corey.We shouldn't be [laugh] just going off on tangents that are completely irrelevant to whoever's listening. And there's ways to make it more interactive. I don't know if you are familiar, or how much you've watched Twitch, but in my mind, the same sorts of techniques, the same sorts of interactivity that Twitch streamers are doing, we should absolutely be bringing that to the business world. If they can keep the attention of 12-year-olds for hours at a time, why can we not capture the attention of business professionals for an hour-long meeting, right? There's all sorts of techniques and learnings that we can do there.Corey: The problem I keep running into is, if you go stumbling down that pathway into the Twitch streaming model, I found it awkward the few experiments I've made with it because unless I have a whole presentation ready to go and I'm monologuing the whole time, the interactive part with the delay built in and a lot of ‘um' and ‘ah' and waiting and not really knowing how it's going to play out and going seat of the pants, it gets a little challenging in some respects.Tom: Yeah, that's fair. Sometimes it can be challenging. It's risky, but it's also higher reward. Because if you are monologuing the entire time, who's to say that halfway through the content that you are presenting is content that they want to actually hear, right? Obviously, we need to start from some sort of fundamental place and set the stage, say this is the agenda, but at some point, we need to get feedback—similar to software development—we need to know if the direction that we're going is the direction they also want to go.Otherwise, we start diverging at minute 10 and by minute 60, we have presented nothing at all that they actually want to see or want to learn about. So, it's so critical to get that sort of feedback and be able to incorporate it in some way, right? Whether that way is something that you're prepared to directly address. Or if it's something that says, “Hey, we're not on the same page. Let's make sure this is actually a good use of time instead of [laugh] me pretending and listening to myself talk and not taking you into account.” That's critical, right? And that is just as important, even if it feels worse in the moment.Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: From where I sit, one of the many, many, many problems confronting us is that there's this belief that everyone is like we are. I think that's something fundamental, where we all learn in different ways. I have never been, for example—this sounds heretical sitting here saying it, but why not—I'm not a big podcast person; I don't listen to them very often, just because it's such a different way of consuming information. I think there are strong accessibility reasons for there to be transcripts of podcasts. That's why every 300-and-however-many-odd episodes that this one winds up being the sequence in, every single one of them has a transcript attached to it done by a human.And there's a reason for that. Not just the accessibility wins which are obvious, but the fact that I can absorb that information way more quickly if I need to review something, or consume that. And I assume other people are like me, they're not. Other people prefer to listen to things than to read them, or to watch a video instead of listening, or to build something themselves, or to go through a formal curriculum in order to learn something. I mean, I'm sitting here with an eighth-grade education, myself. I take a different view to how I go about learning things.And it works for me, but assuming that other people learn the same way that I do will be awesome for a small minority of people and disastrous for everyone else. So, maybe—just a thought here—we shouldn't pattern society after what works for me.Tom: Absolutely. There is a multiple intelligence theory out there, something they teach you when you're going to be a teacher, which is that people learn in different ways. You don't judge a fish by its ability to climb a tree. We all learn in different ways and getting back to what we were talking about presenting effectively, there needs to be multiple approaches to how those people can consume information. I know we're not recording video, but for everyone listening to this, I am waving my hands all over the place because I am a highly visual learner, but you must be able to accept that other people are relying more on the auditory experience, other people need to be able to read that—like you said with the accessibility—or even get their hands on it and interact with it in some way.Whether that is Ctrl-F-ing your way through the transcript—or Command-F I'm sorry, Mac users [laugh]; I am also on a Mac—but we need to make sure that the information is ready to be consumed in some way that allows people to be successful. It's ridiculous to think that everyone is wired to be able to sit in front of a computer or in a little cubicle for eight hours a day, five days a week, and be able to retain concentration and productivity that entire time. Absolutely not. We should be recording everything, allowing people to come back and consume it in small chunks, consume it in different formats, consume it in the way that is most effective to them. And the onus for that is on the person presenting, it is not on the consumer.Corey: I make it a point to make what I am doing accessible to the people I am trying to reach, not to me. And sometimes I'm slacking, for example, we're not recording video today, so whenever it looks like I'm not paying attention to you and staring off to the side, like, oh, God, he's boring. No. I have the same thing mirrored on both of my screens; I just prefer to look at the thing that is large and easy to read, rather than the teleprompter, which is a nine-inch screen that is about four feet in front of my face. It's one of those easier for me type of things.On video, it looks completely off, so I don't do it, but I'm oh good, I get to take the luxury of not having to be presentable on camera in quite the same way. But when I'm doing a video scenario, I absolutely make it a point to not do that because it is off-putting to the people I'm trying to reach. In this case, I'm not trying to reach you; I already have. This is a promoted guest episode you're trying to reach the audience, and I believe from what I can tell, you're succeeding, so please keep at it.Tom: Oh, you bet. Well, thank you. You know this already, but this is the very first podcast I've ever been a guest on. So, thank you also for making it such a welcoming place. For what it's worth, I was not offended and didn't think you weren't listening. Obviously, we're having a great time here.But yeah, it's something that especially in the software space, people need to be aware of because everyone's job is—[laugh]. Whether you like it or not, here's a controversial statement: Everyone's job is sales. Are you selling your good ideas for your product, to your boss, to your product manager? Are you able to communicate with marketing to effectively say, “Hey, this is what, in tech support, I'm seeing. This is what people are coming to me with. This is what they care about.”You are always selling your own performance to your boss, to your customers, to other departments where you work, to your spouse, to everybody you interact with. We're all selling ourselves all the time. And all of that is really just communication. It's really just making sure you're able to meet people where they are and, effectively, bridge your point of view with theirs to make sure that we're on the same page and, you know, we're able to communicate well. That's so especially important now that we're all remote.Corey: Just so you don't think this is too friendly of a place, let's go ahead and finish out the episode with a personal attack. Before you wound up working at LaunchDarkly. You were at Perforce. What's up with that? I mean, that seems like an awfully big company to cater to its single customer, who is of course J. Paul Reed.Tom: [laugh]. Yeah. Well, Perforce is a wonderful place. I have nothing but love for Perforce, but it is a very different landscape than LaunchDarkly, certainly. When I joined Perforce, I was supporting product called Helix ALM, which, they're still headquartered—Perforce is headquartered here in Minneapolis. I just saw some Perforce folks last week. It truly is a great place, and it is the place that introduced me to so many DevOps concepts.But that's a fair statement. Perforce has been around for a while. It has grown by acquisition over the past several years, and they are putting together new offerings by mixing old offerings together in a way that satisfies more modern needs, things like virtual production, and game development, and trying to package this up in a way that you can then have a game development environment in a box, right? So, there's a lot of things to be said for that, but it very much is a different landscape than a smaller cloud-native company. Which it's its own learning curve, let me tell you, but truly, yeah, to your Perforce, there's a lot more complexity to the products themselves because they've been around for a little bit longer.Solid, solid products, but there's a lot going on there. And it's a lot harder to learn them right upfront. As opposed to something like LaunchDarkly, which seems simple on the surface and you can get started with some of the easy concepts in implementation in, like, an hour, but then as you start digging deeper, whoof, suddenly, there's a lot more complexity hidden underneath the surface than just in terms of how this is set up, and some of those edge cases.Corey: I have to say for the backstory, for those who are unfamiliar, is I live about four miles away from J. Paul Reed, who is a known entity in reliability engineering, in the DevOps space, has been for a long time. So, to meet him, of course I had to fly to Israel. And he was keynoting DevOpsDays Tel Aviv. And I had not encountered him before, and it was this is awesome, I loved his talk, it was fun.And then I gave a talk a little while later called, “Terrible Ideas in Git.” And he's sitting there just glaring at me, holding his water bottle that is a branded Perforce thing, and it's like, “Do you work there?” He's like, “No. I just love Perforce.” It's like, “Congratulations. Having used it, I think you might be the only one.”I kid. I kid. It was great and a lot of different things. It was not quite what I needed when I needed it to but that's okay. It's gotten better and everyone else is not me, as we've discussed; people have different use cases. And that started a very long-running joke that J. Paul Reed is the entirety of the Perforce customer base.Tom: [laugh]. Yeah. And to your point, there's definitely use cases—you're talking about Perforce Version Control or Helix Core.Corey: Back in those days, I don't believe it was differentiated.Tom: It was just called Perforce. Exactly right. But yeah, as Perforce has gotten bigger, now there's different product lines; you name it. But yeah, some of those modern scalable problems, being able to handle giant binary files, being able to do automatic edge replication for globally distributed teams so that when your team in APAC comes online, they're not having to spend the first two hours of their day just getting the most recent changes from the team in the Americas and Europe. Those are problems that Perforce is absolutely solving that are out there, but it's not problems that everybody faces and you know, there's just like everybody else, we're navigating the landscape and trying to find out where the product actually fits and how it needs to evolve.Corey: And I really do wish you well on it. I think there's going to be an awful lot of—Tom: Mm-hm.Corey: —future stories where there is this integration. And you'd say, “Oh, well, what are you wishing me well for? I don't work there anymore.” But yeah, but isn't that kind of we're talking about, on some level, of building out things that are easy, that are more streamlined, that are opinionated in the right ways, I suppose. And honestly, that's the thing that I found so compelling about LaunchDarkly. I have a hard time imagining I would build anything for production use that didn't feature it these days if I were, you know, better at computers?Tom: Sure. Yeah. [laugh]. Well, we do have our opinions on how some things should work, right? Where the data is exposed because with any feature flagging system or feature management—LaunchDarkly included—you've got a set of rules, i.e. who should see this, where is it turned on? Where is it turned off? Who in your audience or user base should be able to see these features? That's the rules engine side of it.And on the other side, you've got the context to decide, well, you know, I'm Corey, I'm logging in, I'm in my mid-30s. And I know all this information about Corey, and those rules need to then be able to determine whether something should be on or off or which experience Corey gets. So, we are very opinionated over the architecture, right, and where that evaluation actually happens and how that data is exposed or where that's exposed. Because those two halves need to meet and both halves have the potential to be extremely sensitive. If I'm targeting based off of a list of 10,000 of my premium users' email addresses, I should not be exposing that list of 10,000 email addresses to a web browser or a mobile phone.That's highly insecure. And inefficient; that's a large amount of text to send, over 10,000 email addresses. And so when we're thinking about things like page load times, and people being able to push F12 to inspect the page, absolutely not, we shouldn't be exposing that there. At the same time, it's a scary prospect to say, “Hey, I'm going to send personal information about Corey over to some third-party service, some edge worker that's going to decide whether Corey should see a feature or not.” So, there's definitely architectural considerations of different use cases, but that's something that we think through all the time and make sure is secure.There's a reason—I'm going to put on my sales engineer hat here—which is to say that there is a reason that the Center for Medicare and Medicaid Services is our sponsor for FedRAMP moderate certification, in process right now, expected to be completed mid-2022. I don't know. But anybody who is unfamiliar with that, if you've ever had to go through high trust certification, you know, any of these compliances to make your regulators happy, you know that FedRAMP is so incredibly stringent. And that comes down to evaluating where are we exposing the data? Who gets to see that? Is security built in and innate into the architecture? Is that something that's been thought through?I have went so far afield from the original point that you made, but I agree, right? We've got to be opinionated about some things while still providing the freedom to use it in a way that is actually useful to you and [laugh] and we're not, you know, putting up guardrails, that mean that you've got such a narrow set of use cases.Corey: I'd like to hope—maybe I'm wrong on this—that it gets easier the more that we wind up doing these things because I don't think that it necessarily has been easy enough for an awful lot of us.Tom: When you say ‘it,' what do you mean?Corey: All of it. That's the best part, I suppose the easy parts of working on computers, which I guess might be typing if you learn it early enough.Tom: Sure. [laugh] yeah. Mario Teaches Typing, or Starcraft taught me how to type quickly. You can't type slowly or else your expansion is going to get destroyed. No, so for someone who got their formal education in music or for someone with an eighth-grade education, I agree there needs to be resources out there.And there are. Not every single StackOverflow post with a question that's been asked has the response, “That's a dumb question.” There are some out there. There's definitely a community or a group of folks who think that there is a correct way to do things and that if you're asking a question, that it's a dumb question. It really isn't. It's getting back to the diverse backgrounds and diverse schools of thought that are coming in.We don't know where someone is coming from that led them to that question without the context, and so we need to continue providing resources to folks to make it easy to self-enable and continue abstracting away the machine code parts of it in friendlier and friendlier ways. I love that there are services like Squarespace out there now, right, that allow anybody to make a website. You don't have to have a degree in computer science to spin something up and share it with the world on the web. We're going to continue to see that type of abstraction, that type of on-ramp for folks, and I'm excited to be part of it.Corey: I really look forward to it. I'm curious to see what happens next for you, especially as you continue—‘you' being the corporate ‘you' here; that's like the understood ‘you' are the royal ‘you.' This is the corporate ‘you'—continue to refine the story of what it is LaunchDarkly does, where you start, where you stop, and how that winds up playing out.Tom: Yeah, you bet. Well, in the meantime, I'm going to continue to play with things like GitHub Copilot, see how much I can autofill, and see which paths that takes me down?Corey: Oh, I've been using it for a while. It's great. Just tab-complete my entire life. It's amazing.Tom: Oh, yeah. Absolutely.Corey: [unintelligible 00:36:08] other people's secrets start working, great, that makes my AWS bill way lower when I use someone else's keys. But that's neither here nor there.Tom: Yeah, exactly. That's a next step of doing that npm install or, you know, bringing in somebody else's [laugh] tools that they've already made. Yeah, just a couple weeks ago, I was playing around with it, and I typed in two lines: I imported the LaunchDarkly SDK and the configuration for the LaunchDarkly SDK, and then I just let it autofill, whatever it wanted. It came out with about 100 lines of something or other. [laugh]. And not all of it made sense, but hey, I saw where the thought process was. It was pretty cool to see.Corey: I really want to thank you for spending as much time and energy as you have talking about how you see the world and where you folks are going. If people want to learn more. Where's the best place to find you?Tom: At launchdarkly.com. Of course, any other various different booths, DevOpsDays, we're at re:Invent, we're at QCon right now. We're at all sorts of places, so come stop by, say hi, get a demo. Maybe we'll talk.Corey: Excellent. We will be tossing links to that into the [show notes 00:37:09]. Thanks so much for your time. I really appreciate it.Tom: Corey, Thank you.Corey: Tom Totenberg, senior solutions engineer at LaunchDarkly. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry and insulting comment, and then I'll sing it to you.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Isaac Levin from Elastic joins Scott Hanselman to discuss Elastic Cloud on Azure. Elastic Cloud is an Elasticsearch and Kibana managed service - with solutions for enterprise search, observability, and security. Running Elastic on Azure enables you to take data from any source - reliably and securely, in any format - then search, analyze, and visualize that data in real time. Elastic on Azure users experience frictionless integration directly within the Azure portal, allowing for faster time to market. With deployment models to meet your unique use case, you'll gain the speed, scale, and relevance you need to react quickly to support your rapidly evolving business needs. Chapters 00:00 - Introduction 01:04 - Getting started with Elasticsearch 04:05 - Enterprise search 05:10 - App Search: Engines 06:06 - App Search: Analytics 06:58 - App Search: Web crawler 08:16 - App Search: Search UI 10:17 - App Search: Relevance tuning 12:13 - App Search: Synonyms 14:56 - App Search: Curations 17:15 - Wrap-up Recommended resources Elastic on Azure Elastic Enterprise Search Elastic Search UI Create a free account Connect Scott Hanselman | Twitter: @shanselman Isaac Levin | Twitter: @isaacrlevin Elastic | Twitter: @elastic Azure Friday | Twitter: @azurefriday

Isaac Levin from Elastic joins Scott Hanselman to discuss Elastic Cloud on Azure. Elastic Cloud is an Elasticsearch and Kibana managed service - with solutions for enterprise search, observability, and security. Running Elastic on Azure enables you to take data from any source - reliably and securely, in any format - then search, analyze, and visualize that data in real time. Elastic on Azure users experience frictionless integration directly within the Azure portal, allowing for faster time to market. With deployment models to meet your unique use case, you'll gain the speed, scale, and relevance you need to react quickly to support your rapidly evolving business needs. Chapters 00:00 - Introduction 01:04 - Getting started with Elasticsearch 04:05 - Enterprise search 05:10 - App Search: Engines 06:06 - App Search: Analytics 06:58 - App Search: Web crawler 08:16 - App Search: Search UI 10:17 - App Search: Relevance tuning 12:13 - App Search: Synonyms 14:56 - App Search: Curations 17:15 - Wrap-up Recommended resources Elastic on Azure Elastic Enterprise Search Elastic Search UI Create a free account Connect Scott Hanselman | Twitter: @shanselman Isaac Levin | Twitter: @isaacrlevin Elastic | Twitter: @elastic Azure Friday | Twitter: @azurefriday

This week we talk about the completion and release of the big project Kevin has been working on for the last several weeks. We also talk about how a single appointment takes up an entire day in Ursula's mind, compared to how it appears to Kevin. We also talk about the loneliest whale, the Bloop, and then we read your letters! Links for this Episode: Charity Spotlight: Bread for the City Charity Spotlight: Tech by Choice New to Elastic Cloud on AWS: Optimized hardware profiles for improved performance Small Changes by Alicia Witt The Bloop Clockwise EA Podcasts Interview with Ursula Vernon The 52-hertz Whale

Conversamos sobre os desafios de Monitoramento e Observabilidade de Produtos Digitais e como preparar adequadamente nossas aplicações para serem. Com participação do Ricardo Ferreira, Developer Advocate da Elastic, uma empresa que revolucionou a forma como gerenciamos logs e busca em Produtos Digitais. [ONE PLATFORM] Monitore seu produto digital em uma solução all-in-one: https://1p.elven.works/sign-up [ELASTIC] Evento ElasticON Global 2021: https://events.elastic.co/elasticon/global/register [ELASTIC] Free trial do Elastic Cloud: https://cloud.elastic.co/registration

TranscriptCorey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: You know what really grinds my gears? Well, lots of things, but in this case, let's talk about multi-cloud. Not my typical rant about multi-cloud not ever being a good best practice—because it's not—but rather how companies talk about multi-cloud. HashiCorp just did a whole survey on how multi-cloud is the future, and at no point during that entire process did they define the term. So, you wind up with a whole bunch of people responding, each one talking about different things.Are we talking about multiple clouds and we have a workload that flows between them? Are we talking about, “Well, we have some workloads on one cloud provider and a different set of workloads on other cloud providers?” Did they break it down as far as SaaS companies go of, “Yeah, we have an application and we'd like to run it all on one cloud, but it's data-heavy and we have to put it where our customers are, so of course we're on multiple cloud providers.” And then you wind up with the stories that other companies talk about, where you have a bunch of folks where their sole contribution to the ecosystem is, “Ah, you get a single pane of glass between different cloud providers.”You know who wants that? No one. The only people who really care about those things are the folks who used to sell those items and realized that if this dries up and blows away, they have nothing left to sell you. There's also a lot of cloud providers who are deep into the whole multi-cloud is the way and the light and the future because they know if you go all-in on a single cloud provider, it will certainly not be them. And then you have the folks who say, “Go in on one cloud provider and don't worry about it. It'll be fine. If you need to migrate down the road, you can do that.”And I believe that that's generally the way that you should approach things, but it gets really annoying and condescending when AWS tells that story because from their perspective, yeah, just go all-in and use Dynamo as your data store for everything even though there's really no equivalent on other cloud providers. Or, “Yeah, go ahead and just tie all of your data warehousing to some of the more intricate and non-replicable parts of S3.” And so on and so forth. And it just feels like they're pushing a lock-in narrative in many respects. I like having the idea of a strategic Exodus, where if I have to move a thing down the road, I don't have to reinvent the data model.And a classic example of what I would avoid in that case is something like Google Spanner—or Google Cloud Spanner, or whatever the one they sell us is—because yeah, it's great, and it's awesome. And you wind up with, effectively, what looks like an ACID-compliant SQL database that spans globally. But there's nothing else quite like that, so if I have to migrate off, it's not just a matter of changing APIs, I have to re-architect my entire application to be aware of the fact that I can't really have that architecture anymore, just from a data flow perspective. And looking at this across the board, I find that this is also a bit esoteric because generally speaking, the people who are talking the most about multi-cloud and wanting to avoid lock-in, are treating the cloud like it's fundamentally an extension of their own crappy data center where they run a bunch of VMs and that's it.They say they want to be multi-cloud, but they're only ever building for one cloud, and everything that they're building on top of it is just reinventing baseline primitives. “Oh, we don't trust their load balancers. We're going to run our own with Nginx or HAProxy.” Great. While you're doing that, your competitors are getting further ahead.You're not even really in the cloud: you basically did the lift part of it, declined to shift, declared victory, and really the only problem you solve for is you suck at dealing with hard drive failure, so you used to deal with outages in your data center and now your cloud provider handles it for you at a premium that's eye-wateringly high.Corey: I really love installing, upgrading, and fixing security agents in my cloud estate. Why do I say that? Because I sell things for a company that deploys an agent. There's no other reason. Because let's face it; agents can be a real headache. Well, Orca Security now gives you a single tool to detect basically every risk in your cloud environment that's as easy to install and maintain as a smartphone app. It is agentless—or my intro would have gotten me in trouble here—but it can still see deep into your AWS workloads while guaranteeing 100% coverage. With Orca Security there are no overlooked assets, no DevOps headaches—and believe me, you will hear from those people if you cause them headaches—and no performance hits on live environment. Connect your first cloud account in minutes and see for yourself at orca dot security. That's orca—as in whale—dot security as in that thing your company claims to care about but doesn't until right after it really should have.Corey: Look, I don't mean to be sitting here saying that this is how every company operates because it's not. But we see a lot of multi-cloud narrative out there, and what's most obnoxious about all of it is that it's coming from companies that are strong enough to stand on their own. And by pushing this narrative, it's increasingly getting to a point where if you're not in a multi-cloud environment, you start to think, “Maybe I'm doing something wrong.” You're not. There's no value to this.Remember, you have a business that you're trying to run, in theory. Or for those of us who are still learning things, yeah, we want to learn a cloud provider before we learn all the cloud providers, let's not kid ourselves. Pick one, go all-in on for the time being, and don't worry about what the rest of the industry is doing. We're not trying to collect them all. There is no Gartner Magic Quadrant for Pokemons and I don't think the cloud providers should be one of them.I know I've talked about this stuff before, but people keep making the same fundamental errors and it's time for me to rant on it just a smidgen more than I have already.Thank you for listening, as always to Fridays From the Field on the AWS Morning Brief. And as always, I'm Chief Cloud Economist Corey Quinn, imploring you to continue to make good choices.Announcer: This has been a HumblePod production. Stay humble.

Links: AWS's Egregious Egress: https://blog.cloudflare.com/aws-egregious-egress/ TranscriptCorey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: Hi there. Chief Cloud Economist Corey Quinn from the Duckbill Group here to more or less rant for a minute about something it's been annoying the heck out of me for a while, as anyone who follows me on Twitter or subscribes to the lastweekinaws.com newsletter, or passes me in a crowded elevator will attest to, and that is AWS's data transfer story.Back on July 23rd—of 2021, for those listening to this in future years—CloudFlare did a blog post titled AWS's Egregious Egress, and that was co-authored by Matthew Prince—CloudFlare's CEO—and Nitin Rao—who is one of their employees. Presumably. That was somewhat unclear—and it effectively tears down the obnoxious—and I mean deeply obnoxious—level of AWS data transfer pricing for egress to the outside world.And there's a bunch of things to unpack in this blog post, where they wind up comparing AWS pricing to the wholesale bandwidth market. And they go into a whole depth for those who aren't aware of how bandwidth is generally charged for. And the markups that they come up with for AWS are, in many cases, almost 8,000%, which is just ludicrous, in some respects, because—spoiler—every year, give or take, the wholesale cost of network bandwidth winds up dropping by about 10%, give or take. And the math that they've done that I'm too lazy to check, says that in effect, given that they don't tend to reduce egress bandwidth pricing, basically ever, while the wholesale market has dropped 93%, what we pay AWS hasn't. And that's obnoxious.They also talk—rather extensively—about how ingress is generally free. Now, there's a whole list of reasons that this could be true, but let's face it, when you're viewing bandwidth into AWS as being free, you start to think of it that way of, “Oh, it's bandwidth, how expensive could it possibly be?” But when you see data coming out and it charges you through the nose, you start to think that it's purely predatory. So, it already starts off with customers not feeling super great about this. Then diving into it, of course; they're pushing for the whole bandwidth alliance that CloudFlare spun up, and good for them; that's great.They have a bunch of other providers willing to play games with them and partner. Cool, I get it. It's a sales pitch. They're trying to more or less bully Amazon into doing the right thing here, in some ways. Great, not my actual point.My problem is that it's not just that data transfer is expensive in AWS land, but it's also inscrutable because, ignoring for a second what it costs to send things to the outside world, it's more obnoxious trying to figure out what it costs to send things inside of AWS. It ranges anywhere from free to very much not free. If you have a private subnet that's talking to something in the public subnet that needs to go through a managed NAT gateway, whatever your transfer price is going to be has four and a half cents per gigabyte added on to it with no price breaks for volume. So, it's very easy to wind up accidentally having some horrifyingly expensive bills for these things and not being super clear as to why. It's very challenging to look at this and not come away with the conclusion that someone at the table is the sucker.And, as anyone who plays poker is able to tell you, if you can't spot the sucker, it's you. Further—and this is the part that I wish more people paid attention to—if I'm running an AWS managed service—maybe RDS, maybe DynamoDB, maybe ElastiCache, maybe Elasticsearch—none of these things are necessarily going to be best-to-breed for the solution I'm looking at, but their replication traffic between AZs in the same region is baked into the price and you don't pay a per-gigabyte fee for this. If you want to run something else, either run it yourself on top of EC2 instances or grab something from the AWS marketplace that a partner has provided to you. There is no pattern in which that cross-AZ replication traffic is free; you pay for every gigabyte, generally two cents a gigabyte, but that can increase significantly in some places.Corey: I really love installing, upgrading, and fixing security agents in my cloud estate. Why do I say that? Because I sell things for a company that deploys an agent. There's no other reason. Because let's face it; agents can be a real headache. Well, Orca Security now gives you a single tool to detect basically every risk in your cloud environment that's as easy to install and maintain as a smartphone app. It is agentless—or my intro would have gotten me in trouble here—but it can still see deep into your AWS workloads while guaranteeing 100% coverage. With Orca Security there are no overlooked assets, no DevOps headaches—and believe me, you will hear from those people if you cause them headaches—and no performance hits on live environment. Connect your first cloud account in minutes and see for yourself at orca dot security. That's orca—as in whale—dot security as in that thing your company claims to care about but doesn't until right after it really should have.Corey: It feels predatory, it feels anti-competitive, and you look at this and you can't shake the feeling that somehow their network group is being evaluated on how much profit it can turn, as opposed to being the connective tissue that makes all the rest of their services work. Whenever I wind up finding someone who has an outsized data transfer bill when I'm doing the deep-dive analysis on what they have in their accounts, and I talk to them about this, they come away feeling, on some level, ripped off, and they're not wrong. Now, if you take a look at other providers—like Oracle Cloud is a great example of this—their retail rate is about 10% of what AWS's for the same level of traffic. In other words, get a 90% discount without signing any contract and just sign the dotted line and go with Oracle Cloud. Look, if what you're doing is bandwidth-centric, it's hard to turn your nose up at that, especially if you start kicking the tires and like what you see over there.This is the Achilles heel of what happens in the world of AWS. Now, I know I'm going to wind up getting letters about this because I always tend to whenever I rant about this that no one at any significant scale is paying retail rate for AWS bandwidth. Right, but that's sort of the point because when I'm sitting here doing back-of-the-envelope calculations on starting something new and that thing tends to be fairly heavy on data transfer—like video streaming—and I look at the retail published rates, it doesn't matter what the discount is going to be because I'm still trying to figure out if this thing has any baseline level of viability, and I run the numbers and realize, wow, 95% of my AWS bill is going to be data transfer. Well, I guess my answer is not AWS. That's not a pure hypothetical.I was speaking to someone years ago, and they have raised many tens of millions of dollars for their company since, and it's not on AWS because it can't be given their public pricing. Look, this is not me trying to beat up unnecessarily on AWS. I'm beating them up on something that frankly, has been where it is for far too long and needs to be addressed. This is not customer obsession; this is not earning trust; this is not in any meaningful way aligned with where customers are and the problems customers are trying to solve. In many cases, customers are going to be better served by keeping two copies of the data, one in each availability zone rather than trying to replicate back and forth between them because that's what the economics dictate.That's ludicrous. It should never be that way. But here we are. And here I am. I'm Chief Cloud Economist Corey Quinn here at the Duckbill Group. Thank you for listening to my rant about AWS data transfer pricing.Announcer: This has been a HumblePod production. Stay humble.

Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: And I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways that we've seen AWS used and abused in the wild. Today, we're going to be talking about the relationship between cost optimization work and investing in reservations or private pricing with AWS. This is kind of a situation conversation. Let's say you've got three months left on your EDP, or maybe your spend is reaching the point where you're starting to think about investing in, or signing an EDP. But you've also got some cost optimization opportunities that you want to work on. How do you prioritize those two ideas?Tim: I think when we're talking about this, first it's important to talk about what goes into an EDP, like, what it is and what it involves. So, EDP for AWS is Enterprise Discount Program, and what it involves is you making a monetary commitment to AWS to spend a certain amount over a certain amount of time. So, a three year EDP, you're going to spend X amount in one year, X amount the next year, and X amount the third year for a total of whatever you decide on. So, you know, AWS typically going to want 20% year-over-year growth, so you're going to say—you're going to spend a million dollars, and then a million dollars plus 20% is something like $1.2 million; then, you know, 20% of that and so forth and so on.And then so your total commit will be somewhere around, like, $3.6, $3.7 million, we'll say, right? Once you signed the EDP, that's how much you're going to get billed for, minimum. So, it's important to cost optimize before you make that commitment because if AWS is expecting you and you're on the hook to make 20% year-over-year growth, but then you optimize and you save 20% of your bill, it won't matter because you're still going to owe AWS the same amount of money even if you cost-optimize.Jesse: Yeah, I want to take a step back and talk about EDP—as we mentioned, Enterprise Discount Program—also has—there's a couple other flavors that give you a variety of different types of discounts. EDP generally focuses on a cross-service discount for a certain annual commit, but there are also private pricing agreements or private pricing addendums, and other private pricing, generally speaking, offered by AWS. All of those basically expect some amount of either spend on a yearly basis or some amount of usage on a yearly basis, in exchange for discounts on that usage. And really, that is something that, broadly speaking, we do recommend you focus on, we do recommend that you invest in those reservations, but it is important to think about that—I agree—I would say after cost optimization work.Amy: The thing is that AWS also provides discounts that are commandment required, that you don't need an EDP for, namely in reservations and savings plans. So, you would similarly be on the hook if you decide, “I have this much traffic, and I want to savings plan or reservation for it.” And then suddenly you don't have that requirement anymore, but you still have to make up that commitment.Tim: I'll say, I think too, that also matters when you're looking at things like reservations. If you're going to reserve instances, you're going to get an idea of how many you're specifically going to need, so that way you're not reserving too many, and then you optimize, you downsize, and all of a sudden, now you have all these reservations that you're not going to use.Jesse: One thing to also call out: when renewing an EDP, or private pricing, or when entering into a new agreement for any kind of private pricing with AWS, they will generally look at the last six months of your usage—either broadly speaking if it's an EDP, or specifically within a specific AWS service if it's private pricing for a specific service—and they will double, basically, that spend over the last six months and expect you to continue spending that. So, if you spent a high amount of money over the last six months, they're going to expect that kind of trend to continue, and if you enter into an agreement with that 12-month spend, essentially, going forward, and then make cost optimization changes, you're ultimately going to be on the hook for this higher level of spending you're not spending any more. So, if you focus on that cost optimization work first, it will ultimately give you the opportunity to approach AWS with a lower commit level, which may ultimately mean a lower tier of percentage discount, but ultimately, then you're not on the hook for spend that you wouldn't otherwise be spending.Tim: I think one of the main things people see, too, is when they've looked at, like, oh, what's the low hanging fruit for me to get lower the cost? They'll think, “Oh, well, I can do EDP,” because AWS is going to want you to sign on; they would love to have that guaranteed money, right? And a lot of times, that's going to be a much easier thing to do, organizationally, than the work of cost optimization because almost always, that involves engineering hours, it involves planning, it involves some changes that are going to have to be made that's probably going to be harder than just signing a contract. But again, it's super necessary because you really need to know, have eyes open, when you're going to go, and figure out what you're going to commit, whether it's private pricing agreement, or an EDP, or reservations. You want to go in there and at least decide what you want to do, what it should look like, get as optimized and as lean as you can, then make your commitments. And then once you get to an EDP, that's when you're going to want to do your reservation or savings plans purchases and things like that, so you do that with a discount across those.Jesse: Yeah, that's another important thing to point out: focus on the cost optimization work first. Get your architecture, your workloads, as optimized as possible, or as optimized as you can within the given timeframe, then focus on the investment because then you'll be able to have a much better idea of what your growth is going to look like year-over-year for an EDP or any kind of private pricing. And then after that, purchase any reservations, like reserved instances or savings plans because ultimately, then you get not only the discount from the EDP that you just signed, but any upfront payments that you make, or partial upfront payments that you make for those reservations applied towards your first year EDP. So ultimately, not only are you getting a discount on that, but you are also able to put money towards that first-year commit; you're essentially giving yourself a little bit more wiggle room by purchasing reservations after you've signed an EDP.Tim: And another way to game that system is if you know that you're going to be undertaking some projects, especially that you want to get discounts around, and you're going to need to utilize software or service or anything like that involves an AWS partner on the AWS marketplace, you're going to want to do that after you sign your EDP, too, because even though you may not get a discount on it, that money will still count towards your commit.Corey: I really love installing, upgrading, and fixing security agents in my cloud estate. Why do I say that? Because I sell things for a company that deploys an agent. There's no other reason. Because let's face it; agents can be a real headache. Well, Orca Security now gives you a single tool to detect basically every risk in your cloud environment that's as easy to install and maintain as a smartphone app. It is agentless—or my intro would have gotten me in trouble here—but it can still see deep into your AWS workloads while guaranteeing 100% coverage. With Orca Security there are no overlooked assets, no DevOps headaches—and believe me, you will hear from those people if you cause them headaches—and no performance hits on live environment. Connect your first cloud account in minutes and see for yourself at orca dot security. That's orca—as in whale—dot security as in that thing your company claims to care about but doesn't until right after it really should have.Tim: It is important to talk about the future goals for your company, from a financial perspective, both at an architectural level but also at a strategic level, so you can make good quality decisions. And, you know, to toot our own horn, that's a lot of where our expertise comes in, where we can say, “These are the order you're going to do these things in, and these are what you should prioritize.” I mean, everyone knows that in the end, the net result should still be the same. You're going to have to do the engineering and architecture work to optimize; you're going to have to do the administrative stuff to sign these agreements to get discounts, but you need to know what to prioritize and what's going to be most important, and sometimes you don't have the insight on that. And that's where if you don't, get someone in there to help you figure out what's what, what's going to give you the best, most bang for your buck, but also what's going to make the most sense for you going forward, six months, a year, two years, three years, and so forth and so on. So, it is okay to not know these things. Nobody's an expert on everything, but it behooves you to rely on the people who are experts when it's a blind spot for you.Jesse: I think that's a really good point that you make, Tim. One of the things that we see in a number of organizations that we work with is essentially a disconnect between the folks who are—well, two disconnects really: one between the folks who are doing the work day-to-day, and another between the folks who are purchasing reservations. But that also a disconnect between the people who are purchasing the reservations and potentially the people who are purchasing or investing in some kind of Enterprise Discount Program or private pricing. And to Tim's point, it's really important to get all of those people in a conversation together, get everybody in a room together, so to speak, to make sure that everybody understands what everybody else is doing so that finance and engineering and product and leadership all understand together that the cost optimization work is going on, that reservations are being purchased, that we're having a conversation about investing in some kind of private pricing with AWS. So collaboratively, collectively, everybody can make a decision together, make a data-driven decision together, that's going to ultimately help everybody, essentially, win and accomplish their goals.Amy: Speaking of collaboration, we often talk about having a good relationship with your AWS account manager, and this is one of those places that having a good rapport really works in your favor because if you are in a lot of communications with your account manager, and you know each other well, and you have a good working relationship, and they are good at their job, then they'll know that you are using XYZ service, and you're using at a high volume, they will be able to tell you, it's like, “Hey, you hit a threshold. Let's see if we can get you some extra discounts.” They'll be the ones who can actually know what those discount programs are and be able to facilitate them.Jesse: All right, well, that will do it for us this week, folks. If you've got questions you'd like us to answer please go to lastweekinaws.com/QA; fill out the form and we'll answer those questions on a future episode of the show. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us how you would cost-optimize your organization.Announcer: This has been a HumblePod production. Stay humble.

TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: And I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways we've seen AWS used and abused in the wild. Today, we're going to be talking about AWS, an open-source software. Now, that's kind of a broad topic, but there have been some specific, recent events I'll say, over the last year maybe or maybe even less, related to AWS and open-source software that really got us talking, and I wanted to have a deeper conversation with both of you on this topic.Tim: Well, you should probably start by going over some of the things that you're mentioning, when you say ‘some of these things,' what are those things, Jesse?Jesse: Yeah. So, I think the best place to start is what constitutes open-source software. And specifically, I think, not just what constitutes open-source software, but how does that differ from an open-source company?Tim: So, open-source software can be anything: Linux kernel, bash, anything like that, any Python functioning module. If you make a piece of software, whatever it is, and you license it with one of the various open-source licenses, or your own open-source license or whatever, it's something that the community kind of owns. So, when they get big, they have maintainers, everything like that, but at its essence, it's a piece of software that you can freely download and use, and then you're free to modify it as you need, and then it's up to the specifics of the license to whether you're required to send those modifications back, to include them, or to whatever. But the essence is that it's a piece of software that's free for me to use and free for me to modify under it's license.Jesse: And one of the other things I want to add to that is, correct me if I'm wrong here, but isn't a lot of open-source software is very community-owned, so there's a lot of focus on folks from the community that is using this software giving back not because they need to under the licensing, necessarily, but because they want to continue using this and making it better over time.Amy: I think one of the issues is that becomes a very opinionated kind of statement where there are a lot of people in the open-source community who feel that if you're going to use something and make changes to better suit what your needs are, that you should be able to submit those changes back to the community, or back to whoever owns the base of the software. But that said, it's like the community edition of MySQL before Microsoft bought it, where the assumption was that there's essentially a candidate of it that anyone can use without the expectation of submitting it back.Jesse: So, that's a broad definition of open-source software, but how does open-source software, broadly speaking, differ from an open-source company? I'm thinking specifically there is the open-source software of Elasticsearch, for example, or I should say, previously the open-source software of Elasticsearch that was owned by the open-source company, Elastic. So, what does that relationship look like? How does an open-source company like that differ from the open-source software itself?Tim: So, there are typically a couple of ways. Usually, a company that is the owner of an open-source product still has some kind of retention of the IP in their various licenses that they can do that with, but essentially—and this is in the words of one of the founders of Elastic—that they're benevolent dictators over the software. And so they allow folks to contribute, but they don't have to. And most of those open-source software companies will have a commercial version of that software that has other features that are not available, packages with support or some of the things like that, some kind of value-added thing that you're going to wind up paying for. The best way to describe—like you said—there's the company Elastic and then the product Elasticsearch.I relate back to before: there was Red Hat Linux, which was open-source, and then the company Red Hat. And I remember when they went public and everyone was shocked that a company can make profit off of something they gave away for free. But while the core of the software itself was free, the support was not free, nor was the add-on features that enterprises wanted. And so that tends to be kind of what the business model is, is that you create the software, it's open-source for a while to get a big user base, and then when it gets adopted by enterprises or people that really would pay for support or for other features, that's when the license tends to change, or there's a fork between the open-source version and then the commercial version.Jesse: And it definitely sounds like there can be benefits to an open-source company essentially charging for not just the open-source software, but these extra benefits like supports and additional features because I know I've traced multiple code bugs back to a piece of open-source software that there's a PR or an issue that has been sitting open for months, if not longer because the community just doesn't have the time to look into the issue, doesn't have the time to work on the issue, they are managing it on their own, separate as a side job, separate from their day-to-day work. Whereas if that is a bug that I'm tracing back to a feature in an open-source piece of software, or I should say software that I am paying for through an open-source company, I have a much clearer support path to a resolution to resolving that issue.Tim: And I think what the end up doing is then you see it more like a traditional core software model, like, you know, a la Oracle, or something like that where you pay for the software essentially, but it comes packaged with these things that you get because of it, and then there's a support contract on top of it, and then there's hosting or cloud, whatever it is, on top of that, now, but you would still end up paying for the software and then support as part of the same deal. But as you know, these are for-profit companies. People get paid for them; they are publicly traded; they sell this software; they sell this product, whether it's the services or the hosting, for profit. That is not open-source software. So, if company X that makes software X, goes under, they are acting like the software would then go under as if the software doesn't belong to the community.So, a business that goes after a business is always going to be fair play; I believe they call it capitalism. But when you talk about going after open-source software, you're looking at what Microsoft was doing in the '90s and early 2000s, with Linux and other open-source challenges to the Windows and the other paid commercial enterprise software market. When folks started using Linux and servers because it was free, customizable, and they could do pretty much everything they wanted to or version of it that they were using commercial Unices for, or even replacing Windows for, you didn't really see the commercial Unices going after it because that very specialized use cases; the user had specialized hardware. What folks were doing, they're buying Wintel machines and putting Linux on them, they were getting them without Windows licenses, or trial licenses, throwing Linux on it. And Microsoft really went after open-source; they really went after open-source.They were calling it insecure, they were calling it flash in the pan, saying it would never happen. They ran a good marketing campaign for a long time against open-source software so that people would not use it and would instead use their closed-source software. That is going after open-source, not going after quote-unquote, “Open-source companies.”Jesse: Yeah, I think that's ultimately what I want to dive into next, which is, there's been a lot of buzz about AWS going after open-source, being a risk to open-source software, specifically, with the release of AWS Managed Services for software like Elasticsearch, for example, Kubernetes, Prometheus vs. Other open-source packages that you can now run as a managed service in AWS. There's a lot of concern that AWS is basically a risk to all of these pieces of open-source software, but that doesn't necessarily seem to be the case, based on what we're talking about. One of the things that I want to dive into really specifically here is this licensing idea. Is it important to end-users? How would they know about what license they're using, or if the license changes?Tim: I'll let Amy dig in on it because she's probably the expert of three of them, but I will say one case in point, I remember where licensing did become very important was Java. JDK licenses, when Oracle started cornering the market on enclosing all the licenses, you had to use different types of Javas. So, you had to get, like, open JDK; you couldn't use Sun, Oracle Java, or whatever it was. And so that became a heavy lift of replacing packages and making sure all that stuff was in compliance, and while tracking packages, replacing them, doing all the necessary things because if you're running Java, you're probably running it in production. Why you would, I don't know, but there are those things that you would have to do in order to be able to just replace a package. The impact of the license, even if it doesn't cost a dime for usage, it still matters, and in real dollars and real engineering time.Amy: Even free licensing will cost you money if you do it wrong. The reason why I love talking about licensing is because I used to work for the government—Jesse: [laugh].Amy: —and if you think a large company like Amazon or Microsoft loves doing anything to rattle the cage of smaller businesses, it's not nearly as much as they love doing it to the government. So, any company that has a government-specific license, and the government is not using it, they will get sued and fined for a bunch of money, which sounds like a conflict between a super-large company and the government and who the hell cares about that, but this also translates the way they handle licensing for end-users and for smaller companies. So, for the most part for the end-user, you're going to look at what is generally sent to you to use any piece of licensing, the EULA, the End-User License Agreement, and you're just going to say, “Yeah, fine, this thing is 20 pages long; I'm not going to read this, it's fine.” And for most end-users, that is actually, you're good to go because they're not going to be coming after small, single-person users. What these licenses do is restrict the way larger organizations—be it the government or mid to larger companies—actually use their software, so that—this is a little dating—someone does not buy a single disk that does not report home, and then install that one disk on 20 computers, which is a thing that everyone has seen done if they've been in the industry long enough.Jesse: Yeah.Amy: Yeah. And it means things like licensing inventory is important, to the single you're using this license at home and you install Adobe on three computers, you would think it's not… would not hurt their value very much, but they also make it so that you can't even do that anymore. So, in purchased software, it makes a big deal for end-users; if it's just something free like being able to use some community SQL workbench just to mess around with stuff at home or on personal projects, you're usually going to be okay.Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like HubSpot, Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Jesse: Yeah, this is a really big issue. There's so much complexity in this space because Tim, like you said, there's some amount of capitalism here of AWS competing with open-source companies; there's business opportunities to change licensing, which can be a good thing for a company or it could be a terrible thing for a company's user base. There's lots of complexity to this issue. And I mean, in the amount of time that we've been talking, we've only really scratched the surface. I think there's so much more to this space to talk about.Tim: There really is, and there's a lot of history that we really need to cover to really paint an accurate picture. I think back when web hosting first became a thing, and everyone was running LAMP stacks and nobody was saying, “Oh, no, using cPanel is going to kill Apache.” That wasn't a thing because, yeah, it was a for-profit company that was using open-source software to make money and yet Apache still lived, and [unintelligible 00:15:00] still lived; MySQL still made it; PHP was still around. So, to say that utilizing open-source software to provide a service, to provide a paid service, is going to kill the open-source softwares, at best it's misrepresentation and omits a lot of things. So, yeah, there's a lot of stuff we can dig into, a lot of things we can cover.And the topic is broad, and so this is why it's important for us to talk about it, I think, in the context of AWS and the AWS, kind of, ecosystem is that when you see companies with big crocodile tears, saying, “Oh, yeah, AWS is trying to kill open-source,” it's like, “No, they're not trying to kill open-source.” They may be trying to go after your company, but they aren't the same.Jesse: And it feels to me like that is part of the way that the business world works. And I'm not saying that it's a great part of the way the business world works, but how can you differentiate your company in such a way that you still retain your user base if AWS releases a competing product? I'm not thrilled with the fact that AWS is releasing all these products that are competing with open-source companies, but I'm also not going to say that it's not beneficial, in some ways, for AWS customers. So, I see both sides of the coin here and I don't have a clear idea of what the best path forward is.Amy: As much as I hate the market demands it type of argument, a lot of the libraries, and open-source software, and all of these other things that AWS has successfully gone after, they've gone after ones that weren't entirely easy to use in the first place. Things like Kubernetes, and Prometheus, and MongoDB, and Elastic. These are not simple solutions to begin with, so if they didn't do it, there are a lot of other management companies that will help you deal with these very specific products. The only difference is, one of them is AWS.Jesse: [laugh]. One of them is a multibillion-dollar company.Amy: Oh, they've all got money, man.Jesse: [laugh].Amy: I mean, let's be real. At our pay grade, the difference between a multimillion-dollar and a billion-dollar company, I don't think affects you at your level at all.Jesse: No.Amy: I'm not seeing any of that difference. I am not. [laugh].Tim: Yeah, I definitely think if you all want us to dig into more of this—and we could do a lot more—let us know. If there are things you think we're wrong on, or things that you think we need to dig deeper on, yeah, we'd love to do that. Because this is a complex and nuanced topic that does have a lot of information that should be discussed so that folks can have a clear view of what the picture looks like.Jesse: Well, that'll do it for us this week, folks. If you've got questions you'd like us to answer please go to lastweekinaws.com/QA, fill out the form and we'll answer those questions on a future episode of the show.If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us your thoughts on this conversation, on AWS versus open-source software versus open-source companies.Announcer: This has been a HumblePod production. Stay humble.

TranscriptCorey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: And I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways we've seen AWS used and abused in the wild with a healthy dose of complaining about AWS for good measure. Today, we're going to be talking about a recent addition to the AWS family: AWS Application Cost Profiler.Tim: But hold on for a second, Jesse, because AWS Application Cost Profiler we can get to; that's rather unremarkable. I really want to talk about how impressed I am with AWS InfiniDash. I've been benchmarking this thing, and it is fan… tastic. It's so good. And we could probably talk about for a while, but suffice to say that I am far more impressed with AWS InfiniDash than I am with AWS Application Cost Profiler.Jesse: You know, that's fair. And I feel like InfiniDash should absolutely get credit where credit is due. I want to make sure that everybody can really understand the full breadth of everything that InfiniDash is able to accomplish. So, I want to make sure that we do get to that; maybe in a future episode, we can touch on that one. But for right now, I have lots of feelings about AWS Application Cost Profiler, and what better place to share those feelings than with two of my favorite people, Amy and Tim, and then all of you listeners who are listening in to this podcast. I can't wait to dive into this. But I think we should probably start with, what is AWS Application Cost Profiler?Amy: It is [unintelligible 00:01:54] in a trench coat.Jesse: [laugh].Amy: Which is the way AWS likes to solve problems sometimes. And in this case, it's talking about separating billing costs by tenants by service, which is certainly a lot of things that people have problems with.Jesse: That is a lot of buzzwords.Amy: A lot of words there.Jesse: Yeah. Looking at the documentation, the sales page, “AWS Application Cost Profiler is a managed service that helps us separate your AWS billing and costs by the tenants of your service.” That has a lot of buzzwords.Tim: Well, to be fair, that's also a majority of the documentation about service.Jesse: Yeah, that is fair. That is a lot of what we saw, and I think we'll dive into that with documentation in a minute. But I do want to call out before we dive into our thoughts on this service—because we did kick the tires on this service and we want to share what our experience was like, but I do want to call out that this problem that AWS Application Cost Profiler is trying to solve. This idea of cost allocation of shared resources, it is a real, valid problem and it is one that is difficult to solve.Amy: And we've had clients that have had this very explicit problem and our findings have been that it's very difficult to accurately splice usage and spend against what's essentially consumption-based metrics—which is how much a user or request is using all the way along your pipeline—if they're not using dedicated resources.Jesse: Yeah, when we talk about cost allocation, generally speaking, we talk about cost allocation from the perspective of tagging resources, broadly speaking, and moving resources into linked accounts and separating spend by linked accounts, or allocating spend by linked accounts. But if you've got a shared compute cluster, a shared database, any kind of shared resources where multiple tenants are using that infrastructure, slapping one tag on it isn't going to solve the issue. Even putting all of those shared resources in a single linked account isn't going to solve that issue. So, the problem of cost allocation for shared resource is real; it is a valid problem. So, let's talk specifically about AWS Application Cost Profiler as a solution for this problem. As I mentioned, we kicked the tires on this solution earlier this week and we have some thoughts to share.Tim: I think one of the main things around this AWS Application Profiler like I said, there's some problems that can be solved there, there's some insights that people really want to gain here, but the problem is people don't want to do a lot more work or rewrite their observability stack to do it. The problem is, that's exactly what AWS Cost Profiler seems to be doing or seems to want you to do. It doesn't get data from, I think it only gets data from certain EC2 services, and it's just, it's doing things that you can already do in other tools to do aggregation. And if I'm going to do all the work to rewrite that stack, to be able to use the Profiler, am I going to want to spend that time doing something else? I mean, that kind of comes to the bottom line about it.Jesse: Yeah, the biggest thing that I ran into, or that I experienced when we were setting up the Cost Profiler, is that documentation basically said, “Okay, configure Cost Profiler and then submit your data.” And [unintelligible 00:05:54] stop, like wait, what? Wait, what do you mean, ‘submit data?' And it said, “Okay, well now that you've got Cost Profiler as a service running, you need to upload all of the data that Cost Profiler is going to profile for you.” It boggles my mind.Tim: And it has to be in this format, and it has to have these specific fields. And so if you're not already emitting data in that format with those fields, now you have to go back and do that. And it's not really solving any problems, but it offers to create more problems.Amy: And also, if you're going to have to go through the work of instrumenting and managing all that data anyway, you could send it anywhere you wanted to. You could send it to your own database to your own visualization. You don't need Profiler after that.Jesse: Yeah, I think that's a really good point, Amy. AWS Cost Profiler assumes that you already have this data somewhere. And if not, it explicitly says—in its documentation it says, to generate reports you need to submit tenant usage data of your software applications that use shared AWS resources. So, it explicitly expects you to already have this data. And if you are going to be looking for a solution that is going to help you allocate the cost of shared resources and you already have this data somewhere else, there are better solutions out there than AWS Application Cost Profiler. As Amy said, you can send that data anywhere. AWS Application Cost Profiler probably isn't going to be the first place that you think of because it probably doesn't have as many features as other solutions.Amy: If you were going to instrument things to that level, and let's say you were using third-party services, you could normalize your own data and build out your own solution, or you can send it to a better data and analytics service. There are more mature solutions out there that require you to do less work.Corey: This episode is sponsored in part by ChaosSearch. You could run Elastic Search or Elastic Cloud or Open Search, as they're calling it now, or a self hosted out stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for App performance monitoring, cyber security. If you're using ElasticSearch consider not running ElasticSearch. They're also available now on the AWS market place, if you prefer not to go direct and have half of whatever you pay them count toward your EDP commitment. Discover what companies like, Klarna, Equifax, Armor Security and Blackboard already have. To learn more visit chaossearch.io and tell them I sent you just so you can see them facepalm yet again.Jesse: I feel like I'd missed something, broadly speaking. I get that this is a preview, I get that this is a step on the road for this solution, and I'm hoping that ultimately AWS Application Cost Profiler can automatically pull data from resources. And also, not just from EC2 compute resources, but from other shared services as well. I would love this service to be able to automatically dynamically pull this data from multiple AWS services that I already use. But this just feels like a very minimal first step to me.Tim: And let's be honest; AWS has a history of putting out services before they're ready for primetime, even if they're GA—Jesse: Yeah.Tim: —but this seems so un-useful that I'm not sure how it made it past the six-pager or the press release. It's disappointing for a GA service from AWS.Amy: What would you both like to see, other than it just being… more natively picked up by other services?Tim: I would like to see either a UI for creating the data tables that you're going to need, or a plugin that you can automatically put with those EC2 resources: an agent you can run, or a sidecar, or a collector that you just enable to gather that data automatically. Because right now, it's not really useful at all. What it's doing is basically the same thing you can do in an Excel spreadsheet. And that's being very, very honest.Jesse: Yeah, I think that's a really good point that ultimately, a lot of this data is not streamlined and that's ultimately the thing that is the most frustrating for me right now. It is asking a lot of the customer in terms of engineering time, in terms of design work, in terms of implementation details, and I would love AWS to iterate on this service by providing that dynamically, making it easier to onboard and use this service.Amy: Personally, what I would like is some either use case, or demonstration, or tutorial that shows how to track consumption costs using non-compute resources like Kinesis especially, because you're shoving a lot of things in there and you just need to be able to track these things and have that show up in some sort of visualization that's like Cost Explorer. Or even have that wired directly to Cost Explorer so that you can, from Cost Explorer, drill down to a request and be able to see what it is actually doing, and what it's actually costing. I want a lot of things.Jesse: [laugh]. But honestly, I think that's why we're here, you know? I want to make these services better. I want people to use the services. I want people to be able to allocate costs of shared resources. But it is still a hard problem to solve, and no one solution has quite solved it cleanly and easily yet.You know what? Amy, to get back to your question, that's ultimately what I would love to see, not just specifically with an AWS Application Cost Profiler necessarily, but I would love to see better native tools in AWS to help break out the cost of shared resources, to help break out and measure how tenants are using shared resources in AWS, natively. More so than this solution.Amy: I would love that. It would make so many things so much easier.Jesse: Mm-hm. I'm definitely going to be adding that to my AWS wishlist for a future episode.Tim: How many terabytes is your AWS wishlist right now?Jesse: Oh… it is long. I, unfortunately, have made so many additions to my AWS wishlist that are qualitative things—more so than quantitative things—that just aren't going to happen.Amy: You become that kid at Christmas that, they get onto Santa's lap in the mall, and it's a roller page that just hops off the platform, and just goes down the wall, and all the other kids are staring at you and ready to punch you in the face when you get off. [laugh].Jesse: [laugh]. All right, well that'll do it for us this week, folks. If you've got questions you'd like us to answer please go to lastweekinaws.com/QA, fill out the form and we'd be happy to answer that question on a future episode. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us how you allocate the costs of shared resources.Announcer: This has been a HumblePod production. Stay humble.

LinksPete and Jesse Talk Account ManagersTranscriptCorey: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the Cloud: low effort, high visibility and detection. To learn more, visit lacework.com.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: And I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways we've seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today, we're going to be talking about, really, a couple things; building your relationship with AWS, really. This stems from one of the questions that we got from a listener from a previous event. The question is, “How do the different companies that we've worked with work with AWS? Is the primary point of contact for AWS at a company usually the CTO, the VP of engineering, an architect, an ops person, a program manager, or somebody from finance, a [unintelligible 00:01:00] trainer? Who ultimately owns that relationship with AWS?”And so we're going to talk about that today. I think there's a lot of really great content in this space. Pete and I, back in the day, recorded an episode talking about building your relationship with your account manager, and with your TAM, and with AWS in general. I'll link that in the show notes. That's a great precursor to this conversation. But I think there's a lot of great opportunities to build your relationship and build rapport with AWS, as you work with AWS and as you put more things on the platform.Amy: I think one of the things we always say right off the bat is that you should introduce yourself and make a good relationship with your account manager and your technical account manager, just because they're the ones who, if you need help, they're going to be the ones to help you.Jesse: Yeah, I think one of the things that we should also take a step back and add is that if you are listening to this and you're saying to yourself, “I don't have an account manager,” that's actually wrong; you do have an account manager. Anybody who's running workloads on AWS has an account manager. Your account manager might not have reached out to you yet because usually speaking, account managers don't reach out unless they see that you're spending a certain amount of money. They usually don't start a conversation with you unless you specifically are spending a certain amount of money, have reached a certain threshold, and then they want to start talking to you about opportunities to continue using AWS, opportunities to save money, invest in AWS. But you definitely have an account manager and you should definitely start building that rapport with them as soon as possible.Amy: First question. How do you actually engage your account manager?Tim: So, there's a couple ways to do it. If you have reached a certain spend threshold where your account manager will reach out to you, it's real simple: you just reply back to them. And it kind of depends. The question most people are going to have is, “Well, why do I need to reach out to my account manager? If I just have, like, a demo account, if I'm just using free tier stuff.”You probably don't ever need to reach out to your account manager, so what are the things, typical things that people need to reach out to their account manager for? Well, typically because they want to grow and want to see what kind of discounts are offered for growth, and I want to see what I can do. Now, you can open a support ticket, you can open a billing ticket, but what will end up happening is once you reach a spend threshold, your account manager will reach out to you because they want to talk to you about what programs they have, they want to see how they can help you grow your account, they want to see what things they can do for you because for them, that means you're going to spend more money. Most account managers within a little bit of time of you opening your account and reaching a lower spend threshold, they're going to send you an email and say, “Hey, this is my name, this is how you reach me,” et cetera, et cetera. And they'll send you some emails with links to webinars or other events and things like that, and you can typically reply back to those and you'll be able to get your account manager sometimes as well. But like I said, the easiest way to get a hold of your account manager or find out who it is, is to start increasing your spend on AWS.Jesse: So, then if you're a small company, maybe a startup or maybe just a student's using AWS for the first time, likely that point of contact within a company is going to be you. From a startup perspective, maybe you are the lead engineer, maybe you are the VP of engineering, maybe you are the sole engineer in the company. We have seen most organizations that we talk to have a relationship with AWS, or build that relationship or own that relationship with AWS at a engineering management or senior leadership level. Engineering management seems to be the sweet spot because usually, senior leadership has a larger view of things on their plate than just AWS so they're focused on larger business moves for the company, but the engineering manager normally has enough context and knowledge of all of the day-to-day specifics of how engineering teams are using AWS to really be involved in that conversation with your account manager, with your technical account manager, or with your solutions architect, or whatever set of folks you have from AWS's side for an account team. And I think that's another thing that we should point out as well, which is, you will always have an account manager; you won't always have a technical account manager.The technical account manager generally comes in once you have signed an enterprise discount program agreement. So, generally speaking, that is one of the perks that comes with an EDP, but obviously, there are other components to the EDP to be mindful of as well.Tim: So, let me clarify that. You get a technical account manager when you sign up for enterprise support. You don't have to have an EDPs to have enterprise support, but when you sign up for enterprise support, you automatically get a technical account manager.Jesse: And, Tim, if you could share with everybody, what kind of things can you expect from a technical account manager?Tim: So, a technical account manager, I mean, they will do—like, all TAMs everywhere pretty much can liaise with support to escalate tickets or investigate them and see what's going on with them, try and, kind of, white-glove them into where they need to be. AWS TAM's, they also have the same—or a lot of the same access to the backend. Not your data because no one at AWS actually has access to your data or inside your systems, but they have access to the backend so they can see API calls, they can see logs, and they can see other things like that to get insight into what's going on in your system and so they can do analytics. They have insight to your billing, they can see your Cost Explorer, they can see what your contract spends are, they can see all the line items in your bills, they have access to the roadmaps, they have access to the services and the service teams so that if you need to talk to someone at a particular service team, they can arrange that meeting for you. If you need to talk to specialists SAs, they can arrange those meetings for you.With a TAM, you—and if you have enterprise support, and they're looking you for an EDP, you can have what's called an EBC or an Executive Briefing Council, where they, in non-pandemic times, they will bring you to Seattle, put you up for a couple of days and you'll have a couple of days of meetings with service teams to go over, kind of like, what the roadmap looks like, what your strategy for working with those teams are or working with those services are. And you can get good steps on how to utilize these services, whether it's going to be some more deep dives on-site, or whether it's going to be some key roadmap items that the service team is going to prioritize and other things like that. And the EBC is actually pretty neat, but you know, you have to be larger spender to get access to those. Another thing that a TAM can do is they can actually enter items on the roadmap for you. They have access to and can provide you access to betas, or pilot programs, or private releases for various services.You'll have access to a weekly email that include what launches are pending, or what releases are pending over the next week or two weeks. You'll have access to quarterly or monthly business reviews where you get access to see what your spend looks like, what your spending trends are, support ticket trends, you know, usage and analytics, and things like that. So, a TAM can be quite useful. They can do quite a lot for you, especially in the realm of cloud economics. That said, every TAM has their specialty.I mean, depending on how many customers they have, the level of engagement you may get. And, you know, some TAMs are super, super, really good at the financial aspects, some are better at the technical aspects. So, to be fair because the TAM org is so large at AWS, you don't always have the same experience with all your TAMs, and the level of depth to which they can dive is going to vary somewhat.Corey: This episode is sponsored in part by ChaosSearch. You could run Elastic Search or Elastic Cloud or Open Search, as they're calling it now, or a self hosted out stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for App performance monitoring, cyber security. If you're using ElasticSearch consider not running ElasticSearch. They're also available now on the AWS market place, if you prefer not to go direct and have half of whatever you pay them count toward your EDP commitment. Discover what companies like, Klarna, Equifax, Armor Security and Blackboard already have. To learn more visit chaossearch.io and tell them I sent you just so you can see them facepalm yet again. Amy: So, let's say we got the best TAM—even though he technically works for us now—when trying to envision what our relationship with the world's best TAM is going to be—and I just imagine that as a nice little block text on a white mug—what is that relationship going to look like? How are we going to engage with them? And even, how often should we talk to them?Jesse: I used to work for an organization that had, I believe, quarterly meetings with our account manager and our TAM, and every time we met with them, it felt like this high stakes poker game where we didn't want to show our cards and they didn't want to show their cards, but then nobody really was able to do anything productive together. And I have to say that is the exact opposite of how to engage your account manager and your TAM.Tim: Yeah, that doesn't sound great.Jesse: No, it was not great. I do not recommend that. You want to have an open, honest conversation about your roadmap, about what you want to do with AWS.Amy: They're not getting that mug.Tim: No, no.Jesse: [laugh].Tim: So, if you have a super-engaged TAM—and I will use my own experience as a TAM at AWS—that we had office hours, routinely, bi-weekly. One customer I had, I would have onsite office hours at their offices in LA, and I would have virtual office hours in offices in London. And those office hours, sometimes I'd have—we—that—we would use those to bring in, whether it was specialist SAs, whether we go over roadmap items, or tickets, or something like that, or we do architectural reviews, or cost reviews, we would schedule quarterly business reviews aside from that, typically sometimes the same day or on the same group of days, but there was typically be different than office hours. I was in their Slack channel so they needed to ping me on something that's not a ticket but a question, we could have conversations in there. A couple of their higher points of contact there had my phone number, so they would call me if something was going on. They would page me—because AWS TAMS have pagers—if they had a major issue, or, like, an outage or something [unintelligible 00:11:05] that would affect them.Jesse: I'm sorry, I just have to ask really quick. Are we talking, like, old school level pager?Tim: No, no, no. Like on your phone, like PagerDuty.Jesse: Okay, okay. I was really excited for a minute there because I kind of miss those old-school pagers.Tim: Let me say, it was like PagerDuty; it wasn't actual PagerDuty because AWS did not actually use PagerDuty. They had something internal, but PagerDuty was the closest analog.Amy: Internal PagerDuty as a Service.Tim: Something like that.Jesse: Oh, no.Amy: So, you know, if you have a very engaged TAM, you would have regular, several times a week, contact if not daily, right? Additionally, the account team will also meet internally to go over strategy, go over issues, and action items, and things like that once or twice a week. Some accounts have multiple TAM, in which case then, you know, the touchpoints are even more often.Jesse: I feel like there's so much opportunity for engagement with your AWS account team, your account manager, your TAM. It's not entirely up to you to build that relationship, but it is a relationship; it definitely requires investment and energy from both sides.Tim: And I would say in the context of who's working with a TAM, ideally, the larger contact paths you have at an org with your TAM, the better off it's going to be. So, you don't want your TAM or account team to only talk to the VP of engineering, or the DevOps manager, or the lead architect; you want them to be able to talk to your devs, and your junior devs, and your finance people, and your CTO, and other folks like that, and pretty much anyone who's a stakeholder because they can have various conversations, and they can bring concerns around. If they're talking about junior devs, your TAM can actually help them how to use CloudFormation, and how to use a AWS CLI, or do a workshop on the basics of using Kubernetes, or something like that. Whereas if you're going to have a conversation with the VP of engineering, they're going to talk about strategies, they're going to talk about roadmap items, they're going to talk about how things can affect the company, they're going to talk about EDPs and things like that. So ideally, in a successful relationship with your TAM, your TAM is going to have several people in your org are going to have that TAM's contact information and will talk with them regularly.Jesse: One of the clients that we worked with actually brought us in for a number of conversations, and brought their TAM in as part of those conversations, too. And I have to say, having the TAM involved in those conversations was fantastic because as much as I love the deep, insightful work that we do, there were certain things about AWS's roadmap that we just don't have visibility into sometimes. And the TAM had that visibility and was able to be part of those conversations on multiple different levels. The TAM was able to communicate to multiple audiences about both roadmap items from a product perspective, from a finance perspective, from an engineering architecture perspective; it was really great to have them involved in the conversation and share insights that were beneficial for multiple parties in that meeting.Tim: And oftentimes, too, involving your TAM when you do have this one thing in your bill you can't figure out, saying, “We've looked and this spend is here, but we don't know exactly why it is.” Your TAM can go back and look at the logs, or go back and look at some of the things that were spun up at the specific time and say, “Oh, here was the problem. It was when you deploy this new AMI, it caused your CPU hours to go way, way up so you had to spin up more instances.” Or a great one was a few years back when Datadog changed its API calls and a lot of people's CloudWatch costs went through the roof. And then several TAMs had to through and figure out, it was this specific call and this is how you fix that and give that guidance back to their customers to reduce their spend. So, being able to have that backend access is very, very useful, even when you are working with an optimization group like ourselves or other folks, to say, “Hey, we've noticed these things. These are the line items we want to get some insight into.” I mean, your TAM can definitely be a good partner in that.Jesse: All right, folks, well, that'll do it for us this week. If you've got questions that you'd like us to answer, please go to lastweekinaws.com/QA. Fill out the form; we'd be happy to answer those on a future show. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review. Give it a five-star rating on your podcast platform of choice and tell us, did Tim pronounce the shortening of ‘Amazon Machine Image' correctly as ‘ah-mi' or should he have said ‘A-M-I?'Amy: I heard it and I wasn't going to say it. [laugh].Jesse: [laugh].Amy: I was just going to wait for someone to send him the t-shirt.Tim: Just to note, if you put beans in your chili, you can keep your comments to yourself.Jesse: [laugh].Amy: You're just going to keep fighting about everything today, is all I'm—[laugh].Jesse: [laugh]. Oh, no.Announcer: This has been a HumblePod production. Stay humble.

Links: https://www.duckbillgroup.com/blog/aws-cost-allocation-guide-tagging-best-practices/ https://www.duckbillgroup.com/blog/aws-cost-allocation-guide-identifying-your-costs/ TranscriptCorey: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the Cloud: low effort, high visibility and detection. To learn more, visit lacework.com. Jesse: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: And I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways we've seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today, we're actually going to talk about a very specific listener question that we didn't get to last week, but really, we had so many thoughts on this topic that we wanted to break it out into its own episode. So, today we're going to be talking about tagging, and the importance of tagging, and how tagging can be used. And when I say tagging, specifically we're talking about user-defined cost allocation tags. The original question that I'll read off was from [Aaron 00:00:58].Aaron asks, “Is tagging over-recommended as a cost reporting mechanism? I recently took on managing my company's AWS bill and when talking to AWS and reading third-party blog posts about cost management, a solid tagging strategy is often extolled this step zero for understanding AWS costs. Based on what I know about AWS so far, this approach seems like it may work for some aspects of cost management, but does not seem to be a sound strategy for more formal cost reporting, like budgeting or calculating total spend for a given product or cost center. To me, these activities require complete or near-complete accuracy the tags just don't seem to be able to provide since there are some costs like data transfer that aren't tagged, and the fact that the tags are not retroactive—” that's a big one that I can say is super frustrating for me. “Is there something I'm missing here? Is there in fact, a way to use these tags to ensure that 100% of an AWS account's costs are in fact attributed back to a specific cost center accurately? It seems drastically simpler to embrace a multi-account strategy where each account is simply billed to whatever cost center makes sense to the organization.” So, Amy and Tim, again, the main question here is, is tagging over-recommended as a cost reporting mechanism?Tim: The simple answer is no, it is not over-recommended. And the question makes a lot of good points around some of the heartaches and some the problems that come with tagging, specifically about tags not being retroactive, but, if you're going to make changes to reflect changes in the past, I mean, you know, I don't really have a good answer for that, if we're being honest. But if we're talking about going forward, tracking costs from this point forward, tagging is going to be a much more concise solution than using multi-account strategy. That said, there are a lot of reasons you should use multi-account strategy and tagging together. Multi-account strategy and tagging strategies should definitely be an ‘and' situation, not an ‘or' situation. That's like pizza or steak. No. It's both pizza and steak.And I feel like because there are a number of non-cost reasons to use multiple accounts, especially in AWS, the biggest concern of which are service limits, right? Service limits, as you know, are done by account by region, so, if I have a service limit of S3 buckets that I can create—and I think that the hard limit is, like, one thousand—once I need that one thousandth and one S3 bucket, I have to create another account. That account can still be production, it can still be for all the same things that I've used for anything else, but I had to add another account so I can spin up S3 buckets. So, how do I track those, what those buckets are for, what those costs are going to be? I'm going to track those with tags.And I'm going to track those tags from the payer account, or from up in the organization. So, as you set up multiple accounts, you can have—even if they're all production, they still need to be tagged. Even if they're all dev, they still need to be tagged. If you're using the account vending machine style stuff from Control Tower where you spin up a sandbox account, you run some stuff, and then you throw it away, tagging is going to be the best way to track those costs, not just the fact that this account is named a certain thing. Names are arbitrary; they don't really reflect necessarily what they're going to be for, accounts can come and go.So, I don't necessarily like the use of name. Plus, sometimes it's hard to do that if you're doing, like, [unintelligible 00:04:21] various countries and things like that, various languages. Different things can impart different meanings. Tags also still probably use language problems, but they are arbitrary values. You know you're going to try and lump these all together; that's all that matters.So, I definitely think that, if we're using tagging, tagging is going to let you be more concise with your costs, it's going to let you apply costs across different accounts more readily, it's going to let you apply costs across different cloud providers, especially if you use one of the CMP tools like CloudHealth, or Cloudcheckr, or something like that and you run production workloads from a single cost center across multiple clouds, you're going to want to tag those in those tools, so, that way, you can keep a consistent track and more concise tracking of costs, versus just using account names. Account names after a while is going to just become unmanageable when it comes to tracking costs.Amy: I totally agree. And one of the big things that I harp on, especially on this podcast, is that if you're worried that it's not going to be as explicit as other billing methods, you will still at least have that data. You will still know per resource—if it's properly tagged—who it's supposed to be charged to and who owns it. You would make that decision on an architectural level, you should also make it for your bill, just to make sure that if you ever need that information in the future, you can go get it. You're not going to get it—since they don't happen retroactively, then you may as well do it as early as possible.Jesse: Yeah. It's super frustrating that a lot of this information is not available retroactively. And while I understand the technical limitations to that, I can't harp enough why starting to tag resources early is super, super critical to understanding that spend, and using that tagging setup, that tagging policy, to better understand your spend in a number of different ways. But again, I also want to call out that, like, I've been saying everything about tagging related to spend, there are other ways that tags can be beneficial to your organization. I've seen organizations where security needs to know, are all of the containers that were running patched to a certain level?Are all of the AMIs that we're running patched to a certain level? Tags can do that; tags can help you understand what resources are using a certain AMI version, or a certain container version, or other security pieces that are important for security to know and be able to understand that all of these resources are patched to the latest available version of whatever we're looking at. One of the things that we talk about a lot in this podcast is having conversations with other teams because I feel like cloud cost management is not just an engineering responsibility. It's a responsibility of finance, and product, and security, and IT because there's all sorts of different groups that may ultimately be using the cloud. And that's kind of important for everybody to be on the same page in terms of how you're using the cloud. And so it's not just about tagging so, you can know the cost of something, but tagging so that you can know all these other important things like security, like product details, like maybe IT details, all these other different use cases for different departments that are also involved in cloud usage.Corey: This episode is sponsored in part by ChaosSearch. You could run Elastic Search or Elastic Cloud or Open Search, as they're calling it now, or a self hosted out stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for App performance monitoring, cyber security. If you're using ElasticSearch consider not running ElasticSearch. They're also available now on the AWS market place, if you prefer not to go direct and have half of whatever you pay them count toward your EDP commitment. Discover what companies like Hubspot, Clarna, Equifax, Armor Security and Blackboard already have. To learn more visit chaossearch.io and tell them I sent you just so you can see them facepalm yet again.Tim: Yeah. I think there's this idea that comes, I think, from very legacy data center operations where you're going to use an account name to, kind of, specify what it does and where it comes from in the same way that you would use, like, a host naming scheme to define what a computer is and what it does and things like that. And I think that can be practical, but sometimes it's often short-sighted, especially as an organization grows, and you create more accounts, and you bubble up other accounts [unintelligible 00:08:21] accounts. It comes time to sign the EDP and you need to have a master payer account, you acquire some other accounts and things like that, and then all of a sudden, whatever naming schemes they used is now integrated into what your naming scheme is. And that becomes, maybe, unmanageable.So, I've always preferred to have account names. I mean, if you need to have it specified, understand it's going to just be for humans to, really quick, find it, but I'm just as content to have an account name be a UUID and then have some other kind of method for looking at what it does or assigning billing to it. Because in the end, like I said, I prefer to use tagged resources to define what they are and where they go. They are obviously going to be exceptions made for things that are, like, dev, test, UAT, or something like that, where [unintelligible 00:09:06] are different, but we're still talking about changes on an account, and then you make the changes on the account as you need. And then if it's for production, then obviously those accounts can be tagged as production. They don't have to necessarily be named production.Amy: Right, and I think, security boundaries and resource permissions aside, if you're just looking at trying to track costs to a resource, an account ID is really just one piece of information as opposed to tags, where you can just overload it with as much information as you need.Jesse: Absolutely. Now, one other thing that I do want to talk about is we're talking about a lot of good use cases for tags. We should also talk about some of the not-so-good use cases for tags, or some of the not-so-great best practices for tags that we have seen. Amy, I know specifically you had some examples that you want to talk about.Amy: Yes. [laugh]. So, this comes from having to do data normalization, back in the day. First thing you never want to do when developing your tag strategy is you want it to just determine things like casing, or whether or not you're allowed to use spaces because I've seen in different places, not just on resource tagging, but also the way information is meta-keyed, where they have their key name identical to a completely different key name, like you have ‘product owner' except ‘product owner' is capitalized in one instance and not capitalized in another instance, and these are considered to be different things within the system. Whether or not that's your intention, they will show up as different things on some visualizations. On other visualizations, they will get normalized and turned into the same thing. So, it really depends on what it is that you want your reports to look like and what you want these resources to be able to tell you.Jesse: Yeah, that's a really great point that one of the things that we haven't potentially touched on for this episode, and is covered in a number of other podcast episodes and blog posts in general is a good tagging strategy is equally as important as tagging coverage. Knowing that the tags should all be uppercase or all lowercase, or use these types of characters and not those types of characters is equally as important as making sure that those tags are applied accurately across all of your resources. So, as you are talking about tagging, as you are thinking about tagging, even in the multi-account situation, it is important to think about, what are the best practices? What are the standards that you want for your tagging? And again, this may not be a conversation that you have in a silo by yourself; this may be a conversation that you have with a number of other teams because there may be a number of other teams that need certain information from tags and need to use certain letters or special characters. And you need to incorporate all of that; you need to include all of that in the tagging policy that you create.Tim: I think it's also important, though too, that with most analytics tools, even if it's just, you know, Cost Explorer within AWS Console, you can still aggregate those tags together, especially if you're doing costs, you can absolutely aggregate multiple cases and things like that. CloudHealth, I know you can select multiples or anything that matches a pattern regardless of case and do it that way. So, it is possible to work around those mistakes. It's not a, “Oh, we didn't have our tagging schema set up correctly, so, throw your hands up and give up.” It's just something else you have to consider, and hopefully, you can normalize going forward.Jesse: Yeah, absolutely.Amy: And really, the other thing is to make sure that the tags that you choose makes sense for what you're doing. So, if you are tagging the environment and that is the only tag that you put on a resource, then just know that when you start pulling things up in Cost Explorer or Cost and Usage Report, that's the only thing you're going to see. So, you're only going to see things split up between your production account and your dev account; you're not going to be able to see what service is actually costing you more money, or what storage, as associated to a team, has suddenly decided to grow beyond the usual predictive usage patterns.Jesse: Yeah, we have some recommendations we can make if you are just getting started on your tagging journey, and I will make sure that information is shared in the [show notes 00:13:53]. But ultimately, again, it becomes a strategy conversation. It becomes a question of what are you trying to accomplish? What are the goals that you're trying to accomplish? What is the information that you want out of tagging? Because that's ultimately going to drive what you tag and why you tag.All right, that'll do it for us this week, folks. If you've got questions you'd like us to answer, please go to lastweekinaws.com/QA, fill out the form and we'd be happy to answer your question on a future episode. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review. Give it a five-star rating on your podcast platform of choice and tell us what are the most important things that you focus on in your tagging strategies? What are the things that you tag for your company?Announcer: This has been a HumblePod production. Stay humble.

About BradAuthor and Senior Executive Editor, Bloomberg TechnologyBrad Stone is the author of four books, including Amazon Unbound: Jeff Bezos and the Invention of a Global Empire,published by Simon & Schuster in May 2021. It traces the transformation of Amazon into one of the largest and most feared companies of the world and the accompanying emergence of Jeff Bezos as the richest man alive. Brad is also the author of The Everything Store: Jeff Bezos and the Age of Amazon, which chronicled the foundational early years of the company and its founder. The book, a New York Times and Wall Street Journal bestseller, was translated into more than 35 languages and won the 2013 Financial Times/Goldman Sachs Business Book of the Year Award. In 2017, he also published The Upstarts: Uber, Airbnb, and the Battle for the New Silicon Valley.Brad is Senior Executive Editor for Global Technology at Bloomberg Newswhere he oversees a team of 65 reporters and editors that covers high-tech companies, startups, cyber security and internet trends around the world. Over the last ten years, as a writer for Bloomberg Businessweek, he’s authored over two dozen cover stories on companies such as Apple, Google, Amazon, Softbank, Twitter, Facebook and the Chinese internet juggernauts Didi, Tencent and Baidu. He’s a regular contributor to Bloomberg’s technology newsletter Fully Charged, and to the daily Bloomberg TV news program, Bloomberg Technology. He was previously a San Francisco-based correspondent for The New York Times and Newsweek. A graduate of Columbia University, he is originallyfrom Cleveland, Ohio and lives in the San Francisco Bay Area with his wifeand three daughtersLinks: The Everything Store: https://www.amazon.com/Everything-Store-Jeff-Bezos-Amazon/dp/0316219282/ Amazon Unbound: https://www.amazon.com/Amazon-Unbound-Invention-Global-Empire/dp/1982132612/ Andy Jassy book review: https://www.amazon.com/gp/customer-reviews/R1Q4CQQV1ALSN0/ref=cm_cr_getr_d_rvw_ttl?ie=UTF8&ASIN=B00FJFJOLC TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It’s an awesome approach. I’ve used something similar for years. Check them out. But wait, there’s more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It’s awesome. If you don’t do something like this, you’re likely to find out that you’ve gotten breached, the hard way. Take a look at this. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That’s canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I’m a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored in part by VMware. Because let’s face it, the past year hasn’t been kind to our AWS bills, or honestly any cloud bills. The pandemic had a bunch of impacts: it forced us to move workloads to the cloud sooner than we would have otherwise, we saw strange patterns such as user traffic drops off but infrastructure spend doesn’t. What do you do about it? Well, the CloudLIVE 2021 virtual conference is your chance to connect with people wrestling with the same type of thing, be they practitioners, vendors in the space, leaders of thought—ahem, ahem—and get some behind the scenes look into various ways different companies are handling this. Hosted by CloudHealth by VMware on May 20, the CloudLIVE 2021 conference will be 100% virtual and 100% free to attend, so you really have no excuses for missing out on this opportunity to deal with people who care about cloud bills. Visit cloudlive.com/coreyto learn more and save your virtual seat today. That’s cloud L-I-V-E slash Corey. C-O-R-E-Y. Drop the E, we’re all in trouble. My thanks to VMware for sponsoring this ridiculous episode.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. Sometimes people tell me that I should write a book about Amazon. And that sounds awful. But to be sure, today, my guest is Brad Stone, someone who has written not one, but two books about Amazon, one of which coming out on May 11th, or as most of you will know while listening to this, today. Brad, thanks for joining me.Brad: Corey, thanks for having me.Corey: So, what on earth would inspire you to not just write a book about one of what is in many ways an incredibly secretive company, but then to go back and do it again?Brad: Yeah. I’m a glutton for punishment. And Corey, my hair right now is completely white way before it should be, and I think that Amazon might be responsible for some of that. So, as you contemplate your own project, consider that this company—will you already know: it can age you. They are sometimes resistant to scrutiny.So, to answer your question, I set out to write The Everything Store back in 2011, and this was a much smaller company. It was a cute little tiny internet company of about $100 billion in market value. And poor, impoverished Jeff Bezos maybe had, I’d be guessing maybe $50 billion.So anyway, it was a much different time. And that was a great experience. The company was kind of flowering as the book came out. And to my surprise, it was embraced not by Bezos or the management team, who maybe we’ll talk about didn’t love it, but by Amazon employees, and customers, and competitors, and prospective employees. And I was really proud of it that this had become a kind of definitive account of the early years of the company.And then a funny thing happened. The little cute little internet company became a juggernaut, a $1.5 trillion market cap. Bezos is the wealthiest guy in the world now with a $200 billion fortune, and Alexa, and the rise of AWS, and the Go store, and incursions into India and Mexico and other countries, I mean, so much had changed, and my definitive history felt a little out of date. And so back in 2017—also a different world, Bezos is a happily married man; he’s the CEO of Amazon, Amazon’s headquarters are in Seattle only—I set out to research and write Amazon Unbound. And as I was writing the story, yeah, just, like, the ground kept shifting under my feet.Corey: Not a lot changes in the big sphere. I mean, one of the things that Bezos said is, “Oh, what’s going to be different in 10 years? I think the better question is, ‘what’s going to not be different in 10 years?’” but watching the company shift, watching it grow, just from the outside has been a real wild ride, I’ve got to say. And I restrict myself primarily to the AWS parts because well, there’s too much to cover if you go far beyond that, and two, it’s a very different place with very different challenges around it.I viewed The Everything Store when it came out and I read that, almost like it was a biography of Jeff Bezos himself. And in some respects, Amazon Unbound feels like it hews in that direction as well, but it also goes beyond that. How do you approach separating out the story of Amazon from the story of Jeff Bezos?Brad: Yeah, you’re putting your finger on almost the core challenge, and the adjoining challenge, which is how do you create a narrative, a linear story? Often readers want a chronological story out of a miasma of overlapping events, and initiatives, and challenges. Amazon’s really decentralized; everything is happening at once. Bezos is close to some things, he was very close to Alexa. He is really distant from other things.Andy Jassy for years had a lot of independence to run AWS. So, how do you tell that story, and then keep Bezos in the center? I mean, Andy Jassy and Jeff Wilke and everyone, I mean, those are great business people. Not necessarily dynamic personalities as, Corey, you know well, but people want to read about Jeff Bezos. He is a larger-than-life figure.He’s a pioneer. He’s an innovator. He’s controversial. And so the challenge all along is to, kind of, keep him in the center. And so that’s just, like, a writing challenge. It’s a narrative challenge.And the lucky thing is that Amazon does tend to orbit around Jeff Bezos’s brain. And so in all the storytelling, even the AWS bits of the book, which we can talk about, as an author, you can always bring Bezos back just by following the facts. You’ll eventually get, in the evolution of any story, to an S Team meeting, or to an acquisition discussion where Jeff had an impact, said something insightful, walked out of a meeting, raise the bar, had impossibly high standards. So, the last thing I’ll say is, because Amazon’s so decentralized, when you write these books you have to talk to a lot of people. And then you get all the pieces of the puzzle, and you start to assemble them, and the challenge as a writer is to, kind of, keep Bezos, your main character in the lens at all times, never let him drift too far out.Corey: One of the things that I learned from it was just the way that Bezos apparently talks to his senior executives, as far as, “I will invest in this project, more than you might think I would.” I guess I’ve never really heard of a budget meeting talking about, “I”—in the first person—“Will invest.” Like, that is what happens, but for some reason the business books never put it quite that starkly or frame it quite that way. But in hindsight, it made a lot of things of my own understanding of Amazon fall into place. That makes sense.Brad: He’s got a lot of levers, ways in which he’ll back a new initiative or express his support. And one of them is simply how he spends his time. So, with Alexa in the early years, he would meet once or twice a week with that team. But another lever is just the amount of investment. And oftentimes teams will come to him—the India team is a great example—they’ll come to the S Team with a budget, and they’ll list out their priorities and their goals for the coming year, and he’ll say, “You know, you’re thinking about this all wrong. Don’t constrain yourself. Tell us what the goals are, tell us what the opportunity is, then we’ll figure out how much it costs.”And his mindset is like you can kind of break up opportunity into two categories: one are the land grabs, the big immediate opportunities where he will go all out, and India was a great example of that, I think the failed fire phone was another example, Prime Video, he doesn’t cap the investment, he wants to win. And then there are the more greenfield opportunities that he thinks he can go slower on and groceries for a long time was in that category. And there the budgets might be more constrained. The other example is the much older businesses, just like the retail business. That’s 20 years old—I have a chapter about that—and the advertising business, and he recognized that the retail business wasn’t profitable and it was depending on advertising as a crutch, and he blew it up because he thinks that those older divisions shouldn’t require investment; they should be able to stand on their own.Corey: One quote you had as well, that just really resonated with me, as far as basically my entire ethos of how I make fun of Amazon is—and I’m going to read the excerpt here. My apologies. You have to listen to your own words being read back toward you—Brad: [laugh].Corey: These were typically Amazonian names: geeky, obscure, and endlessly debated inside AWS since—according to an early AWS exec—Bezos had once mused, “You know, the name is about 3% of what matters, but sometimes 3% is the difference between winning and losing.” And I just want to call that out because I don’t think I’ve ever seen an AWS exec ever admit that names might be even 3% worth of important. Looking at how terrible some of their service names are, I would say that 3% might be an aspirational target for their worldview.Brad: [laugh]. Let me throw this back at you, Corey. Have you ever figured out why certain AWS services are Amazon and why others are AWS?Corey: I did. I got to sit down—in the before times—with then the VP of Global Marketing, Ariel Kelman—who’s now Oracle’s Chief Marketing Officer—and Jeff Barr. And the direction that they took that in was that if you could use an AWS service without getting into the AWS weeds of a bunch of other services, then it was called Amazon whatever. Amazon S3, for example, as a primitive service doesn’t need a bunch of other AWS services hooked into it, so that gets the Amazon moniker. Whereas if you’re dealing with a service that requires the integration of a whole bunch of AWS in the weeds stuff—Brad: Mmm, right.Corey: —then it’s AWS. For example, AWS Systems Manager is useless without a whole bunch of other Amazon services. And they say they don’t get it perfectly right all the time, but that is the direction that it’s gone in. And for better or worse, I still have to look a lot of them up myself because I don’t care nearly as much as their branding people do.Brad: Right. Well, I’ll tell you in the chapter about AWS, that quote comes up when the team is contemplating the names of the databases. And they do go into long debates, and I remember talking to Charlie Bell about the search for Redshift, and they go back and forth on it, and the funny thing about that one was, of course, Oracle interpreted it as a competitive slight. Its corporate color, I guess, being red, which he intended it more as a physics term. But yeah, when they were launching Aurora and Redshift, they contemplated those names quite a bit. And I don’t know if it’s 3%. I don’t know if it does matter, but certainly, those services have become really important to a lot of businesses.Corey: Oh, yeah. And once you name something, it’s really hard to rename it. And AWS does view for—better or worse—APIs as a promise, so when you build something and presented a certain way, they’re never going to turn it off. Our grandkids are going to have to deal with some of these decisions once they get into computers. That’s a problem.And I understand the ethos behind it, but again, it’s easy to make fun of names; it’s an accessible thing because let’s be very real here, a lot of what AWS does is incredibly inaccessible to people who don’t live in this space. But naming is something that everyone can get behind making fun of.Brad: Absolutely. Yep. And [laugh] it’s perhaps why they spend a lot of time on it because they know that this is going to be the shingle that they hang out to the world. I don’t know that they’re anticipating your ridicule, but it’s obviously key to the marketing process for them.Corey: Some of the more aware ones do. But that’s a different topic for a different time. One question I have for you that I wrestle with myself is I’ve been spending the last four years or so basically studying AWS all the time. And there’s a lot of things they get right; there’s a lot of things that they get wrong. But for better or worse, it’s very difficult not to come away from an in-depth study with an appreciation for an awful lot of the things that they do. At least for me.I’m not saying that I fall in love with the company and will excuse them their wrongs; I absolutely do not do that. But it is hard, bordering on impossible for me, to not come away with a deep respect for a lot of the things that they do and clearly believe. How do you feel about that? Looking at Amazon, do you come away with this with, “Ooh. Remind me to never to become a Prime member and get rid of everything with an Amazon logo in my house,” versus the you’re about to wind up wondering if they can hire you for some esoteric role? Where do you fall on that spectrum?Brad: I think I’m probably with you. I come away with an admiration. And look, I mean, let me say upfront, I am a Prime member. I have a Alexas in my home, probably more than my wife and kids are comfortable with. We watch Prime Video, we have Prime Video.We order from Amazon all the time, we ordered from Whole Foods. I’m an Amazon customer, and so part of my appreciation comes from, like all other customers, the fact that Amazon uniquely restores time to our lives rather than extracts it. I wouldn’t say that about the social networks, right? You know, those can be time-wasters. Amazon’s a great efficiency machine.But in terms of my journalism, you know, now two books and this big in-depth study in Amazon Unbound, and you have to admire what they have built. I mean, a historic American institution that has not only changed our economic reality, in ways good and bad but over the last year and a half, in the pandemic was among the few institutions that functioned properly and served as a kind of lifeline. And there is a critique in Amazon Unbound and we can talk about it, but it’s hard to come away—I think you said it well—it’s hard to come away after studying this company and studying the top executives, and how Jeff Bezos, thinks and how he has conceived products without real admiration for what they have built over the last 25 years.Corey: Well, let’s get into your critique of Amazon. What do you think is, from what you’ve seen with all of the years of research you put into this company, what’s the worst thing about them?Brad: Well, that’s a good way to put it, Corey. [laugh]. Let me—Corey: [laugh]. It’s like, talk about a target-rich opportunity. Like, “Oh, wow. It’s like my children. I can’t stand any of them. How in the world could I pick just one?” But give it a shot.Brad: Right. Well, let me start this way, which is I often will listen to their critiques from Amazon critics—and I’m sure you might feel this way as well—and just think, like, “Do they get it?” They’ll argue that Amazon exercised its size and might to buy the companies that led to Alexa. As I write in the Alexa chapter, that’s not true at all. They bought a couple of small companies, and those executives were all horrified at what Amazon was trying to do, and then they made it work.Or the critics will say, “Fifty percent or more of internet users start their product searches on Amazon. Amazon has lock-in.” That’s not true either. Lock-in on the internet is only as strong as a browser window that remains open. And you could always go find a competitor or search on a search engine.So, I find at least some of the public criticism to be a little specious. And often, these are people that complained about Walmart for ten years. And now Amazon’s the big, bad boogeyman.Corey: Oh, I still know people who refuse to do business with Walmart but buy a bunch of stuff from Amazon, and I’m looking at these things going, any complaints you have about Walmart are very difficult to avoid mapping to Amazon.Brad: Here’s maybe the distillation of the critique that’s an Amazon Unbound. We make fun of Facebook for, “Move fast and break things.” And they broke things, including, potentially, our democracy. When you look at the creation of the Amazon Marketplace, Jeff wanted a leader who can answer the question, “How would you bring a million sellers into the Amazon Marketplace?” And what that tells you is he wanted to create a system, a self-service system, where you could funnel sellers the world over into the system and sell immediately.And that happened, and a lot of those sellers, there was no friction, and many of them came from the Wild West of Chinese eCommerce. And you had—inevitably because there were no guardrails—you had fraud and counterfeit, and all sorts of lawsuits and damage. Amazon moved fast and broke things. And then subsequently tried to clean it up. And if you look at the emergence of the Amazon supply chain and the logistics division, the vans that now crawl our streets, or the semi-trailers on our highways, or the planes.Amazon moved fast there, too. And the first innings of that game were all about hiring contractors, not employees, getting them on the road with a minimum of guidance. And people died. There were accidents. You know, there weren’t just drivers flinging packages into our front yards, or going to the bathroom on somebody’s porch.That happened, but there were also accidents and costs. And so I think some of the critique is that Amazon, despite its profession that it focuses only on customers, is also very competitor-aware and competitor-driven, and they move fast, often to kind of get ahead of competitors, and they build the systems and they’re often self-service systems, and they avoid employment where it’s possible, and the result have been costs to society, the cost of moving quickly. And then on the back-end when there are lawsuits, Amazon attempts to either evade responsibility or settle cases, and then hide those from the public. And I think that is at the heart of what I show in a couple of ways in Amazon Unbound. And it’s not just Amazon; it’s very typical right now of corporate America and particularly tech companies.And part of it is the state of the laws and regulations that allow the companies to get away with it, and really restrict the rights of plaintiffs, of people who are wronged from extracting significant penalties from these companies and really changing their behavior.Corey: Which makes perfect sense. I have the luxury of not having to think about that by having a mental division and hopefully one day a real division between AWS and Amazon’s retail arm. For me at least, the thing I always had an issue with was their treatment of staff in many respects. It is well known that in the FAANG constellation of tech companies, Facebook, Amazon, Apple, Netflix, and Google, apparently, it’s an acronym and it’s cutesy. People in tech think they’re funny.But the problem is that Amazon’s compensation is significantly below that. One thing I loved in your book was that you do a breakdown of how those base salaries work, how most of it is stock-based and with a back-loaded vesting and the rest, and looking through the somewhat lengthy excerpt—but I will not read your own words to you this time—it more or less completely confirms what I said in my exposé of this, which means if we’re wrong, we’re both wrong. And we’ve—and people have been very convincing and very unified across the board. We’re clearly not wrong. It’s nice to at least get external confirmation of some of the things that I stumble over.Brad: But I think this is all part of the same thing. What I described as the move fast and break things mentality, often in a race with competition, and your issues about the quality, the tenor of work, and the compensation schemes, I think maybe and this might have been a more elegant answer to your question, we can wrap it all up under the mantle of empathy. And I think it probably starts with the founder and soon-to-be-former CEO. And look, I mean, an epic business figure, a builder, an inventor, but when you lay out the hierarchy of qualities, and attributes, and strengths, maybe empathy with the plight of others wasn’t near the top. And when it comes to the treatment of the workforce, and the white-collar employees, and the compensation schemes, and how they’re very specifically designed to make people uncomfortable, to keep them running fast, to churn them out if they don’t cut it, and the same thing in the workforce, and then the big-scale systems and marketplace and logistics—look, maybe empathy is a drag, and not having it can be a business accelerant, and I think that’s what we’re talking about, right?That some of these systems seem a little inhumane, and maybe to their credit, when Amazon recognizes that—or when Jeff has recognized it00, he’s course-corrected a little bit. But I think it’s all part of that same bundle. And maybe perversely, it’s one of the reasons why Amazon has succeeded so much.Corey: I think that it’s hard to argue against the idea of culture flowing from the top. And every anecdote I’ve ever heard about Jeff Bezos, never having met the man myself, is always filtered through someone else; in many cases, you. But there are a lot of anecdotes from folks inside Amazon, folks outside Amazon, et cetera, and I think that no one could make a serious argument that he is not fearsomely intelligent, penetratingly insightful, and profoundly gifted in a whole bunch of different ways. People like to say, “Well, he started Amazon with several $100,000 and loan from his parents, so he’s not really in any ways a self-made anything.” Well, no one is self-made. Let’s be very clear on that.But getting a few $100,000 to invest in a business, especially these days, is not that high of a stumbling block for an awful lot of folks similarly situated. He has had outsized success based upon where he started and where he wound up ending now. But not a single story that I’ve ever heard about him makes me think, yeah, that’s the kind of guy I want to be friends with. That’s the kind of guy I want to invite to a backyard barbecue and hang out with, and trade stories about our respective kids, and just basically have a social conversation with. Even a business conversation doesn’t feel like it would be particularly warm or compelling.It would be educational, don’t get me wrong, but he doesn’t strike me as someone who really understands empathy in any meaningful sense. I’m sure he has those aspects to him. I’m sure he has a warm, wonderful relationship with his kids, presumably because they still speak to him, but none of that ever leaks through into his public or corporate persona.Brad: Mmm, partially agree, partially disagree. I mean, certainly maybe the warmth you’re right on, but this is someone who’s incredibly charismatic, who is incredibly smart, who thinks really deeply about the future, and has intense personal opinions about current events. And getting a beer with him—which I have not done—with sound fantastic. Kicking back at the fireplace at his ranch in Texas, [laugh] to me, I’m sure it’s tremendously entertaining to talk to him. But when it comes to folks like us, Corey, I have a feeling it’s not going to happen, whether you want to or not.He’s also incredibly guarded around the jackals of the media, so perhaps it doesn’t make a difference one way or another. But, yeah, you’re right. I mean, he’s all business at work. And it is interesting that the turnover in the executive ranks, even among the veterans right now, is pretty high. And I don’t know, I mean, I think Amazon goes through people in a way, maybe a little less on the AWS side. You would know that better than me. But—Corey: Yes and no. There’s been some turnover there that you can also pretty easily write down to internal political drama—for lack of a better term—palace intrigue. For lack of a better term. When, for example, Adam Selipsky is going to be the new CEO of AWS as Andy Jesse ascends to be the CEO of all Amazon—the everything CEO as it were. And that has absolutely got to have rubbed some people in unpleasant ways.Let’s be realistic here about what this shows: he quit AWS to go be the CEO of Tableau, and now he’s coming back to run AWS. Clearly, the way to get ahead there is to quit. And that might not be the message they’re intending to send, but that’s something that people can look at and take away, that leaving a company doesn’t mean you can’t boomerang and go back there at a higher level in the future.Brad: Right.Corey: And that might be what people are waking up to because it used to be a culture of once you’re out, you’re out. Clearly not the case anymore. They were passed over for a promotion they wanted, “Well, okay, I’m going to go talk to another company. Oh, my God, they’re paying people in yachts.” And it becomes, at some level, time for something new.I don’t begrudge people who decide to stay; I don’t begrudge people who decide to leave, but one of my big thrusts for a long time has been understand the trade-offs of either one of those decisions and what the other side looks like so you go into it with your eyes open. And I feel like, on some level, a lot of folks there didn’t necessarily feel that they could have their eyes open in the way that they can now.Brad: Mm-hm. Interesting. Yeah. Selipsky coming back, I never thought about that, sends a strong message. And Amazon wants builders, and operators, and entrepreneurial thinking at the top and in the S Team. And the fact that Andy had a experienced leadership team at AWS and then went outside it for the CEO could be interpreted as pretty demotivating for that team. Now, they’ve all worked with Adam before, and I’ve met him and he seems like a great guy so maybe there are no hard feelings, but—Corey: I never have. He left a few months before I started this place. So, it—I get the sense that he knew I was coming and said, “Well, better get out of here. This isn’t going to go well at all.”Brad: [laugh]. I actually went to interview him for this book, and I sat in his office at Tableau thinking, “Okay, here’s a former AWS guy,” and I got to tell you, he was really on script and didn’t say anything bad, and I thought, “Okay, well, that wasn’t the best use of my time.” He was great to meet, and it was an interesting conversation, but the goss he did not deliver. And so when I saw that he got this job, I thought, well, he’s smart. He smartly didn’t burn any bridges, at least with me.Corey: This episode is sponsored in part by our friends at ChaosSearch., you could run Elasticsearch or Elastic Cloud—or OpenSearch as they’re calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you’ve come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you’re using Elasticsearch, consider not running Elasticsearch. They’re also available now in the AWS marketplace if you’d prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like HubSpot, Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: No. And it’s pretty clear that you don’t get to rise to those levels without being incredibly disciplined with respect to message. I don’t pity Andy Jesse’s new job wherein a key portion of the job description is going to be testifying before Congress. Without going into details, I’ve been in situations where I’ve gotten to ask him questions before in a real-time Q&A environment, and my real question hidden behind the question was, “How long can I knock him off of his prepared talking points?” Because I—Brad: Good luck. [laugh].Corey: Yeah. I got the answer: about two and a half seconds, which honestly was a little bit longer than I thought I would get. But yeah, incredibly disciplined and incredibly insightful, penetrating answers, but they always go right back to talking points. And that’s what you have to do at that level. I’ve heard stories—it may have been from your book—that Andy and Adam were both still friendly after Adam’s departure, they would still hang out socially and clearly, relationships are still being maintained, if oh, by the way, you’re going to be my successor. It’s kind of neat. I’m curious to see how this plays out once that transition goes into effect.Brad: Yeah, it’ll be interesting. And then also, Andy’s grand homecoming to the other parts of the business. He started in the retail organization. He was Jeff’s shadow. He ran the marketing department at very early Amazon.He’s been in all those meetings over the years, but he’s also been very focused on AWS. So, I would imagine there’s a learning curve as he gets back into the details of the other 75% of Amazon.Corey: It turns out that part of the business has likely changed in the last 15 years, just a smidgen when every person you knew over there is now 10,000 people. There was an anecdote in your book that early on in those days, Andy Jesse was almost let go as part of a layoff or a restructuring, and Jeff Bezos personally saved his job. How solid is that?Brad: Oh, that is solid. An S Team member told me that, who was Andy’s boss at the time. And the story was, in the late 90s—I hope I remember this right—there was a purge of the marketing department. Jeff always thought that marketing—in the early days marketing was purely satisfying customers, so why do we need all these people? And there was a purge of the marketing department back when Amazon was trying to right-size the ship and get profitable and survive the dotcom bust.And Jeff intervened in the layoffs and said, “Not Andy. He’s one of the most—yeah, highest ceiling folks we have.” And he made him his first full-time shadow. Oh, and that comes right from an S Team member. I won’t say the name because I can’t remember if that was on or off the record.But yeah, it was super interesting. You know what? I’ve always wondered how good of a identifier of talent and character is Bezos. And he has some weaknesses there. I mean, obviously, in his personal life, he certainly didn’t identify Lauren Sánchez’s brother as the threat that he became.You know, I tell the story in the book of the horrific story of the CEO of Amazon Mexico, who Jeff interviewed, and they hired and then later ended up what appears to be hiring an assassin to kill his wife. I tell the story in the book. It’s a horrible story. So, not to lay that at the feet of Jeff Bezos, of course, but he often I think, moves quickly. And I actually have a quote from a friend of his in the book saying, “It’s better to not be kind of paranoid, and the”—sort of—I can’t remember what the quote is.It’s to trust people rather than be paranoid about everyone. And if you trust someone wrongly, then you of course-correct. With Andy, though, he somehow had an intuitive sense that this guy was very high potential, and that’s pretty impressive.Corey: You’re never going to bet a thousand. There’s always going to be people that slip through the cracks. But learning who these people are and getting different angles on them is always interesting. Every once in a while—and maybe I’m completely wrong on this, but never having spent time one on one with Andy Jassy, I have to rely on other folks and different anecdotes, most of them, I can’t disclose the source of, but every time that I wind up hearing about these stories, and maybe I’m projecting here, but there are aspects of him where it seems like there is a genuinely nice person in there who is worried, on some level, that people are going to find out that he’s a nice person.Brad: [laugh]. I think he is. He’s extraordinarily nice. He seems like a regular guy, and what’s sort of impressive is that obviously he’s extraordinarily wealthy now, and unlike, let’s say Bezos, who’s obviously much more wealthy, but who, who really has leaned into that lifestyle, my sense is Andy does not. He’s still—I don’t know if he’s on the corporate jet yet, but at least until recently he wasn’t, and he presents humbly. I don’t know if he’s still getting as close from wherever, [unintelligible 00:32:50] or Nordstroms.Corey: He might be, but it is clear that he’s having them tailored because fit is something—I spent a lot of time in better years focusing on sartorial attention, and wherever he’s sourcing them from aside, they fit well.Brad: Okay, well, they didn’t always. Right?Corey: No. He’s, he’s… there’s been a lot of changes over the past decade. He is either discovered a hidden wellspring of being one of the best naturally talented speakers on the planet, or he’s gone through some coaching to improve in those areas. Not that he was bad at the start, but now he’s compelling.Brad: Okay. Well, now we’re talking about his clothes and his speaking style. But—Corey: Let’s be very honest here. If he were a woman, we would have been talking about that as the beginning topic of this. It’s on some level—Brad: Or we wouldn’t have because we’d know it’s improper these days.Corey: We would like to hope. But I am absolutely willing to turn it back around.Brad: [laugh]. Anyway.Corey: So, I’m curious, going back a little bit to criticisms here, Amazon has been criticized roundly by regulators and Congress and the rest—folks on both sides of the aisle—for a variety of things. What do you see is being the fair criticisms versus the unfair criticisms?Brad: Well, I mean, I think we covered some of the unfair ones. But there’s one criticism that Amazon uses AWS to subsidize other parts of the business. I don’t know how you feel about that, but until recently at least, my reading of the balance sheet was that the enormous profits of AWS were primarily going to buy more AWS. They were investing in capital assets and building more data centers.Corey: Via a series of capital leases because cash flow is king in how they drive those things there. Oh, yeah.Brad: Right. Yeah. You know, and I illustrate in the book how when it did become apparent that retail was leaning on advertising, Jeff didn’t accept that. He wanted retail to stand on its own, and it led to some layoffs and fiercer negotiations with brands, higher fees for sellers. Advertising is the free cash flow that goes in Prime movies, and TV shows, and Alexa, and stuff we probably don’t know about.So, this idea that Amazon is sort of improperly funneling money between the divisions to undercut competitors on price, I think we could put that in the unfair bucket. In the fair bucket, those are the things that we can all look at and just go, “Okay, that feels a little wrong.” So, for an example, the private brand strategy. Now, of course, every supermarket and drugstore is going to line their shelves with store brands. But when you go to an Amazon search results page these days, and they are pockmarked with Amazon brands, and Whole Foods brands, and then sponsored listings, the pay-to-play highest bidder wins.And then we now know that, at least for a couple of years, Amazon managers, private label managers were kind of peeking at the third-party data to figure out what was selling and what they should introduce is a private Amazon brand. It just feels a little creepy that Amazon as the everything store is so different than your normal Costco or your drugstore. The shelves are endless; Amazon has the data, access to the data, and the way that they’re parlaying their valuable real estate and the data at their disposal to figure out what to launch, it just feels a little wrong. And it’s a small part of their business, but I think it’s one where they’re vulnerable. The other thing is, in the book, I tried to figure out how can I take the gauge of third-party sellers?There’s so many disgruntled voices, but do they really speak for everyone? And so instead of going to the enemies, I went to every third-party seller that had been mentioned in Jeff Bezos’s shareholder letters over the past decade. And these were the allies. These were the success stories that Bezos was touting in his sacrosanct investor letter, and almost to a one, they had all become disgruntled. And so the way in which the rules of the marketplace change, the way that the fees go up, and the difficulty that sellers often have in getting a person or a guiding hand at Amazon to help them with those changes, that kind of feels wrong.And I think that maybe that’s not a source of regulation, but it could be a source of disruptive competition. If somebody can figure out how to create a marketplace that caters to sellers a little better with lower fees, then they could do to Amazon with Amazon years ago did to eBay. And considering that Marketplace is now a preponderance of sales more than even retail on amazon.com, that can end up hurting the company.Corey: Yeah, at some point, you need to continue growing things, and you’ve run out of genuinely helpful ways, and in turn in start to have to modify customer behavior in order to continue doing things, or expand into brand new markets. We saw the AWS bleeding over into Alexa as an example of that. And I think there’s a lot of interesting things still to come in spaces like that. It’s interesting watching how the Alexa ecosystem has evolved. There’s still some very basic usability bugs that drive me nuts, but at the same token, it’s not something that I think we’re going to see radically changing the world the next five years. It feels like a hobby, but also a lucrative one, and keeps people continuing to feed into the Amazon ecosystem. Do you see that playing out differently?Brad: Wait, with Alexa?Brad: Absolutely.Brad: Yeah. I agree with you. I mean, it feels like there was more promise in the early years, and that maybe they’ve hit a little bit of a wall in terms of the AI and the natural language understanding. It feels like the ecosystem that they tried to build, the app store-like ecosystem of third-party skills makers, that hasn’t crystallized in the way we hoped—in the way they hoped. And then some of these new devices like the glasses or the wristband that have Alexa feel, just, strange, right?Like, I’m not putting Alexa on my face. And those haven’t done as well. And so yeah, I think they pioneered a category: Alexa plays music and answers basic queries really well, and yet it hasn’t quite been conversational in the way that I think Jeff Bezos had hoped in the early days. I don’t know if it’s a profitable business now. I mean, they make a lot of money on the hardware, but the team is huge.I think it was, like, 10,000 people the last I checked. And the R&D costs are quite large. And they’re continuing to try to improve the AI, so I think Jeff Bezos talks about the seeds, and then the main businesses, and I don’t think Alexa has graduated yet. I think there’s still a little bit of a question mark.Corey: It’s one of those things that we remain to see. One last thing that I wanted to highlight and thank you for, was that when you wrote the original book, The Everything Store, Andy Jassy wrote a one-star review. It went into some depth about all the things that, from his perspective, you got wrong, were unfair about, et cetera.And that can be played off as a lot of different things, but you can almost set that aside for a minute and look at it as the really only time in recent memory that Andy Jassy has sat down and written something, clearly himself, and then posted it publicly. He writes a lot—Amazon has a writing culture—but they don’t sign their six-pagers. It’s very difficult to figure out where one person starts and one person stops. This shows that he is a gifted writer in many respects, and I don’t think we have another writing sample from him to compare it to.Brad: So, Corey, you’re saying I should be honored by his one-star review of The Everything Store?Corey: Oh, absolutely.Brad: [laugh].Corey: He, he just ignores me. You actually got a response.Brad: I got a response. Well.Corey: And we’ll put a link to that review in the [show notes 00:40:10] because of course we will.Brad: Yes, thank you. Do you—remember, other Amazon executives also left one-star reviews. And Jeff’s wife, and now ex-wife Mackenzie left a one-star review. And it was a part of a, I think a little bit of a reflexive reaction and campaign that Jeff himself orchestrated at my—this was understanding now, in retrospect. After the book came out, he didn’t like it.He didn’t like aspects of his family life that were represented in the book, and he asked members of the S Team to leave bad reviews. And not all of them did, and Andy did. So, you wonder why he’s CEO now. No, I’m kidding about that. But you know what?It ended up, kind of perversely, even though that was uncomfortable in the moment, ended up being good for the first book. And I’ve seen Andy subsequently, and no hard feelings. I don’t quite remember what his review said. Didn’t it, strangely, like, quote a movie or something like that?Corey: I recall that it did. It went in a bunch of different directions, and at the end—it ended with, “Well, maybe someday he’ll write the actual story. And I’m not trying to bait anyone into doing it, but this book isn’t it.” Well, in the absence of factual corrections, that’s what we go with. That is also a very Amazonian thing. They don’t tell their own story, but they’re super quick to correct the record—Brad: Yeah.Corey: —after someone says a thing.Brad: But I don’t recall him making many specific claims of anything I got wrong. But why don’t we hope that there’s a sequel review for Amazon Unbound? I will look forward to that from Andy.Corey: I absolutely hope so. It’s one of those things that we just really, I guess, hope goes in a positive direction. Now, I will say I don’t try to do any reviews that are all positive. And that’s true. There’s one thing that you wrote that I vehemently disagree with.Brad: Okay, let’s hear it.Corey: Former Distinguished Engineer and VP at AWS, Tim Bray, who resigned on conscientious objector grounds, more or less, has been a guest on the show, and I have to say, you did him dirty. You described him—Brad: How did I—what did I do? Mm-hm.Corey: Oh, I quote, “Bray, a fedora-wearing software developer”—which is true, but still is evocative in an unpleasant way—“And one of the creators of the influential web programming language, XML”—which is true, but talk about bringing up someone’s demons to haunt them. Oh, my starts.Brad: [laugh]. But wait. How is the fedora-wearing pejorative?Corey: Oh, it has a whole implication series of, and entire subculture of the gamer types, people who are misogynist, et cetera. It winds up being an unfair characterization—Brad: But he does wear a fedora.Corey: He does. And he can pull it off. He has also mentioned that he is well into retirement age, and it was a different era when he wore one. But that’s not something that people often will associate with him. It’s—Brad: I’m so naive. You’re referring to things that I do not understand what the implication was that I made. But—Corey: Oh, spend more time with the children of Reddit. You’ll catch on quickly.Brad: [laugh]. I try, I try not to do that. But thank you, Corey.Corey: Of course. So, thank you so much for taking the time to go through what you’ve written. I’m looking forward to seeing the reaction once the book is published widely. Where can people buy it? There’s an easy answer, of course, of Amazon itself, but is there somewhere you would prefer them to shop?Brad: Well, everyone can make their own decisions. I flattered if anyone decides to pick up the book. But of course, there is always their independent bookstore. On sale now.Corey: Excellent. And we will, of course, throw a link to the book in the [show notes 00:43:31]. Brad, thank you so much for taking the time to speak with me. I really appreciate it.Brad: Corey, it’s been a pleasure. Thank you.Corey: Brad Stone, author of Amazon Unbound: Jeff Bezos and the Invention of a Global Empire, on sale now wherever fine books are sold—and crappy ones, too. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice and then a multi-paragraph, very long screed telling me exactly what I got wrong.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

You can find out more about Suyog and his career here. True story, he once worked on tablets way before tablets were a thing.He's on Twitter here. You can check out Elastic Cloud and it's suite of services here.Suyog talks a bit about data gravity, a concept you can learn more about here.If you're a fan of release notes and want to get a sense of what Suyog worked on at Elastic over the years, check out his blog archives here.Thanks to our lifeboat badge winner of the week, lhf, for anwering the question: How can I get the current UTC time in a Lua script?

For our 13th installation of the data on k8s meetup, we will be talking with Senior Software Engineer Sebastien Guilloux from Elastic about Distributed workloads on k8s and how operators play a part in that! // Abstract: How easily can you run distributed workloads on Kubernetes? The initial deployment of your 10-nodes database might be easy to setup, but day-2 operations (changing the configuration, adding and removing nodes, version upgrades, etc.) are much more complicated. We'll discuss how operators can help you manage distributed workloads, and a few operator tricks we learned while working on ECK (Elastic Cloud on Kubernetes) - an operator for the Elastic stack. // Sebastien Guilloux Bio: Sebastien Guilloux is a senior software engineer at Elastic. He has spent most of his career working with distributed systems, building resilient applications, and orchestrating Apache Kafka and Elasticsearch nodes around the world. He currently works on writing a Kubernetes operator for the Elastic Stack, Elastic Cloud on Kubernetes (ECK). @_sebgl on twitter // Final thoughts This was a damn good chat and Sebastian was sooo easy to talk to and shared so much knowledge with us! thanks to all that joined and watched! ▬▬▬▬▬▬ Connect with us

Elastic est une compagnie réputée pour des produits tels que Elastic Search, Elastic Observability ou encore Elastic Security. Mais Elastic peut également héberger et gérer pour vous cette gamme de produits ; ce qui signifie que vous pouvez les utiliser sans avoir à vous soucier de leur maintenance et tout en bénéficiant pleinement de leurs fonctionnalités.Pour cela, Elastic à développé un orchestrateur pour Amazon Web Services et Google Cloud qui se charge de maintenir l'état de vos clusters. Par exemple, en cas de perte d'un noeud du cluster, un nouveau noeud est automatiquement mis en place pour le remplacer, et toutes les opérations que cela implique sont elles aussi automatiquement gérées pour vous.Dans cet épisode, je reçois François Teychené. François est ingénieur software pour Elastic, où il est en charge de l'orchestration et de l'industrialisation des clusters Elastic. Avec lui, nous allons en apprendre un peu plus sur Elastic Cloud et sur le quotidien d'un employé d'Elastic.Support the show (https://www.patreon.com/electromonkeys)

The Elastic stack is built with distribution in mind, isn't it time we started being more Kubernetes friendly? This episode, Peter Brachwitz and Anurag Gupta join us and dive into the brand new Elastic Stack operator for Kubernetes in our Elastic Cloud on Kubernetes product. Tune in for all the details or jump right into the code on our GitHub repo at https://github.com/elastic/cloud-on-k8s. In the news: Homebrew, RDB synching, and TLS integration.

It's been a while, but Aaron and Mike are back and producing even more Elasticast. This episode they cover the hugeness that is the Elastic Stack 7.0 release, Elastic Cloud Enterprise 2.2.x, a new beta release for App Search On Premises, and catch you up to the big news of licensing changes for security features. That is: more security features made free to use for both 7.x and 6.x stack users. We also tease at the brand new Elastic Cloud for Kubernetes! Stay tuned for more updates.

Christian Saide and Devin Bernosky of NS1 join Aaron and Mike to talk about what NS1 does and how they leverage the Elastic stack to provide data-driven DNS. The latest news regarding Elastic going public and changes to Kibana are mentioned and we answer the question: 'What is an availability zone in the context of Elastic Cloud?'

Mark Rittman is joined by Elastic's Mark Walkom to talk about Elasticsearch, Kibana, Logstash and the Elastic Stack; business models built-around an open-source software core; and their move into cloud services with Elastic Cloud

Mark Rittman is joined by Elastic's Mark Walkom to talk about Elasticsearch, Kibana, Logstash and the Elastic Stack; business models built-around an open-source software core; and their move into cloud services with Elastic Cloud