This week we discuss “Radical Transparency,” State of Crypto, IBM's Cloud Efforts and Zoom's Earnings. Plus, paying at the pump abroad. Runner-up Titles I'm here for the sandwich talk I want to get back to the garbage chairs Paying for Petrol Paying at the Pump Anonymize your trash I don't want to improve Making life hard for HR Radical transparency for you, not me Where'd this public cloud come from? Don't ever fall into the trap of sucking That's a Circle Rundown Coinbase Tests App for Employees to Grade Each Other During Meetings (https://www.theinformation.com/articles/coinbase-tests-app-for-employees-to-grade-each-other-during-meetings) Bridgewater Principles & Culture (https://www.bridgewater.com/principles-and-culture?utm_campaign=BPI-Bridgewater-Capitalism-RR&utm_content=bridgewater%20culture&utm_source=paid-gs) You need meeting knobs like this (https://www.youtube.com/watch?v=yGsHq-mZI8U) Seth Green's Stolen NFT Puts His New Animated Bored Ape Series in Jeopardy (https://www.themarysue.com/seth-green-stolen-nfts/) Adam Neumann's Web3 Startup Raises $70 Million From VCs, ‘Goddess Nature Token' (https://www.vice.com/en/article/88gwax/adam-neumanns-web3-startup-raises-dollar70-million-from-vcs-goddess-nature-token) How Committed Is Big Blue To The IBM Cloud? (https://www.itjungle.com/2022/05/23/how-committed-is-big-blue-to-the-ibm-cloud/) Zoom's QTR (https://twitter.com/jaminball/status/1528831823065026560?s=21&t=3K1nAM4G-1fmOzinHP6TbA) Relevant to your Interests Layoffs.fyi - Tech Layoff Tracker and Startup Layoff Lists (https://layoffs.fyi/) Cloud Foundry Launches Korifi to Ease Kubernetes Development (https://thenewstack.io/cloud-foundry-launches-korifi-to-ease-kubernetes-development/) Get The Report: State of Observability 2022 (https://www.splunk.com/en_us/form/state-of-observability/) Lonestar plans to put datacenters in the Moon's lava tubes (https://www.theregister.com/2022/05/21/lonestar_moon_datacenter/) Alex Ellis, OpenFaaS | Kubecon + Cloudnativecon Europe 2022 (https://www.youtube.com/watch?v=iyMIF4olLWI) Broadcom in talks to acquire VMware (http://reut.rs/3PwnpHv> ,conf June 13-16, 2022), June 13-16, 2022 FinOps X (https://events.linuxfoundation.org/finops-x/), June 20-21, 2022, Matt's there! DevOps Loop (https://devopsloop.io), June 22nd. Free! Coté put the agenda together. Open Source Summit North America (https://events.linuxfoundation.org/open-source-summit-north-america/), June 21-24, 2022, Matt's there! THAT Conference Wisconsin (https://that.us/call-for-counselors/wi/2022/), July 25, 2022 VMware Explore 2022, August 29 – September 1, 2022 (https://www.vmware.com/explore.html?src=so_623a10693ceb7&cid=7012H000001Kb0hQAC) SpringOne Platform (https://springone.io/?utm_source=cote&utm_medium=podcast&utm_content=sdt), SF, December 6–8, 2022. SDT news & hype Join us in Slack (http://www.softwaredefinedtalk.com/slack). Get a SDT Sticker! Send your postal address to email@example.com (mailto:firstname.lastname@example.org) and we will send you free laptop stickers! Follow us on Twitch (https://www.twitch.tv/sdtpodcast), Twitter (https://twitter.com/softwaredeftalk), Instagram (https://www.instagram.com/softwaredefinedtalk/), LinkedIn (https://www.linkedin.com/company/software-defined-talk/) and YouTube (https://www.youtube.com/channel/UCi3OJPV6h9tp-hbsGBLGsDQ/featured). Use the code SDT to get $20 off Coté's book, (https://leanpub.com/digitalwtf/c/sdt) Digital WTF (https://leanpub.com/digitalwtf/c/sdt), so $5 total. Become a sponsor of Software Defined Talk (https://www.softwaredefinedtalk.com/ads)! Recommendations Brandon: After Steve (https://www.audible.com/pd/After-Steve-Audiobook/B09CF3K1XT?qid=1653426845&sr=1-1&ref=a_search_c3_lProduct_1_1&pf_rd_p=83218cca-c308-412f-bfcf-90198b687a2f&pf_rd_r=FEFPFCQR2E9H1MTGRS5D) Matt: REI Anniversary Sales (https://www.rei.com/promotions/anniversary-sale) Screaming in the Cloud: AWS Commerce Platform (https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/stepping-onto-the-aws-commerce-platform-with-james-greenfield/) Reply All: Flying the Coop (https://gimletmedia.com/shows/reply-all/94hk5ag/187-flying-the-coop) Coté: The Baby on the Fire Escape: Creativity, Motherhood, and the Mind-Baby Problem (https://www.goodreads.com/book/show/60900832-the-baby-on-the-fire-escape). Photo Credits Banner (https://unsplash.com/photos/-p1zI3l7MuA) CoverArt (https://unsplash.com/photos/KgiXRaqYVpM)
About SimonFounder and CEO of SnapShooter a backup company Links Referenced: SnapShooter.com: https://SnapShooter.com MrSimonBennett: https://twitter.com/MrSimonBennett TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Finding skilled DevOps engineers is a pain in the neck! And if you need to deploy a secure and compliant application to AWS, forgettaboutit! But that's where DuploCloud can help. Their comprehensive no-code/low-code software platform guarantees a secure and compliant infrastructure in as little as two weeks, while automating the full DevSecOps lifestyle. Get started with DevOps-as-a-Service from DuploCloud so that your cloud configurations are done right the first time. Tell them I sent you and your first two months are free. To learn more visit: snark.cloud/duplo. Thats's snark.cloud/D-U-P-L-O-C-L-O-U-D.Corey: What if there were a single place to get an inventory of what you're running in the cloud that wasn't "the monthly bill?" Further, what if there were a way to compare that inventory to what you were already managing via Terraform, Pulumi, or CloudFormation, but then automatically add the missing unmanaged or drifted parts to it? And what if there were a policy engine to immediately flag and remediate a wide variety of misconfigurations? Well, stop dreaming and start doing; visit snark.cloud/firefly to learn more.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the things that I learned early on in my career as a grumpy Unix systems administrator is that there are two kinds of people out there: those who care about backups an awful lot, and people who haven't lost data yet. I lost a bunch of data once upon a time and then I too fell on the side of backups are super important. Here to talk with me about them a bit today is Simon Bennett, founder and CEO of SnapShooter.com. Simon, thanks for joining me.Simon: Thanks for having me. Thank you very much.Corey: It's fun to be able to talk to people who are doing business in the cloud space—in this sense too—that is not venture-backed, that is not, “Well, we have 600 people here that are building this thing out.” And similar to the way that I handle things at The Duckbill Group, you are effectively one of those legacy things known as a profitable business that self-funds. What made you decide to pursue that model as opposed to, well, whatever the polite version of bilking venture capitalists out of enormous piles of money for [unintelligible 00:01:32]?Simon: I think I always liked the idea of being self-sufficient and running a business, so I always wanted to start a physical business when I was younger, but when I got into software, I realized that that's a really easy way, no capital needed, to get started. And I tried for years and years to build products, all of which failed until finally SnapShooter actually gained a customer. [laugh].Corey: “Oh, wait, someone finally is paying money for this, I guess I'm onto something.”Simon: Yeah.Corey: And it's sort of progressed from there. How long have you been in business?Simon: We started in 2017, as… it was an internal project for a company I was working at who had problems with DigitalOcean backups, or they had problems with their servers getting compromised. So, I looked at DigitalOcean API and realized I could build something. And it took less than a week to build a product [with billing 00:02:20]. And I put that online and people started using it. So, that was how it worked.Every other product I tried before, I'd spent months and months developing it and never getting a customer. And the one time I spent less than [laugh] less than a week's worth of evenings, someone started paying. I mean, admittedly, the first person was only paying a couple of dollars a month, but it was something.Corey: There's a huge turning point where you just validate the ability and willingness for someone to transfer one dollar from their bank account to yours. It speaks to validation in a way that social media nonsense generally doesn't. It's the oh, someone is actually willing to pay because I'm adding value to what they do. That's no small thing.Simon: Yeah. There's definitely a big difference between people saying they're going to and they'd love it, and actually doing it. So.Corey: I first heard about you when Patrick McKenzie—or @patio11, as he goes by on Twitter—wound up doing a mini-thread on you about, “I've now used SnapShooter.com for real, and it was such a joy, including making a server migration easier than it would otherwise have been. Now, I have automatically monitored backups to my own S3 account for a bunch of things, which already had a fairly remote risk of failure.” And he keeps talking about the awesome aspects of it. And okay, when Patrick says, “This is neat,” that usually means it's time for me to at least click the link and see what's going on.And the thing that jumped out at me was a few things about what it is that you offer. You talk about making sure that people can sleep well at night, that it's about why backups are important, about—you obviously check the boxes and talk about how you do things and why you do them the way that you do, but it resonates around the idea of helping people sleep well at night. Because no one wants to think about backups. Because no one cares about backups; they just care an awful lot about restores, usually right after they should have cared about the backups.Simon: Yeah. This is actually a big problem with getting customers because I don't think it's on a lot of people's minds, getting backups set up until, as you said in the intro, something's gone wrong. [laugh]. And then they're happy to be a customer for life.Corey: I started clicking around and looking at your testimonials, for example, on your website. And the first one I saw was from the CEO of Transistor.fm. For those who aren't familiar with what they do, they are the company that hosts this podcast. I pay them as a vendor for all the back issues and whatnot.Whenever you download the show. It's routing through their stuff. So yeah, I kind of want them to have backups of these things because I really don't want to have all these conversations [laugh] again with everyone. That's an important thing. But Transistor's business is not making sure that the data is safe and secure; it's making podcasts available, making it easy to publish to them.And in your case, you're handling the backup portion of it so they can pay their money and they set it up effectively once—set it and forget it—and then they can go back to doing the thing that they do, and not having to fuss with it constantly. I think a lot of companies get it wrong, where they seem to think that people are going to make sustained, engaged efforts in whatever platform or tool or service they build. People have bigger fish to fry; they just want the thing to work and not take up brain sweat.Simon: Yeah. Customers hardly ever log in. I think it's probably a good sign when they don't have to log in. So, they get their report emails, and that's that. And they obviously come back when they got new stuff to set up, but from a support point of view is pretty, pretty easy, really, people don't—[laugh] constantly on there.Corey: From where I sit, the large cloud providers—and some of the small ones, too—they all have backup functionality built into the offering that they've got. And some are great, some are terrible. I assume—perhaps naively—that all of them do what it says on the tin and actually back up the data. If that were sufficient, you wouldn't have any customers. You clearly have customers. What is it that makes those things not work super well?Simon: Some of them are inflexible. So, some of the providers have built-in server backups that only happen weekly, and six days of no backups can be a big problem when you've made a mistake. So, we offer a lot of flexibility around how often you backup your data. And then another key part is that we let you store your data where you want. A lot of the providers have either vendor lock-in, or they only store it in themselves. So… we let you take your data from one side of the globe to the other if you want.Corey: As anyone who has listened to the show is aware, I'm not a huge advocate for multi-cloud for a variety of excellent reasons. And I mean that on a per-workload basis, not, “Oh, we're going to go with one company called Amazon,” and you use everything that they do, including their WorkMail product. Yeah, even Amazon doesn't use WorkMail; they use Exchange like a real company would. And great, pick the thing that works best for you, but backups have always been one of those areas.I know that AWS has great region separation—most of the time. I know that it is unheard of for there to be a catastrophic data loss story that transcends multiple regions, so the story from their side is very often, oh, just back it up to a different region. Problem solved. Ignoring the data transfer aspect of that from a pricing perspective, okay. But there's also a risk element here where everyone talks about the single point of failure with the AWS account that it's there, people don't talk about as much: it's your payment instrument; if they suspend your account, you're not getting into any region.There's also the story of if someone gets access to your account, how do you back that up? If you're going to be doing backups, from my perspective, that is the perfect use case, to put it on a different provider. Because if I'm backing up from, I don't know, Amazon to Google Cloud or vice versa, I have a hard time envisioning a scenario in which both of those companies simultaneously have lost my data and I still care about computers. It is very hard for me to imagine that kind of failure mode, it's way out of scope for any disaster recovery or business continuity plan that I'm coming up with.Simon: Yeah, that's right. Yeah, I haven't—[laugh] I don't have that in my disaster recovery plan, to be honest about going to a different cloud, as in, we'll solve that problem when it happens. But the data is, as you say, in two different places, or more. But yeah, the security one is a key one because, you know, there's quite a lot of surface area on your AWS account for compromising, but if you're using either—even a separate AWS account or a different provider purely for storage, that can be very tightly controlled.Corey: I also appreciate the idea that when you're backing stuff up between different providers, the idea of owning both sides of it—I know you offer a solution where you wind up hosting the data as well, and that has its value, don't get me wrong, but there are also times, particularly for regulated industries, where yeah, I kind of don't want my backup data just hanging out with someone else's account with whatever they choose to do with it. There's also the verification question, which again, I'm not accusing you of in any way, shape, or form of being nefarious, but it's also one of those when I have to report to a board of directors of like, “Are you sure that they're doing what they say they're doing?” It's a, “Well, he seemed trustworthy,” is not the greatest answer. And the boards ask questions like that all the time. Netflix has talked about this where they backup a rehydrate-the-business level of data to Google Cloud from AWS, not because they think Amazon is going to disappear off the face of the earth, but because it's easier to do that and explain it than having to say, “Well, it's extremely unlikely and here's why,” and not get torn to pieces by auditors, shareholders, et cetera. It's the path of least resistance, and there is some validity to it.Simon: Yeah, when you see those big companies who've been with ransomware attacks and they've had to either pay the ransom or they've literally got to build the business from scratch, like, the cost associated with that is almost business-ending. So, just one backup for their data, off-site [laugh] they could have saved themselves millions and millions of pounds. So.Corey: It's one of those things where an ounce of prevention is worth a pound of cure. And we're still seeing that stuff continue to evolve and continue to exist out in the ecosystem. There's a whole host of things that I think about like, “Ooh, if I lost, that would be annoying but not disastrous.” When I was going through some contractual stuff when we were first setting up The Duckbill Group and talking to clients about this, they would periodically ask questions about, “Well, what's your DR policy for these things?” It's, “Well, we have a number of employees; no more than two are located in the same city anywhere, and we all work from laptops because it is the 21st century, so if someone's internet goes out, they'll go to a coffee shop. If everyone's internet goes out, do you really care about the AWS bill that month?”It's a very different use case and [unintelligible 00:11:02] with these things. Now, let's be clear, we are a consultancy that fixes AWS bills; we're not a hospital. There's a big difference in the use case and what is acceptable in different ways. But what I like is that you have really build something out that lets people choose their own adventure in how managed they want it to be, what the source is, what the target should be. And it gives people enough control but without having to worry about the finicky parts of aligning a bunch of scripts that wind up firing off in cron jobs.Simon: Yeah. I'd say a fair few people run into issues running scripts or, you know, they silently fail and then you realize you haven't actually been running backups for the last six months until you're trying to pull them, even if you were trying to—Corey: Bold of you to think that I would notice it that quickly.Simon: [laugh]. Yeah, right. True. Yeah, that's presuming you have a disaster recovery plan that you actually test. Lots of small businesses have never even heard of that as a thing. So, having as us, kind of, manage backups sort of enables us to very easily tell people that backups of, like—we couldn't take the backup. Like, you need to address this.Also, to your previous point about the control, you can decide completely where data flows between. So, when people ask us about what's GDPR policies around data and stuff, we can say, “Well, we don't actually handle your data in that sense. It goes directly from your source through almost a proxy that you control to your storage.” So.Corey: The best answer: GDPR is out of scope. Please come again. And [laugh] yeah, just pass that off to someone else.Simon: In a way, you've already approved those two: you've approved the person that you're managing servers with and you've already approved the people that are doing storage with. You kind of… you do need to approve us, but we're not handling the data. So, we're handling your data, like your actual customer; we're not handling your customer's customer's data.Corey: Oh, yeah. Now, it's a valuable thing. One of my famous personal backup issues was okay, “I'm going to back this up onto the shared drive,” and I sort of might have screwed up the backup script—in the better way, given the two possible directions this can go—but it was backing up all of its data and all the existing backup data, so you know, exponential growth of your backups. Now, my storage vendor was about to buy a boat and name it after me when I caught that. “Oh, yeah, let's go ahead and fix that.”But this stuff is finicky, it's annoying, and in most cases, it fails in silent ways that only show up as a giant bill in one form or another. And not having to think about that is valuable. I'm willing to spend a few hours setting up a backup strategy and the rest; I'm not willing to tend it on an ongoing basis, just because I have other things I care about and things I need to get done.Simon: Yeah. It's such a kind of simple and trivial thing that can quickly become a nightmare [laugh] when you've made a mistake. So, not doing it yourself is a good [laugh] solution.Corey: So, it wouldn't have been a @patio11 recommendation to look at what you do without having some insight into the rest of the nuts and bolts of the business and the rest. Your plans are interesting. You have a free tier of course, which is a single daily backup job and half a gig of storage—or bring your own to that it's unlimited storage—Simon: Yep. Yeah.Corey: Unlimited: the only limits are your budget. Yeah. Zombo.com got it slightly wrong. It's not your mind, it's your budget. And then it goes from Light to Startup to Business to Agency at the high end.A question I have for you is at the high end, what I've found has been sort of the SaaS approach. The top end is always been a ‘Contact Us' form where it's the enterprise scope of folks where they tend to have procurement departments looking at this, and they're going to have a whole bunch of custom contract stuff, but they're also not used to signing checks with fewer than two commas in them. So, it's the signaling and the messaging of, “Reach out and talk to us.” Have you experimented with that at all, yet? Is it something you haven't gotten to yet or do you not have interest in serving that particular market segment?Simon: I'd say we've been gearing the business from starting off very small with one solution to, you know, last—and two years ago, we added the ability to store data from one provider to a different provider. So, we're sort of stair-stepping our way up to enterprise. For example, at the end of last year, we went and got certificates for ISO 27001 and… one other one, I can't remember the name of them, and we're probably going to get SOC 2 at some point this year. And then yes, we will be pushing more towards enterprises. We add, like, APIs as well so people can set up backups on the fly, or so they can put it as part of their provisioning.That's hopefully where I'm seeing the business go, as in we'll become under-the-hood backup provider for, like, a managed hosting solution or something where their customers won't even realize it's us, but we're taking the backups away from—responsibility away from businesses.Corey: For those listeners who are fortunate enough to not have to have spent as long as I have in the woods of corporate governance, the correct answer to, “Well, how do we know that vendor is doing what they say that they're doing,” because the, “Well, he seemed like a nice guy,” is not going to carry water, well, here are the certifications that they have attested to. Here's copies under NDA, if their audit reports that call out what controls they claim to have and it validates that they are in fact doing what they say that they're doing. That is corporate-speak that attests that you're doing the right things. Now, you're going to, in most cases, find yourself spending all your time doing work for no real money if you start making those things available to every customer spending 50 cents a year with you. So generally, the, “Oh, we're going to go through the compliance, get you the reports,” is one of the higher, more expensive tiers where you must spend at least this much for us to start engaging down this rabbit hole of various nonsense.And I don't blame you in the least for not going down that path. One of these years, I'm going to wind up going through at least one of those certification approaches myself, but historically, we don't handle anything except your billing data, and here's how we do it has so far been sufficient for our contractual needs. But the world's evolving; sophistication of enterprise buyers is at varying places and at some point, it'll just be easier to go down that path.Simon: Yeah, to be honest, we haven't had many, many of those customers. Sometimes we have people who come in well over the plan limits, and that's where we do a custom plan for them, but we've not had too many requests for certification. But obviously, we have the certification now, so if anyone ever [laugh] did want to see it under NDA, we could add some commas to any price. [laugh].Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on premises, private cloud, and they just announced a fully managed service on AWS and Azure called BigAnimal, all one word.Don't leave managing your database to your cloud vendor because they're too busy launching another half dozen manage databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications, including Oracle, to the cloud.To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: What I like as well is that you offer backups for a bunch of different things. You can do snapshots from, effectively, every provider. I'm sorry, I'm just going to call out because I love this: AWS and Amazon LightSail are called out as two distinct things. And Amazonians will say, “Oh, well, under the hood, they're really the same thing, et cetera.” Yeah, the user experience is wildly different, so yeah, calling those things out as separate things make sense.But it goes beyond that because it's not just, “Well, I took a disk image. There we go. Come again.” You also offer backup recipes for specific things where you could, for example, back things up to a local file and external storage where someone is. Great, you also backup WordPress and MongoDB and MySQL and a whole bunch of other things.A unified cloud controller, which is something I have in my house, and I keep thinking I should find a way to back that up. Yeah, this is great. It's not just about the big server thing; it's about having data living in managed services. It's about making sure that the application data is backed up in a reasonable, responsible way. I really liked that approach. Was that an evolution or is that something you wound up focusing on almost from the beginning?Simon: It was an evolution. So, we started with the snapshots, which got the business quite far to be honest and it was very simple. It was just DigitalOcean to start with, actually, for the first two years. Pretty easy to market in a way because it's just focused on one thing. Then the other solutions came in, like the other providers and, you know, once you add one, it was easy to add many.And then came database backups and file backups. And I just had those two solutions because that was what people were asking for. Like, they wanted to make sure their whole server snapshot, if you have a whole server snapshot, the point in time data for MySQL could be corrupt. Like, there could be stuff in RAM that a MySQL dump would have pulled out, for example. Like… there's a possibility that the database could be corrupt from a snapshot, so people were asking for a bit of, more, peace of mind with doing proper backups of MySQL.So, that's what we added. And it soon became apparent when more customers were asking for more solutions that we really needed to, like, step back and think about what we're actually offering. So, we rebuilt this whole, kind of like, database engine, then that allowed us to consume data from anywhere. So, we can easily add more backup types. So, the reason you can see all the ones you've listed there is because that's kind of what people have been asking for. And every time someone comes up with a new, [laugh], like, a new open-source project or database or whatever, we'll add support, even ones I've never heard of before. When people ask for some weird file—Corey: All it takes is just waiting for someone to reach out and say, hey, can you back this thing up, please?Simon: Yeah, exactly, some weird file-based database system that I've never ever heard of. Yeah, sure. Just give us [laugh] a test server to mess around with and we'll build, essentially, like, we use bash in the background for doing the backups; if you can stream the data from a command, we can then deal with the whole management process. So, that's the reason why. And then, I was seeing in, like, the Laravel space, for example, people were doing MySQL backups and they'd have a script, and then for whatever reason, someone rotated the passwords on the database and the backup script… was forgotten about.So, there it is, not working for months. So, we thought we could build a backup where you could just point it at where the Laravel project is. We can get all the config we need at the runtime because it's all there with the project anyway, and then thus, you never need to tell us the password for your database and that problem goes away. And it's the same with WordPress.Corey: I'm looking at this now just as you go through this, and I'm a big believer in disclaiming my biases, conflicts of interest, et cetera. And until this point, neither of us have traded a penny in either direction between us that I'm ever aware of—maybe you bought a t-shirt or something once upon a time—but great, I'm about to become a customer of this because I already have backup solutions for a lot of the things that you currently support, but again, when you're a grumpy admin who's lost data in the past, it's, “Huh, you know what I would really like? That's right, another backup.” And if that costs me a few hundred bucks a year for the peace of mind is money well spent because the failure mode is I get to rewrite a whole lot of blog posts and re-record all podcasts and pay for a whole bunch of custom development again. And it's just not something that I particularly want to have to deal with. There's something to be said for a holistic backup solution. I wish that more people thought about these things.Simon: Can you imagine having to pull all the blog posts off [unintelligible 00:22:19]? [laugh]—Corey: Oh, my got—Simon: —to try and rebuild it.Corey: That is called the crappiest summer internship someone has ever had.Simon: Yeah.Corey: And that is just painful. I can't quite fathom having to do that as a strategy. Every once in a while some big site will have a data loss incident or go out of business or something, and there's a frantic archiving endeavor that happens where people are trying to copy the content out of the Google Search Engine's cache before it expires at whatever timeline that is. And that looks like the worst possible situation for any sort of giant backup.Simon: At least that's one you can fix. I mean, if you were to lose all the payment information, then you've got to restitch all that together, or anything else. Like, that's a fixable solution, but a lot of these other ones, if you lose the data, yeah, there's no two ways around it, you're screwed. So.Corey: Yeah, it's a challenging thing. And it's also—the question also becomes one of, “Well, hang on. I know about backups on this because I have this data, but it's used to working in an AWS environment. What possible good would it do me sitting somewhere else?” It's, yeah, the point is, it's sitting somewhere else, at least in my experience. You can copy it back to that sort of environment.I'm not suggesting this is a way that you can run your AWS serverless environment on DigitalOcean, but it's a matter of if everything turns against you, you can rebuild from those backups. That's the approach that I've usually taken. Do you find that your customers understand that going in or is there an education process?Simon: I'd say people come for all sorts of reasons for why they want backup. So, having your data in two places for that is one of the reasons but, you know, I think there's a lot of reasons why people want peace of mind: for either developer mistakes or migration mistakes or hacking, all these things. So, I guess the big one we come up with a lot is people talking about databases and they don't need backups because they've got replication. And trying to explain that replication between two databases isn't the same as a backup. Like, you make a mistake you drop—[laugh] you run your delete query wrong on the first database, it's gone, replicated or not.Corey: Right, the odds of me fat-fingering an S3 bucket command are incredibly likelier than the odds of AWS losing an entire region's S3 data irretrievably. I make mistakes a lot more than they tend to architecturally, but let's also be clear, they're one of the best. My impression has always been the big three mostly do a decent job of this. The jury's still out, in my opinion, on other third-party clouds that are not, I guess, tier one. What's your take?Simon: I have to be careful. I've got quite good relationships with some of these. [laugh].Corey: Oh, of course. Of course. Of course.Simon: But yes, I would say most customers do end up using S3 as their storage option, and I think that is because it is, I think, the best. Like, is in terms of reliability and performance, some storage can be a little slow at times for pulling data in, which could or could not be a problem depending on what your use case is. But there are some trade-offs. Obviously, S3, if you're trying to get your data back out, is expensive. If you were to look at Backblaze, for example, as well, that's considerably cheaper than S3, especially, like, when you're talking in the petabyte-scale, there can be huge savings there. So… they all sort of bring their own thing to the table. Personally, I store the backups in S3 and in Backblaze, and in one other provider. [laugh].Corey: Oh, yeah. Like—Simon: I like to have them spread.Corey: Like, every once in a while in the industry, there's something that happens that's sort of a watershed moment where it reminds everyone, “Oh, right. That's why we do backups.” I think the most recent one—and again, love to them; this stuff is never fun—was when that OVH data center burned down. And OVH is a somewhat more traditional hosting provider, in some respects. Like, their pricing is great, but they wind up giving you what amounts to here as a server in a rack. You get to build all this stuff yourself.And that backup story is one of those. Oh, okay. Well, I just got two of them and I'll copy backups to each other. Yeah, but they're in the same building and that building just burned down. Now, what? And a lot of people learned a very painful lesson. And oh, right, that's why we have to do that.Simon: Yeah. The other big lesson from that was that even if the people with data in a different region—like, they'd had cross-regional backups—because of the demand at the time for accessing backups, if you wanted to get your data quickly, you're in a queue because so many other people were in the same boat as you're trying to restore stored backups. So, being off-site with a different provider would have made that a little easier. [laugh].Corey: It's a herd of elephants problem. You test your DR strategy on a scheduled basis; great, you're the only person doing it—give or take—at that time, as opposed to a large provider has lost a region and everyone is hitting their backup service simultaneously. It generally isn't built for that type of scale and provisioning. One other question I have for you is when I make mistakes, for better or worse, they're usually relatively small-scale. I want to restore a certain file or I will want to, “Ooh, that one item I just dropped out of that database really should not have been dropped.” Do you currently offer things that go beyond the entire restore everything or nothing? Or right now are you still approaching this from the perspective of this is for the catastrophic case where you're in some pain already?Simon: Mostly the catastrophic stage. So, we have MySQL [bin logs 00:27:57] as an option. So, if you wanted to do, like, a point-in-time of store, which… may be more applicable to what you're saying, but generally, its whole, whole website recovery. For example, like, we have a WordPress backup that'll go through all the WordPress websites on the server and we'll back them up individually so you can restore just one. There are ways that we have helped customers in the past just pull one table, for example, from a backup.But yeah, we geared towards, kind of, the set and the forget. And people don't often restore backups, to be honest. They don't. But when they do, it's obviously [laugh] very crucial that they work, so I prefer to back up the whole thing and then help people, like, if you need to extract ten megabytes out of an entire gig backup, that's a bit wasteful, but at least, you know, you've got the data there. So.Corey: Yeah. I'm a big believer in having backups in a variety of different levels. Because I don't really want to do a whole server restore when I remove a file. And let's be clear, I still have that grumpy old Unix admin of before I start making changes to a file, yeah, my editor can undo things and remembers that persistently and all. But I have a disturbing number of files and directories whose names end in ‘.bac' with then, like, a date or something on it, just because it's—you know, like, “Oh, I have to fix something in Git. How do I do this?”Step one, I'm going to copy the entire directory so when I make a pig's breakfast out of this and I lose things that I care about, rather than having to play Git surgeon for two more days, I can just copy it back over and try again. Disk space is cheap for those things. But that's also not a holistic backup strategy because I have to remember to do it every time and the whole point of what you're building and the value you're adding, from my perspective, is people don't have to think about it.Simon: Yes. Yeah yeah yeah. Once it's there, it's there. It's running. It's as you say, it's not the most efficient thing if you wanted to restore one file—not to say you couldn't—but at least you didn't have to think about doing the backup first.Corey: I really want to thank you for taking the time out of your day to talk to me about all this. If people want to learn more for themselves, where can they find you?Simon: So, SnapShooter.com is a great place, or on Twitter, if you want to follow me. I am @MrSimonBennett.Corey: And we will, of course, put links to that in the [show notes 00:30:11]. Thank you once again. I really appreciate it.Simon: Thank you. Thank you very much for having me.Corey: Simon Bennett, founder and CEO of SnapShooter.com. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this episode, please leave a five-star review on your podcast platform of choice, whereas if you've hated this episode, please leave a five-star review on your podcast platform of choice, along with an angry insulting comment that, just like your backup strategy, you haven't put enough thought into.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
If I asked you to list all of the relationships in your life, we'd be here a while, wouldn't we?! From friendships to romances and everything in between. We're human, and God designed us to engage with others! That's why I'm so excited to share this insightful conversation with Eric Barker, the Internet's resident relationship expert. Of course, you know, I just had to ask him what made him decide to dive into the science of relationships in the first place, and his answer did not disappoint my friends! Listen in as Eric talks to us about how he challenged some of the age-old relationship advice we've all received with science and skepticism. And how that experience led him to more fulfilling friendships, love, and community. This is so, so good. Eric Barker is the author of The Wall Street Journal bestseller “Barking Up the Wrong Tree,” which has sold over half a million copies and been translated into 19 languages. It was even the subject of a question on “Jeopardy!” His new book, “Plays Well with Others,” was released earlier this month. Listen in to learn more about: What surprised Eric the most in his research for this book on relationships, [Hint: It has absolutely everything to do with friendships!] How to navigate negative and positive sentiment override and how they impact our relationships Productive ways to begin an argument or hard discussion that preserves both parties' humanity in the conversation Favorite quotes: ~Friendships make us happier than any other relationship yet friendships don't get the kind of respect or attention because friendship is completely voluntary. ~ We are just not that good at reading others. Human beings in general default to trust and cooperation. We do want information but not all the information. In some ways it is our brain trying to protect us. ~ Screaming matches only lead to divorce 40% of the time. The issue is that you scream when you care and when you stop screaming you stop caring. What precedes divorce is living parallel lives where you don't interact and don't yell and scream, because you don't even care anymore. ~ The four things that are more predictive in divorce are all to do with how you talk, how you communicate. They are - Criticism, Defensiveness, Stonewalling and Contempt. ~ Friendship Is fantastic, makes us happier than anything else but there's a whole new level of synergy when friends become a group, a community. Links to great things we discussed: Memento Theme song - David Julyan Severance - TV Series Ozark - TV series Amazon Kindle - App Bose Noise canceling headphones - Amazon Plays well with others - Book Barking up the wrong tree- Eric Barker's previous book Eric Barker's website Doctor Strange - Movie Ray LaMontagne ft. Sierra Ferrell - I Was Born to Love You Bare Minerals - Well rested eye brightener Confident Motherhood Community Hope you loved this episode! Be sure to subscribe in iTunes and slap some stars on a review! :) xo, Alli
A listener posted this on our Facebook page: What is up with my 5-year-old? All of a sudden she is talking back, yelling at the top of her lungs, not listening, kicking and screaming, throwing tantrums, throwing things at me, and giving me teenager vibes. (I do not allow any of this behavior in my house.) I'm at a loss for what to do. Is this boundary-seeking behavior? Is something happening at school? Is it because I'm working more? Is it a phase? We've been sending her to her room to chill out because no one wants to hang with a banshee, but she screams and screams while throwing her stuff everywhere and kicking the walls. She goes immediately into red brain and it's a 30 minute process for her to calm down and talk to me. Really excited to spend summer break with an insane asylum escapee. It's starting to rub off on the 2 year old. If you find yourself saying "What is wrong with this kid?" it's always good to pause and ask if there is actually something wrong. Kids who are having headaches or stomachaches can express that distress in the most baffling of ways. But as children grow, they alternate between stages of equilibrium and disequilibrium, so it's also possible that your suddenly-banshee child is entering a developmentally appropriate phase. Even if it's driving you batty. Figuring out what works to address it– at your child's actual level of maturity– is key. Amy gives advice towards how to assess the situation. Here's the link to the book Amy mentions: Your Six-Year-Old: Defiant But Loving, by Louise Bates Ames Learn more about your ad choices. Visit megaphone.fm/adchoices
Support Topic Lords on Patreon and get episodes a week early! (https://www.patreon.com/topiclords) Lords: * Nate * http://mommysbestgames.com/ * https://twitter.com/MommysBestGames/ * Fred * https://twitter.com/brobbeh Topics: * Why do rice cookers play music but not microwaves? * https://twitter.com/_LucasRizzotto/status/1516205625662836739 * If all your needs are met in a utopian society will we still need sports and competition? * Hot Ice, Cool Sounds * Goblin Time by Emma * https://mastodon.art/@emmajuettner/108103474257806186 * Esper says: "My long-held suspicion has always been that humans are generally way less prone to salt-related health issues than suggested by nutritional science. I think a lot of that stuff was ginned up by cigarette companies trying to deflect blame. But I try not to go full-on conspiracy theory about it." * Nerds candy has a Dungeons and Dragons themed box. Is this good for D&D? * If the four horsemen of the apocalypse were recruiting for a fifth member, how would you convince them to hire you? Microtopics: * A shmup about how Microsoft screwed up their Xbox 360 dashboard. * Screaming for 24 hours straight and having a heart attack, or vice versa. * Remastering your obscure protest game and needing to build a museum explaining the context for the protest. * An educational title about how to find games on the new Xbox 360 dashboard. * An infinite runner with extremely awkward horse controls. * Using all the right dark patterns. * Meeting an indie game developer in San Francisco and recognizing him because he's a white guy with a beard and glasses who likes Dark Souls. * A pot that's specifically for rice and it knows when to school cooking the rice. * Only rice cookers get to sing a song. * Installing an AI in your microwave so your microwave can try to murder you. * A codependent washing machine. * Landlords in California doing an end run around renter's rights. * The keyless entry system on your apartment that has a battery backup just so it can play "These Boots are Made for Walking" when it can't let you in because the power is out. * 3D printing some laundry to fold because people are bored in your Utopia. * Extremely soporific TV. * A mashup of music and sports. * Gathering to watch a live band play while a chef makes crepes. * Whether you can be bored while watching two things at the same time. * An orchestra trying to live-score a basketball game. * Trying to listen to an arcade game in the arcade. * Goblins in our community. * A bullet point list of how you can help a goblin. * Spicy radish waffles. * Blackwheat crepes and whether they're really black. * Lemonade crepes. * Dipping a sandwich in mustard. * The pros and cons of salting food. * Chickens laying pre-salted eggs. * Paying a monthly subscription to Monsanto to keep up the injections that make your tongue exude its own salt. * Getting in line to hate Monsanto. * Whether nerds are more or less likely to eat Nerds candy. * A candy for people who forgot D&D exists. * Candy&D. * A series of Nerds-themed D&D campaigns. * Hiring people and paying them. * Star Trek Jeopardy where 70% of the answers are 1980s pop culture because the federation is obsessed with the 80s. * Seeing yet another college sports question in the New York Times crossword and deciding whether to get irrationally angry at Will Shortz. * What Jim embodies that is the worst thing. * Death, war, famine, pestilence, crypto and late-onset melatonin. * John Cleese teaching sex ed in a boarding school. * Starting with death and going downhill from there. * Convincing Death to not kill anyone and Death just rides off and mopes. * Whether the four horsemen listen to podcasts or if the internet is down during the apocalypse. * The four new horsemen of the new apocalypse: crypto, gaslighting, late-onset melatonin and Elon Musk. * How to find people to follow on Mastodon. * How to be a part of a human-sized community. * The Topic Lords subreddit. * The photo of a middle school basketball team that somebody posted to the Topic Lords subreddit. * How to keep spam bots out of your Discord. * Batbarian.
Imran tempts fate by nipping to the office ahead of Alfie's Naming Ceremony as Elliot prepares to make strong allegations. Fiz is nervous enough ahead of a dinner date with Phill and his friends and she's sent rushing out of the restaurant when details of her past come out over the poppodums. It's a big week for Max as he discovers his fate from the judge for spiking Amy's drink and learns which school he'll be attending after being expelled from Weathy High. Daniel is encouraged to apply for the permanent teaching role at school but issues with his family weigh heavy on his mind. Jacob's personal hygiene issues give Eileen the fright of her life. Tim wins a romantic couple's overnight getaway at one of Debbie's hotels and orders some medicinal enhancement for the occasion. Gail likes Carry on Screaming. The Edinburgh Trip was a success. Michael needs podcast recommendations.
This week on 2G1C we sit back with Chente and make fun of Nic Cage in the Wicker Man!If you like wax melts and smell good stuff! Check out our sponsor at https://www.etsy.com/listing/1214448022/spooky-soy-wax-melt-box-set-highly and use code TFTP20 for 20% offCheck out House of Mysterious Secrets @ HouseofMysteriousSecrets.com and use our discount code 4130 to save 10% on some awesome horror merch now!!!https://instagram.com/tales_from_the_podcasthttps://twitter.com/TalesFromThePodhttps://facebook.com/groups/talesftalesfromthepodcast.comromthepodcastAnd can contact me through my and email us here at email@example.com
Craig and Troy discuss a very familiar parable which is very often interpreted incorrectly. Your works don't count. Oh, and substantival participles. Be sure to subscribe to this podcast and give us a great review on iTunes or wherever you get your podcasts! Hosts: Pr. Craig Donofrio and Pr. Troy Neujahr Email us: ForYouRadio@1517.org www.1517.org/foryou St. James Lutheran Church www.stjameslcms.church St. Peter's Lutheran Church www.Stpeterslc.org
Here we are! Back for another episode! We're talking our trip to Madison Square Garden to see the HAIM sisters, the SheHulk Trailer, all that Doctor Who news, and Conversations with Friends. Plus, there's new music from Florence + the Machine and Kendrick Lamar and what exactly is Hayley Kiyoko announcing with her new music video?
About WesleyWesley Faulkner is a first-generation American, public speaker, and podcaster. He is a founding member of the government transparency group Open Austin and a staunch supporter of racial justice, workplace equity, and neurodiversity. His professional experience spans technology from AMD, Atlassian, Dell, IBM, and MongoDB. Wesley currently works as a Developer Advocate, and in addition, co-hosts the developer relations focused podcast Community Pulse and serves on the board for SXSW.Links Referenced: Twitter: https://twitter.com/wesley83 Polywork: https://polywork.com/wesley83 Personal Website: https://www.wesleyfaulkner.com/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Finding skilled DevOps engineers is a pain in the neck! And if you need to deploy a secure and compliant application to AWS, forgettaboutit! But that's where DuploCloud can help. Their comprehensive no-code/low-code software platform guarantees a secure and compliant infrastructure in as little as two weeks, while automating the full DevSecOps lifestyle. Get started with DevOps-as-a-Service from DuploCloud so that your cloud configurations are done right the first time. Tell them I sent you and your first two months are free. To learn more visit: snark.cloud/duplo. Thats's snark.cloud/D-U-P-L-O-C-L-O-U-D.Corey: What if there were a single place to get an inventory of what you're running in the cloud that wasn't "the monthly bill?" Further, what if there were a way to compare that inventory to what you were already managing via Terraform, Pulumi, or CloudFormation, but then automatically add the missing unmanaged or drifted parts to it? And what if there were a policy engine to immediately flag and remediate a wide variety of misconfigurations? Well, stop dreaming and start doing; visit snark.cloud/firefly to learn more.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I am joined again for a second time this year by Wesley Faulkner. Last time we spoke, he was a developer advocate. And since then, as so many have, he's changed companies. Wesley, thank you for joining me again. You're the Head of Community at SingleStore, now. Congrats on the promotion.Wesley: Thank you. It's been a very welcome change. I love developer advocates and developer advocacy. But I love people, too, so it's almost, I think, very analogous to the ebbs and flow that we all have gone through, through the pandemic, and leaning into my strong suits.Corey: It's a big deal having a ‘head of' in a role title, as opposed to Developer Advocate, Senior Developer Advocate. And it is a different role. It's easy to default into the world of thinking that it's a promotion. Management is in many ways orthogonal to what it takes to succeed in an actual role. And further, you're not the head of DevRel, or DevRelopers or whatever you want to call the term. You are instead the Head of Community. How tied is that to developer relations, developer advocacy, or other things that we are used to using as terms of art in this space?Wesley: If we're talking about other companies, I would say the Head of Community is something that's under the umbrella of developer relations, where it's just a peer to some of the other different elements or columns of developer relations. But in SingleStore specifically, I have to say that developer relations in terms of what you think about whole umbrella is very new to the company. And so, I consider myself the first person in the role of developer relations by being the Head of Community. So, a lot of the other parts are being bolted in, but under the focus of developer as a community. So, I'm liaisoning right now as helping with spearheading some of the design of the activities that the advocates do, as well as architecting the platform and the experiences of people coming in and experiencing SingleStore through the community's perspective.So, all that to say is, what I'm doing is extremely structured, and a lot of stuff that we're doing with the efficacy, I'm using some of my expertise to help guide that, but it's still something that's kind of like an offshoot and not well integrated at the moment.Corey: How has it changed the way that you view the function of someone who's advocating to developers, which is from my cynical perspective, “Oh, it's marketing, but we don't tell people it's marketing because they won't like it.” And yes, I know, I'll get emails about that. But how does it differ from doing that yourself versus being the head of the function of a company? Because leadership is a heck of a switch? I thought earlier in my career that oh, yeah, it's a natural evolution of being a mediocre engineer. Time to be a mediocre manager. And oh, no, no, I aspired to be a mediocre manager. It's a completely different skill set and I got things hilariously wrong. What's it like for you going through that shift?Wesley: First of all, it is kind of like advertising, and people may not think of it that way. Just to give an example, movie trailers is advertising. The free samples at the grocery store is advertising. But people love those because it gives an experience that they like in a package that they are accustomed to. And so, it's the same with developer relations; it's finding the thing that makes the experience worthwhile.On the community side, this is not new to me. I've done several different roles, maybe not in this combination. But when I was at MongoDB, I was a technical community manager, which is like a cog in the whole giant machine. But before that, in my other life, I managed social and community interactions for Walmart, and I had, at the slow period, around 65, but during the holidays, it would ramp up to 95 direct reports that I managed.It's almost—if you're a fan of The Princess Bride, it's different than fighting one person. Sometimes it's easier to fight, like, a squad or a gang of people. So, being Head of Community with such a young company is definitely a lot different than. In some ways, harder to deal with this type of community where we're just growing and emerging, rather than something more well-established.Corey: It probably gives you an interesting opportunity. Because back when I was doing engineering work as an SRE or whatever we call them in that era, it was, “Yeah, wow, my boss is terrible and has no idea what the hell they're doing.” So, then I found myself in the role, and it's, “Cool. Now, do all the things that you said you would do. Put up or shut up.”And it turns out that there's a lot you don't see that our strategic considerations. I completely avoided things like managing up or managing laterally or balancing trade-offs in different ways. Yeah, you're right. If you view the role of management as strictly being something that is between you and your direct reports, you can be an amazing manager from their perspective, but completely ineffective organizationally at accomplishing the goals that have been laid out for you.Wesley: Yeah. The good thing about being head of and the first head of is that you help establish those goals. And so, when you take a role with another company saying, “Hey, we have headcount for this,” and it's an established role, then you're kind of like streamlining into a process that's already underway. What's good about this role specifically, a ‘head of,' is that I help with not only designing what are the goals and the OKRs but deciding what the teams and what the team structure should look like. And so, I'm hiring for a specific position based on how it interacts with everything else.So, when I'm coming in, I don't say, “Well, what do you do?” Or, “How do you do it?” I said, “This is what needs to be done.” And that makes it so much easier just to say that if everything is working the way it should and to give marching orders based on the grand vision, instead of hitting the numbers this quarter or next quarter. Because what is core to my belief, and what's core, too, of how I approach things is at the heart of what I'm trying to do, which is really great, in terms of making something that didn't exist before.Corey: The challenge, too, is that everyone loves to say—and I love to see this at different ways—is the evolution and understanding of the DevRel folks who I work with and I have great relationships with realizing that you have to demonstrate business value. Because I struggle with this my entire career where I know intrinsically, that if I get on stage and tell a story about a thing that is germane to what my company does, that good things are going to happen. But it's very hard to do any form of attribution to it. In a different light, this podcast is a great example of this.We have sponsors. And people are listening. Ideally, they aren't fast-forwarding through sponsor messages; I do have interesting thoughts about the sponsors that I put into these ads. And that's great, but I also appreciate that people are driving while they're listening to this, and they are doing the dishes, they are mowing the lawn, and hopefully not turning up the volume too loudly so it damages their hearing. And the idea that they're going to suddenly stop any of those things and go punch in the link that I give is a little out to lunch there.Instead, it's partially brand awareness and it is occasionally the, “Wait. That resonates exactly with the problem that I have.” So, they get to work or they get back in front of a computer and the odds are terrific they're not going to punch in that URL of whatever I wound up giving; they're going to type in whatever phrases they remember and the company name into Google. Now—and doing attribution on something like that is very hard.It gets even more hard when we're talking about something that is higher up the stack that requires a bit more buy-in than individual developers. There's often a meeting or two about it. And then someone finally approaches the company to have a conversation. Now, does it work? Yes. There are companies that are sponsoring this stuff that spend a lot of time, effort, and money on that.I don't know how you do that sort of attribution; I don't pretend to know, but I know that it works. Because these people whose entire job is making sure that it does tell me it does. So, I smile, I nod, and that's great. But it's very hard to wind up building out a direct, “If you spend X dollars sponsoring this, you will see Y dollars in response.” But in the DevOps world, when your internal doing these things, well, okay because to the company, I look an awful lot like an expensive developer except I don't ever write production code.And then—at least in the before times—“So, what does your job do? Because looking at the achievements and accomplishments last quarter, it looks an awful lot like you traveled to exotic places on the company dime, give talks that are of only vague relevance to what we do, and then hang out at parties with your friends? Nice job, how can I get that?” But it's also first on the chopping block when okay, how do we trim expenses go? And I think it's a mistake to do that. I just don't think that story of the value of developer relations is articulated super-well. And I say that, but I don't know how to do a much better job of it myself.Wesley: Well, that's why corporate or executive buy-in is important because if they know from the get-go while you're there, it makes it a little bit easier to sell. But you do have to show that you are executing. So, there are always two parts to presenting a story, and that's one, the actual quantitative, like, I've done this many talks—so that output part—I've written this many blog posts, or I've stood up this many events that people can attend to. And then there's the results saying, people did read this post, people did show up to my event, people did listen to my talk that I gave. But you also need to give the subjective ones where people respond back and say, “I loved your talk,” or, “I heard you on Corey's podcast,” or, “I read your blog posts,” because even though you might not understand that it goes all the way down in a conversion funnel to a purchase, you can least use that stand-in to say there's probably, like, 20, 30 people behind this person to have that same sentiment, so you can see that your impact is reaching people and that it's having some sort of lasting effect.That said, you have to keep it up. You have to try to increase your output and increase your sphere of influence. Because when people go to solve their problem, they're going to look into their history and their own Rolodex of saying what was the last thing that I heard? What was the last thing that's relevant?There is a reason that Pepsi and Coke still do advertising. It's not because people don't know those brands, but being easily recalled, or a center of relevance based on how many touchpoints or how many times that you've seen them, either from being on American Idol and the logo facing the camera, or seeing a whole display when you go into the grocery store. Same with display advertising. All of this stuff works hand in hand so that you can be front-of-mind with the people and the decision-makers who will make that decision. And we went through this through the pandemic where… that same sentiment, it was like, “You just travel and now you can't travel, so we're just going to get rid of the whole department.”And then those same companies are hunting for those people to come back or to rebuild these departments that are now gone because maybe you don't see what we do, but when it's gone, you definitely notice a dip. And that trust is from the top-up. You have to do not just external advocacy, but you have to do internal advocacy about what impacts you're having so that at least the people who are making that decision can hopefully understand that you are working hard and the work is paying off.Corey: Since the last time that we spoke, you've given your first keynote, which—Wesley: Yes.Corey: Is always an interesting experience to go through. It was at a conference called THAT Conference. And I feel the need to specify that because otherwise, we're going to wind up with a ‘who's on first' situation. But THAT Conference is the name.Wesley: Specify THAT. Yes.Corey: Exactly. Better specify THAT. Yes. So, what was your keynote about? And for a bit of a behind-the-scenes look, what was that like for you?Wesley: Let me do the behind-the-scenes because it's going to lead up to actual the execution.Corey: Excellent.Wesley: So, I've been on several different podcasts. And one of the ones that I loved for years is one called This Week in Tech with Leo Laporte. Was a big fan of Leo Laporte back in the Screen Saver days back in TechTV days. Loved his opinion, follow his work. And I went to a South by Southwest… three, four years ago where I actually met him.And then from that conversation, he asked me to be on his show. And I've been on the show a handful of times, just talking about tech because I love tech. Tech is my passion, not just doing it, but just experiencing and just being on either side of creating or consuming. When I moved—I moved recently also since, I think, from the last time I was on your show—when I moved here to Wisconsin, the organizer of THAT Conference said that he's been following me for a while, since my first appearance on This Week in Tech, and loved my outlook and my take on things. And he approached me to do a keynote.Since I am now Wisconsin—THAT Conference is been in Wisconsin since inception and it's been going on for ten years—and he wanted me to just basically share my knowledge. Clean slate, have enough time to just say whatever I wanted. I said, “Yes, I can do that.” So, my experience on my end was like sheer excitement and then quickly sheer terror of not having a framework of what I was going to speak on or how I was going to deliver it. And knowing as a keynote, that it would be setting the tone for the whole conference.So, I decided to talk on the thing that I knew the most about, which was myself. Talked about my journey growing up and learning what my strengths, what my weaknesses are, how to navigate life, as well as the corporate jungle, and deciding where I wanted to go. Do I want to be the person that I feel like I need to be in order to be successful, which when we look at structures and examples and the things that we hold on a pedestal, we feel that we have to be perfect, or we have to be knowledgeable, and we have to do everything, well rounded in order to be accepted. Especially being a minority, there's a lot more caveats in terms of being socially acceptable to other people. And then the other path that I could have taken, that I chose to take, was to accept my things that are seen as false, but my own quirkiness, my own uniqueness and putting that front and center about, this is me, this is my person that over the years has formed into this version of myself.I'm going to make sure that is really transparent and so if I go anywhere, they know what they're getting, and they know what they're signing up for by bringing me on board. I have an opinion, I will share my opinion, I will bring my whole self, I won't just be the person that is technical or whimsical, or whatever you're looking for. You have to take the good with the bad, you have to take the I really understand technology, but I have ADHD and I might miss some deadlines. [laugh].Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on premises, private cloud, and they just announced a fully managed service on AWS and Azure called BigAnimal, all one word.Don't leave managing your database to your cloud vendor because they're too busy launching another half dozen manage databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications, including Oracle, to the cloud.To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: I have a very similar philosophy, and how I approach these things where it's there is no single speaking engagement that I can fathom even being presented to me, let alone me accepting that is going to be worth me losing the reputation I have developed for authenticity. It's you will not get me to turn into a shill for whatever it is that I am speaking in front of this week. Conversely, whether it's a paid speaking engagement or not, I have a standing policy of not using a platform that is being given to me by a company or organization to make them look foolish. In other words, I will not make someone regret inviting me to speak at their events. Full stop.And I have spoken at events for AWS; I have spoken at events for Oracle, et cetera, et cetera, and there's no company out there that I'm not going to be able to get on stage and tell an entertaining and engaging story, but it requires me to dunk on them. And that's fine. Frankly, if there is a company like that where I could not say nice things about them—such as Facebook—I would simply decline to pursue the speaking opportunity. And that is the way that I view it. And very few companies are on that list, to be very honest with you.Now, there are exceptions to this, if you're having a big public keynote, I will do my traditional live-tweet the keynote and make fun of people because that is, A, expected and, B, it's live-streamed anywhere on the planet I want to be sitting at that point in time, and yeah, if you're saying things in public, you can basically expect that to be the way that I approach these things. But it's a nuanced take, and that is something that is not fully understood by an awful lot of folks who run events. I'll be the first to admit that aspects of who and what I am mean that some speaking engagements are not open to me. And I'm okay with that, on some level, I truly am. It's a different philosophy.But I do know that I am done apologizing for who I am and what I'm about. And at some point that required a tremendous amount of privilege and a not insignificant willingness to take a risk that it was going to work out all right. I can't imagine going back anymore. Now, that road is certainly not what I would recommend to everyone, particularly folks earlier in their career, particularly for folks who don't look just like I do and have a failure mode of a board seat and a book deal somewhere, but figuring out where you will and will not compromise is always an important thing to get straight for yourself before you're presented with a situation where you have to make those decisions, but now there's a whole bunch of incentive to decide in one way or another.Wesley: And that's a journey. You can't just skip sections, right? You didn't get to where you are unless you went through the previous experience that you went through. And it's true for everyone. If you see those success books or how-to books written by people who are extremely rich, and, like, how to become successful and, like, okay, well, that journey is your own. It doesn't make it totally, like, inaccessible to everyone else, but you got to realize that not everyone can walk that path. And—Corey: You were in the right place at the right time, an early employee at a company that did phenomenally well and that catapulted you into reach beyond the wildest dreams of avarice territory. Good for you, but fundamentally, when you give talks like that as a result, what it often presents as is, “I won the lottery, and here's how you can too.” It doesn't work that way. The road you walked was unique to you and that opportunity is closed, not open anyone else, so people have to find their own paths.Wesley: Yeah, and lightning doesn't strike in the same place twice. But there are some things where you can understand some fundamentals. And depending on where you go, I think you do need to know yourself, you do need to know—like, be able to access yourself, but being able to share that, of course, you have to be at a point where you feel comfortable. And so, even if you're in a space where you don't feel that you can be your authentic self or be able to share all parts of you, you yourself should at least know yourself and then make that decision. I agree that it's a point of privilege to be able to say, “Take me how I am.”I'm lucky that I've gotten here, not everyone does, and just because you don't doesn't mean that you're a failure. It just means that the world hasn't caught up yet. People who are part of marginalized society, like, if you are, let's say trans, or if you are even gay, you take the same person, the same stance, the same yearning to be accepted, and then transport it to 50 years ago, you're not safe. You will not necessarily be accepted, or you may not even be successful. And if you have a lane where you can do that, all the power to you, but not everyone could be themselves, and you just need to make sure that at least you can know yourself, even if you don't share that with the world.Corey: It takes time to get there, and I think you're right that it's impossible to get there without walking through the various steps. It's one of the reasons I'm somewhat reluctant to talk overly publicly about my side project gig of paid speaking engagements, for instance, is that the way to get those is you start off by building a reputation as a speaker, and that takes an awful lot of time. And speaking at events where there's no budget even to pay you a speaking fee out of anyway. And part of what gets the keynote invitations to, “Hey, we want you to come and give a talk,” is the fact that people have seen you speak elsewhere and know what you're about and what to expect. Here's a keynote presented by someone who's never presented on stage before is a recipe for a terrifying experience, if not for the speaker or the audience, definitely [laugh] for the event organizers because what if they choke.?Easy example of this, even now hundreds of speaking engagements in, the adrenaline hit right before I go on stage means that sometimes my knees shake a bit before I walk out on stage. I make it a point to warn the people who are standing with me backstage, “Oh, this is a normal thing. Don't worry, it is absolutely expected. It happens every time. Don't sweat it.”And, like, “Thank you for letting us know. That is the sort of thing that's useful.” And then they see me shake, and they get a little skeptical. Like, I thought this guy was a professional. What's the story and I walk on stage and do my thing and I come back. Like, “That was incredible. I was worried at the beginning.” “I told you, we all have our rituals before going on stage. Mine is to shake like a leaf.”But the value there is that people know what to generally expect when I get on stage. It's going to have humor, there's going to be a point interwoven throughout what I tend to say, and in the case of paid speaking engagements, I always make sure I know where the boundaries are of things I can make fun of a big company for. Like, I can get on stage and make fun of service naming or I can make fun of their deprecation policy or something like that, but yeah, making fun of the way that they wind up handling worker relations is probably not going to be great and it could get the person who championed me fired or centered internally. So, that is off the table.Like, even on this podcast, for example, I sometimes get feedback from listeners of, “Well, you have someone from company X on and you didn't beat the crap out of them on this particular point.” It's yeah, you do understand that by having people on the show I'm making a tacit agreement not to attack them. I'm not a journalist. I don't pretend to be. But if I beat someone up with questions about their corporate policy, yeah, very rarely do I have someone who is in a position in those companies to change that policy, and they're certainly not authorized to speak on the record about those things.So, I can beat them up on it, they can say, “I can't answer that,” and we're not going to go anywhere. What is the value of that? It looks like it's not just gotcha journalism, but ineffective gotcha journalism. It doesn't work that way. And that's never been what this show is about.But there's that consistent effort behind the scenes of making sure that people will be entertained, will enjoy what they're seeing, but also are not going to deeply regret giving me a microphone, has always been the balancing act, at least for me. And I want to be clear, my style is humor. It is not for everyone. And my style of humor has a failure mode of being a jerk and making people feel bad, so don't think that my path is the only or even a recommended way for folks who want to get more into speaking to proceed.Wesley: You also mention, though, about, like, punching up versus punching down. And if you really tear down a company after you've been invited to speak, what you're doing is you're punching down at the person who booked you. They're not the CEO; they're not the owner of the company; they're the person who's in charge of running an event or booking speakers. And so, putting that person and throwing them under the bus is punching down because now you're threatening their livelihood, and it doesn't make any market difference in terms of changing the corporate's values or how they execute. So yeah, I totally agree with you in that one.And, like you were saying before, if there's a company you really thought was abhorrent, why speak there? Why give them or lend your reputation to this company if you absolutely feel that it's something you don't want to be associated with? You can just choose not to do that. For me, when I look at speaking, it is important for me to really think about why I'm speaking as well. So, not just the company who's hiring me, but the audience that I'll be serving.So, if I'm going to help with inspiring the next generation of developers, or helping along the thought of how to make the world a better place, or how people themselves can be better people so that we can just change the landscape and make it a lot friendlier, that is also its own… form of compensation and not just speaking for a speaker's fee. So, I do agree that you need to not just be super Negative Nancy, and try to fight all fights. You need to embrace some of the good things and try to make more of those experiences good for everyone, not just the people who are inviting you there, but the people who are attending. And when I started speaking, I was not a good speaker as well. I made a lot of mistakes, and still do, but I think speaking is easier than some people think and if someone truly wants to do it, they should go ahead and get started.What is the saying? If there's something is truly important, you'll be bad at it [laugh] and you'll be okay with it. I started speaking because of my role as a developer advocate. And if you just do a Google search for ‘CFPs,' you can start speaking, too. So, those who are not public speakers and want to get into it, just Google ‘CFP' and then start applying.And then you'll get better at your submissions, you'll get better at your slides, and then once you get accepted, then you'll get better at preparing, then you'll get better at actually speaking. There's a lot of steps between starting and stopping and it's okay to get started doing that route. The other thing I wanted to point out is I feel public speaking is the equivalent of lifting your own bodyweight. If you can do it, you're one of the small few of the population that is willing to do so or that can do it. If you start public speaking, that in itself is an accomplishment and an experience that is something that is somewhat enriching. And being bad at it doesn't take the passion away from you. If you just really want to do it, just keep doing it, even if you're a bad speaker.Corey: Yeah. The way to give a great talk because you have a bunch of terrible talks first.Wesley: Yeah. And it's okay to do that.Corey: And it's not the in entirety of community. It's not even a requirement to be involved with the community. If you're one of those people that absolutely dreads the prospect of speaking publicly, fine. I'm not suggesting that, oh, you need to get over that and get on stage. That doesn't help anyone. Don't do the things you dread doing because you know that it's not going to go well for you.That's the reason I don't touch actual databases. I mean, come on, let's be realistic. I will accidentally the data, and then we won't have a company anymore. So, I know what things I'm good at and things I'm not. I also don't do hostage negotiations, for obvious reasons.Wesley: And also, here's a little, like, secret tip. If you really want to do public speaking and you start doing public speaking and you're not so good at it from other peoples' perspective, but you still love doing it and you think you're getting better, doing public speaking is one of those things where you can say that you do it and no one will really question how good you are at it. [laugh]. If you're just in casual conversation, it's like, “Hey, I wrote a book.” People like, “Oh, wow. This person wrote the book on blah, blah, blah.”Corey: It's a self-published book that says the best way to run Kubernetes. It's a single page; it says, “Don't.” In 150-point type. “The end.” But I wrote a book.Wesley: Yeah.Corey: Yeah.Wesley: People won't probe too much and it'll help you with your development. So, go ahead and get started. Don't worry about doing that thing where, like, I have to be the best before I can present it. Call yourself a public speaker. Check, done.Corey: Always. We are the stories we tell, and nowhere is it more true than in the world of public speaking. I really want to thank you for taking the time out of your day to speak with me about this for a second time in a single year. Oh, my goodness. If people want to learn more about what you're up to, where can they find you?Wesley: I'm on Twitter, @wesley83 on Twitter. And you can find me also on PolyWork. So, polywork.com/wesley83. Or just go to wesleyfaulkner.com which redirects you there. I list pretty much everything that I am working on and any upcoming speaking opportunities, hopefully when they release that feature, will also be on that Polywork page.Corey: Excellent. And of course, I started Polywork recently, and I'm at thoughtleader.cloud because of course I am, which is neither here nor there. Thank you so much for taking the time to speak about this side of the industry that we never really get to talk about much, at least not publicly and not very often.Wesley: Well, thank you for having me on the show. And I wanted to take some time to say thank you for the work that you're doing. Not just elevating voices like myself, but talking truth to power, like we mentioned before, but being yourself and being a great representation of how people should be treating others: being honest without being mean, being snarky without being rude. And other companies and other people who've given me a chance, and given me a platform, I wanted to say thank you to you too, and I wouldn't be here unless it was people like you acknowledging the work that I've been doing.Corey: All it takes is just recognizing what you're doing and acknowledging it. People often want to thank me for this stuff, but it's just, what, for keeping my eyes open? I don't know, I feel like it's just the job; it's not something that is above and beyond any expected normal behavior. The only challenge is I look around the industry and I realize just how wrong that impression is, apparently. But here we are. It's about finding people doing interesting work and letting them tell their story. That's all this podcast has ever tried to be.Wesley: Yeah. And you do it. And doing the work is part of the reward, and I really appreciate you just going through the effort. Even having your ears open is something that I'm glad that you're able to at least know who the people are and who are making noises—or making noise to raise their profile up and then in turn, sharing that with the world. And so, that's a great service that you're providing, not just for me, but for everyone.Corey: Well, thank you. And as always, thank you for your time. Wesley Faulkner, Head of Community at SingleStore. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a rambling comment telling me exactly why DevRel does not need success metrics of any kind.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
You've got a friend in us at Flicking and Screaming. Our challenge this week was drafting a roster of 8 Pixar films each. The only problem is . . . Pixar has released 25 films. What will get left off? Which classic will go first overall? Plus: what happened to Pixar after 2010?
In our final Panic Fest 2022 episode, Chris and Neil link up with director/writer Conor Boru and actor/writer Ed Hartland of the hilarious horror comedy When The Screaming Starts which was available virtually! They have a fun as hell conversation and afterwards, Chris and Neil tack on their review. Panic Fest 2022 was a blast and we already can't wait for next year. #PanicFest2022 #WhenTheScreaningStarts #HorrorComedy #Mockumentary
Bienvenido Bastarnauta a un episodio más de tus Bastardos Favoritos con música para andar un mes con el mismo pantalón, andando en aventón y disfrutando un… chingo. Disfruta rolas como Yo quiero ser un beach boy; Supernature un rolon electrodisco; Un blues rockeron sabroso de kitty, daisy and lewis; Un funky rap de Galiano; Screaming jay en su mero mole, entre otros. Así que chúpale pichón, a huevo chicharrón con pelos y lléguele lléguele al más acá. --- Send in a voice message: https://anchor.fm/los-bastardos-con-suerte/message
A Co Antrim couple who welcomed beautiful conjoined twins into their lives in March have said they are a “miracle”. Hannah and Dan Bateson, from Toomebridge welcomed the twin girls at London's University College Hospital. Annabelle and Isabelle are joined from the chest to the pelvis and share a liver, bladder and bowel, one shared fused leg and one leg each as well as separate hearts. Hannah joined Sean to share her experience and hopes for her twin girls future. Listen and subscribe to Moncrieff on Apple Podcasts, Google Podcasts or Spotify. Download, listen and subscribe on the Newstalk App. You can also listen to Newstalk live on newstalk.com or on Alexa, by adding the Newstalk skill and asking: 'Alexa, play Newstalk'.
Live from the No Panic Zone—I'm Steve Gruber—I am America's Voice—God Bless America—God Bless You and let's do this! This is the Steve Gruber show— the Ultra MAGA express train to truth! Get on lets roll! Here are three big things you need to know right now— ONE— Russia is losing in Ukraine—and they are also running out of soldiers and missiles—depending on who you trust on the matter— TWO— News today—of yet another Democrat Senator suffering a stroke—that will leave him out of action for at least a week or two—what is going on with so many Dems stroking out? THREE— The babies of the nation are screaming again today—because the lack of baby formula nationwide is hitting home OR maybe it's because Joe Biden is President which would make most grown men cry—Just not me—it makes me angry— and it makes me steadfast and I want you to embrace the same resolve I have today —because my friends we are not going to take back this country sitting on the damn couch! We must stand up—we must be proud and we must be heard! Are you with me? It is time to stand up and be heard my friends—it is time to tell Joe Biden—Bernie Sanders—Nancy Pelosi—Barack Obama—AOC—The squad— and the rest of the Socialists clamoring to put the globalists in charge of the United States—we are not buying it— not one bit— The laundry list of failures grows by the day—the failing economy—and the soaring inflation—driving average gas prices above $6 a gallon in California—and above $5 dollars a gallon in several more states— We are truly at a watershed moment my friends and the question you have to ask—is are you going to sit there and run your mouth—and tap out mean messages on social media—or are you going to get into the arena—just like the 26th President of the United States—Teddy Roosevelt talked about— Are you willing to roll up your sleeves—get dirty—get knocked down and get back up? Because I can guarantee you this—I have been watching the left my entire life—and they never quit—no matter how badly they get crushed in elections like 1994 or 2010—they re-group and they start leaching back into the national conversation again— Like your favorite leftist whack job and mine—Neil Young told us—Rust Never Sleeps—and neither do socialists—they will be putting together new rent-a-riots for this coming weekend—to declare America a horrible—racist and unfair place—while at the same time rationalizing the dismemberment of full term babies as somehow protected under the Constitution— they will do that for political power—because they have no shame— It is time my friends to reject the failure of this President and his administration to make sure there is enough formula to feed our nation's children— and enough strength to take this nation back— It's time— We must reject gas prices at all time record highs while the socialists are crushing the American energy sector –that should be setting the standard for the rest of the world— Reject censorship—human trafficking—a dishonest national media—an unchecked invasion at the southern border— Its time to say we will not allow you to teach our kids to hate each other amy more—we will say no to defunding the police and weak willed George Soros prosecutors—who release violent criminals and step aside as violent crime explodes and blood runs in our streets like never before— We will not blindly follow you to war in Ukraine, Somalia or anywhere else—we reject the Ministry of Truth and the idea that the government can do anything better than the rest of us—except to build machines to kill—on a mass scale— We flatly reject government surveillance of our phones, our computers, our cars, our bank accounts and our lives—back off—because we have had enough— We will not sit here quietly while you destroy the American farm and the American family— we will not be intimidated by your thug tactics or your bogus slogans—or your big lies— We demand voter ID—poll watchers—and accountability at every level—we will fight back against your insanity—and we will have election integrity— And we will defend The United States of America—at every turn—no matter what it takes—we will defend the right to free speech—the right to bear arms—the right to be free of government intervention in our lives—and we will treat the Constitution of this nation as it should be treated —as the greatest document for human freedom and liberty in the history of the world—because it is—and we all know that! We will not stand here and let you divide us anymore—and we will reject those that try— We are many things—we are black and we are white—we are men and women—short and tall—some are thin and some are not—we come from backroads and back alleys—and we are in this together— Because my friends—we are Americans—and that means something to me—in fact beyond God and my amazing family—it means everything— So, I say to you again—are you with me? Do not allow yourself to get down—I need you now more than ever—I need you to save this nation with me—together we can do this—we can rise above and we can reject the socialists in favor of that shining city on a hill—because that is today—more than ever what America still is—and always will be—BUT we must defend her from those that would tear her down— Are you with me? Ask yourself— are you with me?
About JamesJames has been part of AWS for over 15 years. During that time he's led software engineering for Amazon EC2 and more recently leads the AWS Commerce Platform group that runs some of the largest systems in the world, handling volumes of data and request rates that would make your eyes water. And AWS customers trust us to be right all the time so there's no room for error.Links Referenced:Email: firstname.lastname@example.orgTranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Optimized cloud compute plans have landed at Vultr to deliver lightning-fast processing power, courtesy of third-gen AMD EPYC processors without the IO or hardware limitations of a traditional multi-tenant cloud server. Starting at just 28 bucks a month, users can deploy general-purpose, CPU, memory, or storage optimized cloud instances in more than 20 locations across five continents. Without looking, I know that once again, Antarctica has gotten the short end of the stick. Launch your Vultr optimized compute instance in 60 seconds or less on your choice of included operating systems, or bring your own. It's time to ditch convoluted and unpredictable giant tech company billing practices and say goodbye to noisy neighbors and egregious egress forever. Vultr delivers the power of the cloud with none of the bloat. “Screaming in the Cloud” listeners can try Vultr for free today with a $150 in credit when they visit getvultr.com/screaming. That's G-E-T-V-U-L-T-R dot com slash screaming. My thanks to them for sponsoring this ridiculous podcast.Corey: Finding skilled DevOps engineers is a pain in the neck! And if you need to deploy a secure and compliant application to AWS, forgettaboutit! But that's where DuploCloud can help. Their comprehensive no-code/low-code software platform guarantees a secure and compliant infrastructure in as little as two weeks, while automating the full DevSecOps lifestyle. Get started with DevOps-as-a-Service from DuploCloud so that your cloud configurations are done right the first time. Tell them I sent you and your first two months are free. To learn more visit: snark.cloud/duplo. Thats's snark.cloud/D-U-P-L-O-C-L-O-U-D. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. And I've been angling to get someone from a particular department at AWS on this show for nearly its entire run. If you were to find yourself in an Amazon building and wander through the various dungeons and boiler rooms and subterranean basements—I presume; I haven't seen nearly as many of you inside of those buildings as people might think—you pass interesting departments labeled things like ‘Spline Reticulation,' or whatnot. And then you come to a very particular group called Commerce Platform.Now, I'm not generally one to tell other people's stories for them. My guest today is James Greenfield, the VP of Commerce Platform at AWS. James, thank you for joining me and suffering the slings and arrows I will no doubt be hurling at you.James: Thanks for having me. I'm looking forward to it.Corey: So, let's start at the very beginning—because I guarantee you, you're going to do a better job of giving the chapter and verse answer than I would from a background mired deeply in snark—what is Commerce Platform? It sounds almost like it's the retail website that sells socks, books, and underpants.James: So, Commerce Platform actually spans a bunch of different things. And so, I'm going to try not to bore you with a laundry list of all of the things that we do—it's a much longer list than most people assume even internal to AWS—at its core, Commerce Platform owns all of the infrastructure and processes and software that takes the fact that you've been running an EC2 instance, or you're storing an object in S3 for some period of time, and turns it into a number at the end of the month. That is what you asked for that service and then proceeds to try to give you as many ways to pay us as easily as possible. There are a few other bits in there that are maybe less obvious. One is we're also responsible for protecting the platform and our customers from fraudulent activity. And then we're also responsible for helping collect all of the data that we need for internal reporting to support some of the back-ends services that a business needs to do things like revenue recognition and general financial reporting.Corey: One of the interesting aspects about the billing system is just how deeply it permeates everything that happens within AWS. I frequently say that when it comes to cloud, cost and architecture are foundationally and fundamentally the same exact thing. If your entire service goes down, a few interesting things happen. One, I don't believe a single customer is going to complain other than maybe a few accountants here and there because the books aren't reconciling, but also you've removed a whole bunch of constraints around why things are the way that they are. Like, what is the most efficient way to run this workload?Well, if all the computers suddenly become free, I don't really care about efficiency, so much is, “Oh, hey. There's a fly, what do I have as a flyswatter? That's right, I'm going to drop a building on it.” And those constraints breed almost everything. I've said, for example, that S3 has infinite storage because it does.They can add drives faster than we're able to fill them—at least historically; they added some more replication services—but they're going to be able to buy hard drives faster than the rest of us are going to be able to stretch our budgets. If that constraint of the budget falls away, all bets are really off, and more or less, we're talking about the destruction of the cloud as a viable business entity. No pressure or anything.James: [laugh].Corey: You're also a recent transplant into AWS billing as a whole, Commerce Platform in general. You spent 15 years at the company, the vast majority of that over an EC2. So, either it was you've been exiled to a basically digital Siberia or it was one of those, “Okay, keeping all the EC2 servers up, this is easy. I don't see what people stress about.” And they say, “Oh, ho ho, try this instead.” How did you find yourself migrating over to the Commerce Platform?James: That's actually one I've had a lot from folks that I've worked with. You're right, I spent the first 15 or so years of my career at AWS in EC2, responsible for various things over there. And when the leadership role in Commerce Platform opened up, the timing was fortuitous, and part of it, I was in the process of relocating my family. We moved to Vancouver in the middle of last year. And we had an opening in the role and started talking about, potentially, me stepping into that role.The reason that I took it—there's a few reasons, but the primary reason is that if I look back over my career, I've kind of naturally gravitated towards owning things where people only really remember that they exist when they're not working. And for some reason, you know, I enjoy the opportunity to try to keep those kinds of services ticking over to the point where people don't notice them. And so, Commerce Platform lands squarely in that space. I've always been attracted to opportunities to have an impact, and it's hard to imagine having much more of an impact than in the Commerce Platform space. It underpins everything, as you said earlier.Every single one of our customers depends on the service, whether they think about it or realize it. Every single service that we offer to customers depends on us. And so, that really is the sort of nexus within AWS. And I'm a platform guy, I've always been a platform guy. I like the force multiplier nature of platforms, and so Commerce Platform, you know, as I kind of thought through all of those elements, really was a great opportunity to step in.And I think there's something to be said for, I've been a customer of Commerce Platform internally for a long time. And so, a chance to cross over and be on the other side of that was something that I didn't want to pass up. And so, you know, I'm digging in, and learning quickly, ramping up. By no means an expert, very dependent on a very smart, talented, committed group of people within the team. That's kind of the long and short of how and why.Corey: Let's say that I am taking on the role of an AWS product team, for the sake of argument. I know, keep the cringe down for a second, as far as oh, God, the wince is just inevitable when the idea of me working there ever comes up to anyone. But I have an idea for a service—obviously, it runs containers, and maybe it does some other things as well—going from idea to six-pager to MVP to barely better than MVP day-one launch, and at some point, various things happen to that service. It gets staff with a team, objectives and a roadmap get built, a P&L and budget, and a pricing model and the rest. One the last thing that happens, apparently, is someone picks the worst name off of a list of candidates, slaps it on the product, and ships it off there.At what point does the billing system and figuring out the pricing dimensions for a given service tend to factor in? Is that a last-minute story? Is that almost from the beginning? Where along that journey does, “Oh, by the way, we're building this thing. Maybe we should figure out, I don't know, how to make money from it.” Factor into the conversation?James: There are two parts to that answer. Pretty early on as we're trying to define what that service is going to look like, we're already typically thinking about what are the dimensions that we might charge along. The actual pricing discussions typically happen fairly late, but identifying those dimensions and, sort of, the right way to present it to customers happens pretty early on. The thing that doesn't happen early enough is actually pulling the Commerce Platform team in. but it is something that we're going to work this year to try to get a little bit more in front of.Corey: Have you found historically that you have a pretty good idea of how a service is going to be priced, everything is mostly thought through, a service goes to either private preview or you're discussing about a launch, and then more or less, I don't know, someone like me crops up with a, “Hey, yeah, let's disregard 90% of what the service does because I see a way to misuse the remaining 10% of it as a database.” And you run some mental math and realize, “Huh. We're suddenly giving, like, eight petabytes of storage per customer away for free. Maybe we should guard against that because otherwise, it's rife with misuse.” It used to be that I could find interesting ways to sneak through the cracks of various services—usually in pursuit of a laugh—those are getting relatively hard to come by and invariably a lot more trouble than they're worth. Is that just better comprehensive diligence internally, is that learning from customers, or am I just bad at this?James: No, I mean, what you're describing is almost a variant of the Defender's Dilemma. They are way more ways to abuse something than you can imagine, and so defending against that is pretty challenging. And it's important because, you know, if you turn the economics of something upside down, then it just becomes harder for us to offer it to customers who want to use it legitimately. I would say 90% of that improvement is us learning. We make plenty of mistakes, but I think, you know, one of the things that I've always been impressed by over my time here is how intentional we are trying to learn from those mistakes.And so, I think that's what you're seeing there. And then we try very hard to listen to customers, talk to folks like you, because one of the best ways to tackle anything it smells of the Defender's Dilemma is to harness that collective creativity of a large number of smart people because you really are trying to cover as much ground as possible.Corey: There was a fun joke going around a while back of what is the most expensive environment you can get running on a free tier account before someone from AWS steps in, and I think I got it to something like half a billion dollars in the first month. Now, I haven't actually tested this for reasons that mostly have to do with being relatively poor compared to, you know, being able to buy Guam. And understanding as well the fraud protections built into something like AWS are largely built around defending against getting service usage for free that in some way, shape or form, benefits the attacker. The easy example of that would be mining cryptocurrency, which is just super-economic as long as you use someone else's AWS account to do it. Whereas a lot of my vectors are, “Yeah, ignore all of that. How do I just make the bill artificially high? What can I do to misuse data transfer? And passing a single gigabyte through, how much can I make that per gigabyte cost be?” And, “Oh, circular replication and the Lambda invokes itself pattern,” and basically every bad architectural decision you can possibly make only this time, it's intentional.And that shines some really interesting light on it. And I have to give credit where due, a lot of that didn't come from just me sitting here being sick and twisted nearly so much as it did having seen examples of that type of misconfiguration—by mistake—in a variety of customer accounts, most confidently my own because it turns out that the way I learn things is by screwing them up first.James: Yeah, you've touched on a couple of different things in there. So, you know, maybe the first one is, I typically try to draw a line between fraud and abuse. And fraud is essentially trying to spend somebody else's money to get something for free. And we spent a lot of time trying to shut that down, and we're getting really good at catching it. And then abuse is either intentional or unintentional. There's intentional abuse: You find a chink in our armor and you try to take advantage of it.But much more commonly is unintentional abuse. It's not really abuse, you know. Abuse has very negative connotations, but it's unintentionally setting something up so that you run up a much larger bill than you intended. And we have a number of different internal efforts, and we're working on a bunch more this year, to try to catch those early on because one of my personal goals is to minimize the frequency with which we surprise customers. And the least favorite kind of surprise for customers is a [laugh] large bill. And so, what you're talking about there is, in a sufficiently complex system, there's always going to be weaknesses and ways to get yourself tied up in knots.We're trying both at the service team level, but also within my teams to try to find ways to make it as hard as possible to accidentally do that to yourself and then catch when you do so that we can stop it. And even more on the intentional abuse side of things, if somebody's found a way to do something that's problematic for our services, then you know, that's pretty much on us. But we will often reach out and engage with whoever's doing and try to understand what they're trying to do and why. Because often, somebody's trying to do something legitimate, they've got a problem to solve, they found a creative way to solve it, and it may put strain on the service because it's just not something we designed for, and so we'll try to work with them to use that to feed into either new services, or find a better place for that workload, or just bolster what they're using. And maybe that's something that eventually becomes a fully-fledged feature that we offer the customers. We're always open to learning from our customers. They have found far more creative ways to get really cool things done with our services than we've ever imagined. And that's true today.Corey: I mean, most of my service criticisms come down to the fact that you have more-or-less built a very late model, high performing iPad, and I'm out there complaining about, “What a shitty hammer this thing is, it barely works at all, and then it breaks in my hand. What gives?” I would also challenge something you said a minute ago that the worst day for some customers is to get a giant surprise bill, but [unintelligible 00:13:53] to that is, yeah, but, on some level, that kind of only money; you do have levers on your side to fix those issues. A worse scenario is you have a customer that exhibits fraud-like behavior, they're suddenly using far more resources than they ever did before, so let's go ahead and turn them off or throttle them significantly, and you call them up to tell them you saved them some money, and, “Our Superbowl ad ran. What exactly do you think you're doing?” Because they don't get a second bite at that kind of Apple.So, there's a parallel on both sides of this. And those are just two examples. The world is full of nuances, and at the scale that you folks operate at. The one-in-a-million events happen multiple times a second, the corner cases become common cases, and I'm surprised—to be direct—how little I see you folks dropping the ball.James: Credit to all of the teams. I think our secret sauce, if anything, really does come down to our people. Like, a huge amount of what you see as hopefully relatively consistent, good execution comes down to people behind the scenes making sure. You know, like, some of it is software that we built and made sure it's robust and tested to scale, but there's always an element of people behind the scenes, when you hit those edge cases or something doesn't quite go the way that you planned, making sure that things run smoothly. And that, if anything, is something that I'm immensely proud of and is kind of amazing to watch from the inside.Corey: And, on some level, it's the small errors that are the bigger concern than the big ones. Back a couple years ago, when they announced GP3 volumes at re:Invent, well, great, well spin up a test volume and kick the tires on it for an hour. And I think it was 80 or 100 gigs or whatnot, and the next day in the bill, it showed up as about $5,000. And it was, “Okay, that's not great. Not great at all.” And it turned out that it was a mispricing error by I think a factor of a million.And okay, at least it stood out. But there are scenarios where we were prepared to pay it because, oops, you got one over on us. Good job. That's never been the mindset I've gotten about AWS's philosophy for pricing. The better example that I love because no one took it seriously, was a few years before that when there was a LightSail bug in the billing system, and it made the papers because people suddenly found that for their LightSail instance, they were getting predicted bills of $4 billion.And the way I see it, you really only had to make that work once and then you've made your numbers for the year, so why not? Someone's going to pay for it, probably. But that was such out-of-the-world numbers that no one saw that and ever thought it was anything other than a bug. It's the small pernicious things that creep in. Because the billing system is vast; I had no idea when I started working with AWS bills just how complicated it really was.James: Yeah, I remember both of those, and there's something in there that you touched on that I think is really important. That's something that I realized pretty early on at Amazon, and it's why customer obsession is our flagship leadership principle. It's not because it's love and butterflies and unicorns; customer obsession is key to us because that's how you build a long-term sustainable business is your customers depend on you. And it drives how we think about everything that we do. And in the billing space, small errors, even if there are small errors in the customer's favor, slowly erode that trust.So, we take any kind of error really seriously and we try to figure out how we can make sure that it doesn't happen again. We don't always get that right. As you said, we've built an enormous, super-complex business to growing really quickly, and really quick growth like that always acts as kind of a multiplier on top of complexity. And on the pricing points, we're managing millions of pricing points at the moment.And our tools that we use internally, there's always room for improvement. It's a huge area of focus for us. We're in the beginning of looking at applying things like formal methods to make sure that we can make very hard guarantees about the correctness of some of those. But at the end of the day, people are plugging numbers in and you need as many belts and braces as possible to make sure that you don't make mistakes there.Corey: One of the things that struck me by surprise when I first started getting deep into this space was the fact that the finalized bill was—what does it mean to have this be ‘finalized?' It can hit the Cost and Usage Report in an S3 bucket and it can change retroactively after the month closed periodically. And that's when I started to have an inkling of a few things: Not just the sheer scale and complexity inherent to something like the billing system that touches everything, but the sheer data retention stories where you clearly have to be able to go back and reconstruct a bill from the raw data years ago. And I know what the output of all of those things are in the form of Cost and Usage Reports and the billing data from our client accounts—which is the single largest expense in all of our AWS accounts; we spent thousands and thousands and thousands of dollars a year just on storing all of that data, let alone the processing piece of it—the sheer scale is staggering. I used to wonder why does it take you a day to record me using something to it's showing up in the bill? And the more I learned the more it became a how can you do that in only a day?James: Yes, the scale is actually mind-boggling. I'm pretty sure that the core of our billing system is—I'm reasonably confident it's the largest or one of the largest data processing systems on the planet. I remember pretty early on when I joined Commerce Platform and was still starting to wrap my head around some of these things, Googling the definition of quadrillion because we measured the number of metering events, which is how we record usage in services, on a daily basis in the quadrillions, which is a billion billions. So, it's just an absolutely staggering number. And so, the scale here is just out of this world.That's saying something because it's not like other services across AWS are small in their own right. But I'm still reasonably sure that being one of a handful of services that is kind of at the nexus of AWS and kind of deals with the aggregate of AWS's scale, this is probably one of the biggest systems on the planet. And that shows up in all sorts of places. You start with that input, just the sheer volume of metering events, but that has to produce as an output pretty fine-grained line item detailed information, which ultimately rolls up into the total that a customer will see in their bill. But we have a number of different systems further down the pipeline that try to do things like analyze your usage, make sensible recommendations, look for opportunities to improve your efficiency, give you the ability to slice and dice your data and allocate it out to different parts of your business in whatever way it makes sense for your business. And so, those systems have to deal with anywhere from millions to billions to recently, we were talking about trillions of data points themselves. And so, I was tangentially aware of some of the scale of this, but being in the thick of it having joined the team really just does underscore just how vast the systems are.Corey: I think it's, on some level, more than a little unfortunate that that story isn't being more widely told, more frequently. Because when Commerce Platform has job postings that are available on the website, you read it and it's very vague. It doesn't tend to give hard numbers about a lot of these things, and people who don't play in these waters can easily be forgiven for thinking the way that you folks do your job is you fire up one of those 24 terabyte of RAM instances that—you know, those monstrous things that you folks offer—and what do you do next? Well, Microsoft Excel. We have a special high memory version that we've done some horse-trading with our friends over at Microsoft for.It's, yeah, you're several steps beyond that, at this point. It's a challenging problem that every one of your customers has to deal with, on some level, as well. But we're only dealing with the output of a lot of the processing that you folks are doing first.James: You're exactly right. And a big focus for some of my teams is figuring out how to help customers deal with that output. Because even if you're talking about couple of orders of magnitude reduction, you're still talking about very large numbers there. So, to help customers make sense of that, we have a range of tools that exist, we're investing in.There's another dimension of complexity in the space that I think is one that's also very easy to miss. And I think of it as arbitrary complexity. And it's arbitrary because some of the rules that we have to box within here are driven by legislative changes. As you operate more and more countries around the world, you want to make sure that we're tax compliant, that we help our customers be tax compliant. Those rules evolve pretty rapidly, and Country A may sit next to Country B, but that doesn't mean that they're talking to one another. They've all got their own ideas. They're trying to accomplish r—00:22:47Corey: A company is picking up and relocating from India to Germany. How do we—James: Exactly.Corey: —change that on the AWS side and the rest? And it's, “Hoo boy, have you considered burning it all down and filing an insurance claim to start over?” And, like, there's a lot of complexity buried underneath that that just doesn't rise to the notice of 99% of your customers.James: And the fact that it doesn't rise to the notice is something that we strive for. Like, these shouldn't be things that customers have to worry about. Because it really is about clearing away the things that, as far as possible, you don't want to have to spend time thinking about so that you can focus on the thing that your business does that differentiates you. It's getting rid of that undifferentiated heavy lifting. And there's a ton of that in this space, and if you're blissfully unaware of it, then hopefully that means that we're doing our job.Corey: What I'm, I think, the most surprised about, and I have been for a long time. And please don't take this as an insult to various other folks—engineers, the rest, not just in other parts of AWS but throughout the other industry—but talking to the people who work within Commerce Platform has always been just a fantastic experience. The caliber of people that you have managed to attract and largely retain—we don't own people, they do matriculate out eventually—but the caliber of people that you've retained on your teams has just been out of this world. And at first, I wondered, why are these awesome people working on something as boring and prosaic as billing? And then I started learning a little bit more as I went, and, “Oh, wow. How did they learn all the stuff that they have to hold in their head in tension at once to be able to build things like this?” It's incredibly inspiring just watching the caliber of the people that you've been able to bring in.James: I've been really, really excited joining this team, as I've gotten other folks on the team because there's some super-smart people here. But what's really jumped out to me is how committed the team is. This is, for the most part, a team that has been in the space for many years. Many of them have—we talk about boomerangs, folks who live AWS, go spend some time somewhere else and come back and there's a surprisingly high proportion of folks in Commerce Platform who have spent time somewhere else and then come back because they enjoy the space, they find that challenging, folks are attracted to the ability to have an impact because it is so foundational. But yeah, there's a super-committed core to this team. And I really enjoy working with teams where you've got that because then you really can take the long view and build something great. And I think we have tons of opportunities to do that here.Corey: It sounds ridiculous, but I've reached out to team members before to explain two-cent variances in my bill, and never once have I been confronted with a, “It's two cents. What do you care?” They understand the requirement that these things be accurate, not just, “Eh, take our word for it.” And also, frankly, they understand that two cents on a $20 bill looks a little different on a $20 million bill. So yeah, let us figure out if this is systemic or something I have managed to break.It turns out the Cost and Usage Report processing systems don't love it when there's a cost allocation tag whose name contains an emoji. Who knew? It's the little things in life that just have this fun way of breaking when you least expect it.James: They're also a surprisingly interesting problem. So like, it turns out something as simple as rounding numbers consistently across a distributed system at this scale, is a non-trivial problem. And if you don't, then you do get small seventh or eighth decimal place differences that add up to something that then shows up as a two-cent difference somewhere. And so, there's some really, really interesting problems in the space. And I think the team often takes these kinds of things as a personal challenge. It should be correct, and it's not, so we should go make sure it is correct. The interesting problems abound here, but at the end of the day, it's the kind of thing that any engineering team wants to go and make sure it's correct because they know that it can be.Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on premises, private cloud, and they just announced a fully managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half dozen manage databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications, including Oracle, to the cloud.To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: On the one hand, I love people who just round and estimate—we all do that, let's be clear; I sit there and I back-of-the-envelope everything first. But then I look at some of your pricing pages and I count the digits after the zeros. Like, you're talking about trillionths of a dollar on some of your pricing points. And you add it up in the course of a given hour and it's like, oh, it's $250 a month, most months. And it's you work backwards to way more decimal places of precision than is required, sometimes.I'm also a personal fan of the bill that counts, for example, number of Route 53 zones. Great. And it counts them to four decimal places of precision. Like, I don't even know what half of it Route 53 zone is at this point, let alone something to, like, ah the 1,000th of the zone is going to cause this. It's all an artifact of what the underlying systems are.Can you by any chance shed a little light on what the evolution of those systems has been over a period of time? I have to imagine that anything you built in the early days, 16 years ago or so from the time of this recording when S3 launched to general availability, you probably didn't have to worry about this scope and scale of what you do, now. In fact, I suspect if you tried to funnel this volume through S3 back then, the whole thing would have collapsed under its own weight. What's evolved over the time that you had the billing system there? Because changes come slowly to your environment. And frankly, I appreciate that as a customer. I don't like surprising people in finance.James: Yeah, you're totally right. So, I joined the EC2 team as an engineer myself, some 16 years ago, and the very first thing that I did was our billing integration. And so, my relationship with the Commerce Platform organization—what was the billing team way back when—it goes back over my entire career at AWS. And at the time, the billing team was similar, you know, [unintelligible 00:28:34] eight people. And that was everything. There was none of the scale and complexity; it was all one system.And much like many of our biggest, oldest services—EC2 is very similar, S3 is as well—there's been significant growth over the last decade-and-a-half. A lot of that growth has been rapid, and rapid growth presents its own challenges. And you live with decisions that you make early on that you didn't realize were significant decisions that have pretty deep implications 15 years later. We're still working through some of those; they present their own challenges. Evolving an existing system to keep up with the growth of business and a customer base that's as varied and complex as ours is always challenging.And also harder but I also think more fun than a clean sheet redo at this point. Like, that's a great thought exercise for, well, if we got to do this again today, what would we do now that we've learned so much over the last 15 years? But there's this—I find it personally fascinating challenge with evolving a live system where it's like, “No, no, like, things exist, so how do we go from there to where we want to be next?”Corey: Turn the billing system off for 18 months, rebuild—James: Yeah. [laugh].Corey: The whole thing from first principles. Light it up. I'm sure you'd have a much better billing system, and also not a company left anymore.James: [laugh]. Exactly, exactly. I've always enjoyed that challenge. You know, even prior to AWS, my previous careers have involved similar kinds of constraints where you've got a live system, or you've got an existing—in the one case, it was an existing SDK that was deployed to tens of thousands of customers around the world, and so backwards compatibility was something that I spent the first five years of my career thinking about it way more detail than I think most people do. And it's a very similar mindset. And I enjoy that challenge. I enjoy that: How do I evolve from here to there without breaking customers along the way?And that's something that we take pretty seriously across AWS. I think SimpleDB is the poster child for we never turn things off. But that applies equally to the services that are maybe less visible to customers, and billing is definitely one of them. Like, we don't get to switch stuff off. We don't get to throw things away and start again. It's this constant state of evolution.Corey: So, let's say that I were to find a way to route data through a series of two Managed NAT Gateways and then egress to internet, and the sheer density of the expense of that traffic tears a hole in the fabric of space-time, it goes back 15 years ago, and you can make a single change to how the billing system was built. What would it be? What pisses you off the most about the current constraints that you have to work within or around?James: I think one of the biggest challenges we've got, actually, is the concept of an account. Because an account means half-a-dozen different things. And way back, when it seemed like a great idea, you just needed an account; an account was your customer, and it was the same thing as the boundary that you put all your resources inside. And of course, it's the same thing that you're going to roll all of your usage up and issue a bill against. And that has been one of the areas that's seen the most evolution and probably still has a pretty long way to go.And what's interesting about that is, that's probably something we could have seen coming because we watched the retail business go through, kind of, the same evolution because they started with, well, a customer is a customer is a customer and had to evolve to support the concept of sellers and partners. And then users are different than customers, and you want to log in and that's a different thing. So, we saw that kind of bifurcation of a single entity into a wide range of different related but separate entities, and I think if we'd looked at that, you know, thought out 15 years, then yeah, we could probably have learned something from that. But at the same time, when AWS first kicked off, we had wild ambitions for it, but there was no guarantee that it was going to be the monster that it is today. So, I'm always a little bit reluctant to—like, it's a great thought exercise, but it's easy to end up second-guessing a pretty successful 15 years, so I'm always a little bit careful to walk that line. But I think account is one of the things that we would probably go back and think about a little bit more.Corey: I want to be very clear with this next question that it is intentionally setting up a question I suspect you get a lot. It does not mirror my own thinking on the matter even slightly, but I get a version of it myself all the time. “AWS bills, that sounds boring as hell. Why would you choose to work on such a thing?” Now, I have a laundry list of answers to that aren't nearly as interesting as I suspect yours are going to be. What makes working on this problem space interesting to you?James: There's a bunch of different things. So, first and foremost, the scale that we're talking about here is absolutely mind-blowing. And for any engineer who wants to get stuck into problems that deal with mind-blowingly large volumes of data, incredibly rich dimensions, problems where, honestly, applying techniques like statistical reasoning or machine learning is really the only way to chip away at it, that exists in spades in the space. It's not always immediately obvious, and I think from the outside, it's easy to assume this is actually pretty simple. So, the scale is a huge part of that.Corey: “Oh, petabytes. How quaint.”James: [laugh]. Exactly. Exactly I mean, it's mind-blowing every time I see some of the numbers in various parts of the Commerce Platform space. I talked about quadrillions earlier. Trillions is a pretty common unit of measure.The complexity that I talked about earlier, that's a result of external environments is another one. So, imposed by external entities, whether it's a government or a tax authority somewhere, or a business requirement from customers, or ourselves. I enjoy those as well. Those are different kinds of challenge. They really keep you on your toes.I enjoy thinking of them as an engineering problem, like, how do I get in front of them? And that's something we spend a lot of time doing in Commerce Platform. And when we get it right, customers are just unaware of it. And then the third one is, I personally am always attracted to the opportunity to have an impact. And this is a space where we get to hopefully positively impact every single customer every day. And that, to me is pretty fulfilling.Those are kind of the three standout reasons why I think this is actually a super-exciting space. And I think it's often an underestimated space. I think once folks join the team and sort of start to dig in, I've never heard anybody after they've joined, telling me that what they're doing is boring. Challenging, yes. Is frustrating, sometimes. Hard, absolutely, but boring never comes up.Corey: There's almost no service, other than IAM, that I can think of that impacts every customer simultaneously. And it's easy for me to sit in the cheap seats and say, “Oh, you should change this,” or, “You should change that.” But every change you have is so massive in scale that it's going to break a whole bunch of companies' automations around the bill processing in different ways. You have an entire category of user persona who is used to clicking a certain button in this certain place in the console to generate the report every month, and if that button moves or changes color, or has a different font, suddenly that renders their documentation invalid, and they're scrambling because it's not their core competency—nor should it be—and every change you make is so constricted, just based upon all the different concerns that you've got to be juggling with. How do you get anything done at all? I find that to be one of the most impressive aspects about your organization, bar none.James: Yeah, I'm not going to lie and say that it isn't a challenge, but a lot of it comes down to the talent that we have on the team. We have a super-motivated, super-smart, super-engaged team, and we spend a lot of time figuring out how to make sure that we can keep moving, keep up with the business, keep up with a world that's getting more complicated [laugh] with every passing day. So, you've kind of hit on one of the core challenges there, which is, how do we keep up with all of those different dimensions that are demanding an increasing amount of engineering and new support and new investment from us, while we keep those customers happy?And I think you touched on something else a little bit indirectly there, which is, a lot of our customers are actually pretty technical across AWS. The customers that Commerce Platform supports, are often the least technical of our customers, and so often need the most help understanding why things are the way they are, where the constraints are.Corey: “A big bill from Amazon. How many books did you people buy last month?”—James: [laugh]. Exactly.Corey: —is still very much level of understanding in some cases. And it's not because they're dumb; far from it. It's just, imagine that some people view there as being more to life than understanding the nuances and intricacies of cloud computing. How dare they?James: Exactly. Who would have thought?Corey: So, as you look now over all of your domain, such as it is, what sucks the most? What are you looking to fix as far as impactful changes that the rest of the world might experience? Because I'm not going to accept one of those questions like, “Oh, yeah, on the back-end, we have this storage subsystem for a tertiary thing that just annoys me because it wakes us up once in a whi”—no, no, I want something customer-facing. What's the painful thing you're looking at fixing next?James: I don't like surprising customers. And free tier is, sort of, one of those buckets of surprises, but there are others. Another one that's pretty squarely in my sights is, whether we like it or not, customer accounts get compromised. Usually, it's a password got reused somewhere or was accidentally committed into a GitHub repository somewhere.And we have pretty established, pretty effective mechanisms for finding all of those, we'll scan for passwords and credentials, and alert customers to those, and help them correct that pretty quickly. We're also actually pretty good at detecting when an account does start to do something that suggests that it's been compromised. Usually, the first thing that a compromised account starts to do is cryptocurrency mining. We're pretty quick to catch those; we catch those within a matter of hours, much faster most days.What we haven't really cracked and where I'm focused at the moment is getting back to the customer in a way that's effective. And by that I mean specifically, we detect an account compromised super-quickly, we reach out automatically. And so, you know, a customer has got some kind of contact from us usually within a couple of hours. It's not having the effect that we need it to. Customers are still being surprised a month later by a large bill. And so, we're digging into how much of that is because they never saw the contact, they didn't know what to do with the contact.Corey: It got buried with all the other, “Hey, we saw you spun up an S3 bucket. Have you heard of what S3 is?” Again, that's all valuable, but you have 300-some-odd services. If you start doing that for every service, you're going to hit mail sending limits for Gmail.James: Exactly. It's not just enough that we detect those and notify customers; we have to reduce the size of the surprise. It's one thing to spend 100 bucks a month on average, and then suddenly find that your spend has jumped $250 because you reused the password somewhere and somebody got ahold of it and it's cryptocurrency-mining your account. It's a whole different ballgame to spend 100 bucks a month and then at the end of the month discover that your bill is suddenly $2,000 or $20,000. And so, that's something that I really wanted to make some progress on this year. Corey: I've really enjoyed our conversation. If people want to learn more about how you view these things, how you're approaching some of these problems, or potentially are just the right kind of warped to consider joining up, where's the best place for them to go?James: They should drop me an email at email@example.com. That is the most direct way to get hold of me, and I promise I will get back to you. I try to stay on top of my email as much as possible. But that will come straight to me, and I'm always happy to talk to folks about the space, talk to folks about opportunities in this team, opportunities across AWS, or just hear what's not working, make sure that it's something that we're aware of and looking at.Corey: Throughout Amazon, but particularly within Commerce Platform, I've always appreciated the response of, whenever I report something, no matter how ridiculous it is—and I assure you there's an awful lot of ridiculousness in my bug reports—the response has always been the same: “Tell me more. Help me understand what it is you're trying to achieve—even if it is ridiculous—so we can look at this and see what is actually going on.” Every Amazonian team has been great about that or you're not at Amazon very long, but you folks have taken that to an otherworldly level. I just want to thank you for doing that.James: I appreciate you for calling that out. We try, you know, we really do. We take listening to our customers very seriously because, at the end of the day, that's what makes us better, and that's how we make sure we're in it for the long haul.Corey: Thanks once again for being so generous with your time. I really appreciate it.James: Yeah, thanks for having me on. I've enjoyed it.Corey: James Greenfield, VP of Commerce Platform at AWS. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment—possibly on YouTube as well—about how you aren't actually giving this five-stars at all; you have taken three trillions of a star off of the rating.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Episode 311 is about the 2005 movie “Man with the Screaming Brain”. It was written and directed by Bruce Campbell. Yes, the guy from “Evil Dead 2”. It's nothing like that movie. Find us on Instagram where we are @chewingthescenery or easily find us on Facebook. CTS can be found on Soundcloud, Stitcher, Apple Music and anywhere fine podcasts can be found. Please rate, review, subscribe- it really does help new listeners find us! #horror #horrormovies #horrornerd #horroraddict #horrorjunkie #monsterkid #bmovie #scarymovies #monstermovie #podcast #chewingthescenery #zombies #zombie #VHS #moviemonsters #freepodcast #denver #colorado #manwiththescreamingbrain #brucecampbell #tedraimi
Bill Frost (SLUGMag.com & X96 Radio From Hell), Tommy Milagro (SlamWrestling.net), and special guest star Dr. Paul "Zil" White (molder of university minds) talk Mr. Mayor (R.I.P.), Grand Crew, Angelyne' George Carlin's American Dream, The Kids in the Hall: Comedy Punks, What's Dr. White Watching (Jeopardy, Abbott Elementary, Mr. Mayor, Star Trek: Picard, Star Trek: Strange New Worlds, Moon Knight, Better Call Saul, The Late Show with Stephen Colbert, Matter of Fact with Soledad O'Brien, The Wonder Years, Ghosts, Bob's Burgers, Top Chef, The Baby, and Yellowjackets), The Secret of Skinwalker Ranch, The Flight Attendant, Made for Love, Smoking Aces, Welcome to Flatch, Rasslin' News, I Love That for You, The Wilds, The Equalizer, Woke, Hacks, Mayans, and The Kids in the Hall. Drinking: Vanilla Bean Rum and Zero Gin from OFFICIAL TV Tan sponsors Outlaw Distillery and Boozetique.* Yell at us (or order a TV Tan T-shirt) @TVTanPodcast on Twitter, Facebook, Gmail.* Rate us: Spotify, Stitcher, Apple Podcasts, Google Podcasts, YouTube, Amazon Podcasts, Audible, etc.
Episode 81- Hyper Inflation, Nickels & the Screaming Aztec Death Whistle Also Available On Podcast Transcript Gun Lawyer Episode 81 SUMMARY KEYWORDS nickel, blade, crock, knife, sharpening, gun, property, sticks, knives, scabbard, carrying, firearm, cold steel, evan, edge, inflation, magnet, lawyer, piece,
It will take a miracle for our economy to survive the attacks it has undergone. But, will God step-in to save an economy of a Nation that has turned its back on Him? THE SCRIPTURE & SCRIPTURAL RESOURCES: Jeremiah 6:16 [NIV] - "This is what the LORD says: "Stand at the crossroads and look; ask for the ancient paths, ask where the good way is, and walk in it, and you will find rest for your souls. But you said, 'We will not walk in it.' Jeremiah 6:14-15 14 They dress the wound of my people as though it were not serious. ‘Peace, peace,' they say, when there is no peace. 15 Are they ashamed of their detestable conduct? No, they have no shame at all; they do not even know how to blush. So they will fall among the fallen; they will be brought down when I punish them,” says the Lord. THE NEWS & COMMENT: [AUDIO] - Biden starts SCREAMING about food shortages... which are currently happening under his administration [AUDIO] - Bernie Sanders says Food Lines are a Good Thing [AUDIO] - Yellen - more abortions will save the economy [AUDIO] - To his credit, I guess, Joe Biden straight up told us he was going to do this to gas/energy prices and thus tank the entire economy in the final presidential debate of 2020. Remember? THE LISTENERS: Jason - Question for Zach. Will the feds do all they can to keep the economy somewhat stable for midterm elections? Well lot of people do not realize, is a cost of diesel right now. Diesel moves Freight. Cost of freight going through the roof. More inflation this summer on Goods. --- --- --- --- Hi Todd, I've heard you initially on Rush and was thrilled when my coworker Jesse told me about your podcasts. I am a mail carrier so I'm blessed to have the seat time to listen.I've been on board since around episode 50 (ish). I've had a thought that's been eating at me for quite some time. When Donald Trump was in office and the flu with given name Covid broke out, I feel DT had the best interest of the country and was eager to fix things so he got the "shot" pushed thru quickly. Not something done in his best interest, but I also don't hold him accountable as he never forced a mandate. It was each individuals choice. I have chosen NOT to receive ANY shots and I also have never gotten the flu with given name covid. Fast forward to "vaccine " mandates and a now Radical Democrat Administration..... I've done some Bible reading over the years (not a lot) and the book of Revelation - Chapter 13 vs 11-18 speaks of the mark of the beast (666). I feel like the covid shot is a method being used to see how many "sheep" will follow for the future plan of implementing humans with the mark of the beast. I hear Zach A. and yourself talk about a paperless money future and I see so many ways implemented already to buy and sell products/goods without "money in hand" that quite honestly is making me a little frightened. I'd love to hear your thoughts on this. Am I crazy to think this way? Faithful, (hooked on Bonefrog coffee listener) Colleen aka Connie Staeck Gleason Wisconsin See omnystudio.com/listener for privacy information.
Yesterday marked what Stigall believes was the darkest vote ever cast in Congress. Hear all about it and why he believes it's the best possible opportunity for Republicans to counter-message. Emerald Robinson is in today's show (We promise. Sorry for yesterday's error.) Then Stigall talks with Fox News Host Dana Perino about the what she calls one of Biden's worst weeks yet and what being a woman and womanhood really means given she's promoting a paperback version of her bestseller “Everything Will Be OK: Life Lessons for Young Women From a Former Young Woman.” And one of the most incredibly passionate women we've ever interviewed on the subject of life - meet Christina Bennett from Live Action.
IT'S OUR SECOND BIRTHDAY! Happy Birthday I'm Screaming!! To celebrate we're talking the Aly & AJ concert and exciting casting news for Doctor Who & Percy Jackson! We're also wrapping up our Moon Knight chats and giving you our full, spoiler filled Doctor Strange in the Multiverse of Madness review!
The Slamfest Podcast brings the premier rock concert pregaming experience from the parking lot to the podcasting airwaves. Episode 100 - The "Birth" of Slafmest. Brad, Matt, Andy, Mike, Brad C. and Jay got together and saw Ozzfest 2004 - Black Sabbath, Judas Priest, Slayer and Black Label Society - at Fiddler's Green Amphitheatre in Greenwood Village, CO, just outside Denver. Brad welcomes everyone who attended this concert - Matt, Andy, Mike, Brad C. and Jay - to recap the show and discuss the topics. For the Band on the Bill Spotlight, they pin Judas Priest's eighth studio album, Screaming for Vengeance up against their ninth studio album, Defenders of Faith, but with a twist - specific songs up against eachother. After a Slamfest Tip of the Week, they are faced with a "Which Side are you On?" - Side 1 of Sabbath's third studio album, Master of Reality, from 1971 vs. Side 1 of Sabbath's fourth studio album, Vol 4 from 1972.Music in this episode by:Velvet RevolverBlack Label SocietyJudas PriestBlack SabbathKissOzzy OsbourneVisit the Slamfest Podcast online at: https://slamfest-podcast.simplecast.comRequest to join the Slamfest Podcast private Facebook page here:https://www.facebook.com/groups/slamfestpodcastE-mail us at : firstname.lastname@example.org
About AmyAmy Tobey has worked in tech for more than 20 years at companies of every size, working with everything from kernel code to user interfaces. These days she spends her time building an innovative Site Reliability Engineering program at Equinix, where she is a principal engineer. When she's not working, she can be found with her nose in a book, watching anime with her son, making noise with electronics, or doing yoga poses in the sun.Links Referenced: Equinix Metal: https://metal.equinix.com Personal Twitter: https://twitter.com/MissAmyTobey Personal Blog: https://tobert.github.io/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Optimized cloud compute plans have landed at Vultr to deliver lightning-fast processing power, courtesy of third-gen AMD EPYC processors without the IO or hardware limitations of a traditional multi-tenant cloud server. Starting at just 28 bucks a month, users can deploy general-purpose, CPU, memory, or storage optimized cloud instances in more than 20 locations across five continents. Without looking, I know that once again, Antarctica has gotten the short end of the stick. Launch your Vultr optimized compute instance in 60 seconds or less on your choice of included operating systems, or bring your own. It's time to ditch convoluted and unpredictable giant tech company billing practices and say goodbye to noisy neighbors and egregious egress forever. Vultr delivers the power of the cloud with none of the bloat. “Screaming in the Cloud” listeners can try Vultr for free today with a $150 in credit when they visit getvultr.com/screaming. That's G-E-T-V-U-L-T-R dot com slash screaming. My thanks to them for sponsoring this ridiculous podcast.Corey: Finding skilled DevOps engineers is a pain in the neck! And if you need to deploy a secure and compliant application to AWS, forgettaboutit! But that's where DuploCloud can help. Their comprehensive no-code/low-code software platform guarantees a secure and compliant infrastructure in as little as two weeks, while automating the full DevSecOps lifestyle. Get started with DevOps-as-a-Service from DuploCloud so that your cloud configurations are done right the first time. Tell them I sent you and your first two months are free. To learn more visit: snark.cloud/duplo. Thats's snark.cloud/D-U-P-L-O-C-L-O-U-D.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while I catch up with someone that it feels like I've known for ages, and I realize somehow I have never been able to line up getting them on this show as a guest. Today is just one of those days. And my guest is Amy Tobey who has been someone I've been talking to for ages, even in the before-times, if you can remember such a thing. Today, she's a Senior Principal Engineer at Equinix. Amy, thank you for finally giving in to my endless wheedling.Amy: Thanks for having me. You mentioned the before-times. Like, I remember it was, like, right before the pandemic we had beers in San Francisco wasn't it? There was Ian there—Corey: Yeah, I—Amy: —and a couple other people. It was a really great time. And then—Corey: I vaguely remember beer. Yeah. And then—Amy: And then the world ended.Corey: Oh, my God. Yes. It's still March of 2020, right?Amy: As far as I know. Like, I haven't checked in a couple years.Corey: So, you do an awful lot. And it's always a difficult question to ask someone, so can you encapsulate your entire existence in a paragraph? It's—Amy: [sigh].Corey: —awful, so I'd like to give a bit more structure to it. Let's start with the introduction: You are a Senior Principal Engineer. We know it's high level because of all the adjectives that get put in there, and none of those adjectives are ‘associate' or ‘beginner' or ‘junior,' or all the other diminutives that companies like to play games with to justify paying people less. And you're at Equinix, which is a company that is a bit unlike most of the, shall we say, traditional cloud providers. What do you do over there and both as a company, as a person?Amy: So, as a company Equinix, what most people know about is that we have a whole bunch of data centers all over the world. I think we have the most of any company. And what we do is we lease out space in that data center, and then we have a number of other products that people don't know as well, which one is Equinix Metal, which is what I specifically work on, where we rent you bare-metal servers. None of that fancy stuff that you get any other clouds on top of it, there's things you can get that are… partner things that you can add-on, like, you know, storage and other things like that, but we just deliver you bare-metal servers with really great networking. So, what I work on is the reliability of that whole system. All of the things that go into provisioning the servers, making them come up, making sure that they get delivered to the server, make sure the API works right, all of that stuff.Corey: So, you're on the Equinix cloud side of the world more so than you are on the building data centers by the sweat of your brow, as they say?Amy: Correct. Yeah, yeah. Software side.Corey: Excellent. I spent some time in data centers in the early part of my career before cloud ate that. That was sort of cotemporaneous with the discovery that I'm the hardware destruction bunny, and I should go to great pains to keep my aura from anything expensive and important, like, you know, the SAN. So—Amy: Right, yeah.Corey: Companies moving out of data centers, and me getting out was a great thing.Amy: But the thing about SANs though, is, like, it might not be you. They're just kind of cursed from the start, right? They just always were kind of fussy and easy to break.Corey: Oh, yeah. I used to think—and I kid you not—that I had a limited upside to my career in tech because I sometimes got sloppy and I was fairly slow at crimping ethernet cables.Amy: [laugh].Corey: That is very similar to growing up in third grade when it became apparent that I was going to have problems in my career because my handwriting was sloppy. Yeah, it turns out the future doesn't look like we predicted it would.Amy: Oh, gosh. Are we going to talk about, like, neurological development now or… [laugh] okay, that's a thing I struggle with, too right, is I started typing as soon as they would let—in fact, before they would let me. I remember in high school, I had teachers who would grade me down for typing a paper out. They want me to handwrite it and I would go, “Cool. Go ahead and take a grade off because if I handwrite it, you're going to take two grades off my handwriting, so I'm cool with this deal.”Corey: Yeah, it was pretty easy early on. I don't know when the actual shift was, but it became more and more apparent that more and more things are moving towards a world where you could type. And I was almost five when I started working on that stuff, and that really wound up changing a lot of aspects of how I started seeing things. One thing I think you're probably fairly well known for is incidents. I want to be clear when I say that you are not the root cause as—“So, why are things broken?” “It's Amy again. What's she gotten into this time?” Great.Amy: [laugh]. But it does happen, but not all the time.Corey: Exa—it's a learning experience.Amy: Right.Corey: You've also been deeply involved with SREcon and a number of—a lot of aspects of what I will term—and please don't yell at me for this—SRE culture—Amy: Yeah.Corey: Which is sometimes a challenging thing to wind up describing or putting a definition around. The one that I've always been somewhat partial to is, “SRE is DevOps, except you worked at Google for a while.” I don't know how necessarily accurate that is, but it does rile people up.Amy: Yeah, it does. Dave Stanke actually did a really great talk at SREcon San Francisco just a couple weeks ago, about the DORA report. And the new DORA report, they split SRE out into its own function and kind of is pushing against that old model, which actually comes from Liz Fong-Jones—I think it's from her, or older—about, like, class SRE implements DevOps, which is kind of this idea that, like, SREs make DevOps happen. Things have evolved, right, since then. Things have evolved since Google released those books, and we're all just figured out what works and what doesn't a little bit.And so, it's not that we're implementing DevOps so much. In fact, it's that ops stuff that kind of holds us back from the really high impact work that SREs, I think, should be doing, that aren't just, like, fixing the problems, the symptoms down at the bottom layer, right? Like what we did as sysadmins 20 years ago. You know, we'd go and a lot of people are SREs that came out of the sysadmin world and still think in that mode, where it's like, “Well, I set up the systems, and when things break, I go and I fix them.” And, “Why did the developers keep writing crappy code? Why do I have to always getting up in the middle of the night because this thing crashed?”And it turns out that the work we need to do to make things more reliable, there's a ceiling to how far away the platform can take us, right? Like, we can have the best platform in the world with redundancy, and, you know, nine-way replicated data storage and all this crazy stuff, and still if we put crappy software on top, it's going to be unreliable. So, how do we make less crappy software? And for most of my career, people would be, like, “Well, you should test it.” And so, we started doing that, and we still have crappy software, so what's going on here? We still have incidents.So, we write more tests, and we still have incidents. We had a QA group, we still have incidents. We send the developers to training, and we still have incidents. So like, what is the thing we need to do to make things more reliable? And it turns out, most of it is culture work.Corey: My perspective on this stems from being a grumpy old sysadmin. And at some point, I started calling myself a systems engineer or DevOps or production engineer, or SRE. It was all from my point of view, the same job, but you know, if you call yourself a sysadmin, you're just asking for a 40% pay cut off the top.Amy: [laugh].Corey: But I still tended to view the world through that lens. I tended to be very good at Linux systems internals, for example, understanding system calls and the rest, but increasingly, as the DevOps wave or SRE wave, or Google-isation of the internet wound up being more and more of a thing, I found myself increasingly in job interviews, where, “Great, now, can you go wind up implementing a sorting algorithm on the whiteboard?” “What on earth? No.” Like, my lingua franca is shitty Bash, and no one tends to write that without a bunch of tab completions and quick checking with manpages—die.net or whatnot—on the fly as you go down that path.And it was awful, and I felt… like my skill set was increasingly eroding. And it wasn't honestly until I started this place where I really got into writing a fair bit of code to do different things because it felt like an orthogonal skill set, but the fullness of time, it seems like it's not. And it's a reskilling. And it made me wonder, does this mean that the areas of technology that I focused on early in my career, was that all a waste? And the answer is not really. Sometimes, sure, in that I don't spend nearly as much time worrying about inodes—for example—as I once did. But every once in a while, I'll run into something and I looked like a wizard from the future, but instead, I'm a wizard from the past.Amy: Yeah, I find that a lot in my work, now. Sometimes things I did 20 years ago, come back, and it's like, oh, yeah, I remember I did all that threading work in 2002 in Perl, and I learned everything the very, very, very hard way. And then, you know, this January, did some threading work to fix some stability issues, and all of it came flooding back, right? Just that the experiences really, more than the code or the learning or the text and stuff; more just the, like, this feels like threads [BLEEP]-ery. Is a diagnostic thing that sometimes we have to say.And then people are like, “Can you prove it?” And I'm like, “Not really,” because it's literally thread [BLEEP]-ery. Like, the definition of it is that there's weird stuff happening that we can't figure out why it's happening. There's something acting in the system that isn't synchronized, that isn't connected to other things, that's happening out of order from what we expect, and if we had a clear signal, we would just fix it, but we don't. We just have, like, weird stuff happening over here and then over there and over there and over there.And, like, that tells me there's just something happening at that layer and then have to go and dig into that right, and like, just basically charge through. My colleagues are like, “Well, maybe you should look at this, and go look at the database,” the things that they're used to looking at and that their experiences inform, whereas then I bring that ancient toiling through the threading mines experiences back and go, “Oh, yeah. So, let's go find where this is happening, where people are doing dangerous things with threads, and see if we can spot something.” But that came from that experience.Corey: And there's so much that just repeats itself. And history rhymes. The challenge is that, do you have 20 years of experience, or do you have one year of experience repeated 20 times? And as the tide rises, doing the same task by hand, it really is just a matter of time before your full-time job winds up being something a piece of software does. An easy example is, “Oh, what's your job?” “I manually place containers onto specific hosts.” “Well, I've got news for you, and you're not going to like it at all.”Amy: Yeah, yeah. I think that we share a little bit. I'm allergic to repeated work. I don't know if allergic is the right word, but you know, if I sit and I do something once, fine. Like, I'll just crank it out, you know, it's this form, or it's a datafile I got to write and I'll—fine I'll type it in and do the manual labor.The second time, the difficulty goes up by ten, right? Like, just mentally, just to do it, be like, I've already done this once. Doing it again is anathema to everything that I am. And then sometimes I'll get through it, but after that, like, writing a program is so much easier because it's like exponential, almost, growth in difficulty. You know, the third time I have to do the same thing that's like just typing the same stuff—like, look over here, read this thing and type it over here—I'm out; I can't do it. You know, I got to find a way to automate. And I don't know, maybe normal people aren't driven to live this way, but it's kept me from getting stuck in those spots, too.Corey: It was weird because I spent a lot of time as a consultant going from place to place and it led to some weird changes. For example, “Oh, thank God, I don't have to think about that whole messaging queue thing.” Sure enough, next engagement, it's message queue time. Fantastic. I found that repeating myself drove me nuts, but you also have to be very sensitive not to wind up, you know, stealing IP from the people that you're working with.Amy: Right.Corey: But what I loved about the sysadmin side of the world is that the vast majority of stuff that I've taken with me, lives in my shell config. And what I mean by that is I'm not—there's nothing in there is proprietary, but when you have a weird problem with trying to figure out the best way to figure out which Ruby process is stealing all the CPU, great, turns out that you can chain seven or eight different shell commands together through a bunch of pipes. I don't want to remember that forever. So, that's the sort of thing I would wind up committing as I learned it. I don't remember what company I picked that up at, but it was one of those things that was super helpful.I have a sarcastic—it's a one-liner, except no sane editor setting is going to show it in any less than three—of a whole bunch of Perl, piped into du, piped into the rest, that tells you one of the largest consumers of files in a given part of the system. And it rates them with stars and it winds up doing some neat stuff. I would never sit down and reinvent something like that today, but the fact that it's there means that I can do all kinds of neat tricks when I need to. It's making sure that as you move through your career, on some level, you're picking up skills that are repeatable and applicable beyond one company.Amy: Skills and tooling—Corey: Yeah.Amy: —right? Like, you just described the tool. Another SREcon talk was John Allspaw and Dr. Richard Cook talking about above the line; below the line. And they started with these metaphors about tools, right, showing all the different kinds of hammers.And if you're a blacksmith, a lot of times you craft specialized hammers for very specific jobs. And that's one of the properties of a tool that they were trying to get people to think about, right, is that tools get crafted to the job. And what you just described as a bespoke tool that you had created on the fly, that kind of floated under the radar of intellectual property. [laugh].So, let's not tell the security or IP people right? Like, because there's probably billions and billions of dollars of technically, like, made-up IP value—I'm doing air quotes with my fingers—you know, that's just basically people's shell profiles. And my God, the Emacs automation that people have done. If you've ever really seen somebody who's amazing at Emacs and is 10, 20, 30, maybe 40 years of experience encoded in their emacs settings, it's a wonder to behold. Like, I look at it and I go, “Man, I wish I could do that.”It's like listening to a really great guitar player and be like, “Wow, I wish I could play like them.” You see them just flying through stuff. But all that IP in there is both that person's collection of wisdom and experience and working with that code, but also encodes that stuff like you described, right? It's just all these little systems tricks and little fiddly commands and things we don't want to remember and so we encode them into our toolset.Corey: Oh, yeah. Anything I wound up taking, I always would share it with people internally, too. I'd mention, “Yeah, I'm keeping this in my shell files.” Because I disclosed it, which solves a lot of the problem. And also, none of it was even close to proprietary or anything like that. I'm sorry, but the way that you wind up figuring out how much of a disk is being eaten up and where in a more pleasing way, is not a competitive advantage. It just isn't.Amy: It isn't to you or me, but, you know, back in the beginning of our careers, people thought it was worth money and should be proprietary. You know, like, oh, that disk-checking script as a competitive advantage for our company because there are only a few of us doing this work. Like, it was actually being able to, like, manage your—[laugh] actually manage your servers was a competitive advantage. Now, it's kind of commodity.Corey: Let's also be clear that the world has moved on. I wound up buying a DaisyDisk a while back for Mac, which I love. It is a fantastic, pretty effective, “Where's all the stuff on your disk going?” And it does a scan and you can drive and collect things and delete them when trying to clean things out. I was using it the other day, so it's top of mind at the moment.But it's way more polished than that crappy Perl three-liner. And I see both sides, truly I do. The trick also, for those wondering [unintelligible 00:15:45], like, “Where is the line?” It's super easy. Disclose it, what you're doing, in those scenarios in the event someone is no because they believe that finding the right man page section for something is somehow proprietary.Great. When you go home that evening in a completely separate environment, build it yourself from scratch to solve the problem, reimplement it and save that. And you're done. There are lots of ways to do this. Don't steal from your employer, but your employer employs you; they don't own you and the way that you think about these problems.Every person I've met who has had a career that's longer than 20 minutes has a giant doc somewhere on some system of all of the scripts that they wound up putting together, all of the one-liners, the notes on, “Next time you see this, this is the thing to check.”Amy: Yeah, the cheat sheet or the notebook with all the little commands, or again the Emacs config, sometimes for some people, or shell profiles. Yeah.Corey: Here's the awk one-liner that I put that automatically spits out from an Apache log file what—the httpd log file that just tells me what are the most frequent talkers, and what are the—Amy: You should probably let go of that one. You know, like, I think that one's lifetime is kind of past, Corey. Maybe you—Corey: I just have to get it working with Nginx, and we're good to go.Amy: Oh, yeah, there you go. [laugh].Corey: Or S3 access logs. Perish the thought. But yeah, like, what are the five most high-volume talkers, and what are those relative to each other? Huh, that one thing seems super crappy and it's coming from Russia. But that's—hmm, one starts to wonder; maybe it's time to dig back in.So, one of the things that I have found is that a lot of the people talking about SRE seem to have descended from an ivory tower somewhere. And they're talking about how some of the best-in-class companies out there, renowned for their technical cultures—at least externally—are doing these things. But there's a lot more folks who are not there. And honestly, I consider myself one of those people who is not there. I was a competent engineer, but never a terrific one.And looking at the way this was described, I often came away thinking, “Okay, it was the purpose of this conference talk just to reinforce how smart people are, and how I'm not,” and/or, “There are the 18 cultural changes you need to make to your company, and then you can do something kind of like we were just talking about on stage.” It feels like there's a combination of problems here. One is making this stuff more accessible to folks who are not themselves in those environments, and two, how to drive cultural change as an individual contributor if that's even possible. And I'm going to go out on a limb and guess you have thoughts on both aspects of that, and probably some more hit me, please.Amy: So, the ivory tower, right. Let's just be straight up, like, the ivory tower is Google. I mean, that's where it started. And we get it from the other large companies that, you know, want to do conference talks about what this stuff means and what it does. What I've kind of come around to in the last couple of years is that those talks don't really reach the vast majority of engineers, they don't really apply to a large swath of the enterprise especially, which is, like, where a lot of the—the bulk of our industry sits, right? We spend a lot of time talking about the darlings out here on the West Coast in high tech culture and startups and so on.But, like, we were talking about before we started the show, right, like, the interior of even just America, is filled with all these, like, insurance and banks and all of these companies that are cranking out tons of code and servers and stuff, and they're trying to figure out the same problems. But they're structured in companies where their tech arm is still, in most cases, considered a cost center, often is bundled under finance, for—that's a whole show of itself about that historical blunder. And so, the tech culture is tend to be very, very different from what we experience in—what do we call it anymore? Like, I don't even want to say West Coast anymore because we've gone remote, but, like, high tech culture we'll say. And so, like, thinking about how to make SRE and all this stuff more accessible comes down to, like, thinking about who those engineers are that are sitting at the computers, writing all the code that runs our banks, all the code that makes sure that—I'm trying to think of examples that are more enterprise-y right?Or shoot buying clothes online. You go to Macy's for example. They have a whole bunch of servers that run their online store and stuff. They have internal IT-ish people who keep all this stuff running and write that code and probably integrating open-source stuff much like we all do. But when you go to try to put in a reliability program that's based on the current SRE models, like SLOs; you put in SLOs and you start doing, like, this incident management program that's, like, you know, you have a form you fill out after every incident, and then you [unintelligible 00:20:25] retros.And it turns out that those things are very high-level skills, skills and capabilities in an organization. And so, when you have this kind of IT mindset or the enterprise mindset, bringing the culture together to make those things work often doesn't happen. Because, you know, they'll go with the prescriptive model and say, like, okay, we're going to implement SLOs, we're going to start measuring SLIs on all of the services, and we're going to hold you accountable for meeting those targets. If you just do that, right, you're just doing more gatekeeping and policing of your tech environment. My bet is, reliability almost never improves in those cases.And that's been my experience, too, and why I get charged up about this is, if you just go slam in these practices, people end up miserable, the practices then become tarnished because people experienced the worst version of them. And then—Corey: And with the remote explosion as well, it turns out that changing jobs basically means their company sends you a different Mac, and the next Monday, you wind up signing into a different Slack team.Amy: Yeah, so the culture really matters, right? You can't cover it over with foosball tables and great lunch. You actually have to deliver tools that developers want to use and you have to deliver a software engineering culture that brings out the best in developers instead of demanding the best from developers. I think that's a fundamental business shift that's kind of happening. If I'm putting on my wizard hat and looking into the future and dreaming about what might change in the world, right, is that there's kind of a change in how we do leadership and how we do business that's shifting more towards that model where we look at what people are capable of and we trust in our people, and we get more out of them, the knowledge work model.If we want more knowledge work, we need people to be happy and to feel engaged in their community. And suddenly we start to see these kind of generational, bigger-pie kind of things start to happen. But how do we get there? It's not SLOs. It maybe it's a little bit starting with incidents. That's where I've had the most success, and you asked me about that. So, getting practical, incident management is probably—Corey: Right. Well, as I see it, the problem with SLOs across the board is it feels like it's a very insular community so far, and communicating it to engineers seems to be the focus of where the community has been, but from my understanding of it, you absolutely need buy-in at significantly high executive levels, to at the very least by you air cover while you're doing these things and making these changes, but also to help drive that cultural shift. None of this is something I have the slightest clue how to do, let's be very clear. If I knew how to change a company's culture, I'd have a different job.Amy: Yeah. [laugh]. The biggest omission in the Google SRE books was [Ers 00:22:58]. There was a guy at Google named Ers who owns availability for Google, and when anything is, like, in dispute and bubbles up the management team, it goes to Ers, and he says, “Thou shalt…” right? Makes the call. And that's why it works, right?Like, it's not just that one person, but that system of management where the whole leadership team—there's a large, very well-funded team with a lot of power in the organization that can drive availability, and they can say, this is how you're going to do metrics for your service, and this is the system that you're in. And it's kind of, yeah, sure it works for them because they have all the organizational support in place. What I was saying to my team just the other day—because we're in the middle of our SLO rollout—is that really, I think an SLO program isn't [clear throat] about the engineers at all until late in the game. At the beginning of the game, it's really about getting the leadership team on board to say, “Hey, we want to put in SLIs and SLOs to start to understand the functioning of our software system.” But if they don't have that curiosity in the first place, that desire to understand how well their teams are doing, how healthy their teams are, don't do it. It's not going to work. It's just going to make everyone miserable.Corey: It feels like it's one of those difficult to sell problems as well, in that it requires some tooling changes, absolutely. It requires cultural change and buy-in and whatnot, but in order for that to happen, there has to be a painful problem that a company recognizes and is willing to pay to make go away. The problem with stuff like this is that once you pay, there's a lot of extra work that goes on top of it as well, that does not have a perception—rightly or wrongly—of contributing to feature velocity, of hitting the next milestone. It's, “Really? So, we're going to be spending how much money to make engineers happier? They should get paid an awful lot and they're still complaining and never seem happy. Why do I care if they're happy other than the pure mercenary perspective of otherwise they'll quit?” I'm not saying that it's not worth pursuing; it's not a worthy goal. I am saying that it becomes a very difficult thing to wind up selling as a product.Amy: Well, as a product for sure, right? Because—[sigh] gosh, I have friends in the space who work on these tools. And I want to be careful.Corey: Of course. Nothing but love for all of those people, let's be very clear.Amy: But a lot of them, you know, they're pulling metrics from existing monitoring systems, they are doing some interesting math on them, but what you get at the end is a nice service catalog and dashboard, which are things we've been trying to land as products in this industry for as long as I can remember, and—Corey: “We've got it this time, though. This time we'll crack the nut.” Yeah. Get off the island, Gilligan.Amy: And then the other, like, risky thing, right, is the other part that makes me uncomfortable about SLOs, and why I will often tell folks that I talk to out in the industry that are asking me about this, like, one-on-one, “Should I do it here?” And it's like, you can bring the tool in, and if you have a management team that's just looking to have metrics to drive productivity, instead of you know, trying to drive better knowledge work, what you get is just a fancier version of more Taylorism, right, which is basically scientific management, this idea that we can, like, drive workers to maximum efficiency by measuring random things about them and driving those numbers. It turns out, that doesn't really work very well, even in industrial scale, it just happened to work because, you know, we have a bloody enough society that we pushed people into it. But the reality is, if you implement SLOs badly, you get more really bad Taylorism that's bad for you developers. And my suspicion is that you will get worse availability out of it than you would if you just didn't do it at all.Corey: This episode is sponsored by our friends at Revelo. Revelo is the Spanish word of the day, and its spelled R-E-V-E-L-O. It means “I reveal.” Now, have you tried to hire an engineer lately? I assure you it is significantly harder than it sounds. One of the things that Revelo has recognized is something I've been talking about for a while, specifically that while talent is evenly distributed, opportunity is absolutely not. They're exposing a new talent pool to, basically, those of us without a presence in Latin America via their platform. It's the largest tech talent marketplace in Latin America with over a million engineers in their network, which includes—but isn't limited to—talent in Mexico, Costa Rica, Brazil, and Argentina. Now, not only do they wind up spreading all of their talent on English ability, as well as you know, their engineering skills, but they go significantly beyond that. Some of the folks on their platform are hands down the most talented engineers that I've ever spoken to. Let's also not forget that Latin America has high time zone overlap with what we have here in the United States, so you can hire full-time remote engineers who share most of the workday as your team. It's an end-to-end talent service, so you can find and hire engineers in Central and South America without having to worry about, frankly, the colossal pain of cross-border payroll and benefits and compliance because Revelo handles all of it. If you're hiring engineers, check out revelo.io/screaming to get 20% off your first three months. That's R-E-V-E-L-O dot I-O slash screaming.Corey: That is part of the problem is, in some cases, to drive some of these improvements, you have to go backwards to move forwards. And it's one of those, “Great, so we spent all this effort and money in the rest of now things are worse?” No, not necessarily, but suddenly are aware of things that were slipping through the cracks previously.Amy: Yeah. Yeah.Corey: Like, the most realistic thing about first The Phoenix Project and then The Unicorn Project, both by Gene Kim, has been the fact that companies have these problems and actively cared enough to change it. In my experience, that feels a little on the rare side.Amy: Yeah, and I think that's actually the key, right? It's for the culture change, and for, like, if you really looking to be, like, do I want to work at this company? Am I investing my myself in here? Is look at the leadership team and be, like, do these people actually give a crap? Are they looking just to punt another number down the road?That's the real question, right? Like, the technology and stuff, at the point where I'm at in my career, I just don't care that much anymore. [laugh]. Just… fine, use Kubernetes, use Postgres, [unintelligible 00:27:30], I don't care. I just don't. Like, Oracle, I might have to ask, you know, go to finance and be like, “Hey, can we spend 20 million for a database?” But like, nobody really asks for that anymore, so. [laugh].Corey: As one does. I will say that I mostly agree with you, but a technology that I found myself getting excited about, given the time of the recording on this is… fun, I spent a bit of time yesterday—from when we're recording this—teaching myself just enough Go to wind up being together a binary that I needed to do something actively ridiculous for my camera here. And I found myself coming away deeply impressed by a lot of things about it, how prescriptive it was for one, how self-contained for another. And after spending far too many years of my life writing shitty Perl, and shitty Bash, and worse Python, et cetera, et cetera, the prescriptiveness was great. The fact that it wound up giving me something I could just run, I could cross-compile for anything I need to run it on, and it just worked. It's been a while since I found a technology that got me this interested in exploring further.Amy: Go is great for that. You mentioned one of my two favorite features of Go. One is usually when a program compiles—at least the way I code in Go—it usually works. I've been working with Go since about 0.9, like, just a little bit before it was released as 1.0, and that's what I've noticed over the years of working with it is that most of the time, if you have a pretty good data structure design and you get the code to compile, usually it's going to work, unless you're doing weird stuff.The other thing I really love about Go and that maybe you'll discover over time is the malleability of it. And the reason why I think about that more than probably most folks is that I work on other people's code most of the time. And maybe this is something that you probably run into with your business, too, right, where you're working on other people's infrastructure. And the way that we encode business rules and things in the languages, in our programming language or our config syntax and stuff has a huge impact on folks like us and how quickly we can come into a situation, assess, figure out what's going on, figure out where things are laid out, and start making changes with confidence.Corey: Forget other people for a minute they're looking at what I built out three or four years ago here, myself, like, I look at past me, it's like, “What was that rat bastard thinking? This is awful.” And it's—forget other people's code; hell is your own code, on some level, too, once it's slipped out of the mental stack and you have to re-explore it and, “Oh, well thank God I defensively wound up not including any comments whatsoever explaining what the living hell this thing was.” It's terrible. But you're right, the other people's shell scripts are finicky and odd.I started poking around for help when I got stuck on something, by looking at GitHub, and a few bit of searching here and there. Even these large, complex, well-used projects started making sense to me in a way that I very rarely find. It's, “What the hell is that thing?” is my most common refrain when I'm looking at other people's code, and Go for whatever reason avoids that, I think because it is so prescriptive about formatting, about how things should be done, about the vision that it has. Maybe I'm romanticizing it and I'll hate it and a week from now, and I want to go back and remove this recording, but.Amy: The size of the language helps a lot.Corey: Yeah.Amy: But probably my favorite. It's more of a convention, which actually funny the way I'm going to talk about this because the two languages I work on the most right now are Ruby and Go. And I don't feel like two languages could really be more different.Syntax-wise, they share some things, but really, like, the mental models are so very, very different. Ruby is all the way in on object-oriented programming, and, like, the actual real kind of object-oriented with messaging and stuff, and, like, the whole language kind of springs from that. And it kind of requires you to understand all of these concepts very deeply to be effective in large programs. So, what I find is, when I approach Ruby codebase, I have to load all this crap into my head and remember, “Okay, so yeah, there's this convention, when you do this kind of thing in Ruby”—or especially Ruby on Rails is even worse because they go deep into convention over configuration. But what that's code for is, this code is accessible to people who have a lot of free cognitive capacity to load all this convention into their heads and keep it in their heads so that the code looks pretty, right?And so, that's the trade-off as you said, okay, my developers have to be these people with all these spare brain cycles to understand, like, why I would put the code here in this place versus this place? And all these, like, things that are in the code, like, very compact, dense concepts. And then you go to something like Go, which is, like, “Nah, we're not going to do Lambdas. Nah”—[laugh]—“We're not doing all this fancy stuff.” So, everything is there on the page.This drives some people crazy, right, is that there's all this boilerplate, boilerplate, boilerplate. But the reality is, I can read most Go files from top to the bottom and understand what the hell it's doing, whereas I can go sometimes look at, like, a Ruby thing, or sometimes Python and e—Perl is just [unintelligible 00:32:19] all the time, right, it's there's so much indirection. And it just be, like, “What the [BLEEP] is going on? This is so dense. I'm going to have to sit down and write it out in longhand so I can understand what the developer was even doing here.” And—Corey: Well, that's why I got the Mac Studio; for when I'm not doing A/V stuff with it, that means that I'll have one core that I can use for, you know, front-end processing and the rest, and the other 19 cores can be put to work failing to build Nokogiri in Ruby yet again.Amy: [laugh].Corey: I remember the travails of working with Ruby, and the problem—I have similar problems with Python, specifically in that—I don't know if I'm special like this—it feels like it's a SRE DevOps style of working, but I am grabbing random crap off a GitHub constantly and running it, like, small scripts other people have built. And let's be clear, I run them on my test AWS account that has nothing important because I'm not a fool that I read most of it before I run it, but I also—it wants a different version of Python every single time. It wants a whole bunch of other things, too. And okay, so I use ASDF as my version manager for these things, which for whatever reason, does not work for the way that I think about this ergonomically. Okay, great.And I wind up with detritus scattered throughout my system. It's, “Hey, can you make this reproducible on my machine?” “Almost certainly not, but thank you for asking.” It's like ‘Step 17: Master the Wolf' level of instructions.Amy: And I think Docker generally… papers over the worst of it, right, is when we built all this stuff in the aughts, you know, [CPAN 00:33:45]—Corey: Dev containers and VS Code are very nice.Amy: Yeah, yeah. You know, like, we had CPAN back in the day, I was doing chroots, I think in, like, '04 or '05, you know, to solve this problem, right, which is basically I just—screw it; I will compile an entire distro into a directory with a Perl and all of its dependencies so that I can isolate it from the other things I want to run on this machine and not screw up and not have these interactions. And I think that's kind of what you're talking about is, like, the old model, when we deployed servers, there was one of us sitting there and then we'd log into the server and be like, I'm going to install the Perl. You know, I'll compile it into, like, [/app/perl 558 00:34:21] whatever, and then I'll CPAN all this stuff in, and I'll give it over to the developer, tell them to set their shebang to that and everything just works. And now we're in a mode where it's like, okay, you got to set up a thousand of those. “Okay, well, I'll make a tarball.” [laugh]. But it's still like we had to just—Corey: DevOps, but [unintelligible 00:34:37] dev closer to ops. You're interrelating all the time. Yeah, then Docker comes along, and add dev is, like, “Well, here's the container. Good luck, asshole.” And it feels like it's been cast into your yard to worry about.Amy: Yeah, well, I mean, that's just kind of business, or just—Corey: Yeah. Yeah.Amy: I'm not sure if it's business or capitalism or something like that, but just the idea that, you know, if I can hand off the shitty work to some other poor schlub, why wouldn't I? I mean, that's most folks, right? Like, just be like, “Well”—Corey: Which is fair.Amy: —“I got it working. Like, my part is done, I did what I was supposed to do.” And now there's a lot of folks out there, that's how they work, right? “I hit done. I'm done. I shipped it. Sure. It's an old [unintelligible 00:35:16] Ubuntu. Sure, there's a bunch of shell scripts that rip through things. Sure”—you know, like, I've worked on repos where there's hundreds of things that need to be addressed.Corey: And passing to someone else is fine. I'm thrilled to do it. Where I run into problems with it is where people assume that well, my part was the hard part and anything you schlubs do is easy. I don't—Amy: Well, that's the underclass. Yeah. That's—Corey: Forget engineering for a second; I throw things to the people over in the finance group here at The Duckbill Group because those people are wizards at solving for this thing. And it's—Amy: Well, that's how we want to do things.Corey: Yeah, specialization works.Amy: But we have this—it's probably more cultural. I don't want to pick, like, capitalism to beat on because this is really, like, human cultural thing, and it's not even really particularly Western. Is the idea that, like, “If I have an underclass, why would I give a shit what their experience is?” And this is why I say, like, ops teams, like, get out of here because most ops teams, the extant ops teams are still called ops, and a lot of them have been renamed SRE—but they still do the same job—are an underclass. And I don't mean that those people are below us. People are treated as an underclass, and they shouldn't be. Absolutely not.Corey: Yes.Amy: Because the idea is that, like, well, I'm a fancy person who writes code at my ivory tower, and then it all flows down, and those people, just faceless people, do the deployment stuff that's beneath me. That attitude is the most toxic thing, I think, in tech orgs to address. Like, if you're trying to be like, “Well, our liability is bad, we have security problems, people won't fix their code.” And go look around and you will find people that are treated as an underclass that are given codes thrown over the wall at them and then they just have to toil through and make it work. I've worked on that a number of times in my career.And I think just like saying, underclass, right, or caste system, is what I found is the most effective way to get people actually thinking about what the hell is going on here. Because most people are just, like, “Well, that's just the way things are. It's just how we've always done it. The developers write to code, then give it to the sysadmins. The sysadmins deploy the code. Isn't that how it always works?”Corey: You'd really like to hope, wouldn't you?Amy: [laugh]. Not me. [laugh].Corey: Again, the way I see it is, in theory—in theory—sysadmins, ops, or that should not exist. People should theoretically be able to write code as developers that just works, the end. And write it correct the first time and never have to change it again. Yeah. There's a reason that I always like to call staging environments in places I work ‘theory' because it works in theory, but not in production, and that is fundamentally the—like, that entire job role is the difference between theory and practice.Amy: Yeah, yeah. Well, I think that's the problem with it. We're already so disconnected from the physical world, right? Like, you and I right now are talking over multiple strands of glass and digital transcodings and things right now, right? Like, we are detached from the physical reality.You mentioned earlier working in data centers, right? The thing I miss about it is, like, the physicality of it. Like, actually, like, I held a server in my arms and put it in the rack and slid it into the rails. I plugged into power myself; I pushed the power button myself. There's a server there. I physically touched it.Developers who don't work in production, we talked about empathy and stuff, but really, I think the big problem is when they work out in their idea space and just writing code, they write the unit tests, if we're very lucky, they'll write a functional test, and then they hand that wad off to some poor ops group. They're detached from the reality of operations. It's not even about accountability; it's about experience. The ability to see all of the weird crap we deal with, right? You know, like, “Well, we pushed the code to that server, but there were three bit flips, so we had to do it again. And then the other server, the disk failed. And on the other server…” You know? [laugh].It's just, there's all this weird crap that happens, these systems are so complex that they're always doing something weird. And if you're a developer that just spends all day in your IDE, you don't get to see that. And I can't really be mad at those folks, as individuals, for not understanding our world. I figure out how to help them, and the best thing we've come up with so far is, like, well, we start giving this—some responsibility in a production environment so that they can learn that. People do that, again, is another one that can be done wrong, where it turns into kind of a forced empathy.I actually really hate that mode, where it's like, “We're forcing all the developers online whether they like it or not. On-call whether they like it or not because they have to learn this.” And it's like, you know, maybe slow your roll a little buddy because the stuff is actually hard to learn. Again, minimizing how hard ops work is. “Oh, we'll just put the developers on it. They'll figure it out, right? They're software engineers. They're probably smarter than you sysadmins.” Is the unstated thing when we do that, right? When we throw them in the pit and be like, “Yeah, they'll get it.” [laugh].Corey: And that was my problem [unintelligible 00:39:49] the interview stuff. It was in the write code on a whiteboard. It's, “Look, I understood how the system fundamentally worked under the hood.” Being able to power my way through to get to an outcome even in language I don't know, was sort of part and parcel of the job. But this idea of doing it in artificially constrained environment, in a language I'm not super familiar with, off the top of my head, it took me years to get to a point of being able to do it with a Bash script because who ever starts with an empty editor and starts getting to work in a lot of these scenarios? Especially in an ops role where we're not building something from scratch.Amy: That's the interesting thing, right? In the majority of tech work today—maybe 20 years ago, we did it more because we were literally building the internet we have today. But today, most of the engineers out there working—most of us working stiffs—are working on stuff that already exists. We're making small incremental changes, which is great that's what we're doing. And we're dealing with old code.Corey: We're gluing APIs together, and that's fine. Ugh. I really want to thank you for taking so much time to talk to me about how you see all these things. If people want to learn more about what you're up to, where's the best place to find you?Amy: I'm on Twitter every once in a while as @MissAmyTobey, M-I-S-S-A-M-Y-T-O-B-E-Y. I have a blog I don't write on enough. And there's a couple things on the Equinix Metal blog that I've written, so if you're looking for that. Otherwise, mainly Twitter.Corey: And those links will of course be in the [show notes 00:41:08]. Thank you so much for your time. I appreciate it.Amy: I had fun. Thank you.Corey: As did I. Amy Tobey, Senior Principal Engineer at Equinix. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, or on the YouTubes, smash the like and subscribe buttons, as the kids say. Whereas if you've hated this episode, same thing, five-star review all the platforms, smash the buttons, but also include an angry comment telling me that you're about to wind up subpoenaing a copy of my shell script because you're convinced that your intellectual property and secrets are buried within.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.