Podcasts about orca security

ITSPmagazine | Technology. Cybersecurity. Society

Play Episode Listen Later May 3, 2022 34:13

About YoavYoav is a security veteran recognized on Microsoft Security Response Center's Most Valuable Research List (BlackHat 2019). Prior to joining Orca Security, he was a Unit 8200 researcher and team leader, a chief architect at Hyperwise Security, and a security architect at Check Point Software Technologies. Yoav enjoys hunting for Linux and Windows vulnerabilities in his spare time.Links Referenced: Orca Security: https://orca.security Twitter: https://twitter.com/yoavalon TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Optimized cloud compute plans have landed at Vultr to deliver lightning fast processing power, courtesy of third gen AMD EPYC processors without the IO, or hardware limitations, of a traditional multi-tenant cloud server. Starting at just 28 bucks a month, users can deploy general purpose, CPU, memory, or storage optimized cloud instances in more than 20 locations across five continents. Without looking, I know that once again, Antarctica has gotten the short end of the stick. Launch your Vultr optimized compute instance in 60 seconds or less on your choice of included operating systems, or bring your own. It's time to ditch convoluted and unpredictable giant tech company billing practices, and say goodbye to noisy neighbors and egregious egress forever. Vultr delivers the power of the cloud with none of the bloat. "Screaming in the Cloud" listeners can try Vultr for free today with a $150 in credit when they visit getvultr.com/screaming. That's G E T V U L T R.com/screaming. My thanks to them for sponsoring this ridiculous podcast.Corey: Finding skilled DevOps engineers is a pain in the neck! And if you need to deploy a secure and compliant application to AWS, forgettaboutit! But that's where DuploCloud can help. Their comprehensive no-code/low-code software platform guarantees a secure and compliant infrastructure in as little as two weeks, while automating the full DevSecOps lifestyle. Get started with DevOps-as-a-Service from DuploCloud so that your cloud configurations are done right the first time. Tell them I sent you and your first two months are free. To learn more visit: snark.cloud/duplocloud. Thats's snark.cloud/D-U-P-L-O-C-L-O-U-D. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Periodically, I would say that I enjoy dealing with cloud platform security issues, except I really don't. It's sort of forced upon me to deal with much like a dead dog is cast into their neighbor's yard for someone else to have to worry about. Well, invariably, it seems like it's my yard.And I'm only on the periphery of these things. Someone who's much more in the trenches in the wide world of cloud security is joining me today. Yoav Alon is the CTO at Orca Security. Yoav, thank you for taking the time to join me today and suffer the slings and arrows I'll no doubt be hurling your way.Yoav: Thank you, Corey, for having me. I've been a longtime listener, and it's an honor to be here.Corey: I still am periodically surprised that anyone listens to these things. Because it's unlike a newsletter where everyone will hit reply and give me a piece of their mind. People generally don't wind up sending me letters about things that they hear on the podcast, so whenever I talk to somebody listens to it as, “Oh. Oh, right, I did turn the microphone on. Awesome.” So, it's always just a little on the surreal side.But we're not here to talk necessarily about podcasting, or the modern version of an AM radio show. Let's start at the very beginning. What is Orca Security, and why would folks potentially care about what it is you do?Yoav: So, Orca Security is a cloud security company, and our vision is very simple. Given a customer's cloud environment, we want to detect all the risks in it and implement mechanisms to prevent it from occurring. And while it sounds trivial, before Orca, it wasn't really possible. You will have to install multiple tools and aggregate them and do a lot of manual work, and it was messy. And we wanted to change that, so we had, like, three guiding principles.We call it seamless, so I want to detect all the risks in your environment without friction, which is our speak for fighting with your peers. We also want to detect everything so you don't have to install, like, a tool for each issue: A tool for vulnerabilities, a tool for misconfigurations, and for sensitive data, IAM roles, and such. And we put a very high priority on context, which means telling you what's important, what's not. So, for example, S3 bucket open to the internet is important if it has sensitive data, not if it's a, I don't know, static website.Corey: Exactly. I have a few that I'd like to get screamed at in my AWS account, like, “This is an open S3 bucket and it's terrible.” I look at it the name is assets.lastweekinaws.com. Gee, I wonder if that's something that's designed to be a static hosted website.Increasingly, I've been slapping CloudFront in front of those things just to make the broken warning light go away. I feel like it's an underhanded way of driving CloudFront adoption some days, but not may not be the most charitable interpretation thereof. Orca has been top-of-mind for a lot of folks in the security community lately because let's be clear here, dealing with security problems in cloud providers from a vendor perspective is an increasingly crowded—and clouded—space. Just because there's so much—there's investment pouring into it, everyone has a slightly different take on the problem, and it becomes somewhat challenging to stand out from the pack. You didn't really stand out from the pack so much as leaped to the front of it and more or less have become the de facto name in a very short period of time, specifically—at least from my world—when you wound up having some very interesting announcements about vulnerabilities within AWS itself. You will almost certainly do a better job of relating the story, so please, what did you folks find?Yoav: So, back in September of 2021, two of my researchers, Yanir Tsarimi and Tzah Pahima, each one of them within a relatively short span of time from each other, found a vulnerability in AWS. Tzah found a vulnerability in CloudFormation which we named BreakingFormation and Yanir found a vulnerability in AWS Glue, which we named SuperGlue. We're not the best copywriters, but anyway—Corey: No naming things is hard. Ask any Amazonian.Yoav: Yes. [laugh]. So, I'll start with BreakingFormation which caught the eyes of many. It was an XXE SSRF, which is jargon to say that we were able to read files and execute HTTP requests and read potentially sensitive data from CloudFormation servers. This one was mitigated within 26 hours by AWS, so—Corey: That was mitigated globally.Yoav: Yes, globally, which I've never seen such quick turnaround anywhere. It was an amazing security feat to see.Corey: Particularly in light of the fact that AWS does a lot of things very right when it comes to, you know, designing cloud infrastructure. Imagine that, they've had 15 years of experience and basically built the idea of cloud, in some respects, at the scale that hyperscalers operate at. And one of their core tenets has always been that there's a hard separation between regions. There are remarkably few global services, and those are treated with the utmost of care and delicacy. To the point where when something like that breaks as an issue that spans more than one region, it is headline-making news in many cases.So it's, they almost never wind up deploying things to all regions at the same time. That can be irksome when we're talking about things like I want a feature that solves a problem that I have, and I have to wait months for it to hit a region that I have resources living within, but for security, stuff like this, I am surprised that going from, “This is the problem,” to, “It has been mitigated,” took place within 26 hours. I know it sounds like a long time to folks who are not deep in the space, but that is superhero speed.Yoav: A small correction, it's 26 hours for, like, the main regions. And it took three to four days to propagate to all regions. But still, it's speed of lighting in for security space.Corey: When this came out, I was speaking to a number of journalists on background about trying to wrap their head around this, and they said that, “Oh yeah, and security is always, like, the top priority for AWS, second only to uptime and reliability.” And… and I understand the perception, but I disagree with it in the sense of the nightmare scenario—that every time I mention to a security person watching the blood drain from their face is awesome—but the idea that take IAM, which as Werner said in his keynote, processes—was it 500 million or was it 500 billion requests a second, some ludicrous number—imagine fails open where everything suddenly becomes permitted. I have to imagine in that scenario, they would physically rip the power cables out of the data centers in order to stop things from going out. And that is the right move. Fortunately, I am extremely optimistic that will remain a hypothetical because that is nightmare fuel right there.But Amazon says that security is job zero. And my cynical interpretation is that well, it wasn't, but they forgot security, decided to bolt it on to the end, like everyone else does, and they just didn't want to renumber all their slides, so instead of making it point one, they just put another slide in front of it and called the job zero. I'm sure that isn't how it worked, but for those of us who procrastinate and building slide decks for talks, it has a certain resonance to it. That was one issue. The other seemed a little bit more pernicious focusing on Glue, which is their ETL-as-a-Service… service. One of them I suppose. Tell me more about it.Yoav: So, one of the things that we found when we found the BreakingFormation when we reported the vulnerability, it led us to do a quick Google search, which led us back to the Glue service. It had references to Glue, and we started looking around it. And what we were able to do with the vulnerability is given a specific feature in Glue, which we don't disclose at the moment, we were able to effectively take control over the account which hosts the Glue service in us-east-1. And having this control allowed us to essentially be able to impersonate the Glue service. So, every role in AWS that has a trust to the Glue service, we were able to effectively assume a role into it in any account in AWS. So, this was more critical a vulnerability in its effect.Corey: I think on some level, the game of security has changed because for a lot of us who basically don't have much in the way of sensitive data living in AWS—and let's be clear, I take confidentiality extremely seriously. Our clients on the consulting side view their AWS bills themselves as extremely confidential information that Amazon stuffs into a PDF and emails every month. But still. If there's going to be a leak, we absolutely do not want it to come from us, and that is something that we take extraordinarily seriously. But compared to other jobs I've had in the past, no one will die if that information gets out.It is not the sort of thing that is going to ruin people's lives, which is very often something that can happen in some data breaches. But in my world, one of the bad cases of a breach of someone getting access to my account is they could spin up a bunch of containers on the 17 different services that AWS offers that can run containers and mine cryptocurrency with it. And the damage to me then becomes a surprise bill. Okay, great. I can live with that.Something that's a lot scarier to a lot of companies with, you know, serious problems is, yep, fine, cost us money, whatever, but our access to our data is the one thing that is going to absolutely be the thing that cannot happen. So, from that perspective alone, something like Glue being able to do that is a lot more terrifying than subverting CloudFormation and being able to spin up additional resources or potentially take resources down. Is that how you folks see it too, or is—I'm sure there's nuance I'm missing.Yoav: So yeah, the access to data is top-of-mind for everyone. It's a bit scary to think about it. I have to mention, again, the quick turnaround time for AWS, which almost immediately issued a patch. It was a very fast one and they mitigated, again, the issue completely within days. About your comment about data.Data is king these days, there is nothing like data, and it has all the properties of everything that we care about. It's expensive to store, it's expensive to move, and it's very expensive if it leaks. So, I think a lot of people were more alarmed about the Glue vulnerability than the CloudFormation vulnerability. And they're right in doing so.Corey: I do want to call out that AWS did a lot of things right in this area. Their security posture is very clearly built around defense-in-depth. The fact that they were able to disclose—after some prodding—that they checked the CloudTrail logs for the service itself, dating back to the time the service launched, and verified that there had never been an exploit of this, that is phenomenal, as opposed to the usual milquetoast statements that companies have. We have no evidence of it, which can mean that we did the same thing and we looked through all the logs in it's great, but it can also mean that, “Oh, yeah, we probably should have logs, shouldn't we? But let's take a backlog item for that.” And that's just terrifying on some level.It becomes a clear example—a shining beacon for some of us in some cases—of doing things right from that perspective. There are other sides to it, though. As a customer, it was frustrating in the extreme to—and I mean, no offense by this—to learn about this from you rather than from the provider themselves. They wound up putting up a security notification many hours after your blog post went up, which I would also just like to point out—and we spoke about it at the time and it was a pure coincidence—but there was something that was just chef's-kiss perfect about you announcing this on Andy Jassy's birthday. That was just very well done.Yoav: So, we didn't know about Andy's birthday. And it was—Corey: Well, I see only one of us has a company calendar with notable executive birthdays splattered all over it.Yoav: Yes. And it was also published around the time that AWS CISO was announced, which was also a coincidence because the date was chosen a lot of time in advance. So, we genuinely didn't know.Corey: Communicating around these things is always challenging because on the one hand, I can absolutely understand the cloud providers' position on this. We had a vulnerability disclosed to us. We did our diligence and our research because we do an awful lot of things correctly and everyone is going to have vulnerabilities, let's be serious here. I'm not sitting here shaking my fist, angry at AWS's security model. It works, and I am very much a fan of what they do.And I can definitely understand then, going through all of that there was no customer impact, they've proven it. What value is there to them telling anyone about it, I get that. Conversely, you're a security company attempting to stand out in a very crowded market, and it is very clear that announcing things like this demonstrates a familiarity with cloud that goes beyond the common. I radically changed my position on how I thought about Orca based upon these discoveries. It went from, “Orca who,” other than the fact that you folks have sponsored various publications in the past—thanks for that—but okay, a security company. Great to, “Oh, that's Orca. We should absolutely talk to them about a thing that we're seeing.” It has been transformative for what I perceive to be your public reputation in the cloud security space.So, those two things are at odds: The cloud provider doesn't want to talk about anything and the security company absolutely wants to demonstrate a conversational fluency with what is going on in the world of cloud. And that feels like it's got to be a very delicate balancing act to wind up coming up with answers that satisfy all parties.Yoav: So, I just want to underline something. We don't do what we do in order to make a marketing stand. It's a byproduct of our work, but it's not the goal. For the Orca Security Research Pod, which it's the team at Orca which does this kind of research, our mission statement is to make cloud security better for everyone. Not just Orca customers; for everyone.And you get to hear about the more shiny things like big headline vulnerabilities, but we also have very sensible blog posts explaining how to do things, how to configure things and give you more in-depth understanding into security features that the cloud providers themselves provide, which are great, and advance the state of the cloud security. I would say that having a cloud vulnerability is sort of one of those things, which makes me happy to be a cloud customer. On the one side, we had a very big vulnerability with very big impact, and the ability to access a lot of customers' data is conceptually terrifying. The flip side is that everything was mitigated by the cloud providers in warp speed compared to everything else we've seen in all other elements of security. And you get to sleep better knowing that it happened—so no platform is infallible—but still the cloud provider do work for you, and you'll get a lot of added value from that.Corey: You've made a few points when this first came out, and I want to address them. The first is, when I reached out to you with a, “Wow, great work.” You effectively instantly came back with, “Oh, it wasn't me. It was members of my team.” So, let's start there. Who was it that found these things? I'm a huge believer giving people credit for the things that they do.The joy of being in a leadership position is if the company screws up, yeah, you take responsibility for that, whether the company does something great, yeah, you want to pass praise onto the people who actually—please don't take this the wrong way—did the work. And not that leadership is not work, it absolutely is, but it's a different kind of work.Yoav: So, I am a security researcher, and I am very mindful for the effort and skill it requires to find vulnerabilities and actually do a full circle on them. And the first thing I'll mention is Tzah Pahima, which found the BreakingFormation vulnerability and the vulnerability in CloudFormation, and Yanir Tsarimi, which found the AutoWarp vulnerability, which is the Azure vulnerability that we have not mentioned, and the Glue vulnerability, dubbed SuperGlue. Both of them are phenomenal researcher, world-class, and I'm very honored to work with them every day. It's one of my joys.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: It's very clear that you have built an extraordinary team for people who are able to focus on vulnerability research. Which, on some level, is very interesting because you are not branded as it were as a vulnerability research company. This is not something that is your core competency; it's not a thing that you wind up selling directly that I'm aware of. You are selling a security platform offering. So, on the one hand, it makes perfect sense that you would have a division internally that works on this, but it's also very noteworthy, I think, that is not the core description of what it is that you do.It is a means by which you get to the outcome you deliver for customers, not the thing that you are selling directly to them. I just find that an interesting nuance.Yoav: Yes, it is. And I would elaborate and say that research informs the product, and the product informs research. And we get to have this fun dance where we learn new things by doing research. We [unintelligible 00:18:08] the product, and we use the customers to teach us things that we didn't know. So, it's one of those happy synergies.Corey: I want to also highlight a second thing that you have mentioned and been very, I guess, on message about since news of this stuff first broke. And because it's easy to look at this and sensationalize aspects of it, where, “See? The cloud providers security model is terrible. You shouldn't use them. Back to data centers we go.” Is basically the line taken by an awful lot of folks trying to sell data center things.That is not particularly helpful for the way that the world is going. And you've said, “Yeah, you should absolutely continue to be in cloud. Do not disrupt your cloud plan as a result.” And let's be clear, none of the rest of us are going to find and mitigate these things with anything near the rigor or rapidity that the cloud providers can and do demonstrate.Yoav: I totally agree. And I would say that the AWS security folks are doing a phenomenal job. I can name a few, but they're all great. And I think that the cloud is by far a much safer alternative than on-prem. I've never seen issues in my on-prem environment which were critical and fixed in such a high velocity and such a massive scale.And you always get the incremental improvements of someone really thinking about all the ins and outs of how to do security, how to do security in the cloud, how to make it faster, more reliable, without a business interruptions. It's just phenomenal to see and phenomenal to witness how far we've come in such a relatively short time as an industry.Corey: AWS in particular, has a reputation for being very good at security. I would argue that, from my perspective, Google is almost certainly slightly better at their security approach than AWS is, but to be clear, both of them are significantly further along the path than I am going to be. So great, fantastic. You also have found something interesting over in the world of Azure, and that honestly feels like a different class of vulnerability. To my understanding, the Azure vulnerability that you recently found was you could get credential material for other customers simply by asking for it on a random high port. Which is one of those—I'm almost positive I'm misunderstanding something here. I hope. Please?Yoav: I'm not sure you're misunderstanding. So, I would just emphasize that the vulnerability again, was found by Yanir Tsarimi. And what he found was, he used a service called Azure Automation which enables you essentially to run a Python script on various events and schedules. And he opened the python script and he tried different ports. And one of the high ports he found, essentially gave him his credentials. And he said, “Oh, wait. That's a really odd port for an HTTP server. Let's try, I don't know, a few ports on either way.” And he started getting credentials from other customers. Which was very surprising to us.Corey: That is understating it by a couple orders of magnitude. Yes, like, “Huh. That seems sub-optimal,” is sort of like the corporate messaging approved thing. At the time you discover that—I'm certain it was a three-minute-long blistering string of profanity in no fewer than four languages.Yoav: I said to him that this is, like, a dishonorable bug because he worked very little to find it. So it was, from start to finish, the entire research took less than two hours, which, in my mind, is not enough for this kind of vulnerability. You have to work a lot harder to get it. So.Corey: Yeah, exactly. My perception is that when there are security issues that I have stumbled over—for example, I gave a talk at re:Invent about it in the before times, one of them was an overly broad permission in a managed IAM policy for SageMaker. Okay, great. That was something that obviously was not good, but it also was more of a privilege escalation style of approach. It wasn't, “Oh, by the way, here's the keys to everything.”That is the type of vulnerability I have come to expect, by and large, from cloud providers. We're just going to give you access credentials for other customers is one of those areas that… it bugs me on a visceral level, not because I'm necessarily exposed personally, but because it more or less shores up so many of the arguments that I have spent the last eight years having with folks are like, “Oh, you can't go to cloud. Your data should live on your own stuff. It's more secure that way.” And we were finally it feels like starting to turn a cultural corner on these things.And then something like that happens, and it—almost have those naysayers become vindicated for it. And it's… it almost feels, on some level, and I don't mean to be overly unkind on this, but it's like, you are absolutely going to be in a better security position with the cloud providers. Except to Azure. And perhaps that is unfair, but it seems like Azure's level of security rigor is nowhere near that of the other two. Is that generally how you're seeing things?Yoav: I would say that they have seen more security issues than most other cloud providers. And they also have a very strong culture of report things to us, and we're very streamlined into patching those and giving credit where credit's due. And they give out bounties, which is an incentives for more research to happen on those platforms. So, I wouldn't say this categorically, but I would say that the optics are not very good. Generally, the cloud providers are much safer than on-prem because you only hear very seldom on security issues in the cloud.You hear literally every other day on issues happening to on-prem environments all over the place. And people just say they expect it to be this way. Most of the time, it's not even a headline. Like, “Company X affected with cryptocurrency or whatever.” It happens every single day, and multiple times a day, breaches which are massively bigger. And people who don't want to be in the cloud will find every reason not to be the cloud. Let us have fun.Corey: One of the interesting parts about this is that so many breaches that are on-prem are just never discovered because no one knows what the heck's running in an environment. And the breaches that we hear about are just the ones that someone had at least enough wherewithal to find out that, “Huh. That shouldn't be the way that it is. Let's dig deeper.” And that's a bad day for everyone. I mean, no one enjoys those conversations and those moments.And let's be clear, I am surprisingly optimistic about the future of Azure Security. It's like, “All right, you have a magic wand. What would you do to fix it?” It's, “Well, I'd probably, you know, hire Charlie Bell and get out of his way,” is not a bad answer as far as how these things go. But it takes time to reform a culture, to wind up building in security as a foundational principle. It's not something you can slap on after the fact.And perhaps this is unfair. But Microsoft has 30 years of history now of getting the world accustomed to oh, yeah, just periodically, terrible vulnerabilities are going to be discovered in your desktop software. And every once a month on Tuesdays, we're going to roll out a whole bunch of patches, and here you go. Make sure you turn on security updates, yadda, yadda, yadda. That doesn't fly in the cloud. It's like, “Oh, yeah, here's this month's list of security problems on your cloud provider.” That's one of those things that, like, the record-scratch, freeze-frame moment of wait, what are we doing here, exactly?Yoav: So, I would say that they also have a very long history of making those turnarounds. Bill Gates famously did his speech where security comes first, and they have done a very, very long journey and turn around the company from doing things a lot quicker and a lot safer. It doesn't mean they're perfect; everyone will have bugs, and Azure will have more people finding bugs into it in the near future, but security is a journey, and they've not started from zero. They're doing a lot of work. I would say it's going to take time.Corey: The last topic I want to explore a little bit is—and again, please don't take this as anyway being insulting or disparaging to your company, but I am actively annoyed that you exist. By which I mean that if I go into my AWS account, and I want to configure it to be secure. Great. It's not a matter of turning on the security service, it's turning on the dozen or so security services that then round up to something like GuardDuty that then, in turn, rounds up to something like Security Hub. And you look at not only the sheer number of these services and the level of complexity inherent to them, but then the bill comes in and you do some quick math and realize that getting breached would have been less expensive than what you're spending on all of these things.And somehow—the fact that it's complex, I understand; computers are like that. The fact that there is—[audio break 00:27:03] a great messaging story that's cohesive around this, I come to accept that because it's AWS; talking is not their strong suit. Basically declining to comment is. But the thing that galls me is that they are selling these services and not inexpensively either, so it almost feels, on some level like, shouldn't this on some of the built into the offerings that you folks are giving us?And don't get me wrong, I'm glad that you exist because bringing order to a lot of that chaos is incredibly important. But I can't shake the feeling that this should be a foundational part of any cloud offering. I'm guessing you might have a slightly different opinion than mine. I don't think you show up at the office every morning, “I hate that we exist.”Yoav: No. And I'll add a bit of context and nuance. So, for every other company than cloud providers, we expect them to be very good at most things, but not exceptional at everything. I'll give the Redshift example. Redshift is a pretty good offering, but Snowflake is a much better offering for a much wider range of—Corey: And there's a reason we're about to become Snowflake customers ourselves.Yoav: So, yeah. And there are a few other examples of that. A security company, a company that is focused solely on your security will be much better suited to help you, in a lot of cases more than the platform. And we work actively with AWS, Azure, and GCP requesting new features, helping us find places where we can shed more light and be more proactive. And we help to advance the conversation and make it a lot more actionable and improve from year to year. It's one of those collaborations. I think the cloud providers can do anything, but they can't do everything. And they do a very good job at security; it doesn't mean they're perfect.Corey: As you folks are doing an excellent job of demonstrating. Again, I'm glad you folks exist; I'm very glad that you are publishing the research that you are. It's doing a lot to bring a lot I guess a lot of the undue credit that I was giving AWS for years of, “No, no, it's not that they don't have vulnerabilities like everyone else does. It just that they don't ever talk about them.” And they're operationalizing of security response is phenomenal to watch.It's one of those things where I think you've succeeded and what you said earlier that you were looking to achieve, which is elevating the state of cloud security for everyone, not just Orca customers.Yoav: Thank you.Corey: Thank you. I really appreciate your taking the time out of your day to speak with me. If people want to learn more, where's the best place they can go to do that?Yoav: So, we have our website at orca.security. And you can reach me out on Twitter. My handle is at @yoavalon, which is @-Y-O-A-V-A-L-O-N.Corey: And we will of course put links to that in the [show notes 00:29:44]. Thanks so much for your time. I appreciate it.Yoav: Thank you, Corey.Corey: Yoav Alon, Chief Technology Officer at Orca Security. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, or of course on YouTube, smash the like and subscribe buttons because that's what they do on that platform. Whereas if you've hated this podcast, please do the exact same thing, five-star review, smash the like and subscribe buttons on YouTube, but also leave an angry comment that includes a link that is both suspicious and frightening, and when we click on it, suddenly our phones will all begin mining cryptocurrency.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

A Conversation With Andy Ellis | Securing Bridges With Alyssa Miller | Episode 7

Play Episode Listen Later Apr 25, 2022 45:33

In this episode, Alyssa talks to Andy Ellis about presenting security to the business in a way that encourages them to participate.________________________________It is a podcast, yes, but you can join us as we record each episode live on Twitter, LinkedIn, Facebook, and Youtube.Live, Every Wednesday at 1pm PDT | 4pm EDT (USA) | The Recorded Podcast version is published a few days later.Our ability to improve the security posture of our organizations depends heavily on connecting the security function with the various aspects of the business. Join our host, Alyssa Miller, as she and her guests examine key ways to build and secure the bridges between security, product development, the executive suite, and beyond.Listen in as Alyssa sits down with senior and executive security leaders from various industries to share stories of successes and failures we experience working across business teams. Explore practical strategies for building sponsorship and gaining buy-in for security initiatives.It's time to build and secure the bridge to the business.________________________________GuestAndy EllisAdvisory CISO at Orca Security [@orcasec]On Twitter | https://twitter.com/csoandyOn LinkedIn | https://www.linkedin.com/in/csoandy/________________________________HostAlyssa MillerOn ITSPmagazine

A Conversation With Andy Ellis | Securing Bridges With Alyssa Miller | Episode 7

Securing Bridges

Play Episode Listen Later Apr 25, 2022 45:33

156: The Cloud Pod Takes Back Everything It Said About Windows vs Linux Security

The Cloud Pod

Play Episode Listen Later Mar 17, 2022 52:14

On The Cloud Pod this week, the team reminisces about dealing with awful database technologies, which Ryan luckily managed to avoid. Plus all things cybersecurity as Linux gets hit with a huge security emergency, Google acquires Mandiant for $5.4 billion, and Orca Security catches a major Azure cross-tenant vulnerability. A big thanks to this week's sponsor, Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. This week's highlights

amazon google spanish public security windows oracle generally wallet aws linux azure crowdstrike google cloud lambda okta rds mysql gcp mandiant cncf purview registration open jdbc orca security linux security cybergrx las vegas october microsoft pr cloud spanner amazon fsx cloud pod foghorn consulting

The Surprise Mandoogle

Play Episode Listen Later Mar 17, 2022 5:55

Links: Links Referenced: Couchbase Capella: https://couchbase.com/screaminginthecloud couchbase.com/screaminginthecloud: https://couchbase.com/screaminginthecloud blog post: https://awsteele.com/blog/2022/02/03/aws-vpc-data-exfiltration-using-codebuild.html AutoWarp: https://orca.security/resources/blog/autowarp-microsoft-azure-automation-service-vulnerability/ “Google Announces Intent to Acquire Mandiant”: https://www.googlecloudpresscorner.com/2022-03-08-mgc password table: https://www.hivesystems.io/blog/are-your-passwords-in-the-green New Relic: http://newrelic.com newrelic.com/morningbrief: http://newrelic.com/morningbrief newrelic.com/morningbrief: http://newrelic.com/morningbrief DirtyPipe: https://www.theregister.com/2022/03/08/in_brief_security/ “Manage AWS resources in your Slack channels with AWS Chatbot”: https://aws.amazon.com/blogs/mt/manage-aws-resources-in-your-slack-channels-with-aws-chatbot/ “How to set up federated single-sign-on to AWS using Google Workspace”: https://aws.amazon.com/blogs/security/how-to-set-up-federated-single-sign-on-to-aws-using-google-workspace/ Cloudsaga: https://github.com/awslabs/aws-cloudsaga lastweekinaws.com: https://lastweekinaws.com TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it's nobody in particular's job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured, and fully managed with built-in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: Make your data sing.Hello and welcome to Last Week in AWS Security. A lot has happened; let's tear into it.So, there was a “Sort of yes, sort of no” security issue with CodeBuild that I've talked about previously. The blog post I referenced has, in fact, been updated. AWS has stated that, “We have updated the CodeBuild service to block all outbound network access for newly created CodeBuild projects which contain a customer-defined VPC configuration,” which indeed closes the gap. I love happy endings.On the other side, oof. Orca Security found a particularly nasty Azure breach called AutoWarp. You effectively could get credentials for other tenants by simply asking a high port on localhost for them via curl or netcat. This is bad enough; I'm dreading the AWS equivalent breach in another four months of them stonewalling a security researcher if the previous round of their nonsense silence about security patterns is any indicator.“Google Announces Intent to Acquire Mandiant”. This is a big deal. Mandiant has been a notable center of excellent cybersecurity talent for a long time. Congratulations or condolences to any Mandoogles in the audience. Please let me know how the transition goes for you.Hive Systems has updated its password table for 2022, which is just a graphic that shows how long passwords of various levels of length and complexity would take to break on modern systems. The takeaway here is to use long passwords and use a password manager.Corey: You know the drill: You're just barely falling asleep and you're jolted awake by an emergency page. That's right, it's your night on call, and this is the bad kind of Call of Duty. The good news is, is that you've got New Relic, so you can quickly run down the incident checklist and find the problem. You have an errors inbox that tells you that Lambdas are good, RUM is good, but something's up in APM. So, you click the error and find the deployment marker where it all began. Dig deeper, there's another set of errors. What is it? Of course, it's Kubernetes, starting after an update. You ask that team to roll back and bam, problem solved. That's the value of combining 16 different monitoring products into a single platform: You can pinpoint issues down to the line of code quickly. That's why the Dev and Ops teams at DoorDash, GitHub, Epic Games, and more than 14,000 other companies use New Relic. The next late-night call is just waiting to happen, so get New Relic before it starts. And you can get access to the whole New Relic platform at 100 gigabytes of data free, forever, with no credit card. Visit newrelic.com/morningbrief that's newrelic.com/morningbrief.And of course, another week, another terrifying security concern. This one is called DirtyPipe. It's in the Linux kernel, and the name is evocative of something you'd expect to see demoed onstage at re:Invent.Now, what did AWS have to say? Two things. The first is “Manage AWS resources in your Slack channels with AWS Chatbot”. A helpful reminder that it's important to restrict access to your AWS production environment down to just the folks at your company who need access to it. Oh, and to whomever can access your Slack workspace who works over at Slack, apparently. We don't talk about that one very much, now do we?And the second was, “How to set up federated single-sign-on to AWS using Google Workspace”. This is super-aligned with what I want to do, but something about the way that it's described makes it sounds mind-numbingly complicated. This isn't a problem that's specific to this post or even to AWS; it's industry-wide when it comes to SSO. I'm starting to think that maybe I'm the problem here.And lastly, AWS has open-sourced a tool called Cloudsaga, designed to simulate security events in AWS. This may be better known as, “Testing out your security software,” and with sufficiently poor communication, “Giving your CISO a heart attack.”And that's what happened last week in AWS security. If you've enjoyed it, please tell your friends about this place. I'll talk to you next week.Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

The Hari Seldon of Third Party Tooling with Aidan Steele

Play Episode Listen Later Mar 16, 2022 33:39

About AidanAidan is an AWS enthusiast, due in no small part to sharing initials with the cloud. He's been writing software for over 20 years and getting paid to do it for the last 10. He's still not sure what he wants to be when he grows up.Links: Stedi: https://www.stedi.com/ GitHub: https://github.com/aidansteele Blog posts: https://awsteele.com/ Ipv6-ghost-ship: https://github.com/aidansteele/ipv6-ghost-ship Twitter: https://twitter.com/__steele TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by someone who is honestly, feels like they're after my own heart. Aidan Steele by day is a serverless engineer at Stedi, but by night, he is an absolute treasure and a delight because not only does he write awesome third-party tooling and blog posts and whatnot around the AWS ecosystem, but he turns them into the most glorious, intricate, and technical shit posts that I think I've ever seen. Aidan, thank you for joining me.Aidan: Hi, Corey, thanks for having me. It's an honor to be here. Hopefully, we get to talk some AWS, and maybe also talk some nonsense as well.Corey: I would argue that in many ways, those things are one in the same. And one of the things I always appreciated about how you approach things is, you definitely seem to share that particular ethos with me. And there's been a lot of interesting content coming out from you in recent days. The thing that really wound up showing up on my radar in a big way was back at the start of January—2022, for those listening to this in the glorious future—about using IPv6 to use multi-factor auth, which it is so… I don't even have the adjectives to throw at this because, first it is ridiculous, two, it is effective, and three, it is just who thinks like that? What is this and what did you—what monstrosity have you built?Aidan: So, what did I end up calling it? I think it was ipv6-ghost-ship. And I think I called it that because I'd recently watched, oh, what was that series that was recently on Apple TV? Uh, the Isaac Asimov—Corey: If it's not Paw Patrol, I have no idea what it is because I have a four-year-old who is very insistent about these things. It is not so much a TV show as it is a way of life. My life is terrible. Please put me out of my misery.Aidan: Well, at least it's not Bluey. That's the one I usually hear about. That's Australia's greatest export. But it was one of the plot devices was a ship that would teleport around the place, and you could never predict where it was next. And so no one could access it. And I thought, “Oh, what about if I use the IPv6 address space?”Corey: Oh, Foundation?Aidan: That's the one. Foundation. That's how the name came about. The idea, honestly, it was because I saw—when was it?—sometime last year, AWS added support for those IP address prefixes. IPv4 prefixes were small; very useful and important, but IPv6 with more than 2 trillion IP addresses, per instance, I thought there's got to be fun to be had there.Corey: 281 trillion, I believe is the—Aidan: 281 trillion.Corey: Yeah. It is sarcastically large space. And that also has effectively, I would say in InfoSec sense, killed port scanning, the idea I'm going to scan the IP range and see what's there, just because that takes such a tremendous amount of time. Now here, in reality, you also wind up with people using compromised resources, and yeah, it turns out, I can absolutely scan trillions upon trillions of IP addresses as long as I'm using your AWS account and associated credit card in which to do it. But here in the real world, it is not an easily discoverable problem space.Aidan: Yeah. I made it as a novelty, really. I was looking for a reason to learn more about IPv6 and subnetting because it's the term I'd heard, a thing I didn't really understand, and the way I learn things is by trying to build them, realizing I have no idea what I'm doing, googling the error messages, reluctantly looking at the documentation, and then repeating until I've built something. And yeah, and then I built it, published it, and seemed to be pretty popular. It struck a chord. People retweeted it. It tickled your fancy. I think it spoke something in all of us who are trying not to take our jobs too seriously, you know, know we can have a little fun with this ludicrous tech that we get to play with.Corey: The idea being, you take the multi-factor auth code that your thing generates, and that is the last series of octets for the IP address you wind up going towards and that is such a large problem space that you're not going to find it in time, so whatever it is automatically connect to that particular IP address because that's the only one that's going to be listening for a 30 to 60-second span for the connection to be established. It is a great idea because SSH doesn't support this stuff natively. There's no good two-factor auth approach for this. And I love it. I'd be scared to death to run this in production for something that actually matters.And we also start caring a lot more about how accurate are the clocks on those instances, all of a sudden. But, oh, I just love the concept so much because it hits on the ethos of—I think—what so much of the cloud does were these really are fundamental building blocks that we can use to build incredible, awe-inspiring things that are globe-spanning, and also ridiculousness. And there's so much value of being able to do the same thing, sometimes at the same time.Aidan: Yeah, it's interesting, you mentioned, like, never using in prod, and I guess when I was building it, I thought, you know, that would be apparent. Like, “Yes, this is very neat, but surely no one's going to use it.” And I did see someone raised an issue on the GitHub project which was talking about clock skew. And I mentioned—Corey: Here at the bank where I'm running this in production, we're—Aidan: [laugh].Corey: —having some trouble with the clock. Yeah, it's—Aidan: You know, I mentioned that the underlying 2FA library did account for clock scheme 30 seconds either way, but it made me realize, I might need to put a disclaimer on the project. While the code is probably reasonably sound, I personally wouldn't run it in production, and it was more meant to be a piece of performance art or something to tickle one's fancy and to move on, not to roll it out. But I don't know, different strokes for different folks.Corey: I have gotten a lot better about calling out my ridiculous shitpost things when I do them. And the thing that really drove that home for me was talking about using DNS TXT records to store information about what server a virtual machine lives on—or container or whatnot—thus using Route 53 is a database. And that was a great gag, and then someone did a Reddit post of “This seems like a really good idea, so I'm going to start doing it, and I'm having these questions.”And at that point is like, “Okay, I've got a break character at that point.” And is, yeah, “Hi. That's my joke. Don't do it because X, Y, and Z are your failure modes, there are better tools for it. So yeah, there are ways you can do this with DNS, but it's not generally a great idea, and there are some risk factors to it. And okay, A, B, and C are the things you don't want to do, so let's instead do it in a halfway intelligent way because it's only funny if everyone's laughing. Otherwise, we fall into this trap of people take you seriously and they feel bad as a result when it doesn't work in production. So, calling it out as this is a joke tends to put a lot of that aside. It also keeps people from feeling left out.Aidan: Yeah. I realized that because the next novelty project I did a few days later—not sure if you caught it—it was a Rick Roll over ICMPv6 packets, where if you had run ping six to a certain IP range, it would return the lyrics to music's greatest treasure. So, I think that was hopefully a bit more self-evident that this should never be taken seriously. Who knows, I'm sure someone will find a use for it in prod.Corey: And I was looking through this, this is great. I love some of the stuff that you're doing because it's just fantastic. And I started digging a bit more to things you had done. And at that point, it was whoa, whoa, whoa, wait a minute. Back in 2020, you found an example of an issue with AWS's security model where CloudTrail would just start—if asked nicely—spewing other people's credential sets and CloudTrail events and whatnot into your account.And, A, that's kind of a problem. B, it was something that didn't make that big of a splash when it came out—I don't even think I linked to it at the time—and, C, it was examples of after the recent revelations around CloudFormation and Glue that the fine folks at Orca Security found out. That wasn't a one-off because you'd done this a year beforehand. We have now an established track record of cross-account data sharing and, potentially, exploits, and I'm looking at this and I got to level with you I felt incredibly naive because I had assumed that since we hadn't heard of this stuff in any real big sense that it simply didn't happen.So, when we heard about Azure; obviously, it's because Azure is complete clown shoes and the excellent people that AWS would never make these sorts of mistakes. Except we now have evidence that they absolutely did and didn't talk about it publicly. And I've got a level with you. I feel more than a little bit foolish, betrayed, naive for all this. What's your take on it?Aidan: Yeah, so just to clarify, it wasn't actually in your account. It was the new AWS custom resource execution model was you would upload a Lambda function that would run in an Amazon-managed account. And so that immediately set off my spidey sense because executing code in someone else's account seems fraught with peril. And so—Corey: Yeah, you can do all kinds of horrifying things there, like, use it to run containers.Aidan: Yeah. [laugh]. Thankfully, I didn't do anything that egregious. I stayed inside the Lambda function, but I look—I poked around at what credentials have had, and it would use CloudWatch to reinvoke itself and CloudWatch kept recording CloudTrail. And I won't go into all the details, but it ended up being that you could see credentials being recorded in CloudTrail in that account, and I could, sort of, funnel them out of there.When I found this, I was a little scared, and I don't think I'd reported an issue to AWS before, so I didn't want to go too far and do anything that could be considered malicious. So, I didn't actively seek out other people's credentials.Corey: Yeah, as a general rule, it's best once you discover things like that to do the right thing and report it, not proceed to, you know, inadvertently commit felonies.Aidan: Yeah. Especially because it was my first time. I felt better safe than sorry. So, I didn't see other credentials, but I had no reason to believe that, I wouldn't see it if I kept looking. I reported it to Amazon. Their security team was incredibly professional, made me feel very comfortable reporting it, and let me know when, you know, they'd remediated it, which was a matter of days later.But afterwards, it left me feeling a little surprised because I was able to publish about it, and a few people responded, you know, the sorts of people who pay close attention to the industry, but Amazon didn't publish anything as far as I was aware. And it changed the way I felt about AWS security, because like you, I sort of felt that AWS, more or less had a pretty perfect track record. They would have advisories about possible [Zen 00:12:04] exploits, and so on. But they'd never published anything about potential for compromise. And it makes me wonder how many of the things might have been reported in the past where either the third-party researcher either didn't end up publishing, or they published and it just disappeared into the blogosphere, and I hadn't seen it.Corey: They have a big earn trust principle over there, and I think that they always focus on the trust portion of it, but I think what got overlooked is the earn. When people are giving you trust that you haven't earned, on some level, the right thing to do is to call it out and be transparent around these things. Yes, I know, Wall Street's going to be annoyed and headlines, et cetera, et cetera, but I had always had the impression that had there been a cross-account vulnerability or a breach of some sort, they would communicate this and they would have their executives go on a speaking tour about it to explain how defense-in-depth mitigated some of it, and/or lessons learned, and/or what else we can learn. But it turns out that wasn't was happening at all. And I feel like they have been given trust that was unearned and now I am not happy with it.I suddenly have a lot more of a, I guess, skeptical position toward them as a result, and I have very little tolerance left for what has previously been a staple of the AWS security discussions, which is an executive getting on stage for a while and droning on about the shared responsibility model with the very strong implication that “Oh, yeah, we're fine. It's all on your side of the fence that things are going to break.” Yeah, turns out, that's not so true. Just you know, about the things on your side of the fence in a way that you don't about the things that are on theirs.Aidan: Yeah, it's an interesting one. Like, I think about it and I think, “Well, they never made an explicit promise that they would publish these things,” so, on one hand, I say to myself, “Oh, maybe that's on me for making that assumption.” But, I don't know, I feel like the way we felt was justified. Maybe naive in hindsight, but then, you know, I guess… I'm still not sure how to feel because of, like, I think about recent issues and how a couple of AWS Distinguished Engineers jumped on Twitter, and to their credit were extremely proactive in engaging with the community.But is that enough? It might be enough for say, to set my mind at ease or your mind at ease because we are, [laugh] to put it mildly, highly engaged, perhaps a little too engaged in the AWS space, but Twitter's very ephemeral. Very few of AWS's customers—Corey: Yeah, I can't link to tweets by distinguished engineers to present to an executive leadership team as an official statement from Amazon. I just can't.Aidan: Yeah. Yeah.Corey: And so the lesson we can take from this is okay, so “Well, we never actually said this.” “So, let me get this straight. You're content to basically let people assume whatever they want until they ask you an explicit question around these things. Really? Is that the lesson you want me to take from this? Because I have a whole bunch of very explicit questions that I will be asking you going forward, if that is in fact, your position. And you are not going to like the fact that I'm asking these questions.”Even if the answer is a hard no, people who did not have this context are going to wonder why are people asking those questions? It's a massive footgun here for them if that is the position that they intend to have. I want to be clear as well; this is also a messaging problem. It is not in any way, a condemnation of their excellent folks working on the security implementation themselves. This stuff is hard and those people are all-stars. I want to be very clear on this. It is purely around the messaging and positioning of the security posture.Aidan: Yeah, yeah. That's a good clarification because like you, my understanding that the service teams are doing a really stellar, above-average job, industry-wide, and the AWS Security Response Teams, I have absolute faith in them. It is a matter of messaging. And I guess what particularly brings it to front-of-mind is, it was earlier this month, or maybe it was last month, I received an email from a company called Sourcegraph. They do code search.I'm not even a customer of theirs yet, you know? I'm on a free trial, and I got an email that—I'm paraphrasing here—was something to the effect of, we discovered that it was possible for your code to appear in other customers' code search results. It was discovered by one of our own engineers. We found that the circumstances hadn't cropped up, but we wanted to tell you that it was possible. It didn't happen, and we're working on making sure it won't happen again.And I think about how radically different that is where they didn't have a third-party researcher forcing their hand; they could have very easily swept under the rug, but they were so proactive that, honestly, that's probably what's going to tipped me over to the edge into me becoming a customer. I mean, other than them having a great product. But yeah, it's a big contrast. It's how I like to see other companies work, especially Amazon.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: The two companies that I can think of that have had security problems have been CircleCI and Travis CI. Circle had an incredibly transparent early-on blog post, they engaged with customers on the forums, and they did super well. Travis basically denied, stonewalled for ages, and now the only people who use Travis are there because they haven't found a good way to get off of it yet. It is effectively DOA. And I don't think those two things are unrelated.Aidan: Yeah. No, that's a great point. Because you know, I've been in this industry long enough. You have to know that humans write code and humans make mistakes—I know I've made more than my fair share—and I'm not going to write off the company for making a mistake. It's entirely in their response. And yeah, you're right. That's why Circle is still a trustworthy business that should earn people's business and why Travis—why I recommend everyone move away from.Corey: Yeah, I like Orca Security as a company and as a product, but at the moment, I am not their customer. I am AWS's customer. So, why the hell am I hearing it from Orca and not AWS when this happens?Aidan: Yeah, yeah. It's… not great. On one hand, I'm glad I'm not in charge of finding a solution to this because I don't have the skills or the expertise to manage that communication. Because like I think you said in the past, there's a lot of different audiences that they have to communicate with. They have to communicate with the stock market, they have to communicate with execs, they have to communicate with developers, and each of those audiences demands a different level of detail, a different focus. And it's tricky. And how do you manage that? But, I don't know, I feel like you have an obligation to when people place that level of trust in you.Corey: It's just a matter of doing right by your customers, on some level.Aidan: Yeah.Corey: How long have you been working on an AWS-side environments? Clearly, this is not like, “Well, it's year two,” because if so I'm going to feel remarkably behind.Aidan: [laugh]. So, I've been writing code in some capacity or another for 20 years. It took about five years to get anyone to pay me to do so. But yeah, I guess the start of my professional career—and by ‘professional,' I want to use it in strictest term, means getting paid for money; not that I [laugh] am necessarily a professional—coincided with the launch of AWS. So, I don't hadn't experienced with the before times of data centers, never had to think about direct connect, but it means I have been using AWS since sometime in 2008.I was just looking at my bill earlier, I saw that my first bill was for $70. It was—I was using a C1xLarge, which was 80 cents an hour, and it had eight-core CPUs. And to put that in context at the time—Corey: Eight vCPUs, technically I believe.Aidan: An it basically is—Corey: —or were they using [eCPU 00:20:31] model back then?Aidan: Yeah, no, that was vCPUs. But to me, that was extraordinary. You know, I was somewhere just after high school. It was—the Netflix Prize was around. If you're not sure what that was, it was Netflix had this open competition where they said anyone who could improve upon their movie recommendation algorithm could win a million dollars.And obviously being a teenager, I had a massive ego and [laugh] no self-doubt, so I thought I could win this, but I just don't have enough CPUs or RAM on my laptop. And so when EC2 launched, and I could pay 80 cents an hour, rather than signing up for a 12-month contract with a colocation company, it was just a dream come true. I was able to run my terrible algorithms, but I could run them eight times faster. Unfortunately and obviously, I didn't win because it turns out, I'm not a world-class statistician. But—Corey: Common mistake. I make that mistake myself all the time.Aidan: [laugh]. Yeah. I mean, you know, I think I was probably 19 at the time, so I had—my ego did make me think I was one, but it turned out not to be so. But I think that was what really blew my mind was that me, a nobody, could create an account with Amazon and get access to these incredibly powerful machines for less than the dollar. And so I was hooked.Since then, I've worked at companies that are AWS customers since then. I've worked at places that have zero EC2 service, worked at places that have had thousands, and places in between. And it's got to a point, actually, where, I guess, my career is so entwined with AWS that one, my initials are actually AWS, but also—and this might sound ridiculous, and it's probably just a sign of my privilege—that I wouldn't consider working somewhere that used another cloud. Not—Corey: No, I think that's absolutely the right approach.Aidan: Yeah.Corey: I had a Twitter thread on this somewhat recently, and I'm going to turn it into a blog post because I got some pushback. If I were looking at doing something and I would come into the industry right now, my first choice would be Google Cloud because its developer experience is excellent. But I'm not coming to this without any experience. I have spent a decade or so learning not just how it was works, but also how it breaks, understanding the failure mode and what that's going to look like and what it's good at and what it's not. That's the valuable stuff for running things in a serious way.Aidan: Yeah. It's an interesting one. And I mean, for better or worse, AWS is big. I'm sure you will know much better than I do the exact numbers, but if a junior developer came to me and said, “Which cloud should I learn, or should I learn all of them?” I mean, you're right, Google Cloud does have a better developer experience, especially for new developers, but when I think about the sheer number of jobs that are available for developers, I feel like I would be doing them a disservice by not suggesting AWS, at least in Australia. It seems they've got such a huge footprint that you'll always be able to find a job working as an AWS-familiar engineer. It seems like that would be less the case with Google Cloud or Azure.Corey: Again, I am not sitting here, suggesting that anyone should, “Oh, clouds are insecure. We're going to run our own stuff in our own data centers.” That is ridiculous in this era. They are still going to do a better job of security than any of us will individually, let's be clear here. And it empowers and unlocks an awful lot of stuff.But with their privileged position as these hyperscale providers that are the default choice for building things, I think comes with a significant level of responsibility that I am displeased to discover that they've been abdicating. And I don't love that.Aidan: Yeah, it's an interesting one, right, because, like you're saying, they have access and the expertise that people doing it themselves will never match. So, you know, I'm never going to hesitate to recommend people use AWS on account security because your company's security posture will almost always be better for using AWS and following their guidelines, and so on. But yeah, like you say, with great power comes significant responsibility to earn trust and retain that trust by admitting and publicizing when mistakes are made.Corey: One last topic I want to get into with you is one that you and I have talked about very briefly elsewhere, that I feel like you and I are both relatively up-to-date on AWS intricacies. I think that we are both better than the average bear working with the platform. But I know that I feel this way, and I suspect you do too that VPCs have gotten confusing as hell. Is that just me? Am I a secret moron that no one bothered to ever tell me this, and I should update my own self-awareness?Aidan: [laugh]. Yeah, it's… I mean, that's been the story of my career with AWS. When I started, VPCs didn't exist. It was EC2 Classic—well, I guess at the time, it was just EC2—and it was simple. You launched an instance and you had an IP address.And then along came VPCs, and I think at the time, I thought something to the effect of “This seems like needless complexity. I'm not going to bother learning this. It will never be relevant.” In the end that wasn't true. I worked in much large deployments when VPCs made fantastic sense made a lot of things possible, but I still didn't go into the weeds.Since then, AWS has announced that EC2 Classic will be retired; an end of an era. I'm not personally still running anything in EC2 Classic, and I think they've done an incredible job of maintain support for this long, but VPC complexity has certainly been growing year-on-year since then. I recently was using the AWS console—like we all do and no one ever admits to—to edit a VPC subnet route table. And I clicked the drop-down box for a target, and I was overwhelmed by the number of options. There were NAT gateways, internet gateways, carrier gateways, I think there was a thing called a wavelength gateway, ENI, and… I [laugh] I think I was surprised because I just scroll through the list, and I thought, “Wow, that is a lot of different options. Why is that?”Especially because it's not so relevant to me. But I realized a big thing of what AWS has been doing lately is trying to make themselves available to people who haven't used the cloud yet. And they have these complicated networking needs, and it seems like they're trying to—reasonably successfully—make anything possible. But with that comes, you know, additional complexity.Corey: I appreciate that the capacity is there, but there has to be an abstraction model for getting rid of some of this complexity because otherwise, the failure mode is you wind up with this amazingly capable thing that can build marvels, but you also need to basically have a PhD in some of these things to wind up tying it all together. And if you bring someone else in to do it, then you have no idea how to run the thing. You're effectively a golden retriever trying to fly a space shuttle.Aidan: Yeah. It's interesting, like, clearly, they must be acutely aware of this because they have default VPCs, and for many use cases, that's all people should need. But as soon as you want, say a private subnet, then you need to either modify that default VPC or create a new one, and it's sort of going from 0 to 100 complexity extremely quickly because, you know, you need to create route tables to everyone's favorite net gateways, and it feels like the on-ramp needs to be not so steep. Not sure what the solution is, I hope they find one.Corey: As do I. I really want to thank you for taking the time to speak with me about so many of these things. If people want to learn more about what you're up to, where's the best place to find you?Aidan: Twitter's the best place. On Twitter, my username is @__Steele, which is S-T-E-E-L-E. From there, that's where I'll either—I'll at least speculate on the latest releases or link to some of the silly things I put on GitHub. Sometimes they're not so silly things. But yeah, that's where I can be found. And I'd love to chat to anyone about AWS. It's something I can geek out about all day, every day.Corey: And we will certainly include links to that in the [show notes 00:29:50]. Thank you so much for taking the time to speak with me today. I really appreciate it.Aidan: Well, thank you so much for having me. It's been an absolute delight.Corey: Aidan Steele, serverless engineer at Stedi, and shit poster extraordinaire. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an immediate request to correct the record about what I'm not fully understanding about AWS's piss-weak security communications.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

tv amazon netflix australia service phd foundation wall street circle reddit cloud apple tv ip route ram zen screaming steele aws github devops glue azure third party google cloud bluey dns orca s3 sql doa kubernetes paw patrol lambda capella infosec 2fa cpus tooling ssh eni ipv6 rick roll aws lambda ipv4 ec2 circleci vpc corey quinn sysdig minio travis ci cloudformation hari seldon cloudwatch vpcs duckbill group orca security netflix prize chief cloud economist last week in aws vcpus humblepod

From A to Z in Alphabet's Soup with Seth Vargo

ceo challenges asset cloud security shua orca security

Play Episode Listen Later Mar 10, 2022 42:08

About SethSeth Vargo is an engineer at Google. Previously he worked at HashiCorp, Chef Software, CustomInk, and some Pittsburgh-based startups. He is the author of Learning Chef and is passionate about reducing inequality in technology. When he is not writing, working on open source, teaching, or speaking at conferences, Seth advises non-profits.Links:Twitter: https://twitter.com/sethvargo TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I have a return guest today, though it barely feels like it qualifies because Seth Vargo was guest number three on this podcast. I've had a couple of folks on since then, and for better or worse, I'm no longer quite as scared of the microphone as I was back in those early days. Seth, thank you for joining me.Seth: Yeah, thank you so much for having me back, Corey. Really excited to figure out whatever we're talking about today.Corey: Well, let's start there because last time we spoke, you were if memory serves a developer advocate at Google Cloud.Seth: Correct.Corey: And you've changed jobs, but not companies—but kind of companies because, welcome to large environments—but over the past few years, you have remained at Google. You are no longer at Google Cloud and you're no longer a developer advocate. In fact, your title is simply ‘Engineer at Google.' And what you've been focusing on, to my understanding, is helping Alphabet companies, namely—you know, the Alphabet, always in parentheses in journalistic styles, Google's parent company because no one thinks of it in terms of Alphabet—is—you're effectively helping companies within the conglomerate umbrella securely and privately consume public cloud.Seth: Yes, that is correct. So, I used to work in what we call the Cloud PA—PA stands for product area. Other product areas are like Chrome and Android—and I moved to the Core PA where I'm helping lead and run an initiative that, like you said, is to help Alphabet companies to, you know, securely and privately use public cloud services.Corey: So, I am going to go out on a limb because my position on multi-cloud has always been pick a cloud—I don't particularly care which one—but pick one and focus on that. I'm going to go out on a limb and presume that given that you are not at Google Cloud anymore, but you are at Google, you probably have a slight preference as far as which public cloud these various companies within the umbrella should be consuming.Seth: Yeah. I mean, obviously, I think most viewers will think the answer is GCP. And if you said GCP, you would be, like, 95% correct.Corey: Well, you'd also be slightly less than that correct, because they're doing a whole rebrand and calling it Google Cloud in public, as opposed to GCP. You really don't work for the same org anymore. You're not up-to-date on the very latest messaging talking points.Seth: I missed—ugh, there's so many TLAs that you lose all your TLAs over time.Corey: Oh, yes.Seth: So, Google Cloud would be, like, 95% correct. But what you have to really understand is, Google has its own, you know, cloud—we didn't call it a cloud at the time, you might call it on-prem or legacy infrastructure, if you will—primarily built on a scheduling system called Borg, which is like Kubernetes version zero. And a lot of the Alphabet companies have workloads that run onboard. So, we're actually talking about hybrid cloud here, which, you know, you may not think of Google is like a hybrid cloud customer, but a workload that runs on our production infrastructure called Borg that needs to interact with a workload that runs on Google Cloud, that is hybrid cloud, it's no different than a customer who has their own data center that needs peering to a public cloud provider, you know, whether that's Google Cloud, or AWS, or Azure.I think the other thing is if you look at, like, the regulatory space, particularly a lot of the Alphabet companies operate in, say, like healthcare, or finance, or FinTech, where certain countries and certain jurisdictions have regulations around, like, you must be multi-cloud. You know, some people might say that means you have to run, you know, the same instance of the same app across clouds, or some people say your data can be here, but your workloads can be over there. That's to be interpreted, but you know, I would say 95% of GCP, but there is a—or sorry, 95% is Google Cloud—Corey: There we go.Seth: But there is a small percentage that is definitely going to be other cloud providers and hybrid cloud as well.Corey: My position on multi-cloud has often—people like to throw it in my face of, “See you gave this general guidance, and therefore whenever you say something that goes against it, you're a giant phony.” And it's yeah, Twitter doesn't do so well with the nuance. My position of pick a provider and go all-in is intended as general guidance for the common case. There are exceptions to this and any individual company or customer is going to have more context than that general guidance will. So, if you say you need to be in multiple clouds for certain reasons, you're probably correct.If you say you need to be in multiple clouds because your regulator demands it, you are certainly correct. I am not arguing against that in any way. I do want to disclaim my one of my biases here as well, and that is specifically that if I were building a startup today and I were not me—by which I mean having spent ten years in the AWS ecosystem learning, not just how it works, but how it breaks because that's important in production, and you know, also having a bunch of service owners at AWS on speed dial—and I, were approaching this from the naive, I need to pick a cloud, which one would I go with, my bias is for Google Cloud. And the reason behind that is the developer experience is spectacular as the primary but not only perspective on that. So, I am curious to know that as you're helping what are effectively internal customers move to Google Cloud, is their interaction with Google Cloud as a platform the same as it would be if I as a random outside customer, were using Google Cloud? Is there a bunch of internal backchannels? “Oh, you get the good kind of internal Google Cloud that most of us don't get access to?” Or something else?Seth: Yeah, so that's a great question. So first, you know, thank you for the kind words on the developer experience—Corey: They were honest words, to be clear. Let me be very direct with you, if I thought your developer experience was trash, I might not say it outright in their effort not to be, you know, actively antagonistic to someone I'm having on the show right now, but I would not say it if I didn't believe it.Seth: Yeah. And I totally—I know you, I've known you for many years. I totally believe you. But I do thank you for saying that because that was the team that I was on before this was largely responsible for that across the platform. But back to your original question around, like, what does the support experience look like? So, it's a little bit of both.So, Alphabet companies, they get a technical account manager, very similar to how, you know, reasonable-sized spend customer would get a technical account manager. That account manager has access to the Cloud support channels. So, all that looks the same. I think we're things look a little bit different is because myself and some of our other leads came from Cloud, you know, I generally don't like this phrase, but we know people. So, we tend not to go directly to Cloud when we can, right?We want Alphabet companies to really behave and act as if they were an external entity, but we're able to help the technical account manager navigate the support process a little bit better by saying like, “You need to ask for this person,” right? You need to say these words to get in front of the right person to get this ticket assigned to the right person. So, the process is still the same, but we're able to leverage our pre-existing knowledge with Cloud. The same way, if you had a [unintelligible 00:07:45] or an ex-Googler who worked for your company, would be able to kind of help move that support process along a little bit faster.Corey: I am quite sincere when I say that this is a problem that goes far beyond simply Google. A disturbing portion of my job as a cloud economist helping my clients consists of nothing other than introducing Amazonians to one another. And these are hard problems at scale. I work at a company with a dozen people in it. And it turns out that yeah, it's pretty easy to navigate who's responsible for what. When you have a hyperscale-size company in the trillion-dollar range, a lot of that breaks down super quickly.Seth: And there's just a lot of churn at all levels of the organization. And, you know, we talked about this when I first joined the show, like, I switched roles, I used to be in Cloud, and now I'm in what we call Core. I still get people who are reaching out to me, at Google and externally, who are saying, “Oh, can you answer this question? Hey, how do I do this?” And I, you know, I've gradually over the past couple of months, you know, convinced people that I don't work on that anymore, and I try to be helpful where I can, but the—Corey: You use the old name and everything. They're eventually going to learn, right?Seth: I know. They'll be like, “What do you call this? GCP? Okay, great. We don't need you anymore.” But it's true, right? Like, there's people leave the organization, people join the organization, there's reorgs, there's strategic changes, people, you know, switch roles within the org, and all of that leads to complexity with, you know, navigating, what is the size of a small nation, in some cases.Corey: Your line in your biography says that you enable Alphabet companies to securely and privately consume public cloud. Now, that would make perfect sense and I would really have no further questions based on what we've already said, except for the words securely and privately, and I want to dive into that, first. Let's work backwards with the second one first. What is ‘privately' mean in this context?Seth: So, privately means, like, privacy-preserving for both the Alphabet company and the users or customers that they have. So, when we look at that from the perspective of the Alphabet company, that means protecting their data from the eyes of the cloud provider. So, that's things like customer-managed encryption keys, you know, bring-your-own-encryption, that's making sure that you have things like, actually, transparency so that if at any point the cloud provider is accessing your data, even for a legitimate purpose, like submitting a support ticket or something—or diagnosing a support ticket, that you have visibility into that. Then the privacy-preserving side on the Alphabet company's customers is about providing that same level of visibility to their customers as well as making sure that any data that they're storing is, you know, private, it's not accessible to certain parties, it's following whether it's like, you know, actual legislation around how long data can be persisted, things like GDPR, or if it's just a general, like, data retention, insider risk management, all of that comes into this idea of, like, building a private system or privacy-preserving system.Corey: Let's be very clear that my position on it is that Google's relationship with privacy has been somewhat challenged, in due to no small part to the sheer scale of how large Google has grown. And let's be clear, I believe firmly that at certain points of scale, yeah, you deserve elevated levels of scrutiny. That is how we want society to function, by and large. And there are times where it feels a little odd on the cloud side. For example, as the time is recording, somewhat recently, there was a bug in some of the copyright detection stuff where Google Drive would start flagging files as having copyright challenges if they contained just the character ‘1' in them.Which, okay, clearly a bug, but it was a bit of a reminder for some folks that wait, but that's right, Google does tend to scan these things. Well, when you have a bunch of end-user customers and in the ways that Google does, that stuff is baked in and it shapes how you wind up seeing things. From Amazon's perspective, historically, they basically sold books and then later underpants. And doing e-commerce transactions was basically the extent of their data work with customers. They weren't really running large-scale, file sharing systems and abilities—in collaboration suites, at least not that really had any of those pesky things called customers.So, that is not built into their approach and their needs in the same way. To be clear, I am sympathetic to the problems, but it's also… it's a challenging problem, especially as you continue to evolve and move things into cloud, you absolutely must be able to trust your cloud provider, or you should not be working on that cloud provider, has been my approach.Seth: Yeah, I mean, there's certainly things that you can do to mitigate. But in general, like, there is some level of trust, forget the data, on the availability side, right? Like when the cloud provider says, “This is our SLA.” And you agree to that SLA, like, yeah, you get money back if they mess it up, but ultimately, you're trusting them to adhere to that SLA, right? And you get recompense if they fail to do so, but that's still, like, trust—trust is far more than just on the privacy side, right? It's on… the promise on the roadmap, it's on privacy, it's on the SLA, right?Corey: Yeah. And you see that concern expressed more articulately from enterprise customers, when there's a matter of trusting companies to do what they say, such as the continued investment that Alphabet slash Google is making in Google Cloud. It's easy to take the approach of well, you've turned off a bunch of consumer services, so therefore, you're going to turn off the cloud at some point, too. No, let me be very clear, for the record, I do not believe that you are going to one day flip a switch and turn off Google Cloud. And neither do your customers.Instead, the approach, the way that enterprises express this, it's not about you flipping the switch and turning it off—that's what contracts are for—their question, and they enshrine this in contracts, in some cases, in the event, not that you turn it off, but that you fail to appropriately continue to invest in the platform. Because at enterprise scale, this is how things tend to die. It is not through flipping a switch, in most cases, it's through, “We're just going to basically mothball it, keep it more or less exactly as it is until it slowly fades into irrelevance for a long period of time.” And when you're providing the infrastructure to run things for serious institutions, that part isn't okay. And credit where due, I have seen every indication that Google means it when they say this is an area of strategic and continued ongoing focus for us as a company.Seth: Yeah, I mean, Google is heavily investing in cloud. I mean, this is a brand new group that I'm working in and we're trying to get Alphabet companies onto cloud, so obviously there's some very high-level top-down executive support for this. I will say that the—a hundred percent agree with everything you're saying—the traditional enterprise approach of build this Java app—because let's be honest, it's always Java—build this Java app, compile it into a JAR and run it forever is becoming problematic. We saw this recently with, like, the log4j—Corey: Yeah, to be in a container. What the hell?Seth: [laugh].Corey: I'm kidding. I'm kidding. Please don't send me email, whatever you do.Seth: What's a container? I'm just kidding. Like, the idea of, like, software rotting is very real and it's becoming more and more of a risk to security, to privacy, to public cloud providers, to enterprises, where when you see something like log4j happen and you can't answer the question, like, do we have any code that uses that? Like, if getting the answer to that question takes you six weeks, [sigh] boy like, a lot of stuff can happen in six weeks while that particular thing is exploited. And you know, kind of gets into software supply chain a little bit, but I do agree that, like, secure, private, and stable APIs are super important, and it's an area where Google is investing. At the same time, I think the industry is moving, the enterprise industry is moving away a little bit from set-it-and-forget-it as a strategy.Corey: I want to talk about the security portion as well as far as securely consuming public cloud goes. And let me start off with a disclaimer here because I don't want people to misconstrue what I'm about to say. If you are migrating to one of the big three cloud providers, their security will be better than anything you will be able to achieve as a company yourself. Not you personally because Google is a bit of an asterisk to that statement, given what you have been doing and have been doing since the '90s in your on-prem world with Borg and the rest, but my philosophy on the relative positioning of the security of cloud providers relative to one another has changed. I spent four months beating the crap out of Azure forever having an issue where there was control plane access and then really saying nothing about it.And after I wound up finding—the day after I put out a blog post on that topic because I was tired of the lack of response, it came out that right at the same time AWS had a very similar problem and had not said anything themselves. And they went back and forth, apparently waiting to wind up doing a release until this happened, Orca Security wound up putting one out there, and it was frustrating on a couple of levels. First, the people at both of these companies who work in security are stars. There is no argument, no bones about that. Problems are going to happen, things are going to occur as a result, and the only saving grace then is the transparency and communication around it, and there was none of it from them.I'm also more than a little bit irked that my friends at AWS were aware of this, basically watched me drag Azure for four months knowing that they'd done the same thing and never bothered to say a word. But okay, that's a choice. I've been saying for a while that of the big three, Google's security posture is the most impressive. And it used to be a slight difference. Like, you nosed ahead of AWS in that respect, not by a huge margin, but by a bit.I don't think it's nearly as close these days, in my mind, and talking to other large companies about these things, and people who are paid to worry about these things all day long, I am very far from alone in that perspective. So, I guess my question for you is, as you look at moving the workload securely to Google Cloud, it feels like security is baked into everything that all aspects of your company have done. Why is that a specific area of focus? Or is that how it gets baked into everything you folks do?Seth: So, you kind of like set up the answer for this perfectly. I swear we didn't talk about this extensively beforehand.Corey: You didn't know any of that was coming, by the way, just to be very clear here. I don't sit here and feed, “All right, I'm going to say this. And here's the right res—” No, this is an impromptu, more or less ad hoc show every time I do it.Seth: Yeah. And I'm going to preface this by saying, like, I don't want this to sound, like, egotistical, but I have never found a company that has as rigorous security and privacy policies, reviews, and procedures as Google.Corey: I thought I had and I was wrong.Seth: Yeah. And—Corey: And I have a lot of apologizing to people to do as a result of that.Seth: And honestly, every time I interact with our internal security engineering teams, or our IP protection teams, I'm that Nathan Fillion meme, where he's like, what—you know, like, “Okay, I get it. I get it.” Right?Corey: And then facepalm it, uh, I should say some—I can't—yeah. Oh, yeah.Seth: The reason that it's hard for Alphabet companies to securely and privately move to cloud specifically for security, is because Alphabet's stance is so much more rigorous than anyone else in the industry, to the point where, in some cases, even our own cloud provider doesn't meet the bar for what we require for an internal workload. And that's really what it comes down to is, like, the reason that Google is the most secure cloud is because our bar is so high that sometimes we can't even meet it.Corey: I have to assume that the correct answer on this is that you then wind up talking to those product teams and figure out how to get them to a point where they can support that bar because the alternative is effectively, it's like, “Oh, yeah, this is Google Cloud and it's absolutely right for multinational banks to use, but you know, not Google workloads. That stuff's important.” And I don't think that is necessarily how you folks tend to view these things.Seth: So, it's a bidirectional stream, right? So, a lot of it is working with a product management team to figure out where we can add these additional security properties into the system—I should say, tri-directional. The second area is where the policy is so specific to Google that Google should actually build its own layer on top of it that adds the security because it's not generally applicable to even big, huge cloud customers. And then the third area is Google's a very big company. Sometimes we didn't write stuff down, and sometimes we have policies where no one can really articulate where that policy came from.And something that's new with this approach that we're taking now is, like, we're actually trying to figure out where that policy came from, and get at the impetus of what it was trying to protect against and make sure that it's still applicable. And I don't know if you've ever worked with governments or you know, large companies, right, they have this spreadsheet of hundreds of thousands of lines—Corey: You are basically describing my client list. Please continue.Seth: I mean, like, sometimes they have to use an Access database because they exhaust the number of rows in an Excel spreadsheet. And it's just checklist upon checklist upon checklist. And that's not how Google does security, right? Security is a very all-encompassing, kind of, 360 type of thing. But we do have policies that are difficult to articulate what they're actually protecting against, and we are constantly re-evaluating those, and saying, like, “This made sense on Borg. Does it actually make sense on Cloud?” And in some cases, it may not. We get the same protections using, say, a GCP-native service, and we can omit that requirement for this particular workload.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: I think that when it comes to things like policies that are intelligently crafted around security, you folks—and to be fair, the AWS security engineers as well—have been doing it right in that, okay, we're going to build a security control to make sure that a thing can't happen. That's not enough. Then there's the defense-in-depth. Okay, let's say that control fails for some variety of ways. Here are the other things we're going to do to prevent cross-account access, for example.And that in turn, winds up continuing to feed on itself and build into a culture of assuming that you can always continue to invest in security. How far is enough? Well, for most folks, they haven't gone far enough yet.Seth: Another way to put this is like, how well do you want to sleep at night? You know, there's folks on the Google security engineering team who are so smart, and they work on, like, our offensive security team, so their full-time job is to try to hack Google and then figure out how to prevent that. And, you know, so I've read some of the reports and some of the ways they think and I'm like, “How do you… how do you pick up a mobile phone and go to like, any website confidently knowing what you know?” Right? [laugh] and like, how do you—Corey: Who said anything about confidently? Yeah.Seth: Yeah. Yeah. How do you use self-checkout at a supermarket and, like, not just, like, wear your entire full-body tinfoil hat suit? But you know, I think the bigger risk is not knowing what the risks are. And this is a lot what we're seeing in software supply chain, too, is a lot of security is around threat modeling and not checklists. But we tend to, like, gravitate toward checklists because they're concrete.But you really have to ask yourself, like, do I need the same security properties on my static blog website that is stored on an S3 bucket or a GCS bucket that's public to the internet, that I do on my credit card processing service? And a lot of times we don't treat those differently, we don't apply a different threat model to them, and then everything has to have the same level of security.Corey: And then everything is in-scope for whatever it is you're trying to defend against. And that is a short path to madness.Seth: Yes. Yes. Your static HTML files and your GCS bucket are in scope for SOC 1 and 2 because you didn't have a way to say they weren't.Corey: Yeah. You've also done some—again, the nice thing about being at a company for a while—from what I can tell, given that I've never done until I started this place—is you move around and work on different projects. You were involved as well, personally, in the exposure notifications project, the joint collaboration thing between a number of companies in the somewhat early days of the pandemic that all of our phones talk to one another and anonymously and in a privacy-preserving way, let us know that hey, by the way, someone you were in close contact with has tested positive for Covid 19 in the previous fixed period of time. What did do you do over there?Seth: Yeah, so the exposure notifications project was a joint effort, primarily between Apple and Google to use Android and iOS devices to help stop the spread of Covid or reduce the spread of Covid as much as possible. The idea being because the incubation period is roughly 14 days, at least pre-Omicron, if we could tell you hey, you might have been exposed and get you to stay at home for three or four days, self-isolate, we could dramatically reduce the spread of Covid. And we know from some of the studies that have come out of, like, the UK and European region that, like, the technology actually reduced the spread of cases by, like, fourteen-hundred percent in some cases. I was one of the tech leads for the server-side. So, the way the system works is it uses the low-energy Bluetooth on iOS and Android devices to basically broadcast random IDs.So, I know this is Screaming into the Cloud, but if we can just quickly Screaming into the Void as a rebrand—Corey: Oh, yeah.Seth: —that's basically what's happening. [laugh]. You're generating these random identifiers, and just, like, yelling them, and there's other phones out there who are listening. And they collect these we'll call RPIs—or Rolling Indicators. They have no data in them.They're like literally, like, a UUID or 32 bytes of random data, they aren't at all, like, associated with your device or your person. So, then what happens is, like, let's say you're in a supermarket, you're near someone for, you know, every so often, and your phones exchange these IDs. If you then test positive, those IDs go up to a centralized server, the server again, also has no idea who you are, so the whole thing is privacy-preserving, end-to-end, then the server basically bundles all of what we call the TEKs, or the Temporary Exposure Keys—into a tarball that go up onto a CDN, and then every night, all of the devices that are participating in EN download this into a local key match. So, at no point does the server ever know that you were in a supermarket with someone else, only your phone knows that you came in contact with this TEK in the past 14 days—or 21 days in some jurisdictions—and it'll generate an exposure notification or an exposure alert, which says, like, “Hey, in the past 14 days, you've come in contact with someone who's confirmed positive for Covid.” And then there's guidance kind of varies by state and by health jurisdiction of, like, self-isolate, or go get tested, or whatever. But the idea—Corey: Or go to the bar in some places, apparently.Seth: Oh. Yeah. The server itself is actually—there's a verification component because ideally, like, we don't want people to just be like, oh, I'm Covid positive, and then like, all their friends get an alert, right? There needs to be some kind of verification mechanism where you either have a positive test, or you have a clinician or a physician who issues you code that you can put into your app so you can then release your keys. And then there's the actual key server component, which I kind of already described.So, it's a pretty complex system and actually is entirely serverless. So, the whole thing, including all, like, background job processing, it was designed to be serverless from the beginning. Total greenfield project, right, like, nothing like this exists, so we're really fortunate there. We made some fun and interesting design decisions to keep costs down while, you know, abusing slash using some of the features of serverless like auto-scaling and, you know, being able to fan out across multiple regions and things like that—Corey: And using DNS as a database. My personal favorite approach to things?Seth: We don't use DNS as a database. We do use Postgres—Corey: A missed opportunity.Seth: —a real database. But we do use DNS, just not for storing information.Corey: So, one question I have for you is that you've been at Google for a while and you've done an awful lot of things there, but previously, you've also done things that don't really directly aligne any of this stuff going on there. You were at HashiCorp and you were at Chef, neither of whom, to my understanding are technologies that Google makes extensive use of internally for their own stuff. It seems like—and even when you're at Google, you have been continually reinventing what it is that you do. I find that admirable because very often, when you see people at a company for a protracted period of time, they sort of get more or less pigeonholed into the role that looks fairly similar from year-to-year. You've been incredibly dynamic. Was it intentional and how do you do it?Seth: So, I have a diagnosed medical condition called Career-DHD. I'm just kidding, but I do. I get bored, and it's actually something that I'm really forward with my managers about. I've always been very straight with my managers and the people I work with it, like, 8 to 12 months from now, I will be doing something different. It will be different.Corey: I wish I'd figured that out earlier on. In my case, the way that I wound up solving for that is I've got to come in, I'm going to solve a interesting problem. When I'm done with that, the consulting engagement is over and then I'm going to go away and everyone knows the score going in. Works out way better than, and then I'm going to go cause problems on purpose in other people's parts of the org because I see problems there. That was where I always went off the rails.Seth: [laugh]. Yeah, I mean, I don't take a dissimilar approach. You know, I try to find high-priority, strategic things that also align with my interest. And it's important to me that there's things that I can provide and things that I can learn. I never like to be the smartest person in the room because you shouldn't be in that room anymore; there's no one for you to learn from. And it's great to share knowledge, but—Corey: I'm not convinced I'm the smartest person in the room right now, despite the fact that right now I'm the only person in the room that I'm sitting in.Seth: I mean, that Minecraft store is pretty intelligent.Corey: I saw Chihuahua wandering around here, too, a—Seth: [laugh].Corey: —minute ago, so there is that.Seth: But, you know, I think from, like, a career advice standpoint, I tell everyone, you should interview somewhere else at least once a year. You never know what's out there, and worst-case scenario, you kept your interview skills up to date.Corey: Keeping those skills in tune is so critically important just because it's a unique skill set that, for many folks, does not have a whole lot of applicability in their day-to-day job. So, if you suddenly have to find a new job, great, you're rusty at this, it's been years, and you're trying to remember, like, okay, when someone asks you what you're looking for in your next job, they're not trying to pick a fight. Don't respond as if they were. Like, the basic stuff. It's a skill, like anything else.Seth: Yeah. And, like, the common questions like, you know, “What do you want to do with your life?” Or like, “What accomplishment are you most proud of?” Like, having those not prepared, but like knowing in general what you want to say from those is very important when you're thinking about interviewing for other jobs. But even in a big company, like the transfer process is, pretty similar for, like, applying externally to other roles; like sometimes there's interviews—Corey: Do they make you code on whiteboards to solve algorithm problems?Seth: Not me. But—Corey: Good.Seth: —in general—Corey: Google has evolved its interview process since the last time I went through that particular brand of corporate hazing. Good, good, good.Seth: Yeah. The interview process has definitely been refactored a lot, especially with Covid and remote, but also just trying to be accessible to folks. I know one of the big changes Google has made is we no longer require, like, eight congruent hours of your time. You can split interviews out over multiple days, which has been really accommodating for folks that have, you know, already have a full-time job or have family obligations at home that don't let them just, like, take eight hours away and devote a hundred percent of their time to interviews. So, I think that is, you know, not a whole lot of positive things that come out of Covid, but the flexibility with, like, interviewing has enabled more people to participate in the interview process that otherwise would not have been able to do so.Corey: And there's something to be said, for making this more accessible to folks who come from backgrounds that don't all look identical. It's incredibly important.Seth: Yep.Corey: One thing that I definitely want to make sure we get to before the end of this is something you've been talking about that's a bit orthogonal, but maybe not entirely so, which is software supply chain security. That has been a common thread of discussion in some circles for a while. What is it, for those who are unfamiliar, like me sometimes, and what does it imply?Seth: Yeah, so I mean, in the past year—but if you look back, you'll find more cases of it—. We live in a world where no company—Google, Amazon, the US government—writes every line of code that they run. And even if you do, right, even if you could find a company that doesn't rely on any external dependencies, what language are they using? Did they write that language? Okay, let's say hypothetically, you write every single line of code and you wrote your own language, and only your employees contribute to that language.What operating system are you running on? Because I guarantee you, Linus probably contributed to it, or Gates contributed to it, and they don't work for you. But let's say you wrote your own operating system, right—so we're getting into, like, crazy Google things now, right? Like, only Google would write their own programming language and their own operating system, right? Who manufactured your CPU, right? Like, did you actually—Corey: There's always dependencies all the way down. We see this sometimes with companies talk about oh, yeah, we're going to go to multiple clouds or a different clouds so that we don't get impacted if there's another AWS outage in us-east-1. Cool, great. Power to you, but are you sure your payment providers not going to go down? Are they taking a dependency on us-east-1?Great, let's say that they're not. Are you sure that their vendors who are in the critical path are also not taking critical and core dependencies on that? And are you sure that they're aware of who all of those critical dependencies and those vendors are, and so on and so forth? It is a vast interconnected web. This is a problem. Dependency sprawl is real and I don't think that there's a good way to get to the bottom of it, particularly across company boundaries like that.Seth: Yeah. And this is where if you look at the non-software supply chain, like, if you look at construction, right? If you're working with a reputable construction agency, they're actually able to tell you, given a granite countertop or, you know, a quartz countertop, from what beach and what lot on what date the grains of sand in that countertop came from. That is a reality of that industry that is natural. You think about, like, automotive, like, VIN, the Vehicle Identification Numbers, like, they tell you exactly what manufacturer, and then there's records that show you exactly what human being on the line put that particular part in that machine.And we don't have that in software today. Like, we have some, you know, bastardized versions of, like, Software Bills of Material, or SBOM, but the simple fact of the matter is like because software has grown organically and because this wasn't ingrained in software from the beginning like it was from, you know, traditional manufacturing, you're going to have an insecure software supply chain for most of my life. Now, what does that actually mean, right—insecure has this negative connotation—it means that you need to make sure that you're aware of everything that you're depending on—which is kind of what you were saying is, like, both the technical dependencies and the process or the people dependencies—and you need to have a rigorous process for how you're going to respond to these incidents. And I think log4j was a really good eye-opening moment for folks when they realized that they didn't have a way to make a large-scale dependency update across their entire fleet of applications.Corey: Because who has to do that on a consistent basis? It happens rarely, but when it happens, it's super important.Seth: But I do think that more and more, we're going to see it happened more and more frequently. And ideally, you know, my opinion is that we're going to get to a point where this is inescapable, but ideally, we get to the point where it's like, “Oh, okay, this dependency is vulnerable. I have a playbook. I follow the playbook. Everything is patched in 30 minutes or less, and I can move on with my life.” And it's not a six-week fire drill with people working late and, you know, going super crazy, trying to mitigate these issues.You know, there's a lot of work happening in this space. We have, like, SLSA, which is an open standard—SLSA—for how you declare, kind of like, your software bill of materials and things like binary authorization and attestations. There's, like, Sigstore, there's Chainguard, there's some companies evolving in this space. Every time I talk to GitHub, I tell them, I'm like, “Hey, if this VP and that VP, like, talked together and, like, worked on something, you could do something amazing in this space.” But I think it's going to be quite a while until we get to a point where we can say the software supply chain is secure.Because like I was saying at the beginning, like, until you manufacture your own CPU, like, you're dependent on Intel and AMD. And until you write your own programming language, you're dependent on Ruby, Python, Go, whatever it might be. And until you take no dependencies on some external system—which by the way, might be a bad business decision, like, if someone did the work for you already in an open-source ecosystem, it's probably a better business decision to evaluate and use that than to build it yourself. Until we have the analysis on that supply chain, and we can in a dashboard, or the click of a button, or the run of a command, very easily see the security status of our supply chain—software supply chain—and determine if a particular vulnerability is or is not relevant, I think we're still going to be in this firefighting mode for at least another couple of years.Corey: And I want to say you're wrong, but I know you're not. And that's what, I guess, keeps a lot of us awake at night for unfortunate reasons. Seth, I really want to thank you for taking the time to speak with me. If people want to learn more, where's the best place to find you?Seth: I'm on Twitter. You can find me at—Corey: I'm sorry to hear that. So, am I. It's the experience.Seth: Yeah, you can find me at @sethvargo. If you say mean and hateful things to me, I actually exercise this finger, and you can click the block button real fast. But yeah, I mean, my DMs are open. If you have any questions, comments, complaints, concerns, you can throw the complaints away and come to me for everything else.Corey: Thank you so much for being so generous with your time. I really appreciate it.Seth: Yeah, thanks for having me. It's always a pleasure.Corey: Seth Vargo, engineer at Google. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment asking how dare I malign the good name of the other cloud provider that isn't Google that also just so coincidentally happens to employ you.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

covid-19 amazon world power google uk apple service european chefs security pittsburgh software android cloud ios engineers ip oracle material dms intel minecraft counting soup excel void fintech omicron alphabet bluetooth chrome screaming gdpr python aws java github apis vpn borg amd devops azure html ids linus chihuahua google drive cpu dependency google cloud dns s3 sql soc jar kubernetes cdn sla capella nathan fillion tek googlers gcp gcs hashicorp oracle cloud sbom corey quinn amazonians teks always free openvpn tlas chainguard duckbill group sigstore orca security chief cloud economist seth vargo last week in aws humblepod

Cloud Security: With Challenges Comes Solutions

Cybersecurity Unplugged

Play Episode Listen Later Feb 9, 2022 23:24

Avi Shua is the brilliant CEO and co-founder of Orca Security and leader of Instant-On Cloud Security. In this episode of Cybersecurity Unplugged, Shua discusses his idea of a Cloud Security solution: Asset and infrastructure visibility, Cloud Orchestration and how Orca Security mediates the problem of extra layers and confusion, and how using the solution helps to identify risk sensitive data and speeds up the process.

Episode 11: Andy Ellis, Advisory CISO, Orca Security & Operating Partner, YL Ventures

FIRST Impressions Podcast

Play Episode Listen Later Feb 4, 2022

Join the interview in progress! Chris, Martin, and Andy chat building teams, navigating within organizations, career change, and interpretive dance. Andy Ellis is the Advisory CISO at Orca Security, where he helps companies embrace secure practices while leaping into the cloud era. He is a 2021 Inductee into the CSO Hall of Fame, an Operating Partner at YL Ventures, the CEO of leadership training company Duha, and was formerly a U.S. Air Force officer and the CSO at Akamai Technologies. You can find him on Twitter at @csoandy. Ellis has received The Spirit of Disneyland Award, The Wine Spectator's Award of Excellence, the Air Force Commendation Medal, and the CSO Compass Award. Disclaimer: The views expressed by the hosts and guests are their own and their participation on the podcast does not imply an endorsement of them or any entity they represent.

ceo spirit security excellence air force ventures advisory cso orca ciso operating partner inductee duha wine spectator akamai technologies andy ellis air force commendation medal orca security yl ventures

McFee and FireEye join forces for XDR

Play Episode Listen Later Jan 26, 2022 3:51

Cloud Security News this week 26 Jan 2022 Early December on Cloud Security News, we shared that Symphony Technology Group had acquired McAfee for 4 Billion along with FireEye for 1.2 Billion. The merger of these two companies has now form Trellix, which aims to be a leader in extended detection and response (XDR). In their blog post Trellix shared that “Customers can expect Trellix's living security platform to deliver bold innovation across the XDR market.” - “with automation, machine learning, extensible architecture, and threat intelligence.” You can find out more about Trellix and read their blog post here and let us know if you are excited about this merger? Orca Security is back in the news this week, not for their funding round or their vulnerability findings in AWS. They have made their 1st acquisition: RapidSec, an Israeli cybersecurity startup that protects web applications from client-side attacks. RapidSec's software allows for detection of web-application misconfigurations and deviations from best practices. Orca has indicated that it plans to integrate these web services and API security technologies into its agentless cloud security platform. You can read more about this acquisition here. Cloud Security Firm Polar Security that has emerged from Stealth With $8.5 Million Seed Funding. They are a Tel Aviv, Israel-based cloud security company that aims to provide visibility into companies' cloud data storage to allow security teams to secure the data and avoid compliance problems. You can find out more about them here Hunters.ai announced that it has raised a $68 million Series C round bringing their total funding to date to $118 million. Hunters share in their blog that “Never before has it been more lucrative to be a cyber criminal” and “On the defenders' side, we see organizations struggling to keep pace. As technology advances and more tools are being used, the attack surface grows and the number of security products used by these organizations increases.” This is where Hunter.ai believes they can help with their Extended Detection and Response (XDR) platform used by Security Operations Center (SOC) teams to detect, investigate and stop threats. You can find out more about them here Podcast Twitter - Cloud Security Podcast (@CloudSecPod) Instagram - Cloud Security News If you want to watch videos of this LIVE STREAMED episode and past episodes, check out: - Cloud Security Podcast: - Cloud Security Academy:

israel billion customers israelis forces api hunters tel aviv aws mcafee orca series c live streamed fireeye xdr mcfee orca security extended detection

McFee and FireEye join forces for XDR

Play Episode Listen Later Jan 26, 2022 3:51

israel billion customers israelis forces api hunters tel aviv aws mcafee orca series c live streamed fireeye xdr mcfee orca security extended detection

The Gruntled Developer

Play Episode Listen Later Jan 20, 2022 6:05

Links: S3 Bucket Negligence Award: http://saharareporters.com/2022/01/10/exclusive-hacker-breaks-nimc-server-steals-over-three-million-national-identity-numbers Anyone in a VPC, any VPC, anywhere: https://Twitter.com/santosh_ankr/status/1481387630973493251 A disgruntled developer corrupts their own NPM libs ‘colors' and ‘faker', breaking thousands of apps: https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/ “Top ten security best practices for securing backups in AWS”: https://aws.amazon.com/blogs/security/top-10-security-best-practices-for-securing-backups-in-aws/ Glue: https://aws.amazon.com/security/security-bulletins/AWS-2022-002/ CloudFormation: https://aws.amazon.com/security/security-bulletins/AWS-2022-001/ S3-credentials: https://simonwillison.net/2022/Jan/18/weeknotes/ TranscriptCorey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it's nobody in particular's job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.Corey: This episode is sponsored in part by my friends at Thinkst Canary. Most companies find out way too late that they've been breached. Thinkst Canary changes this and I love how they do it. Deploy canaries and canary tokens in minutes, and then forget about them. What's great is then attackers tip their hand by touching them, giving you one alert, when it matters. I use it myself and I only remember this when I get the weekly update with a, “We're still here, so you're aware,” from them. It's glorious. There is zero admin overhead to this, there are effectively no false positives unless I do something foolish. Canaries are deployed and loved on all seven continents. You can check out what people are saying atcanary.love. And, their Kube config canary token is new and completely free as well. You can do an awful lot without paying them a dime, which is one of the things I love about them. It is useful stuff and not a, “Oh, I wish I had money.” It is spectacular. Take a look. That'scanary.love because it's genuinely rare to find a security product that people talk about in terms of love. It really is a neat thing to see.Canary.love. Thank you to Thinkst Canary for their support of my ridiculous, ridiculous nonsense.Corey: So, yesterday's episode put the boots to AWS, not so much for the issues that Orca Security uncovered, but rather for its poor communication around the topic. Now that that's done, let's look at the more mundane news from last week's cloud world. Every day is a new page around here, full of opportunity and possibility in equal measure.This week's S3 Bucket Negligence Award goes to the Nigerian government for exposing millions of their citizens to a third party who most assuredly did not follow coordinated disclosure guidelines. Whoops.There's an interesting tweet, and exploring it is still unfolding at time of this writing, but it looks that making an API Gateway ‘Private' doesn't mean, “To your VPCs,” but rather, “To anyone in a VPC, any VPC, anywhere.” This is evocative of the way that, “Any Authenticated AWS User,” for S3 buckets caused massive permissions issues industry-wide.And a periodic and growing concern is one of software supply chain—which is a fancy way of saying, “We're all built on giant dependency chains”—what happens when, say, a disgruntled developer corrupts their own NPM libs ‘colors' and ‘faker', breaking thousands of apps across the industry, including some of the AWS SDKs? How do we manage that risk? How do we keep developers gruntled?Corey: Are you building cloud applications with a distributed team? Check out Teleport, an open-source identity-aware access proxy for cloud resources. Teleport provides secure access for anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps, and databases. Teleport gives engineers superpowers.Get access to everything via single sign-on with multi-factor, list and see all of SSH servers, Kubernetes clusters, or databases available to you in one place, and get instant access to them using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility, and ensuring compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That's goteleport.com.AWS had a couple of interesting things. The first is “Top ten security best practices for securing backups in AWS”. People really don't consider the security implications of their backups anywhere near seriously enough. It's not ‘live' but it's still got—by definition—a full set of your data just waiting to be harvested by nefarious types. Be careful with that.And of course, AWS had two security bulletins, one about its Glue issues, one about its CloudFormation issues. The former allowed cross-account access to other tenants. In theory. In practice, AWS did the responsible thing and kept every access event logged, going back for the full five years of the service's life. That's remarkably impressive.And lastly, I found an interesting tool called S3-credentials last week, and what it does is it helps generate tightly-scoped IAM policies that were previously limited to a single S3 bucket, but now are limited to a single prefix within that bucket. You can also make those credential sets incredibly short-lived. More things like this, please. I just tend to over-scope things way too much. And that's what happened Last Week in AWS: Security. Please feel free to reach out and tell me exactly what my problem is.Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign up for the Last Week in AWS newsletter at lastweekinaws.com.Announcer: This has been a HumblePod production. Stay humble.

amazon spotify cloud apple podcast i am developers nigerians last week aws devops glue canary s3 deploy kubernetes canaries ssh npm teleport kube vpc cloudformation aws security vpcs orca security last week in aws humblepod

Managing Cloud Security Risk Posture Through a Full Stack Approach

The New Stack Podcast

Play Episode Listen Later Jan 19, 2022 9:28

Kubernetes, containers, and cloud-native technologies offer organizations the benefits of portability, flexibility and increased developer productivity but the security risks associated with adopting them continue to be a top concern for companies. In the recent State of Kubernetes Security report, 94% of respondents experienced at least one security incident in their Kubernetes environment in the last 12 months. In this episode of The New Stack Makers podcast, Avi Shua, CEO and Co-Founder of Orca Security talks about how organizations can enhance the security of their cloud environment by acting on the critical risks such as vulnerabilities, malware and misconfigurations by taking a snapshot of Kubernetes clusters and analyzing them, without the need for an agent.

ceo state co founders managing risk makers posture kubernetes fullstack cloud security new stack orca security new stack makers

Remote Access Trojans target Public Cloud Infrastructure

Play Episode Listen Later Jan 19, 2022 7:06

Cloud Security News this week 19 Jan 2022 Cisco Talos Researchers have shared in a blog last week that a trio of remote access Trojans (RATs)—Nanocore, Netwire and AsyncRAT—are being spread in a campaign that taps public cloud infrastructure and is primarily aimed at victims in the U.S., Italy and Singapore. According to the blog “Threat actors are increasingly using cloud technologies to achieve their objectives without having to resort to hosting their own infrastructure,” and “cloud services like Azure and AWS allow attackers to set up their infrastructure and connect to the internet with minimal time or monetary commitments. It also makes it more difficult for defenders to track down the attackers' operations.” Read more about this here. Netskope also released a blog last week about Malwares. Interestingly their research which surveyed millions of users worldwide from January 1, 2020 to November 30, 2021 found that Cloud-delivered malware is now more prevalent than web-delivered malware, accounting for 66%, up from 46% last year. They also found that Google Drive is the top app for most malware downloads and Cloud-delivered malware via Microsoft Office nearly doubled from 2020 to 2021. Read the report here Vulnerability in AWS's cloudformation service that was discovered and shared by Orca Security. Orca Security confirmed that AWS completely mitigated within 6 days of their submission.If you want to know more about their discovery, you can read it here The US government is reportedly reviewing the cloud computing arm of Chinese ecommerce giant Alibaba to determine whether or not it poses a risk to national security.” As reported by Reuters, the Biden administration launched the probe to find out more about how Alibaba Cloud stores the data of US clients including personal information and intellectual property and to see if the Chinese government could gain access to it. You can read Reuters report here Sysdig's platform who were recently valued at 2.5 Billion have expanded their cloud security offering to Azure Cloud aswell. . You can find out more about them here Podcast Twitter - Cloud Security Podcast (@CloudSecPod) Instagram - Cloud Security News If you want to watch videos of this LIVE STREAMED episode and past episodes, check out: - Cloud Security Podcast: - Cloud Security Academy:

Remote Access Trojans target Public Cloud Infrastructure

Play Episode Listen Later Jan 19, 2022 7:06

Orca Security, AWS, and the Killer Whale of a Problem

Play Episode Listen Later Jan 19, 2022 13:01

Want to give your ears a break and read this as an article? You're looking for this link.https://www.lastweekinaws.com/blog/orca-security-aws-and-the-killer-whale-of-a-problemNever miss an episode Join the Last Week in AWS newsletter Subscribe wherever you get your podcasts Help the show Leave a review Share your feedback Subscribe wherever you get your podcasts What's Corey up to? Follow Corey on Twitter (@quinnypig) See our recent work at the Duckbill Group Apply to work with Corey and the Duckbill Group to help lower your AWS bill

amazon killers security cloud last week aws devops orca killer whales duckbill group orca security last week in aws

24 November 2021 - GoDaddy looses 1.2 million user information

Play Episode Listen Later Nov 24, 2021 5:23

Cloud Security News this week 24 November 2021 CSA recently announced that they have now had 1500 Cloud services evaluated across to the STAR registry principles. According to CSA, by publishing to the registry organizations can show current and potential customers their security and compliance posture which may prevent the need for them to complete multiple security questionnaires. You can find more information about CSA and STAR registry here Security researcher Schütz was rewarded a $4,133 bounty by the Google Vulnerability Rewards Program for his Google Internal API vulnerability discovery. Google has now fixed this bug. You can read more about this here and the Schütz has documented his discovery here Palo Alto Networks - a well known cybersecurity Vendor - Their Chairman and CEO Nikesh Arora told investors that they are “18-to-24 months ahead from a competitive platform perspective”. There a few exciting players in the Cloud Security Market right now and you can read more about this here You can also find more about Palo Alto, Orca Security, Wiz and Lacework on the links Lacework, they have recently raised $1.3 billion in fresh capital at a valuation of $8.3 billion, making this one of the largest venture funding rounds of the year in the United States. Nasdaq covered a bit more about this here. In comparison Orca Security raised $550 million in Series C funding to raise their valuation to $1.8 Billion and Wiz raised $250 million on a $6 billion valuation Clubhouse, an audio based chatroom launched in 2020 which gained popularity during the pandemic has launched a BugBounty program on HackerOne. The scope of the Bounty includes their API and websites. The program has upto $3000 on offer for any critical vulnerabilities reported. You can find more about the program here Using a compromised password, an unauthorised third party has managed to infiltrate GoDaddy's systems affecting atleast 1.2 million users. Along with usernames, passwords and emails, the attackers also gained access to SSL private keys for a subset of users. Episode Show Notes on Cloud Security Podcast Website. Podcast Twitter - Cloud Security Podcast (@CloudSecPod) Instagram - Cloud Security News If you want to watch videos of this LIVE STREAMED episode and past episodes, check out: - Cloud Security Podcast: - Cloud Security Academy:

Palo Alto Investors told: "18- 24 months ahead" of competition

Paul's Security Weekly TV

Play Episode Listen Later Nov 24, 2021 5:23

Privacy Engineering Firms, Facebook Outages, Orca Series C, & Gravwell - ESW #245

Play Episode Listen Later Oct 8, 2021 42:15

In the Enterprise Security News for this week: Orca Security raises all the money, Privacy engineering firms hit their funding stride, McAfee and FireEye merge, but where's RSA's dance partner? Akamai acquires Guardicore, NetApp picks up CloudCheckr, SPDX becomes the ISO standard for SBOMs, & Facebook shares details on how they accidentally Thanos snapped themselves! All that, our weekly Squirrel, and more, on this episode of the Enterprise Security Weekly News! Visit https://www.securityweekly.com/esw for all the latest episodes! Show Notes: https://securityweekly.com/esw245

engineering privacy thanos squirrel iso firms mcafee orca rsa series c outages netapp akamai fireeye sboms gravwell orca security guardicore enterprise security news

Privacy Engineering Firms, Facebook Outages, Orca Series C, & Gravwell - ESW #245

Enterprise Security Weekly (Video)

Play Episode Listen Later Oct 8, 2021 42:15

engineering privacy thanos squirrel iso firms mcafee orca rsa series c outages netapp akamai fireeye sboms gravwell orca security guardicore enterprise security news

Complete Nightmare - ESW #245

Enterprise Security Weekly (Audio)

Play Episode Listen Later Oct 7, 2021 101:48

This week, we welcome Richard Reinders, Head of Security at Gravity Payments, to discuss Better Sales, Worse Relationships? In the next segment, we welcome Ryan Kalember, Executive Vice President, Cybersecurity Strategy at Proofpoint, to discuss Shifty Adversaries, Shifting Tactics! In the Enterprise News, Orca Security raises all the money, Privacy engineering firms hit their funding stride, McAfee and FireEye merge, but where's RSA's dance partner? Akamai acquires Guardicore, NetApp picks up CloudCheckr, SPDX becomes the ISO standard for SBOMs, & Facebook shares details on how they accidentally Thanos snapped themselves! Show Notes: https://securityweekly.com/esw245 Visit https://securityweekly.com/proofpoint to learn more about them! Visit https://www.securityweekly.com/esw for all the latest episodes! Follow us on Twitter: https://www.twitter.com/securityweekly Like us on Facebook: https://www.facebook.com/secweekly

head security nightmare privacy thanos iso mcafee rsa netapp akamai fireeye cybersecurity strategy proofpoint sboms gravity payments orca security guardicore ryan kalember enterprise news

Complete Nightmare - ESW #245

Paul's Security Weekly

Play Episode Listen Later Oct 7, 2021 101:48

head security nightmare privacy thanos iso mcafee rsa netapp akamai fireeye cybersecurity strategy proofpoint sboms gravity payments orca security guardicore ryan kalember enterprise news

06 October, 2021 - AWS Launches Cloud Control API

Play Episode Listen Later Oct 6, 2021 3:36

Cloud Security News this week 06 October 2021 AWS has announced the availability of AWS Cloud Control API - a set of common application programming interfaces (APIs) that are designed to make it easy for developers to manage their AWS and third-party services. AWS Cloud Control API can be used to create, read, update, delete, and list (CRUD-L) your cloud resources that belong to a wide range of services—both AWS and third-party. You won't have to generate code or scripts specific to each individual service responsible for those resources.We have linked in the podcast notes a informative video from AWS that explains more about this The inaugural HashiCorp State of Cloud Strategy Survey with about 3200 responses has shared that multi-cloud is no longer aspirational goal but an everyday reality - with ¾ of the respondents noting that they were using 2 clouds or more, top drivers for multicloud adoption are digital transformation, avoid vendor lock in, cost reduction and scaling, many enterprises are yet to realise substantial value from their cloud investment and Cloud skills shortage still remains a major challenge Amazon, Google, Microsoft, Atlassian, CISCO, IBM, Salesforce, Slack and SAP have joined forces to establish the Trusted Cloud Principles as a commitment to protect the rights of their customers. AWS tweeted that this is to “help safeguard the interests of organizations and the basic rights of individuals using cloud services” You can view the Trusted Cloud Principles here. Orca Security has secured $550 million in Series C funding to raise their valuation to $1.8 Billion, investment was led by Temasek, an investment company he adquartered in Singapore. Orca Security has a patent-pending SideScanning™ technology that collects data directly from cloud provider APIs/ cloud configuration and the workload's runtime block storage out-of-band to detect vulnerabilities, malware, misconfigurations, lateral movement risk, weak and leaked passwords, and unsecured personal identifiable information. Episode Show Notes on Cloud Security Podcast Website. Podcast Twitter - Cloud Security Podcast (@CloudSecPod) Instagram - Cloud Security News If you want to watch videos of this LIVE STREAMED episode and past episodes, check out: - Cloud Security Podcast: - Cloud Security Academy:

amazon google microsoft cloud singapore billion ibm slack salesforce launches cisco sap aws apis atlassian series c live streamed temasek cloud control orca security

AWS Launches Cloud Control API - Cloud Security News

Play Episode Listen Later Oct 6, 2021 3:29

A MultiCloud Rant

Play Episode Listen Later Aug 20, 2021 7:29

TranscriptCorey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: You know what really grinds my gears? Well, lots of things, but in this case, let's talk about multi-cloud. Not my typical rant about multi-cloud not ever being a good best practice—because it's not—but rather how companies talk about multi-cloud. HashiCorp just did a whole survey on how multi-cloud is the future, and at no point during that entire process did they define the term. So, you wind up with a whole bunch of people responding, each one talking about different things.Are we talking about multiple clouds and we have a workload that flows between them? Are we talking about, “Well, we have some workloads on one cloud provider and a different set of workloads on other cloud providers?” Did they break it down as far as SaaS companies go of, “Yeah, we have an application and we'd like to run it all on one cloud, but it's data-heavy and we have to put it where our customers are, so of course we're on multiple cloud providers.” And then you wind up with the stories that other companies talk about, where you have a bunch of folks where their sole contribution to the ecosystem is, “Ah, you get a single pane of glass between different cloud providers.”You know who wants that? No one. The only people who really care about those things are the folks who used to sell those items and realized that if this dries up and blows away, they have nothing left to sell you. There's also a lot of cloud providers who are deep into the whole multi-cloud is the way and the light and the future because they know if you go all-in on a single cloud provider, it will certainly not be them. And then you have the folks who say, “Go in on one cloud provider and don't worry about it. It'll be fine. If you need to migrate down the road, you can do that.”And I believe that that's generally the way that you should approach things, but it gets really annoying and condescending when AWS tells that story because from their perspective, yeah, just go all-in and use Dynamo as your data store for everything even though there's really no equivalent on other cloud providers. Or, “Yeah, go ahead and just tie all of your data warehousing to some of the more intricate and non-replicable parts of S3.” And so on and so forth. And it just feels like they're pushing a lock-in narrative in many respects. I like having the idea of a strategic Exodus, where if I have to move a thing down the road, I don't have to reinvent the data model.And a classic example of what I would avoid in that case is something like Google Spanner—or Google Cloud Spanner, or whatever the one they sell us is—because yeah, it's great, and it's awesome. And you wind up with, effectively, what looks like an ACID-compliant SQL database that spans globally. But there's nothing else quite like that, so if I have to migrate off, it's not just a matter of changing APIs, I have to re-architect my entire application to be aware of the fact that I can't really have that architecture anymore, just from a data flow perspective. And looking at this across the board, I find that this is also a bit esoteric because generally speaking, the people who are talking the most about multi-cloud and wanting to avoid lock-in, are treating the cloud like it's fundamentally an extension of their own crappy data center where they run a bunch of VMs and that's it.They say they want to be multi-cloud, but they're only ever building for one cloud, and everything that they're building on top of it is just reinventing baseline primitives. “Oh, we don't trust their load balancers. We're going to run our own with Nginx or HAProxy.” Great. While you're doing that, your competitors are getting further ahead.You're not even really in the cloud: you basically did the lift part of it, declined to shift, declared victory, and really the only problem you solve for is you suck at dealing with hard drive failure, so you used to deal with outages in your data center and now your cloud provider handles it for you at a premium that's eye-wateringly high.Corey: I really love installing, upgrading, and fixing security agents in my cloud estate. Why do I say that? Because I sell things for a company that deploys an agent. There's no other reason. Because let's face it; agents can be a real headache. Well, Orca Security now gives you a single tool to detect basically every risk in your cloud environment that's as easy to install and maintain as a smartphone app. It is agentless—or my intro would have gotten me in trouble here—but it can still see deep into your AWS workloads while guaranteeing 100% coverage. With Orca Security there are no overlooked assets, no DevOps headaches—and believe me, you will hear from those people if you cause them headaches—and no performance hits on live environment. Connect your first cloud account in minutes and see for yourself at orca dot security. That's orca—as in whale—dot security as in that thing your company claims to care about but doesn't until right after it really should have.Corey: Look, I don't mean to be sitting here saying that this is how every company operates because it's not. But we see a lot of multi-cloud narrative out there, and what's most obnoxious about all of it is that it's coming from companies that are strong enough to stand on their own. And by pushing this narrative, it's increasingly getting to a point where if you're not in a multi-cloud environment, you start to think, “Maybe I'm doing something wrong.” You're not. There's no value to this.Remember, you have a business that you're trying to run, in theory. Or for those of us who are still learning things, yeah, we want to learn a cloud provider before we learn all the cloud providers, let's not kid ourselves. Pick one, go all-in on for the time being, and don't worry about what the rest of the industry is doing. We're not trying to collect them all. There is no Gartner Magic Quadrant for Pokemons and I don't think the cloud providers should be one of them.I know I've talked about this stuff before, but people keep making the same fundamental errors and it's time for me to rant on it just a smidgen more than I have already.Thank you for listening, as always to Fridays From the Field on the AWS Morning Brief. And as always, I'm Chief Cloud Economist Corey Quinn, imploring you to continue to make good choices.Announcer: This has been a HumblePod production. Stay humble.

amazon discover field exodus cloud rant pokemon saas api acid aws apis devops elk s3 equifax sql dynamo klarna vms multicloud blackboard hashicorp elasticsearch nginx gartner magic quadrant edb haproxy orca security elastic cloud google cloud spanner last week in aws humblepod

How AWS is Still Egregiously Egressing

Play Episode Listen Later Aug 6, 2021 9:20

Links: AWS's Egregious Egress: https://blog.cloudflare.com/aws-egregious-egress/ TranscriptCorey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: Hi there. Chief Cloud Economist Corey Quinn from the Duckbill Group here to more or less rant for a minute about something it's been annoying the heck out of me for a while, as anyone who follows me on Twitter or subscribes to the lastweekinaws.com newsletter, or passes me in a crowded elevator will attest to, and that is AWS's data transfer story.Back on July 23rd—of 2021, for those listening to this in future years—CloudFlare did a blog post titled AWS's Egregious Egress, and that was co-authored by Matthew Prince—CloudFlare's CEO—and Nitin Rao—who is one of their employees. Presumably. That was somewhat unclear—and it effectively tears down the obnoxious—and I mean deeply obnoxious—level of AWS data transfer pricing for egress to the outside world.And there's a bunch of things to unpack in this blog post, where they wind up comparing AWS pricing to the wholesale bandwidth market. And they go into a whole depth for those who aren't aware of how bandwidth is generally charged for. And the markups that they come up with for AWS are, in many cases, almost 8,000%, which is just ludicrous, in some respects, because—spoiler—every year, give or take, the wholesale cost of network bandwidth winds up dropping by about 10%, give or take. And the math that they've done that I'm too lazy to check, says that in effect, given that they don't tend to reduce egress bandwidth pricing, basically ever, while the wholesale market has dropped 93%, what we pay AWS hasn't. And that's obnoxious.They also talk—rather extensively—about how ingress is generally free. Now, there's a whole list of reasons that this could be true, but let's face it, when you're viewing bandwidth into AWS as being free, you start to think of it that way of, “Oh, it's bandwidth, how expensive could it possibly be?” But when you see data coming out and it charges you through the nose, you start to think that it's purely predatory. So, it already starts off with customers not feeling super great about this. Then diving into it, of course; they're pushing for the whole bandwidth alliance that CloudFlare spun up, and good for them; that's great.They have a bunch of other providers willing to play games with them and partner. Cool, I get it. It's a sales pitch. They're trying to more or less bully Amazon into doing the right thing here, in some ways. Great, not my actual point.My problem is that it's not just that data transfer is expensive in AWS land, but it's also inscrutable because, ignoring for a second what it costs to send things to the outside world, it's more obnoxious trying to figure out what it costs to send things inside of AWS. It ranges anywhere from free to very much not free. If you have a private subnet that's talking to something in the public subnet that needs to go through a managed NAT gateway, whatever your transfer price is going to be has four and a half cents per gigabyte added on to it with no price breaks for volume. So, it's very easy to wind up accidentally having some horrifyingly expensive bills for these things and not being super clear as to why. It's very challenging to look at this and not come away with the conclusion that someone at the table is the sucker.And, as anyone who plays poker is able to tell you, if you can't spot the sucker, it's you. Further—and this is the part that I wish more people paid attention to—if I'm running an AWS managed service—maybe RDS, maybe DynamoDB, maybe ElastiCache, maybe Elasticsearch—none of these things are necessarily going to be best-to-breed for the solution I'm looking at, but their replication traffic between AZs in the same region is baked into the price and you don't pay a per-gigabyte fee for this. If you want to run something else, either run it yourself on top of EC2 instances or grab something from the AWS marketplace that a partner has provided to you. There is no pattern in which that cross-AZ replication traffic is free; you pay for every gigabyte, generally two cents a gigabyte, but that can increase significantly in some places.Corey: I really love installing, upgrading, and fixing security agents in my cloud estate. Why do I say that? Because I sell things for a company that deploys an agent. There's no other reason. Because let's face it; agents can be a real headache. Well, Orca Security now gives you a single tool to detect basically every risk in your cloud environment that's as easy to install and maintain as a smartphone app. It is agentless—or my intro would have gotten me in trouble here—but it can still see deep into your AWS workloads while guaranteeing 100% coverage. With Orca Security there are no overlooked assets, no DevOps headaches—and believe me, you will hear from those people if you cause them headaches—and no performance hits on live environment. Connect your first cloud account in minutes and see for yourself at orca dot security. That's orca—as in whale—dot security as in that thing your company claims to care about but doesn't until right after it really should have.Corey: It feels predatory, it feels anti-competitive, and you look at this and you can't shake the feeling that somehow their network group is being evaluated on how much profit it can turn, as opposed to being the connective tissue that makes all the rest of their services work. Whenever I wind up finding someone who has an outsized data transfer bill when I'm doing the deep-dive analysis on what they have in their accounts, and I talk to them about this, they come away feeling, on some level, ripped off, and they're not wrong. Now, if you take a look at other providers—like Oracle Cloud is a great example of this—their retail rate is about 10% of what AWS's for the same level of traffic. In other words, get a 90% discount without signing any contract and just sign the dotted line and go with Oracle Cloud. Look, if what you're doing is bandwidth-centric, it's hard to turn your nose up at that, especially if you start kicking the tires and like what you see over there.This is the Achilles heel of what happens in the world of AWS. Now, I know I'm going to wind up getting letters about this because I always tend to whenever I rant about this that no one at any significant scale is paying retail rate for AWS bandwidth. Right, but that's sort of the point because when I'm sitting here doing back-of-the-envelope calculations on starting something new and that thing tends to be fairly heavy on data transfer—like video streaming—and I look at the retail published rates, it doesn't matter what the discount is going to be because I'm still trying to figure out if this thing has any baseline level of viability, and I run the numbers and realize, wow, 95% of my AWS bill is going to be data transfer. Well, I guess my answer is not AWS. That's not a pure hypothetical.I was speaking to someone years ago, and they have raised many tens of millions of dollars for their company since, and it's not on AWS because it can't be given their public pricing. Look, this is not me trying to beat up unnecessarily on AWS. I'm beating them up on something that frankly, has been where it is for far too long and needs to be addressed. This is not customer obsession; this is not earning trust; this is not in any meaningful way aligned with where customers are and the problems customers are trying to solve. In many cases, customers are going to be better served by keeping two copies of the data, one in each availability zone rather than trying to replicate back and forth between them because that's what the economics dictate.That's ludicrous. It should never be that way. But here we are. And here I am. I'm Chief Cloud Economist Corey Quinn here at the Duckbill Group. Thank you for listening to my rant about AWS data transfer pricing.Announcer: This has been a HumblePod production. Stay humble.

ceo amazon discover cloud api aws devops elk s3 equifax cloudflare klarna rds blackboard elasticsearch ec2 oracle cloud dynamodb edb azs duckbill group orca security elasticache elastic cloud last week in aws humblepod

A Conversation between Cloud Economists with Amy Arambulo Negrette

Play Episode Listen Later Aug 5, 2021 35:01

About AmyWith over ten years industry experience, Amy Arambulo Negrette has built web applications for a variety of industries including Yahoo! Fantasy Sports and NASA Ames Research Center. One of her projects modernized two legacy systems impacting the entire research center and won her a Certificate of Excellence from the Ames Contractor Council. More recently, she built APIs for enterprise clients for a cloud consulting firms and led a team of Cloud Software Engineers. Amy has survived acquisitions, layoffs, and balancing life with two small children. Links: The Duckbill Group: http://duckbillgroup.com/ @nerdypaws: https://twitter.com/nerdypaws TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Your company might be stuck in the middle of a DevOps revolution without even realizing it. Lucky you! Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy in on DevOps practices? Well, download the 2021 State of DevOps report brought to you annually by Puppet since 2011 to explore the trends and blockers keeping evolution firms stuck in the middle of their DevOps evolution. Because they fail to evolve or die like dinosaurs. The significance of organizational buy in, and oh it is significant indeed, and why team identities and interaction models matter. Not to mention weither the use of automation and the cloud translate to DevOps success. All that and more awaits you. Visit: www.puppet.com to download your copy of the report now!Corey: And now for something completely different!Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by my colleague, Amy Arumbulo Negrette, who's a cloud economist here at The Duckbill Group. Amy, thank you for taking the time to, basically, deal with my slings and arrows instead of the ones that clients throw your way.Amy: It's perfectly fine. It's not as if we are… not the kindest people within the Slack channels anyway. So, I am totally good. [laugh].Corey: [laugh]. So, you've been at The Duckbill Group, as of the time of this recording, which when you're releasing things in the future, it's always a question of how long will you have been here by then? No. We're playing it straight here from a perspective of, as of the time of this recording, you've been here six months, all of which, of course, have been during the global pandemic. So first, what's that been like?Amy: It has been very loud. And that's to say, I live in a house with five other people in it, so it's one thing for me to be a remote worker and just being at my desk, working quietly, but also having to manage noise that you can't really control, it's been an extra level of stress that I could possibly do without. It's fine. [laugh].Corey: One of the whole problems with the pandemic, from our perspective, has been that we've run this place as a full remote operation since it was started, and people come at this from a perspective of, “Well, this whole experience we've had with working remote is awful. It's terrible. No one likes it. I'm not productive.” Let's be very clear here. There's been a global pandemic; this is not like most years, and there are stressors and things that absolutely suck about this that don't normally impact the remote work story quite the way that they have.Amy: I totally agree. At least before, in one of my previous companies had an office in Chicago, so I would be there once a week, but effectively I was remote because that was an all meetings type of day. And the difference between that and now is that you had very explicit work hours; you had client hours; your work sometimes brought you out the house, if you had to go on travel, or on-site. This is just everything is done within the same ten feet of basically where you work, and you eat, and you sleep, if you have really unhealthy living habits like I do. And while I'm trying to get better at it, I'm also not the best at having time-based boundaries. I'm only good with physical boundaries. So, I have to turn off the work computer, to turn on the fun computer, which are physically next to each other, but I have to look in a different direction. And that is as close as I get.Corey: Yeah. It's the good screen versus the bad screen model of, “Oh, yeah, we're going to stop doing work now and just move our gaze slightly to one side and look at the fun screen and work on those things instead.” And at first, I was trying to be militant when we started the whole pandemic thing and working full remote of booting people of, “Hey, all right, it's quitting time. Go home and stop it.” The other side of that, though, is some people are, in some ways, using work to escape.So, we've modified our approach to get the work done. If you're working consistently more than 40 hours to get it done, let us know; that's a problem. But let people work when they want to work, how they want to work and be empathetic humans. And that carries surprisingly far. Now, the question, of course, becomes, does this scale to a company that has 50,000 employees? I don't know. That's never been a problem that anyone has asked me to solve. But it works for us at small scale.Amy: I find that the attitude working here has been really understanding as to, we know what our lives are like and we know what kind of work that we actually have to get done within a certain time period. And all of us make those. We don't feel the need to explain how we were able to get that work done or what time slots happen. You'll see me and Jesse—one of the other cloud economists—it's like we'll be hitting document at the same time at—for my time would be closer to later in the evening, but that's just because I spent most of the day either taking care of kids or handling house management sort of duties. So, having that flexibility on where my schedule goes without having to answer that question of, “Well, why were you working at that hour?” I feel gives me a lot more control and takes one less thing away that I have to worry about.Corey: One approach that we've always taken here has been that we treat people functionally like adults. And that's sort of an insulting way to frame it. Like, “What are you saying. That a lot of employers treat their staff like kids?” Well, basically, yes, is the short answer, where it's, “great, we're going to trust you with root in production, or a bunch of confidential customer data, but we're also not going to trust you to make a $50 purchase on the company credit card without a bunch of scrutiny because we don't trust that you're not embezzling.”The cross-incentives of different organizational structures are so twisted at that point that it's very hard to self-correct. I mean, our approach going into this was always never to go down that dark path, and so far, so good. Will it bite us someday if we continue to grow to that 50,000 person company? Undoubtedly. But I have to believe it can be done.Amy: I would also think the pressure of managing 50,000 people would break every single person in this company. Just thinking about it gives me panic.Corey: Oh, yeah. When every person becomes effectively, what, 500 people, that's divisional stuff. I don't think that anyone wants to stick around and see a small company go through those kinds of transformations because functionally, it becomes such a radically different place. For better or worse, that's not something we have to worry about, at least not anytime soon.Amy: We work really well with our fairly tight teams, I think. And it's I think it's one of the virtues of the type of work that we do. You don't need a team of 15 people looking at one document.Corey: No. And invariably, it seems like that tends to slow things down. Let's talk a little bit about something that makes you a bit of an outlier insofar as when you are only dealing with a small number of people everyone's inherently an outlier. You're the only cloud economist at The Duckbill Group with a background as a software engineer.Amy: Yeah. I did not realize that when I was joining. So, this deal with my infrastructure background is that I've only ever done enough infrastructure to support my applications. Like, Pete, I come from a startup background initially in my career, where you had to manage your application, you had to manage your own routes, you had to manage your own database connections, your own storage connections, and all of that, which, looking back on it and saying it out loud, sounds like a really bad idea now, but this was life before infrastructure as code and it was the quote-unquote, “Wild West” of San Francisco startups, where you can make a product out of basically anything. And the anything I went into was fantasy football of all things.So, I always made sure to have enough knowledge so that if I knew something broke, that I could blame it on networking, and then I would be able to show the paper trail to prove it. That was the extent of my knowledge. And then I started getting into the serverless space, where I started building things out of cloud services, instead of just spinning up more EC2 because I was young and impressionable. And that gave me a lot more understanding of what infrastructure engineering was like. But beyond that, I build APIs, I do business logic; that's really where my comfort levels are.Corey: For the rest of us, we seem to come from a background of grumpy Unix sysadmin types where we were running infrastructures, but as far as the code that was tied into it, eh, that was always the stuff we would kind of hand-wave over or we'd go diving into it as little as possible. And that does shape how you go through your career. For example, most companies are not going to wind up needing someone in that role until they've raised at least a Series B round, whereas in many cases, “Oh, you're an engineer. Great, why don't you start the company yourself?” From a software engineering perspective. It's a different philosophy in many respects, and one that I think is a little bit on the strange side if that makes sense.Amy: It is. It's extremely strange how completely dependent the two are on each other, yet the mindset to get into either and of that level of engineering is completely different. My husband and my father are both on the infrastructure side, and they've tried to explain networking to me my entire life, and I just—the minute the word subnet comes up, my brain is gone. It's like I'm replaying Star Trek episodes in my head because I can no longer handle this [laugh] conversation.Corey: That's how a lot of us feel about various code constructs at some point. For better or worse, we've made our peace with it, and let's be very direct here for a minute, we've learned to talk our way around customer questions that go too deep into the software engineering space, by and large. What's it like on the other side of that, where there's an expectation that you have a lot more in-depth infrastructure experience than perhaps you do? Or isn't there at that expectation?Amy: There is, I think a lot of that is just because of the type of industry this is. Cloud consulting is always infrastructure first because that is what the cloud is selling; they are selling managed infrastructures. They are giving you data center alternatives, but they're not giving you are full-blown apps. And whenever they do—let's say Lightsail—it's an expensive thing that you, somewhere in your mind go, “I could build that cheaper. Why am I paying for this service?”So, when I am on the phone with clients and they have a situation that is obviously going to be a software solution, where their infrastructure is growing, but it's because their software has a specific requirement, either for logging, or for surge, that they're using either Elasticsearch, or Kubernetes, or CloudWatch Metrics for, and it's turning into an expensive solution, it gives me an inside, “Well, this is the kind of engineering effort that's going to need to happen in order for you to write all of these problems and to reduce these costs. And these aren't as simple as hitting an option within AWS console to bring all of that down.” It's always going to be seen as more of an effort, but you also get a bit of empathy from the engineers you're talking to because you now are explaining to them that you understand what they built. You understand why they built it a specific way, and you're just trying to give them a path out.Corey: You mentioned the now antiquated idea of going on-site and talking to clients. I mean, before pandemic, Mike and I would head out to a lot of our clients for the final wrap-up meeting, or even in some cases kickoffs, because it made sense for us to do it and get everyone in the same room and on the same page. And over the past year, we've found ways to solve for these problems in ways that I don't necessarily know are going to go away once the pandemic is over. Is it more effective for us to travel somewhere and sit down in the same room with people, who in many cases have to travel in for wherever they live themselves? I don't know. There is going to be a higher bandwidth story there, of course, and the communication is going to be marginally more effective, but is it going to be so effective that it's worth more or less throwing a wrench into everyone's schedule for that meeting? That leaves me somewhat unconvinced.Amy: One of the strange things is that previously, I would go on-site to clients and fly out to where they are because as many startups as there are within Chicago, and the [unintelligible 00:13:10], and within Illinois, I'm always being sent to New York, or Atlanta, or Denver, for some reason because they're far and there're planes there. But we always end up having to talk to some amount of people that don't even work in that time zone, or maybe even then in this country, so we're talking to resources in Asia, resources in Europe, which meant we were flying people in to be on somebody else's phone. And I'm glad to not do that. I'm glad to not have to hang out in an airport, there is a burrito place in O'Hare that I truly enjoy, but that is the one thing I miss about traveling. I almost have my punch card done and it stopped right before then, but I'm kind of okay with that.Corey: I was chasing the brass ring of airline status and all the rest for a long time. I can't wait to finally go and hit the next tier and the rest, and where, well, the pandemic through all of that into a jumble and I take a step back and look at it and, you know, I don't miss it as much. What I do miss is that the opportunity, in my case, to get away from everyone that I spend all of my time with now, just for a day or two, and clear my head and recenter myself. But there are probably ways to do that that doesn't keep me on the road for 140,000 miles a year.Amy: I think, or at least I hope, that this will give us a chance to as an industry just reevaluate how we treat travel. A lot of clients treated it as essentially a status level that came with your engagement where we need you on-site so we can show off we have consultants coming in on-site so frequently to give us personal reports, even though we are all in a room on a conference call with other people. So hopefully, even if they're not forcing that every other week—sometimes weekly, depending on what your engagement is—type of cadence on travel, then maybe it'll just increase the quality of life for some of us. It would be nice. It would be super nice. I honestly don't see them forcing that anytime soon, but once everyone gets vaccinated, and there's a successful pediatric vaccine that comes out, it's like, I don't see them, just letting us stay at home and continue doing our job the way we have been for the past year, going on two years.Corey: So, dialing back into the mists of the distant past, it's always a question of where do cloud economists—or clouds economist, depending upon how we choose to mis-pluralize things—come from. And everyone here is a different story and there's not a whole lot of common points between those stories. You, for example, spent some time doing work with NASA. What was that about?Amy: There's a lot of misconceptions about working for NASA like you need to be a doctor. [laugh]. And trust me, you don't. I knew a lot of people who work there that they basically got their degree, and then they just did code work forever and they are lifers there. And it's such an interesting place to be because, on one hand, you have that mission of space and exploration and trying to do better by the world, but also, it's still a federal agency and there's still a lot of problems with federal agencies in that how you get paid is essentially at the beginning of the year, that's when all the budgets are done.So, you can't do the startup thing where you go, I'm going to try a bunch of things, and if one doesn't work, I'm going to pivot to something else because you're essentially answering taxpayers and they don't let you do that. No one wants their taxes going to someone who tried a thing and then found out they messed up. Which is unfortunate, but also a really hard reality of the way these work.Corey: I really love installing, upgrading, and fixing security agents in my cloud estate! Why do I say that? Because I sell things, because I sell things for a company that deploys an agent, there's no other reason. Because let's face it. Agents can be a real headache. Well, now Orca Security gives you a single tool that detects basically every risk in your cloud environment -- and that's as easy to install and maintain as a smartphone app. It is agentless, or my intro would've gotten me into trouble here, but it can still see deep into your AWS workloads, while guaranteeing 100% coverage. With Orca Security, there are no overlooked assets, no DevOps headaches, and believe me you will hear from those people if you cause them headaches. and no performance hits on live environments. Connect your first cloud account in minutes and see for yourself at orca.security. Thats “Orca” as in whale, “dot” security as in that things you company claims to care about but doesn't until right after it really should have. Corey: I must confess that I'm somewhat disappointed that you opened with, “You don't really need to be a rocket scientist to work at NASA,” just because, honestly, I was liking the mystique of, “Oh, yeah. You need to be a rocket scientist to understand AWS billing constructs.” But I suppose if I'm being honest, that might be a slight overreach.Amy: I knew a lot of people there who had multiple PhDs, and they could barely keep their computer on, so really, I'm finding that I respect very smart people, but it also does not imply your world intelligence anywhere else outside of that very specific field. One of the really weird things about having worked there—I worked at NASA Ames as part of their IT department—one of the things we did as outreach was, we did a booth over at SiliCon Valley Comic Con once, and it was great. We had a vintage display of old electronics, like CRT monitors, and full keyboards, and all of this nonsense, and kids would go, “These are so old. Why would anyone use this? It's so boring.” [laugh].And the entire IT department showed up to volunteer. We're like, “No, you don't understand why it's interesting. It's great.” It was so, so hard to watch young people just [laugh] not be interested in what the past of digital devices were. Very sad. And on the other hand, we did get a lot of interest, but it was also having to have that conversation in real life can be a little disheartening.Corey: It really is.Amy: But it was fun because it was one of the few things that you don't really get to see NASA at a comic book convention, so that was actually a really cool thing to do. Also, we got free tickets, so that was great.Corey: So, what was your background before you got to the point of, “You know what I want to do? Work for a consultancy, whose entire mascot is a platypus, and from there, go ahead and fix AWS bills,” which sounds like, to folks who aren't steeped in it, the worst thing ever? What series of, I guess, decisions led you here?Amy: I know you don't remember this, but we actually met at Serverlessconf, and you opened your talk with, “I am a cloud economist, a title I completely made up.” Your talk was right before mine, so that's why you didn't remember I was there because I was actually on the backstage getting prepared for my talk.Corey: That's right. I would have been breathing into a paper bag right before or right after my talk, trying not to pass out. People say, “Oh, you won't be nervous once you give enough talks.” I'm still waiting.Amy: It never happens. I did finally stop having blackouts, so that's an improvement. It gets better, but it never goes away. And when you told me that, and I saw the listing, I'm like, “I don't know what this job is. There's an easy way to find out what the job is, and that's to apply.”And that is when I started going through the process of applying, and then you hired me some months afterwards. And the thing that I found out about looking through AWS billing is that I found out I have a very specific skill set, in trying to find a discount while looking through receipts. This is a thing I thought only applied to my personal life because I don't really like paying retail prices for anything, so I'm always looking for a way to squeeze out another 20, 30% out of something because 10% is really just taking taxes off of stuff. And the fact that I was able to apply that very specific skill set to an actual technical job is so much fun for me because I like being able to tell stories out of what people spend, just because it is—as we say around here—it's the sum of all of your engineering decisions. Because everything you do, there's a price tag on it. And knowing how you got there, and that you can optimize the architecture by looking through the bill is super fascinating to me.Corey: So, now that it's been six months, is the job what you expected it to be? Is it something radically different? Is it something else?Amy: The tone of our engagements have actually changed within the past six months, partially because of the way AWS has made some organizational changes internally as far as we can tell. But really, it's also what types of companies are finding out that they need a service like this. Before, when I was interviewing, and when I started, I was talking to Pete and Jesse about the types of engagements you do. It was for larger companies; they're looking for some amount of savings, and then we run some tooling and then we get back to them. And now it's turning into… some are relatively small companies, companies that wouldn't get use out of an EDP, for example, because they're spending so low.But also other companies, they don't even really want this saving specifically, they want validation on what the process is like, they want validation on their unit economics and what the cost allocation strategies are. So, it's fascinating what people actually want now that they understand that that option is out there.Corey: One thing that you mentioned a minute ago, was the idea of going and giving talks—in the before times, at Serverlessconf and things like that—how do you find that that has changed for you over the past year? And how are you viewing a slow, but effectively guaranteed in the long enough timeframe, return to normalcy?Amy: So, when everything went virtual, it was a really hard transition for a lot of communities. I'm part of AWS Chicago, I'm an organizer at a meetup group called Write/Speak/Code where our whole deal was to give women and people of underrepresented genders the opportunity to learn how to do open-source, how do you learn how to do technical speaking, we helped them with their CFPs, and all of that required a physical community in order for us to be able to give each other that type of support. Well, we can't do that anymore. The last big event we did was specifically around getting everyone set up into the organization. So, we do one big one every year where we tell you how to do slides, we tell you how to do everything, and then the rest of the year smaller meetups where we do feedback and prompts and other types of support events.Now, that we can't do that, we found that a lot of the people who ran these events, they're extroverts and they're social to begin with, so they got burnt out very quickly, which meant we not only had to find new ways of supporting them but also reaching out to members and making sure everyone could still do the types of thing we promised we'd help them to do. The other issue is that with things going virtual, there used to be clear lines on the types of events you could apply to, which were events that I could reach, that my company would pay for, what have you. But now everything's virtual, I accidentally applied to a meetup that I spoke at that was based in Australia, which meant I was talking at 10 o'clock at night, just because I explained to them the mix-up and essentially begged to get an earlier slot. And it's interesting because it presents both a wide array of opportunities, but it also means that there's now so much noise, and so much burnout and fatigue from these one direction types of conferences, which are long Zoom meetings, which is basically everyone's workday now, which is just full day's worth of Zoom meetings. And it's hard to get people interested.And really, what these events did best was give people who generally don't have that type of visibility—like me—who, I am a female engineer, and I am a person of color, so my opportunities aren't always going to be as well as they could be, but this visibility also gave me a boost. So, when we lost out on physical events, my organization personally lost out on a lot of things that we could do for those people, which is unfortunate, but it's also what was happening, really, everywhere. Now, we're gearing more towards how to do less Zoom meeting type of events, and we're now using a tool called gather.town, which lets everyone go into a space, you can walk around, you can drop in and out of conversations like you would in a hallway, and it has this cute little eight-bit kind of avatar feel to it so it looks kind of like a game, but it's also—if you go around a large group of people, it pulls up everyone's picture and then you're suddenly speaking to each other over voice without having to physically join in or wait for a breakout room or wait to be let into a different room. So, it's been difficult to try to find ways of managing it, but it's also been very interesting seeing the tools that had come out of this.Now, once we start going back to physical events and on-site events, personally, I have a lot of anxiety about that just because my allergies and everything makes it hard for me to do travel in the first place, but also, I'm not really sure how everyone's going to react being in the same space anymore, or how full they're going to be. So, to me, it does cause me a little bit of anxiety and I'm going to wait a little until things are more settled and are a little more stable than they are right now. That said, I believe AWS Midwest is doing something physical at the end of this year. But I'm not entirely positive about that.Corey: One of the things I want to avoid is going back to an old style of re:Invent, where it's only open to folks who are able to spend $2,000 on a ticket, travel to Las Vegas, and get the time off and afford the hotel stay and the rest. One thing I loved about the 2020 version of that was that everyone was coming from a baseline of its full remote. There was no VIP ticket option that got you a better experience than anyone else had. And I do worry, on some level, that as soon as they can, they're going to go running back to a story like that. I hope not.Amy: I hope not, too, because one of the great things about virtual re:Invent was everyone saw the same things at the same time. And you didn't have to give up one panel because you're too busy being in line for another panel. And I've never liked that type of convention activity where you know you're actively giving something up just because the line for the thing you want to get to is so super long. That said, they could have done better a bit on the website. The website was really hard to navigate and confusing and the schedule was weird.If they get that kind of usability fixed and a little more reasonable so that things are surprisingly searchable and easier to navigate, and they have more social type of events instead of AWS is going to—it's going to announce another service that they're calling a product, even though it's just… it's a service enhancement, and that's all it is. I would like for all of that to kind of be streamlined so it's not just more announcements. I can read announcements. I don't need to watch two hours' worth of announcements.Corey: I really hope that at some point, some of the AWS service teams learn that, “Hey. We could announce services anytime of the year,” rather than in a three-week sprint that leaves no one able to pay attention because there's just too much.Amy: And it's hard because it's hard when you have to hold a release because re:Invent's coming up. Or you have to be part of their Developer Relations Group where you have to do all of your training and all of your docs right beforehand, just so that it's prepared for this launch that people may not hear about because it gets drowned out in all the other noise.Corey: And that's sometimes part of the entire problem.Amy: Yeah.Corey: Thank you so much for taking the time to speak with me. As always, it's a pleasure. If people want to learn more about what you're up to, where can they find you?Amy: They can hire me for engagement here. [laugh] but also—Corey: Good answer.Amy: —I do technical talks. I don't have anything lined up right now, just because it's spring in my brain took a break, like everyone else did. I'm on Twitter as @nerdypaws because that was a handle I had since college and have not changed.Corey: Excellent. And we will, of course, leave a link to that in the [show notes 00:31:10], as we always do. Thank you so much for taking the time to speak with me. It's always a pleasure, and it's deeply appreciated.Amy: It's always a good time talking to you, Corey.Corey: Amy Arumbulo Negrette, cloud economist here at The Duckbill Group. I am Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me why you do in fact need to be a rocket surgeon in order to properly work on AWS bills.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.This has been a HumblePod production. Stay humble.

new york amazon chicago australia europe las vegas work state san francisco phd zoom illinois nasa excellence cloud star trek vip yahoo economists slack wild west certificates puppets screaming aws apis crt devops invent kubernetes fantasy sports series b unix nasa ames research center edp elasticsearch ec2 cfps corey quinn lightsail silicon valley comic con amy it duckbill group orca security amy so chief cloud economist write speak code last week in aws amy they humblepod

The Operations of Operations with Jesse DeRose

Play Episode Listen Later Aug 3, 2021 31:38

About JesseJesse is a seasoned operations engineer with a deep passion for understanding complex technical and organizational systems. He's spent his career helping Engineering teams achieve their business goals by improving how they interact with their technical systems, and with each other. He's currently a Cloud Economist with Duckbill Group, guiding organizations along their journey of cloud cost optimization and management.Links: The Duckbill Group: https://www.duckbillgroup.com/ Jesse's Twitter: https://twitter.com/jesse_derose AWS Morning Brief: https://www.lastweekinaws.com/podcast/aws-morning-brief/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Your company might be stuck in the middle of a DevOps revolution without even realizing it. Lucky you! Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy in on DevOps practices? Well, download the 2021 State of DevOps report brought to you annually by Puppet since 2011 to explore the trends and blockers keeping evolution firms stuck in the middle of their DevOps evolution. Because they fail to evolve or die like dinosaurs. The significance of organizational buy in, and oh it is significant indeed, and why team identities and interaction models matter. Not to mention weither the use of automation and the cloud translate to DevOps success. All that and more awaits you. Visit: www.puppet.com to download your copy of the report now!Corey: Up next we've got the latest hits from Veem. Its climbing charts everywhere and soon its going to climb right into your heart. Here it is!Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Jesse DeRose, my colleague, and cloud economist at The Duckbill Group. Jesse, thank you for joining me, even though when I asked you, it isn't exactly like you felt you had much of a choice.Jesse: [laugh]. I appreciate being on this podcast with you. I think you've had an opportunity to talk to a few other folks from our organization so far, so I'm just happy to be included. I'm hoping that I get the Members Only jacket after this recording.Corey: Oh, absolutely. The swag that goes out to guests is secret and wonderful, all at the same time. So, let's start at the very beginning. It turns out that despite the prevalent narrative that we put out there, are clouds' economist did not just spring fully-formed from the forehead of some God, and appear—fully-formed—ready to slice AWS bills to ribbons. There's a process and there's always an origin story. Where do you come from? Where were you before this?Jesse: So, my background is mixed. I started with a management of information systems degree, which is inherently interdisciplinary. You're linking, sort of, the importance of both technical systems and business goals and outcomes into a single degree. And when I graduated with this degree, I went out into the working world and said, “Okay, I'm here. I'm ready. Here's my degree. Here's what I'm interested in. Let's do this.”And nobody knew what to do with me, Corey. Nobody knew what management of information systems was. They simultaneously said, “You know, you don't have a computer science or a computer engineering degree, so we don't believe that you've got programmer chops.” Especially since this is the golden age of boot camps and quickstart programming books, and this movement that anybody can be a developer, which makes it even stranger that I'm not spending my weekends learning every programming language under the sun and automating the little tasks.Corey: Oh, absolutely. In fact, I got the exact same feedback; I don't have a degree, and, “Oh, you couldn't possibly be a good programmer because you don't have the degree, you didn't go to the right schools, you didn't go to a boot camp. And when I asked you to write some sample code to demonstrate how to program, you just started crying instead.” Because yeah, it turns out, I'm actually not a good programmer, I just, sort of, brute force my way through it. And so far, so good.Unfortunately, people aren't paying me to program these days, for excellent reason. In fact, I strongly suspect some people are paying me not to. But yeah, now it's funny to laugh about, but back when you're getting started, and going out in that space, in the world of operations, which we both came up in, and looking at it through a lens of the SRE movement, suddenly, “Hey, you used to be really, really good at all these Linux things and working on systems and keeping them up. Great. Now, learn to code.” And that was a big lift, at least for me.Jesse: Absolutely. I struggled with that so much because every company that I interviewed with, every company that I even talked to, just assumed that I had some kind of programming experience and didn't want to talk to me if I didn't have programming experience. And to me, looking back, I think I really look at it like I may not have the programming chops to be the software engineer that is going to write the code for you, day in and day out, but I am the person who knows enough that I can have the conversation with the software engineers. I can be the SRE, I can be the ops person that has the conversation with your software engineers and knows the things that they're talking about, but also knows when to get out of their way and let them be the expert and do what they do best.Corey: There's something to be said for valuing expertise in areas that are—how to put this—not the thing that you think you're looking for. I mean, back when I was getting jobs, before The Duckbill Group and I would be the first ops hire into a team of developers—which happened a few times—the process was always the same, where you'd have a bunch of developers asking what they thought were ops questions, or just giving up on that entirely and trying to figure out how decent have an ops person I would be by how badly I programmed.Or, “Oh, okay, cool. You're an ops person. Great. Can you invert a binary tree on a whiteboard?” It's, “No, but I can invert a rack in your data center. I'll go rage-flip the rack. Why not?” And it takes time. You have to guide those interviews and those conversations. But it's always weird.] interviews are always weird because you're being judged on a skill set that only matters when you're interviewing for a job.Jesse: Yes. This is one of the things that I struggled with the most because I knew that most of the people who were interviewing me were either business people themselves—so they assumed that because their engineering team thought a certain way and acted a certain way that I should act that way—specifically, too—the same way as everybody else on the team. Or they were software engineers themselves, so they said, “Okay, I know how to invert a binary tree”—to your point, Corey—“So, do how to invert a binary tree. If you know how to do that, then sure, you can be part of my team because you know how to do these things and think about these things the same way I do.” Whereas because I was coming from this operations space, I knew other things that were equally as important but weren't part of the conversations that they were used to having, day-to-day.So, they didn't understand that just because I didn't have the same engineering chops as them, that I didn't have important information to share and wasn't able to stand on my own two feet in other ways. And that was one of the things that I really struggled with when I was starting out in the industry because I was thinking to myself, well I have such passion to be part of these conversations, to have that conversation between the business side of the organization and the engineering side of the organization, from an ops perspective, from a business perspective, from a technical perspective, and if I can't convince these people of my own volition, of my own passion for the good of the company, maybe data will help. Maybe there's something that I can find from a scientific research perspective. Maybe there's something that—I'm sure somebody else has already researched this topic or found the same problems that I have in this space, and maybe they're already talking about it, and maybe I can ride their coattails, so to speak, or follow in their footsteps and use the information that other people are talking about the industry to help me not just land these jobs, but ultimately better sell myself and help these companies move forward.Corey: That is fundamentally an encapsulation of what I believe the ops role to be of, make things better, and move them forward. But man, do we get stuck in an awful lot of weird and strange places. And interviewing itself is a skill. Giving an interview, very often, it's a, “I know a bunch of things that are trivia, but I know them. And if I know them, everyone must know them, therefore, if you don't know them, you must be bad at things.” And it turns out—for better or worse—being able to memorize the documentation and spit out answers is not indicative of whether someone is a good ops person or a terrible one.Jesse: Absolutely. I think that is one of the biggest problems that I have faced and one of the biggest problems in the interviewing space today because it's not just about, can you regurgitate this information, but it's about how do you think? How do you look at problems? How do you communicate to the rest of your team, and within the rest of the organization? Those technical skills are ultimately important because you do need to understand some amount of technical information to have those conversations, but the soft skills are also super, super important to be able to communicate effectively, to be able to think collaboratively, and help everybody, not just yourself, but help the team build that shared purpose and move forward together.Corey: We're talking right now, so far, about traditional ops roles. Then we have what we do here, which is beyond the rest of all of that, where all of what we just said is necessary but not sufficient. Then it comes down to great, okay. So, you understand how systems work together; we found that, for what we do and how we do it, you need to be a competent ops person as a fundamental tenet.Otherwise, learning what all these AWS services do will occupy you for the next three years. So okay, we start off there. Then on top of that is, okay, there are consulting skills that it turns out are possible to teach, but incredibly challenging and time-consuming because a lot of them boil down to, can you be in a meeting with stakeholders of various levels? Can you deliver bad news in a way that they don't hate you? Because they don't really want to pay you just say yes to whatever they think.And can you do that in such a way without, you know, actively insulting them, which sounds like a strange thing until you realize, oh, wait, that's right. I do that, too. So ooh, yeah. Corey is going to have that problem, isn't he? Yeah. And that's part of the beautiful part about this place is that finally, I'm able to hire people like you.You were the first cloud economist here, which meant suddenly I didn't have to do it all myself and my mouth slowly stopped getting me in trouble in consulting engagements, so I could spend more time having my mouth getting me in trouble on Twitter.Jesse: Yeah, I have to tell you, Corey, when I originally spoke with you and Mike about this role, I had just taken another operations position with a tech startup, and I was about two months into the role. And Mike sat down with me for coffee one morning and said, “Hey, we're thinking about doing this thing. Are you interested?” And I said, “Yes, but I just started this other operations gig. I can't up and leave them; I really care about the team, I really care about the company. And it would look really bad on me if I just, you know, two months left.”Because—unless they were a really, really awful employer, which they weren't. So, I said, “Sure. I'm interested in doing some kind of part-time work.” And that's ultimately where I started with you and your business partner, Mike. And I have to admit to when Mike originally approached me and said, “Hey, this is what we are thinking about; this is what we're doing,” I didn't really think twice about the opportunity because I wanted to work with you and Mike again.But the way that Mike described the work, just didn't stick with me. It didn't resonate with me, it was more about, “Hey, I would love to work with Mike and Corey again,” than, “Oh, my God. This sounds like the dream role that I want to be a part of.” And then, when I came back to Mike, probably, I don't know, a month or two later, after I had started working part-time with both of you, I said, you know, “Mike, I don't think I really made myself clear. I want to make sure that I help you understand, ultimately, the things that I want to do are having these conversations, being that bridge with the business side, and being able to talk tech with the tech side, and being able to talk business, and make sure that both sides of the conversation are aligned.”And he just looked at me and said, “What do you think we're doing? What do you think we sold you on?” And it was that aha moment where I thought, “Oh, my god, yes.” I had already said yes; I was already working with both of you part-time, but that was the moment that really solidified it for me of, this is what I've kind of been moving towards. I've been wanting to be that person that can speak both languages and have a conversation with both sides of the table, and speak to multiple different audiences, and now I'm finally getting the chance to do that. I'm getting the chance to grow both skill sets, which I think is extremely rare in a lot of the smaller tech spaces that we see today.Corey: You've hit on one of the secrets of The Duckbill Group if I can be so grandiose as to claim that. And it's true because we take a look at people that we bring in, and things that they're good at, and things that we do—the things we do publicly and the things that we do, sort of, behind the scenes and there's no reason we don't talk about them publicly, but there's no real reason for us to do so. Easy example, and what I want to talk to you about next is, you are deep into improving understanding of complex systems, both technical—okay, great people expect that—and organizational, which sometimes throws people for a loop. And it sounds like a weird thing to focus on here because we fix the AWS bill. We do not bill ourselves as management consultants, we do not bill ourselves as coming in and we will restructure your organization because that sounds patently ridiculous, and no one in their right mind is going to buy that thing.I wouldn't buy it, at least not for me. My God, there are large consultancies that specialize in these sorts of things. I don't know how they do it because I certainly don't. We're not here to sell that, though. Fixing the AWS bill—I mean actually fixing it. Fixing the business problem tied to it mandates an understanding of those complex systems. And your expertise and interest in that area is incredibly helpful here. Tell me more about it.Jesse: This is one of the things that I've been really fascinated by ever since I joined Duckbill Group. I think everybody in Duckbill Group has a superpower or has a really passionate hobby to some extent, which makes each of us really interesting, unique individuals that can focus in different areas of a client's bill or a client's pain points when it comes to cloud cost management and help, and find the parts that are frustrating, find the levers that can be moved, and point them out and say, “Okay, this is ultimately where you want to pull this lever or not pull this lever to make these changes.” And to your point, Corey, the one that is most interesting and passionate for me is that organizational development space. It is really understanding, not just the small things that we can do today to help you save money on your AWS bill, but how we, and collectively how our clients can think about costs long term to save money on their AWS bill. And I know that sounds really, really broad, and that's part of why I think that there is a lot of nuance in this space, to your point about other organizations or other vendors that are providing these consulting services, and I think is also something that is also difficult to sell, which is why it's not our expertise in terms of what we are on the cover trying to sell to any of our clients.But I definitely think that there is opportunity to have some of those conversations within each of our clients' spaces to talk about some of the pain points that we see that may ultimately lead to better cost management practices long term, things that ultimately might help the engineering teams communicate better with finance on a long term basis, help the finance team and any of the leadership team more collaboratively talk with the engineering teams about understanding how much money is the product costing us? How much can we continue to spend on this product? Or how much can we discount one of our products for our customers before we are losing shares, losing money? What are the fine lines that we understand, based on how much money we're ultimately spending on these features, on these products, that will help us make better data-driven decisions about other parts of the company?Corey: I really love installing, upgrading, and fixing security agents in my cloud estate! Why do I say that? Because I sell things, because I sell things for a company that deploys an agent, there's no other reason. Because let's face it. Agents can be a real headache. Well, now Orca Security gives you a single tool that detects basically every risk in your cloud environment -- and that's as easy to install and maintain as a smartphone app. It is agentless, or my intro would've gotten me into trouble here, but it can still see deep into your AWS workloads, while guaranteeing 100% coverage. With Orca Security, there are no overlooked assets, no DevOps headaches, and believe me you will hear from those people if you cause them headaches. and no performance hits on live environments. Connect your first cloud account in minutes and see for yourself at orca.security. Thats “Orca” as in whale, “dot” security as in that things you company claims to care about but doesn't until right after it really should have.Corey: And I want to call out that this is something that we are comparatively enthusiastic amateurs around. It's valuable; it's important; it's an awful lot of deep work, but I'm not sure that we go more than three working days without referencing Dr. Nicole Forsgren's work, internally, as we think about these things. So, if you're hearing this, and you think that okay, AWS bill, fine, whatever. We really want to talk about organizational challenge and improvement, oh, my God, talk to Dr. Forsgren. Holy crap. She's been on this show, at least I think, three times now, and every time I feel like I'm lucky to get her. Most weeks, you know, I'm stuck with people like you. My God, Jesse.Jesse: [laugh].Corey: But no, her work is seminal in this space. And in seriousness, every time I start to question the value of expertise, I look at how deep she goes on all of these things and the level both of understanding that's baked into this, and the amount of sheer work that it takes for her to take all of that very deep, penetrating analysis, and make it accessible and understandable. But every time I look at her work, I come away more impressed than I started, and that wasn't a low bar, to begin with.Jesse: Yes. And this gets back to my earlier comment about, I just want to be here to help. And in a lot of cases, when I was starting out, folks didn't know what to do with me because I didn't have data, I didn't have any information. But we have folks in the industry like Dr. Nicole Forsgren, and other folks who are doing the research, who are knowledgeable in this space, who are putting in the effort to run these studies, to analyze this data, to share the results—and to your comment, Corey—to share the results in a way that makes sense to everybody, that's easy to read, it's approachable, it's understandable.And I am so thankful to have folks in the industry who are doing that work because that is not my expertise. But that means that I get to say to the folks that I'm working with—internally and with our clients—“Hey, don't take my word for it. There are other folks who have done research, and here's what the data says, and here's how we can help you apply this work, or you can apply this work within your own organization.”Corey: And this is the challenge in some cases, too, where there's a lot of organizational theory, and that is being advanced heavily, and in ways that makes teams more effective is super helpful. The challenge, of course, is that sitting here and talking about the theoretical layout of teams and how to improve functioning as an organization is all well and good, but we're brought in by our clients to help them with their AWS billing situation, so at some point the conversation has to evolve beyond, “Okay, so here's what you could do in theory, in a vacuum, assuming spherical cows, et cetera, et cetera.” And their response is, “That's great. You actually going to fix the bill or just pontificate for a while here?” So, for better or worse, we don't really get to sit there and have deep organizational conversations at length with our clients, just because that's not the problem we're there to solve. Everyone's busy and we want to make sure that we're respectful of their time.Jesse: Yeah, one of the things that I've learned through my time with Duckbill Group and through other similar roles in the past, is that I may have a strong passion, I may have this strong guiding light in my head, but it's not the same guiding light that our clients or our customers have. And that's fine because we don't need to necessarily have the same goal in sight, but that means that I, to best serve our clients or best serve our customers, need to make sure that I am aligning, that Duckbill Group is aligning with the clients that we're working with, with the organizations that we're working with. So, I may be thinking to myself, “Oh, my gosh, I would love to come in with this long list of organizational development practices and share a million different things,” but that's not ultimately what they need. Maybe that's something that they're going to want long-term, but it's not what we're here for today. And it's more important to help serve the client that is in front of me that is asking for things now, today, than try to educate them on quote-unquote, “What their problem is,” and then sell them on a solution.Corey: The thing that I think gets lost as well, whenever I start talking in-depth about what we do on Twitter, for example, is generally from other engineers whose response is, “Okay, yeah, sounds great. You come in and say that it will save a bunch of money before you rearchitect our application. But that's an awful lot of engineering time, so I bet engineers most hate you.” And the honest answer is, “Yeah. We know that you would save some significant money if you rearchitected your application, but we also pay attention to organizational dynamics, and we know you're not going to do it because there's no business justification for doing it. So, we're not going to bring it up, other than, possibly, in passing, just so people are aware of the relative benefit if they want to bake that in.”But we don't go in and suggest nonsense that is abhorrent. We all started as engineers ourselves. We are sensitive to engineering time, both in terms of what engineers enjoy working on—which is more important than people think—as well as the sheer cost of engineering time. People get concerned about the AWS bill, but it's invariably pale compared to the cost of the people working on the AWS infrastructure. You want to optimize the right things, and then you want to get back to doing what your company does, not continue to iterate forward and spend thousands of dollars to save tens of dollars.Jesse: Yeah, it's an extremely difficult balance to find. And it's really important to think about, is the ROI on this change worth the change? How much money am I ultimately going to save for the amount of engineering effort that I'm going to put in? Because we don't want to run in and tell your engineering teams, “Rearchitect everything,” if it's going to maybe save them tens of dollars. We want to make it very clear that here are the different levers that you can pull to affect change within your AWS bill, to optimize your AWS bill.It's up to you which ones you want to pull. We are just giving you that guiding path, and then you have the option to say, “Yes, I want to spend the engineering effort to get this kind of ROI.” Or, “You know what? I don't think that's the priority for us right now.” One of the things that I've noticed with a number of our clients is that balance of, do we focus more time on building new features which brings in new customers, or do we spend time on the existing infrastructure and making changes to the existing workloads that we have?And it's this delicate balance of internal work—the things that you have put on the backburner over time that you ultimately say, “Well, I'll come back to that,” versus the new things that are ultimately going to bring in new customers, bring in new users, potentially bring in more money. Because both are important, but there needs to be a delicate balance of both, and I feel like that's one of the biggest challenges that we face when managing an AWS bill and trying to optimize, and organize, and better manage cloud costs.Corey: And that's fundamentally what it comes down to what is the best outcome for the client? And the answer to ‘what is best?' Varies based upon their constraints, what they're focused on, what they're trying to achieve. You could look at the end result of one of our analyses for a customer, and take issue with it in a vacuum of but they're spending all this money on things that they shouldn't be doing, or don't need to be. Why didn't you suggest this, or this, or this, or this?And the answer is because, based upon our conversations, we knew that they weren't going to do it, and suggesting things that we know they're not going to do is one of the best ways to erode trust. We're there to deliver an outcome. We're not naive software that is just running a pile of tools on an Amazon bill and saying, “Here you go. Have fun.” We're not the billing equivalent of a Nessus scan that someone slaps their logo on top of, drops off, then it's 700 pages long. “Have a good one. Check, please.” It doesn't work. Not well, anyway. It doesn't drive to lasting change.Jesse: Yeah. And I think that's ultimately part of where we come in best because we ultimately want to be that bridge between the engineering teams, and finance, and leadership, and essentially the business side of the business. We want to give both sides the information that they need to be able to speak effectively and collaboratively with the other side. We want to make sure that the finance side understands enough tech that they can work with the engineering teams to guide them in terms of what goals are important for finance, in terms of managing budget, in terms of forecasting spend. And then from the flip side, we want to make sure that engineering understands that the business collaboratively wants to manage these things, and also help build the organization, and engineering has great, great potential to do little things, take little steps to help the organization get the data that they need to make these decisions.And that scratches that itch for me. That scratches that itch of how can I really be both sides of the conversation? How can I flex the business lingo and also flex the tech lingo, maybe not in the same conversation but with the same client over time, to really help both sides understand that both sides are important, and both sides need to understand each other in order to help the business succeed?Corey: And that's ultimately what it comes down to. Jesse, thanks for taking the time to speak with me. If people want to hear more about what you have to say, and how you like to say it, where can they find you, other than go into The Duckbill Group and get me a consulting engagement underway?Jesse: So, there's two places that folks can find me. The first is on Twitter at @jesse_derose. And then the second is our other podcast, the AWS Morning Brief podcasts, on the mornings where we don't get to hear your melodious voice, Corey.My colleague, Pete Cheslock and I are talking about all of the interesting things that we've seen on AWS, from client-specific situations to things that we've seen on the job in previous organizations that we use to work at, to new features that AWS is releasing. There's a whole slew of interesting things that we get to talk about from a more practical perspective. Because there's so many releases coming out day-to-day that you're obviously helping us stay on top of, Corey, but there's so many other things that we want to make sure that we are talking about from the real-world applicable perspective.Corey: And we will, of course, put links to these things in the [show notes 00:27:48], but you should already be aware of most of them, at least the ones that are on the company side. Jesse, thank you so much for taking the time to speak with me.Jesse: Thank you for having me.Corey: Jesse DeRose, cloud economist here at The Duckbill Group. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an insulting comment telling me that organizational dynamics really aren't that hard and you could solve it better than Dr. Forsgren does, in a weekend.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.This has been a HumblePod production. Stay humble.

god amazon giving state holy operations engineering cloud roi fixing puppets screaming aws linux my god devops sre members only veem derose varies corey quinn nessus nicole forsgren duckbill group orca security chief cloud economist jesse yeah last week in aws pete cheslock humblepod

Optimize Yourself Before You Invest Yourself

Play Episode Listen Later Jul 30, 2021 13:35

Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: And I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways that we've seen AWS used and abused in the wild. Today, we're going to be talking about the relationship between cost optimization work and investing in reservations or private pricing with AWS. This is kind of a situation conversation. Let's say you've got three months left on your EDP, or maybe your spend is reaching the point where you're starting to think about investing in, or signing an EDP. But you've also got some cost optimization opportunities that you want to work on. How do you prioritize those two ideas?Tim: I think when we're talking about this, first it's important to talk about what goes into an EDP, like, what it is and what it involves. So, EDP for AWS is Enterprise Discount Program, and what it involves is you making a monetary commitment to AWS to spend a certain amount over a certain amount of time. So, a three year EDP, you're going to spend X amount in one year, X amount the next year, and X amount the third year for a total of whatever you decide on. So, you know, AWS typically going to want 20% year-over-year growth, so you're going to say—you're going to spend a million dollars, and then a million dollars plus 20% is something like $1.2 million; then, you know, 20% of that and so forth and so on.And then so your total commit will be somewhere around, like, $3.6, $3.7 million, we'll say, right? Once you signed the EDP, that's how much you're going to get billed for, minimum. So, it's important to cost optimize before you make that commitment because if AWS is expecting you and you're on the hook to make 20% year-over-year growth, but then you optimize and you save 20% of your bill, it won't matter because you're still going to owe AWS the same amount of money even if you cost-optimize.Jesse: Yeah, I want to take a step back and talk about EDP—as we mentioned, Enterprise Discount Program—also has—there's a couple other flavors that give you a variety of different types of discounts. EDP generally focuses on a cross-service discount for a certain annual commit, but there are also private pricing agreements or private pricing addendums, and other private pricing, generally speaking, offered by AWS. All of those basically expect some amount of either spend on a yearly basis or some amount of usage on a yearly basis, in exchange for discounts on that usage. And really, that is something that, broadly speaking, we do recommend you focus on, we do recommend that you invest in those reservations, but it is important to think about that—I agree—I would say after cost optimization work.Amy: The thing is that AWS also provides discounts that are commandment required, that you don't need an EDP for, namely in reservations and savings plans. So, you would similarly be on the hook if you decide, “I have this much traffic, and I want to savings plan or reservation for it.” And then suddenly you don't have that requirement anymore, but you still have to make up that commitment.Tim: I'll say, I think too, that also matters when you're looking at things like reservations. If you're going to reserve instances, you're going to get an idea of how many you're specifically going to need, so that way you're not reserving too many, and then you optimize, you downsize, and all of a sudden, now you have all these reservations that you're not going to use.Jesse: One thing to also call out: when renewing an EDP, or private pricing, or when entering into a new agreement for any kind of private pricing with AWS, they will generally look at the last six months of your usage—either broadly speaking if it's an EDP, or specifically within a specific AWS service if it's private pricing for a specific service—and they will double, basically, that spend over the last six months and expect you to continue spending that. So, if you spent a high amount of money over the last six months, they're going to expect that kind of trend to continue, and if you enter into an agreement with that 12-month spend, essentially, going forward, and then make cost optimization changes, you're ultimately going to be on the hook for this higher level of spending you're not spending any more. So, if you focus on that cost optimization work first, it will ultimately give you the opportunity to approach AWS with a lower commit level, which may ultimately mean a lower tier of percentage discount, but ultimately, then you're not on the hook for spend that you wouldn't otherwise be spending.Tim: I think one of the main things people see, too, is when they've looked at, like, oh, what's the low hanging fruit for me to get lower the cost? They'll think, “Oh, well, I can do EDP,” because AWS is going to want you to sign on; they would love to have that guaranteed money, right? And a lot of times, that's going to be a much easier thing to do, organizationally, than the work of cost optimization because almost always, that involves engineering hours, it involves planning, it involves some changes that are going to have to be made that's probably going to be harder than just signing a contract. But again, it's super necessary because you really need to know, have eyes open, when you're going to go, and figure out what you're going to commit, whether it's private pricing agreement, or an EDP, or reservations. You want to go in there and at least decide what you want to do, what it should look like, get as optimized and as lean as you can, then make your commitments. And then once you get to an EDP, that's when you're going to want to do your reservation or savings plans purchases and things like that, so you do that with a discount across those.Jesse: Yeah, that's another important thing to point out: focus on the cost optimization work first. Get your architecture, your workloads, as optimized as possible, or as optimized as you can within the given timeframe, then focus on the investment because then you'll be able to have a much better idea of what your growth is going to look like year-over-year for an EDP or any kind of private pricing. And then after that, purchase any reservations, like reserved instances or savings plans because ultimately, then you get not only the discount from the EDP that you just signed, but any upfront payments that you make, or partial upfront payments that you make for those reservations applied towards your first year EDP. So ultimately, not only are you getting a discount on that, but you are also able to put money towards that first-year commit; you're essentially giving yourself a little bit more wiggle room by purchasing reservations after you've signed an EDP.Tim: And another way to game that system is if you know that you're going to be undertaking some projects, especially that you want to get discounts around, and you're going to need to utilize software or service or anything like that involves an AWS partner on the AWS marketplace, you're going to want to do that after you sign your EDP, too, because even though you may not get a discount on it, that money will still count towards your commit.Corey: I really love installing, upgrading, and fixing security agents in my cloud estate. Why do I say that? Because I sell things for a company that deploys an agent. There's no other reason. Because let's face it; agents can be a real headache. Well, Orca Security now gives you a single tool to detect basically every risk in your cloud environment that's as easy to install and maintain as a smartphone app. It is agentless—or my intro would have gotten me in trouble here—but it can still see deep into your AWS workloads while guaranteeing 100% coverage. With Orca Security there are no overlooked assets, no DevOps headaches—and believe me, you will hear from those people if you cause them headaches—and no performance hits on live environment. Connect your first cloud account in minutes and see for yourself at orca dot security. That's orca—as in whale—dot security as in that thing your company claims to care about but doesn't until right after it really should have.Tim: It is important to talk about the future goals for your company, from a financial perspective, both at an architectural level but also at a strategic level, so you can make good quality decisions. And, you know, to toot our own horn, that's a lot of where our expertise comes in, where we can say, “These are the order you're going to do these things in, and these are what you should prioritize.” I mean, everyone knows that in the end, the net result should still be the same. You're going to have to do the engineering and architecture work to optimize; you're going to have to do the administrative stuff to sign these agreements to get discounts, but you need to know what to prioritize and what's going to be most important, and sometimes you don't have the insight on that. And that's where if you don't, get someone in there to help you figure out what's what, what's going to give you the best, most bang for your buck, but also what's going to make the most sense for you going forward, six months, a year, two years, three years, and so forth and so on. So, it is okay to not know these things. Nobody's an expert on everything, but it behooves you to rely on the people who are experts when it's a blind spot for you.Jesse: I think that's a really good point that you make, Tim. One of the things that we see in a number of organizations that we work with is essentially a disconnect between the folks who are—well, two disconnects really: one between the folks who are doing the work day-to-day, and another between the folks who are purchasing reservations. But that also a disconnect between the people who are purchasing the reservations and potentially the people who are purchasing or investing in some kind of Enterprise Discount Program or private pricing. And to Tim's point, it's really important to get all of those people in a conversation together, get everybody in a room together, so to speak, to make sure that everybody understands what everybody else is doing so that finance and engineering and product and leadership all understand together that the cost optimization work is going on, that reservations are being purchased, that we're having a conversation about investing in some kind of private pricing with AWS. So collaboratively, collectively, everybody can make a decision together, make a data-driven decision together, that's going to ultimately help everybody, essentially, win and accomplish their goals.Amy: Speaking of collaboration, we often talk about having a good relationship with your AWS account manager, and this is one of those places that having a good rapport really works in your favor because if you are in a lot of communications with your account manager, and you know each other well, and you have a good working relationship, and they are good at their job, then they'll know that you are using XYZ service, and you're using at a high volume, they will be able to tell you, it's like, “Hey, you hit a threshold. Let's see if we can get you some extra discounts.” They'll be the ones who can actually know what those discount programs are and be able to facilitate them.Jesse: All right, well, that will do it for us this week, folks. If you've got questions you'd like us to answer please go to lastweekinaws.com/QA; fill out the form and we'll answer those questions on a future episode of the show. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us how you would cost-optimize your organization.Announcer: This has been a HumblePod production. Stay humble.

amazon discover field invest cloud optimize api aws qa devops elk xyz s3 equifax klarna blackboard elasticsearch edp edb tim it orca security elastic cloud jesse yeah last week in aws humblepod jesse all

Analyzing Analysts with James Governor

Play Episode Listen Later Jul 29, 2021 41:00

About JamesJames is the Redmonk co-founder, sunshine in a bag, industry analyst loves developers, "motivating in a surreal kind of way". Came up with "progressive delivery". He/HimLinks: RedMonk: https://redmonk.com/ Twitter: https://twitter.com/MonkChips Monktoberfest: https://monktoberfest.com/ Monki Gras: https://monkigras.com/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of Cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Your company might be stuck in the middle of a DevOps revolution without even realizing it. Lucky you! Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy in on DevOps practices? Well, download the 2021 State of DevOps report brought to you annually by Puppet since 2011 to explore the trends and blockers keeping evolution firms stuck in the middle of their DevOps evolution. Because they fail to evolve or die like dinosaurs. The significance of organizational buy in, and oh it is significant indeed, and why team identities and interaction models matter. Not to mention weither the use of automation and the cloud translate to DevOps success. All that and more awaits you. Visit: www.puppet.com to download your copy of the report now!Corey: And now for something completely different!Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by James Governor, analyst and co-founder of a boutique analysis shop called RedMonk. James, thank you for coming on the show.James: Oh, it's my pleasure. Corey.Corey: I've more or less had to continue pestering you with invites onto this for years because it's a high bar, but you are absolutely one of my favorite people in tech for a variety of reasons that I'm sure we're going to get into. But first, let's let you tell the story. What is it you'd say it is that you do here?James: We—industry analysts; we're a research firm, as you said. I think we do things slightly differently. RedMonk has a very strong opinion about how the industry works. And so whilst there are plenty of research firms that look at the industry, and technology adoption, and process adoption through the lens of the purchaser, RedMonk focuses on it through the lens of the practitioner: the developer, the SRE, the people that are really doing the engineering. And so, historically IT was a top-down function: it required a lot of permission; it was something that was slow, you would make a request, you might get some resources six to nine months later, and they were probably the resources that you didn't actually want, but something that was purchased from somebody that was particularly good at selling things.Corey: Yes. And the thing that you were purchasing was aimed at people who are particularly good at buying things, but not using the things.James: Exactly right. And so I think that RedMonk we look at the world—the new world, which is based on the fact there's open-source software, there's cloud-based software, there are platforms like GitHub. So, there's all of this knowledge out there, and increasingly—it's not a permission-free world. But technology adoption is more strongly influenced than ever by developers. That's what RedMonk understands; that's what makes us tick; that's what excites us. What are the decisions that developers are making? When and why? And how can we tap into that knowledge to help everyone become more effective?Corey: RedMonk is one of those companies that is so rare, it may as well not count when you do a survey of a landscape. We've touched on that before on the show. In 2019, we had your colleague, Rachel Stevens on the show; in 2020, we had your business partner Stephen O'Grady on, and in 2021 we have you. Apparently, you're doling out staff at the rate of one a year. That's okay; I will outlast your expansion plans.James: Yeah, I think you probably will. One thing that RedMonk is not good at doing is growing, which may go to some of the uniqueness that you're talking about. We do what we do very well, but we definitely still haven't worked out what we're going to be when we grow up.Corey: I will admit that every time I see a RedMonk blog post that comes across my desk, I don't even need to click on it anymore; I don't need to read the thing because I already get that sinking feeling, because I know without even glancing at it, I'm going to read this and it's going to be depressing because I'm going to wish I had written it instead because the points are always so pitch-perfect. And it feels like the thing that I struggle to articulate on the best of days, you folks—across the board—just wind up putting out almost effortlessly. Or at least that's how it seems from the outside.James: I think Stephen does that.Corey: It's funny; it's what he said about you.James: I like to sell his ideas, sell his work. He's the brains and the talent of the operation in terms of co-founders. Kelly and Rachel are both incredibly smart people, and yeah, they definitely do a fantastic job of writing with clarity, and getting ideas across by stuff just tends to be sort of jumbled up. I do my best, but certainly, those fully formed, ‘I wish I had written that' pieces, they come from my colleagues. So, thank you very much for that praise of them.Corey: One of the central tenets that RedMonk has always believed and espoused is that developers are kingmakers, to use the term—and I steal that term, of course, from your co-founder's book, The New Kingmakers, which, from my read, was talking about developers. That makes a lot of sense for a lot of tools that see bottom-up adoption, but in a world of cloud, where you're seeing massive deals get signed, I don't know too many developers out there who can sign a 50 million dollar cloud services contract more than once because they get fired the first time they outstrip their authority. Do you think that that model is changing?James: So, ‘new kingmakers' is quite a gendered term, and I have been asked to reconsider its use because, I mean, I don't know whether it should be ‘new monarchmakers?' That aside, developers are a fundamentally influential constituency. It's important, I think, to say that they themselves are not necessarily the monarchs; they are not the ones sitting in Buckingham Palace [laugh] or whatever, but they are influences. And it's important to understand the difference between influence and purchase. You're absolutely right, Corey, the cloud is becoming more, like traditional IT. Something I noticed with your good friends at GCP, this was shortly after the article came out that they were going to cut bait if they didn't get to number two after whatever period of time it was, they then went intentionally inside a bunch of 10-year deals with massive enterprises, I guess, to make it clear that they are in it for the long haul. But yeah, were developers making that decision? No. On the other hand, we don't talk to any organizations that are good at creating digital products and services—and increasingly, that's something that pretty much everybody needs to do—that do not pay a lot more attention to the needs and desires of their developers. They are reshoring, they are not outsourcing everything, they want developers that are close to the business, that understand the business, and they're investing heavily in those people. And rather than seeing them as, sort of, oh, we're going to get the cheapest possible people we can that have some Java skills and hope that these applications aren't crap. It may not be Netflix, “Hey, we're going to pay above market rate,” but it's certainly what do they want? What tools do they want to use? How can we help them become more effective? And so yeah, you might sign a really big deal, but you still want to be thinking, “Hang on a minute, what are the skills that people have? What is going to make them happy? What do they know? Because if they aren't productive, if they aren't happy, we may lose them, and they are very, very important talent.” So, they may not be the people with 50 million dollars in budget, but their opinion is indeed important. And I think that RedMonk is not saying there is no such thing as top-down purchasing anymore. What we are saying is that you need to be serving the needs of this very important constituency, and they will make you more productive. The happier they are, the more flow they can have, the more creative they can be with the tools at hand, the better the business outcomes are going to be. So, it's really about having a mindset and an organizational structure that enables you to become more effective by better serving the needs of developers, frankly. It used to just be the only tech companies had to care about that, but now everybody does. I mean, if we look at, whoever it is: Lego, or Capital One, or Branch, the new insurance company—I love Branch, by the way. I mean—Corey: Yeah. They're fantastic people, I love working with them. I wish I got to spend more time talking with them. So far, all I can do is drag them on to the podcast and argue on Twitter, but one of these days, one of these days, they're going to have an AWS bill bigger than 50 cents a month, and then, oh, then I've got them.James: There you go. But I think that the thing of him intentionally saying we're not going to set up—I mean, are they in Columbus, I think?Corey: They are. The greater Ohio region, yes.James: Yes. And Joe is all about, we need tools that juniors can be effective with, and we need to satisfy the needs of those juniors so they can be productive in driving our business forward. Juniors is already—and perhaps as a bad term, but new entrants into the industry, and how can we support them where they are, but also help them gain new skills to become more effective? And I just think it's about a different posture, and I think they're a great example because not everybody is south of Market, able to pay 350 grand a year plus stock options. That's just not realistic for most businesses. So, it is important to think about developers and their needs, the skills they learned, if they're from a non-traditional background, what are those skills? How can we support them and become more effective?Corey: That's really what it comes down to. We're all trying to do more with less, but rather than trying to work twice as hard, how to become more effective with the time we have and still go home in time for dinner every day?James: Definitely. I have to say, I mean, 2020 sucked in lots of ways, but not missing a single meal with my family definitely was not one of them.Corey: Yeah. There are certain things I'm willing to trade and certain things I'm not. And honestly, family time is one of them. So, I met you—I don't even recall what year—because what is even time anymore in this pandemic era?—where we sat down and grabbed a drink, I want to say it was at Google Cloud Next—the conference that Google does every year about their cloud—not that Google loses interest in things, but even their conference is called ‘Next'—but I didn't know what to expect when I sat down and spoke with you, and I got the sense you had no idea what to make of me back then because I was basically what I am now, only less fully formed. I was obnoxious on Twitter, I had barely coherent thoughts that I could periodically hurl into the abyss and see if they resonated, but stands out is one of the seminal grabbing a drink with someone moments in the course of my career.James: Well, I mean, fledgling Corey was pretty close to where he is now. But yeah, you bring something unique to the table. And I didn't totally know what to expect; I knew there would be snark. But yeah, it was certainly a pleasure to meet you, and I think that whenever I meet someone, I'm always interested in if there is any way I can help them. And it was nice because you're clearly a talented fellow and everything else, but it was like, are there some areas where I might be able to help? I mean, I think that's a good position as a human meeting another human. And yeah, it was a pleasure. I think it was in the Intercontinental, I guess, in [unintelligible 00:11:00].Corey: Yes, that's exactly where it was. Good memory. In fact, I can tell you the date: it was April 11 of 2019. And I know that because right after we finished having a drink, you tweeted out a GIF of Snow White carving a pie, saying, “QuinnyPig is an industry analyst.” And the first time I saw that, it was, “I thought he liked me. Why on earth would he insult me that way?”But it turned into something where when you have loud angry opinions, if you call yourself an analyst, suddenly people know what to do with you. I'm not kidding, I had that tweet laser engraved on a piece of wood through Laser Tweets. It is sitting on my shelf right now, which is how I know the date because it's the closest thing I have to a credential in almost anything that I do. So, congratulations, you're the accrediting university. Good job.James: [laugh]. I credentialed you. How about that?Corey: It's true, though. It didn't occur to me that analysts were a real thing. I didn't know what it was, and that's part of what we talked about at lunch, where it seemed that every time I tried to articulate what I do, people got confused. Analyst is not that far removed from an awful lot of what I do. And as I started going to analyst events, and catching up with other analysts—you know, the real kind of analyst, I would say, “I feel like a fake analyst. I have no idea what I'm actually doing.” And they said, “You are an analyst. Welcome to the club. We meet at the bar.” It turns out, no one really knows what is going on, fully, in this zany industry, and I feel like that the thing that we all bond over on some level is the sense of, we each only see a piece of it, and we try and piece it together with our understanding of the world and ideally try and make some sense out of it. At least, that's my off-the-cuff definition of an industry analyst. As someone who's an actual industry analyst, and not just a pretend one on Twitter, what's your take on the subject?James: Well, it's a remarkable privilege, and it's interesting because it is an uncredentialed job. Anybody can be, theoretically at least, an industry analyst. If people say you are and think you are, then then you are; you walk and quack like a duck. It's basically about research and trying to understand a problem space and trying to articulate and help people to basically become more effective by understanding that problem space themselves, more. So, it might be about products, as I say, it might be about processes, but for me, I've just always enjoyed research. And I've always enjoyed advice. You need a particular mindset to give people advice. That's one of the key things that, as an industry analyst, you're sort of expected to do. But yeah, it's the getting out there and learning from people that is the best part of the job. And I guess that's why I've been doing it for such an ungodly long time; because I love learning, and I love talking to people, and I love trying to help people understand stuff. So, it suits me very well. It's basically a job, which is about research, analysis, communication.Corey: The research part is the part that I want to push back on because you say that, and I cringe. On paper, I have an eighth-grade education. And academia was never really something that I was drawn to, excelled at, or frankly, was even halfway competent at for a variety of reasons. So, when you say ‘research,' I think of something awful and horrible. But then I look at the things I do when I talk to companies that are building something, and then I talked to the customers who are using the thing the company's building, and, okay, those two things don't always align as far as conversations go, so let's take this thing that they built, and I'll build something myself with it in an afternoon and see what the real story is. And it never occurred to me until we started having conversations to view that through the lens of well, that is actual research. I just consider it messing around with computers until something explodes.James: Well, I think. I mean, that is research, isn't it?Corey: I think so. I'm trying to understand what your vision of research is. Because from where I sit, it's either something negative and boring or almost subverting the premises you're starting with to a point where you can twist it back on itself in some sort of ridiculous pretzel and come out with something that if it's not functional, at least it's hopefully funny.James: The funny part I certainly wish that I could get anywhere close to the level of humor that you bring to the table on some of the analysis. But look, I mean, yes, it's easy to see things as a sort of dry. Look, I mean, a great job I had randomly in my 20s, I sort of lied, fluked, lucked my way into researching Eastern European art and architecture. And a big part of the job was going to all of these amazing museums and libraries in and around London, trying to find catalogs from art exhibitions. And you're learning about [Anastasi Kremnica 00:15:36], one of the greatest exponents of the illuminated manuscript and just, sort of, finding out about this interesting work, you're finding out that some of the articles in this dictionary that you're researching for had been completely made up, and that there wasn't a bibliography, these were people that were writing for free and they just made shit up, so… but I just found that fascinating, and if you point me at a body of knowledge, I will enjoy learning stuff. So, I totally know what you mean; one can look at it from a, is this an academic pursuit? But I think, yeah, I've just always enjoyed learning stuff. And in terms of what is research, a lot of what RedMonk does is on the qualitative side; we're trying to understand what people think of things, why they make the choices that they do, you have thousands of conversations, synthesize that into a worldview, you may try and play with those tools, you can't always do that. I mean, to your point, play with things and break things, but how deep can you go? I'm talking to developers that are writing in Rust; they're writing in Go, they're writing in Node, they're writing in, you know, all of these programming languages under the sun. I don't know every programming language, so you have to synthesize. I know a little bit and enough to probably cut off my own thumb, but it's about trying to understand people's experience. And then, of course, you have a chance to bring some quantitative things to the table. That was one of the things that RedMonk for a long time, we'd always—we were always very wary of, sort of, quantitative models in research because you see this stuff, it's all hockey sticks, it's all up into the right—Corey: Yeah. You have that ridiculous graph thing, which I'm sorry, I'm sure has an official name. And every analyst firm has its own magic name, whether it's a ‘Magic Quadrant,' or the ‘Forrester Wave,' or, I don't know, ‘The Crushing Pit Of Despair.' I don't know what company is which. But you have the programming language up-and-to-the-right line graph that I'm not sure the exact methodology, but you wind up placing slash ranking all of the programming languages that are whatever body of work you're consuming—I believe it might be Stack Overflow—James: Yeah.Corey: —and people look for that whenever it comes out. And for some reason, no one ever yells at you the way that they would if you were—oh, I don't know, a woman—or someone who didn't look like us, with our over-represented faces.James: Well, yeah. There is some of that. I mean, look, there are two defining forces to the culture. One is outrage, and if you can tap into people's outrage, then you're golden—Corey: Oh, rage-driven development is very much a thing. I guess I shouldn't be quite as flippant. It's kind of magic that you can wind up publishing these things as an organization, and people mostly accept it. People pay attention to it; it gets a lot of publicity, but no one argues with you about nonsense, for the most, part that I've seen.James: I mean, so there's a couple of things. One is outrage; universal human thing, and too much of that in the culture, but it seems to work in terms of driving attention. And the other is confirmation bias. So, I think the beauty of the programming language rankings—which is basically a scatterplot based on looking at conversations in StackOverflow and some behaviors in GitHub, and trying to understand whether they correlate—we're very open about the methodology. It's not something where—there are some other companies where you don't actually know how they've reached the conclusions they do. And we've been doing it for a long time; it is somewhat dry. I mean, when you read the post the way Stephen writes it, he really does come across quite academic; 20 paragraphs of explication of the methodology followed by a few paragraphs explaining what we found with the research. Every time we publish it, someone will say, “CSS is not a programming language,” or, “Why is COBOL not on there?” And it's largely a function of methodology. So, there's always raged to be had.Corey: Oh, absolutely. Channeling rage is basically one of my primary core competencies.James: There you go. So, I think that it's both. One of the beauties of the thing is that on any given day when we publish it, people either want to pat themselves on the back and say, “Hey, look, I've made a really good choice. My programming language is becoming more popular,” or they are furious and like, “Well, come on, we're not seeing any slow down. I don't know why those RedMonk folks are saying that.” So, in amongst those two things, the programming language rankings was where we began to realize that we could have a footprint that was a bit more quantitative, and trying to understand the breadcrumbs that developers were dropping because the simple fact is, is—look, when we look at the platforms where developers do their work today, they are in effect instrumented. And you can understand things, not with a survey where a lot of good developers—a lot of people in general—are not going to fill in surveys, but you can begin to understand people's behaviors without talking to them, and so for RedMonk, that's really thrilling. So, if we've got a model where we can understand things by talking to people, and understand things by not talking to people, then we're cooking with gas.Corey: I really love installing, upgrading, and fixing security agents in my cloud estate! Why do I say that? Because I sell things, because I sell things for a company that deploys an agent, there's no other reason. Because let's face it. Agents can be a real headache. Well, now Orca Security gives you a single tool that detects basically every risk in your cloud environment -- and that's as easy to install and maintain as a smartphone app. It is agentless, or my intro would've gotten me into trouble here, but it can still see deep into your AWS workloads, while guaranteeing 100% coverage. With Orca Security, there are no overlooked assets, no DevOps headaches, and believe me you will hear from those people if you cause them headaches. and no performance hits on live environments. Connect your first cloud account in minutes and see for yourself at orca.security. Thats “Orca” as in whale, “dot” security as in that things you company claims to care about but doesn't until right after it really should have.Corey: One of the I think most defining characteristics about you is that, first, you tend to undersell the weight your words carry. And I can't figure out, honestly, whether that is because you're unaware of them, or you're naturally a modest person, but I will say you're absolutely one of my favorite Twitter follows; @monkchips. If you're not following James, you absolutely should be. Mostly because of what you do whenever someone gives you a modicum of attention, or of credibility, or of power, and that is you immediately—it is reflexive and clearly so, you reach out to find someone you can use that credibility to lift up. It's really an inspirational thing to see. It's one of the things that if I could change anything about myself, it would be to make that less friction-full process, and I think it only comes from practice. You're the kind of person I think—I guess I'm trying to say that I aspire to be in ways that are beyond where I already am.James: [laugh]. Well, that's very charming. Look, we are creatures of extreme privilege. I mean, I say you and I specifically, but people in this industry generally. And maybe not enough people recognize that privilege, but I do, and it's just become more and more clear to me the longer I've been in this industry, that privilege does need to be more evenly distributed. So, if I can help someone, I naturally will. I think it is a muscle that I've exercised, don't get me wrong—Corey: Oh, it is a muscle and it is a skill that can absolutely be improved. I was nowhere near where I am now, back when I started. I gave talks early on in my speaking career, about how to handle a job interview. What I accidentally built was, “How to handle a job interview if you're a white guy in tech,” which it turns out is not the inclusive message I wanted to be delivering, so I retired the talk until I could rebuild it with someone who didn't look like me and give it jointly.James: And that's admirable. And that's—Corey: I wouldn't say it's admirable. I'd say it's the bare minimum, to be perfectly honest.James: You're too kind. I do what I can, it's a very small amount. I do have a lot of privilege, and I'm aware that not everybody has that privilege. And I'm just a work in progress. I'm doing my best, but I guess what I would say is the people listening is that you do have an opportunity, as Corey said about me just now, maybe I don't realize the weight of my words, what I would say is that perhaps you have privileges you can share, that you're not fully aware that you have. In sharing those privileges, in finding folks that you can help it does make you feel good. And if you would like to feel better, trying to help people in some small way is one of the ways that you can feel better. And I mentioned outrage, and I was sort of joking in terms of the programming language rankings, but clearly, we live in a culture where there is too much outrage. And so to take a step back and help someone, that is a very pure thing and makes you feel good. So, if you want to feel a bit less outraged, feel that you've made an impact, you can never finish a day feeling bad about the contribution you've made if you've helped someone else. So, we do have a rare privilege, and I get a lot out of it. And so I would just say it works for me, and in an era when there's a lot of anger around, helping people is usually the time when you're not angry. And there's a lot to be said for that.Corey: I'll take it beyond that. It's easy to cast this in a purely feel-good, oh, you'll give something up in order to lift people up. It never works that way. It always comes back in some weird esoteric way. For example, I go to an awful lot of conferences during, you know, normal years, and I see an awful lot of events and they're all—hmm—how to put this?—they're all directionally the same. The RedMonk events are hands down the exception to all of that. I've been to Monktoberfest once, and I keep hoping to go to—I'm sorry, was it Monki Gras is the one in the UK?James: Monki Gras, yeah.Corey: Yeah. It's just a different experience across the board where I didn't even speak and I have a standing policy just due to time commitments not to really attend conferences I'm not speaking at. I made an exception, both due to the fact that it's RedMonk, so I wanted to see what this event was all about, and also it was in Portland, Maine; my mom lived 15 minutes away, it's an excuse to go back, but not spend too much time. So, great. It was more or less a lark, and it is hands down the number one event I will make it a point to attend. And I put that above re:Invent, which is the center of my cloud-y universe every year, just because of the stories that get told, the people that get invited, just the sheer number of good people in one place is incredible. And I don't want to sound callous, or crass pointing this out, but more business for my company came out of that conference from casual conversations than any other three conferences you can name. It was phenomenal. And it wasn't because I was there setting up an expo booth—there isn't an expo hall—and it isn't because I went around harassing people into signing contracts, which some people seem to think is how it works. It's because there were good people, and I got to have great conversations. And I kept in touch with a lot of folks, and those relationships over time turned into business because that's the way it works.James: Yeah. I mean, we don't go big, we go small. We focus on creating an intimate environment that's safe and inclusive and makes people feel good. We strongly curate the events we run. As Stephen explicitly says in terms of the talks that he accepts, these are talks that you won't hear elsewhere. And we try and provide a platform for some different kind of thinking, some different voices, and we just had some magical, magical speakers, I think, at both events over the years. So, we keep it down to sort of the size of a village; we don't want to be too much over the Dunbar number. And that's where rich interactions between humans emerge. The idea, I think, at our conference is, is that over a couple of days, you will actually get to know some people, and know them well. And we have been lucky enough to attract many kind, and good, and nice people, and that's what makes the event so great. It's not because of Steve, or me, or the others on the team putting it together. It's about the people that come. And they're wonderful, and that's why it's a good event. The key there is we focus on amazing food and drink experiences, really nice people, and keep it small, and try and be as inclusive as you can. One of the things that we've done within the event is we've had a diversity and inclusion sponsorship. And so folks like GitHub, and MongoDB, and Red Hat have been kind enough—I mean, Red Hat—interestingly enough the event as a whole, Red Hat has sponsored Monktoberfest every year it's been on. But the DNI sponsorship is interesting because what we do with that is we look at that as an opportunity. So, there's a few things. When you're running an event, you can solve the speaker problem because there is an amazing pipeline of just fantastic speakers from all different kinds of backgrounds. And I think we do quite well on that, but the DNI sponsorship is really about having a program with resources to make sure that your delegates begin to look a little bit more diverse as well. And that may involve travel stipends, as well as free tickets, accommodation, and so on, which is not an easy one to pull off.Corey: But it's necessary. I mean, I will say one of the great things about this past year of remote—there have been a lot of trials and tribulations, don't get me wrong—but the fact that suddenly all these conferences are available to anyone with an internet connection is a huge accessibility story. When we go back to in-person events, I don't want to lose that.James: Yeah, I agree. I mean, I think that's been one of the really interesting stories of the—and it is in so many dimensions. I bang on about this a lot, but so much talent in tech from Nigeria. Nigeria is just an amazing, amazing geography, huge population, tons of people doing really interesting work, educating themselves, and pushing and driving forward in tech, and then we make it hard for them to get visas to travel to the US or Europe. And I find that to be… disappointing. So, opening it up to other geographies—which is one of the things that free online events does—is fantastic. You know, perhaps somebody has some accessibility needs, and they just—it's harder for them to travel. Or perhaps you're a single parent and you're unable to travel. Being able to dip into all of these events, I think is potentially a transformative model vis-à-vis inclusion. So, yeah, I hope, A) that you're right, and, B) that we as an industry are intentional because without being intentional, we're not going to realize those benefits, without understanding there were benefits, and we can indeed lower some of the barriers to entry participation, and perhaps most importantly, provide the feedback loop. Because it's not enough to let people in; you need to welcome them. I talked about the DNI program: we have—we're never quite sure what to call them. We call them mentors or things like that, but people to welcome people into the community, make introductions, this industry, sometimes it's, “Oh, great. We've got new people, but then we don't support them when they arrive.” And that's one of the things as an industry we are, frankly, bad at, and we need to get better at it.Corey: I could not agree with you more strongly. Every time I wind up looking at building an event or whatnot or seeing other people's events, it's easy to criticize, but I try to extend grace as much as possible. But whenever I see an event that is very clearly built by people with privilege, for people with privilege, it rubs me the wrong way. And I'm getting worse and worse with time at keeping my mouth shut about that thing. I know, believe it or not, I am capable of keeping my mouth shut from time to time or so I'm told. But it's irritating, it rankles because it's people not taking advantage of their privileged position to help others and that, at some point, bugs me.James: Me too. That's the bottom line, we can and must do better. And so things that, sort of, make you proud of every year, I change my theme for Monki Gras, and, you know, it's been about scaling your craft, it's been about homebrews—so that was sort of about your side gig. It wasn't about the hustle so much as just things people were interested in. Sometimes a side project turns into something amazing in its own right. I've done Scandinavian craft—the influence of the Nordics on our industry. We talk about privilege: every conference that you go to is basically a conference about what San Francisco thinks. So, it was nice to do something where I looked at the influence of Scandinavian craft and culture. Anyway, to get to my point, I did the conference one year about accessibility. I called it ‘accessible craft.' And we had some folks from a group called Code Your Future, which is a nonprofit which is basically training refugees to code. And when you've got a wheelchair-bound refugee at your conference, then you may be doing something right. I mean, the whole wheelchair thing is really interesting because it's so easy to just not realize. And I had been doing these conferences in edgy venues. And I remember walking with my sister, Saffron, to check out one of the potential venues. It was pretty cool, but when we were walking there, there were all these broken cobblestones, and there were quite a lot of heavy vehicles on the road next to it. And it was just very clear that for somebody that had either issues with walking or frankly, with their sight, it just wasn't going to fly anymore. And I think doing the accessibility conference was a watershed for me because we had to think through so many things that we had not given enough attention vis-à-vis accessibility and inclusion.Corey: I think it's also important to remember that if you're organizing a conference and someone in a wheelchair shows up, you don't want to ask that person to do extra work to help accommodate that person. You want to reach out to experts on this; take the burden on yourself. Don't put additional labor on people who are already in a relatively challenging situation. I feel like it's one of those basic things that people miss.James: Well, that's exactly right. I mean, we offered basically, we were like, look, we will pay for your transport. Get a cab that is accessible. But when he was going to come along, we said, “Oh, don't worry, we've made sure that everything is accessible.” We actually had to go further out of London. We went to the Olympic Park to run it that year because we're so modern, and the investments they made for the Olympics, the accessibility was good from the tube, to the bus, and everything else. And the first day, he came along and he was like, “Oh, I got the cab because I didn't really believe that the accessibility would work.” And I think on the second day, he just used the shuttle bus because he saw that the experience was good. So, I think that's the thing; don't make people do the work. It's our job to do the work to make a better environment for as many people as possible.Corey: James, before we call it a show, I have to ask. Your Twitter name is @monkchips and it is one of the most frustrating things in the world trying to keep up with you because your Twitter username doesn't change, but the name that goes above it changes on what appears to be a daily basis. I always felt weird asking you this in person, when I was in slapping distance, but now we're on a podcast where you can't possibly refuse to answer. What the hell is up with that?James: Well, I think if something can be changeable, if something can be mutable, then why not? It's a weird thing with Twitter is that it enables that, and it's just something fun. I know it can be sort of annoying to people. I used to mess around with my profile picture a lot; that was the thing that I really focused on. But recently, at least, I just—there are things that I find funny, or dumb, or interesting, and I'll just make that my username. It's not hugely intentional, but it is, I guess, a bit of a calling card. I like puns; it's partly, you know, why you do something. Because you can, so I've been more consistent with my profile picture. If you keep changing both of them all the time, that's probably suboptimal. Sounds good.Corey: Sounds good. It just makes it hard to track who exactly—“Who is this lunatic, and how did they get into my—oh, it's James, again.” Ugh, branding is hard. At least you're not changing your picture at the same time. That would just be unmanageable.James: Yeah, no, that's what I'm saying. I think you've got to do—you can't do both at the same time and maintain—Corey: At that point, you're basically fleeing creditors.James: Well, that may have happened. Maybe that's an issue for me.Corey: James, I want to thank you for taking as much time as you have to tolerate my slings, and arrows, and other various vocal devices. If people want to learn more about who you are, what you believe, what you're up to, and how to find you. Where are you hiding?James: Yeah, I mean, I think you've said already, that was very kind: I am at @monkchips. I'm not on topic. I think as this conversation has shown, I [laugh] don't think we've spoken as much about technology as perhaps we should, given the show is normally about the cloud.Corey: The show is normally about the business of cloud, and people stories are always better than technology stories because technology is always people.James: And so, yep, I'm all over the map; I can be annoying; I wear my heart on my sleeve. But I try and be kind as much as I can, and yeah, I tweet a lot. That's the best place to find me. And definitely look at redmonk.com. But I have smart colleagues doing great work, and if you're interested in developers and technology infrastructure, we're a great place to come and learn about those things. And we're very accessible. We love to talk to people, and if you want to get better at dealing with software developers, yeah, you should talk to us. We're nice people and we're ready to chat.Corey: Excellent. We will, of course, throw links to that in the [show notes 00:37:03]. James, thank you so much for taking the time to speak with me. I really do appreciate it.James: My pleasure. But you've made me feel like a nice person, which is a bit weird.Corey: I know, right? That's okay. You can go for a walk. Shake it off.James: [laugh].Corey: It'll be okay. James Governor, analyst and co-founder at RedMonk. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you hated this podcast, please leave a five-star review on your podcast platform of choice along with an insulting comment in which you attempt to gatekeep being an industry analyst.Announcer: This has been this week's episode of Screaming in the Cloud. You can also find more Corey at screaminginthecloud.com, or wherever fine snark is sold.This has been a HumblePod production. Stay humble.

11 Job Titles in 8 Years at 1 Company with Sean Kilgore

Play Episode Listen Later Jul 27, 2021 34:34

About SeanSean Kilgore is an Architect at Twilio, where he draws boxes, lighthouses and soapboxes. In Sean's spare time, he enjoys reading, walking, gaming, and a well-made drink.Links: Twilio: https://www.twilio.com/ Silvia Botros's Twitter: https://twitter.com/dbsmasher Sean's Twitter: https://twitter.com/log1kal TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Your company might be stuck in the middle of a DevOps revolution without even realizing it. Lucky you! Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy in on DevOps practices? Well, download the 2021 State of DevOps report brought to you annually by Puppet since 2011 to explore the trends and blockers keeping evolution firms stuck in the middle of their DevOps evolution. Because they fail to evolve or die like dinosaurs. The significance of organizational buy in, and oh it is significant indeed, and why team identities and interaction models matter. Not to mention weither the use of automation and the cloud translate to DevOps success. All that and more awaits you. Visit: www.puppet.com to download your copy of the report now!Corey: Up next we've got the latest hits from Veem. Its climbing charts everywhere and soon its going to climb right into your heart. Here it is!Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Sean Kilgore who's an architect at a small company called Twilio. Sean, welcome to the show.Sean: Corey, it's a pleasure to be here.Corey: It really is. You're one of those fun people that I always mean to catch up with and really do a deep dive in, but we keep passing like ships in the night. And in fact, I want to go back to more or less what is pretty damn close to your first real job in technology. You were a network administrator at Lutheran High School in Orange, California.Sean: I was.Corey: And at that same time, I was a network administrator down the street at Chapman University, also in Orange, California. And despite that, we have traveled in many of the same circles since, but we have never met in person despite copious opportunities to do so.Sean: That is amazing.Corey: Talking to you is like looking into a funhouse mirror of what would it have been like if I could, you know, hold down a job, and was actually good at things. It's really fun. Apparently, I'd be able to grow a better beard.Sean: I don't know about that. My beard is pretty patchy.Corey: Yeah, I look like an angry 14-year-old trying to prove a point to Mommy and Daddy. But that's not really the direction that we need to take this in today. And you've done a lot of stuff that aligns with things that are near and dear to my heart. For the last—what is it now?—six years and change? Seven years and change? You've been at SendGrid, then Twilio through acquisition?Sean: Mm-hm.Corey: And you have done basically every operations-looking job at that company. You've had a bunch of titles. You wound up going from DevOps engineer to a team lead, then to a senior DevOps engineer again, and you call—you voluntarily move back to an individual contributor role. Let's start there. What was management like?Sean: The management was interesting. My first go at that, I had no idea what I was doing, and so I didn't know how to ask for the help that I needed. And so my wife and I refer to that time as the time that I played a lot of video games. Just, I wasn't prepared for the emotional outlay that managing humans costs. And so I would end up spending my nights just playing video games trying to unwind and unpack from all that.I've managed twice, now. The second time was—it went much better. I knew more of what I was doing, I had more support. The manager of the team that had left, I had worked with that team very closely in the past, I'd been part of it. And so my whole purpose there was to make sure that we didn't lose anybody after that until we found a new manager.And that actually worked out pretty well. I had to have some really difficult conversations with some people along the way, but they all stayed, they all told me that they really enjoyed me managing them. And I've had people ask me to manage them again after that, which was super bonkers.Corey: It's always flattering when you have an impact as a manager and people seek you out to work with you again. I dabbled in management as well; no one ever asked me to do it twice, and I have effectively avoided managing people here at The Duckbill Group, just because my belief of what a good manager is—and I think it aligns with what you've already said—requires a certain selflessness and ability to focus on others and grow them, whereas my role here is very much as face of the company, and it's about me. That's not a recipe for a successful outcome for managing people and not having them rage-quit.Sean: Definitely.Corey: One thing that I find is interesting about management, the higher you rise in an organization, it's counterintuitive but the more responsibility you get, but the less you can directly affect yourself. Your entire world becomes about effectively delegating work to others and about influence. In your case now, you are an architect, which means different things at different companies. So, I'm not entirely sure I know what it means at Twilio. So first, what is an architect at Twilio? And where does your responsibility start and stop, I guess is where I want to go, there?Sean: So, architecture at Twilio is kind of different. Some of the architects that I've worked with in the past is the ‘I design everything, and then engineers go and build the thing that I designed.'Corey: Oh, yes. The enterprise architect approach where I'm going to sit in my ivory tower and dispatch, effectively, winged monkeys to implement things that I don't fully understand, but I have the flashy title. You're saying that's not what it is?Sean: There's a fine distinction here because I think that some of the people I work with would definitely say that I am up in my ivory tower. It is more about—if I'm looking five years out—what capabilities my teams need to be able to provide to execute against a business strategy. That landscape is going to change immensely along the way, and so my job isn't to say, “We're going to use Kubernetes because that's what we need to do five years out.” It happens to be what I'm saying right now, and I'm sure we're going to go into that, Corey, but it's less about this is how each of these things should be strung together to achieve that goal and more direction setting. So, I worked on something that I call the ‘Lighthouse,' and it's a vision of the future of where we want to go but the caveat is that if you actually go to the Lighthouse, it means you hit the rocks.It's describing what I call the ‘Bay of Appropriate Futures,' and you want to land somewhere in that Bay, but it's not going to be the thing I wrote down five years before we get there. And so it's much more of a technical leadership position, trying to help other technologists make good technology decisions. And so it's more about making sure that all of the right questions get asked, not having all the right answers. That's the difference between some architects that I've worked with in the past.Corey: And one of the challenges in that role is that you're not managing people directly. So, what that means is, you are, on some level, not doing a lot of the hands-on keyboard implementation work yourself and, unless I dramatically misunderstand your corporate culture, you're not empowered to unilaterally fire people, which means that you can only really lead via example and influence. Tell me about that.Sean: When people ask me if they want to be architects, I ask them if they can influence without authority, or if that's even interesting to them. That is definitely the thing. And so when it comes down to, “Man, I really wish I could fire this person.” A, that never happens. But, B, it's definitely about modeling behaviors. And there's a bit of management here in that making expectations clear of senior engineers is part of my job, and helping them also be examples for other engineers is definitely a thing I get graded on.Corey: Influence without authority is sort of the definitional characteristic of being a consultant. It turns out, you can't even force anything; it has to be the strength of your ideas combined, in some shops, and I admit I have a bit of a somewhat cynical view on this, but also the ability for the client to commit to their sunk-cost fallacy of, “Well, we paid a lot of money to hear this advice, we should probably do something with it.” And there's always a story of making sure that you're serving the organization with which you work, well. But when you can only influence rather than direct, it becomes a much more nuanced thing, and I feel like the single differentiator between success and failure in that role is, fundamentally, empathy. Am I wrong on that?Sean: Not at all. When I'm working with a very in-the-code engineer who comes to me and is trying to convince their team that they should do something, one of the things that's a stumbling block for them is that they don't realize that other people need to be influenced in ways that might be different than the way that they're influenced. So, as an example, I work with a team of very senior people; I know that some people will respond well, if I site Accelerate, for example. And some people want to hear, “Well, Google does this, so it's obvious that we should do that.” And trying to thread that needle with everyone in a way that gets everyone on board in the best way for all of us, when you can do that, you can be an architect.Corey: Some weeks, I feel like I'm closer to architect than I do others. It seems that the idea now—solutions architect being a job role that a lot of companies have they hurl out into the universe—is in many ways vastly misunderstood. You want to talk about some kind of architecture story, where I'm going to go ahead and design an architecture that solves some business problem on a whiteboard. That's not hard to do.The hard part is then controlling for constraints such as, “Yeah, we already have a thing that doesn't look anything like that, and we want to get it to that point. And oh, by the way, 18 months of downtime while we do that is not acceptable.” Nothing is ever truly greenfield. And adapting to constraints, and making compromises, and being realistic seems to separate the folks who are good at that from the folks who are playacting.Sean: There's another thing in there where people who've worked on the same thing for a long time sometimes have a problem seeing where you could go. And so the constraints there can be really useful in designing things. A lot of people think that greenfield is awesome, but greenfield just means the possible outcomes are the entire universe. I like working in constraints, essentially.Corey: So, I want to talk to you a little bit about your tenure at Twilio, where you started off at SendGrid and then there was an acquisition, and everyone I know got super-quiet for a while because it turns out when there's a pending acquisition, talking to people about it is frowned upon, and that goes doubly so in the context of someone who basically shitposts for a living. And I get that; I respect confidentiality, but I also don't want people to jeopardize their own positions. So, it's one of those, “Yeah, whatever you're comfortable telling me, or not, is fine.” So, we didn't talk for a while. And then the acquisition happened, and now you're there. And you've been there at Twilio for a couple of years or so and haven't rage-quit, so apparently, it's worked. What was the transition like?Sean: The transition was really interesting. A lot of people were telling me that acquisitions were universally horrible. And that's not how it worked. This is the first acquisition I've been through, so I have no context. People I trust told me that this one went well.So, in my role as architect, this acquisition was kind of interesting because SendGrid had a very robust, and we've done architecture for years. And Twilio's architecture was a little bit different. It was more like, “There are some really, really senior people at Twilio who have seen some things, and you should probably ask them their opinion on stuff.” But there wasn't really, like, an architecture review process. There was definitely, “A you need to write down a bunch of stuff and get some people to look at it,” but it wasn't a, you need to get approved by your local architect or a group of architects.Part of that is to provide visibility across the org so that we're not duplicating work and stuff. But Twilio basically adopted SendGrid's architecture process, but it grew 10x. So, at SendGrid, we had, I think, six architects. At Twilio, we have, like, 40 now, so not quite 10x. But trying to copy and paste that process was kind of rough.We're still, kind of, making that better. And then there were a couple things—as the acquired company, you kind of expect, I don't know, maybe some housecleaning to happen. And that isn't what happened. We saw a lot of like really senior leaders move into positions at Twilio of leadership. So, on day one, I think SendGrid's sales leader became the sales leader for all of Twilio.And that sounded—I haven't done this before, but it sounded like that's not normal. And that's happened in a couple different spots. It's been pretty neat. And when I think about the acquisition, not just of acquiring another channel for Twilio, but kind of doing an acqui-hire of a bunch of key positions, that was a pretty valuable one.Corey: Let's talk about one aspect of working at Twilio that I profoundly envy you for, which is working with one of the greatest people in the world: Silvia. Let's talk about Silvia.Sean: She interviewed me at SendGrid. She's been here almost a year longer than I have, and it's been such a joy to work with her. Not just because everything gets Botros'ed around her, and so we have our own built-in chaos monkeys, but also, there's no one that cares more about making sure that what teams are building won't come back and bite the team later. She's worked with, I think, maybe a 10th of the company now—and at Twilio, that's a lot of teams—trying to just help them do better and make sure that the stuff that they're building is not going to page them all the time, is actually going to serve the customer in ways that isn't surprising to the customer. I can probably talk for half an hour about my appreciation for Silvia.Corey: Well, she was a great guest in the early days of this podcast. Silvia Botros is phenomenal. She has the Twitter handle of @dbsmasher so she's my default go-to on misusing things as databases. And she also was just one of the most genuinely kind people I know.She also has an aura effect, where she is basically a walking EMP, and every time someone tries to show her a piece of technology, it explodes in novel and interesting ways, which, frankly, as an acceptance gate for technology is a terrific skill set to have. Does it cause problems in the office?Sean: Not normally. It causes more problems in the office when we are actually in an office together because Silvia, maybe predictively, is also a giant klutz. And so the joke is that she also EMPs herself. In the office, she does break things, but it's never in an intended way. Or it's just like a fun, “Oh, man, the WiFi's down. It must be Silvia.”Corey: Exactly. It's always nice to have someone you can blame for these things.Sean: It's SOP.Corey: Yeah, oh, absolutely. At some point do you ever wind up missing things such as, “Oh, it's probably just Silvia. No, it was actually a problem somewhere?”Sean: So, we actually determine that Silvia's EMP works at a distance. She flew somewhere close to one of our hardware data centers, and at the time that she passed it, we had an outage. Like, the data center went dark, kind of thing. And so it still happens even if she's not around. We're pretty sure it's not a local phenomenon.Corey: So, the thing that I know is probably going to sound completely boring and ridiculous to half the audience while the other half the audience sits and listens raptly; before I started this place, I never stayed anywhere for longer than two years, because as previously disclosed in multiple directions, I am a terrible employee. First, why did you stay at the same place for as long as you have, and what's it like? And I'm really hoping you have an answer that isn't just, “Oh, I have a complete lack of ambition,” because I won't believe that for a second. But it is a tempting cop-out so let me just shut that down now.Sean: No, it's more I've been here for almost eight years now, and I've never done the same thing. The fun fact that I tell people when they onboard or I'm interviewing them is, I think I've had more titles than anyone at Twilio. I'm up to 11, I think. And so 11 titles in eight years? It hasn't been the same company.When I started, I was employee number 150. There were 80 of us when I started at SendGrid. I work at a company with 4500 people now and going through that growth, the company that I work for today, and even pre-acquisition, you would not recognize from the day I started at SendGrid. And so if I had been doing the same thing all the time, I wouldn't still be here. There was a point before we started architecture at SendGrid, I was definitely in a spot kind of a rut, like, “Cool. I can continue to do the same thing over and over,” but I felt like there wasn't a lot of growth to do.I needed to go see something else, kind of thing. I knew really well how to do our mail stuff, and I felt like I needed to broaden my horizons a bit, or I needed to level up. And at the time, the only place to go up was to management. And then we brought in a chief architect, J.R. Jasperson, and I I remember very clearly, it was like his third day or something, we had an all-company meeting—like a lunch thing—and I walked up to him and said, “You don't know this yet, but you're my new mentor because it seems like what you're talking about is really interesting to me.” And he didn't know it but the subtext there was like, “And if you don't, I'm out.”And since then, the work that I do day-to-day is completely different. Like, I work for a platform org. This platform org is 130 people right now; it spans everything from building EC2 instances to, recently, it was, like, Twilio API Edge. There's such a breadth in there that I never do the same thing every day.Corey: I really love installing, upgrading, and fixing security agents in my cloud estate! Why do I say that? Because I sell things, because I sell things for a company that deploys an agent, there's no other reason. Because let's face it. Agents can be a real headache. Well, now Orca Security gives you a single tool that detects basically every risk in your cloud environment -- and that's as easy to install and maintain as a smartphone app. It is agentless, or my intro would've gotten me into trouble here, but it can still see deep into your AWS workloads, while guaranteeing 100% coverage. With Orca Security, there are no overlooked assets, no DevOps headaches, and believe me you will hear from those people if you cause them headaches. and no performance hits on live environments. Connect your first cloud account in minutes and see for yourself at orca.security. Thats “Orca” as in whale, “dot” security as in that things you company claims to care about but doesn't until right after it really should have.Corey: That's functionally, I think, the problem that I had in working in environments as a DevOps type because for the first three months in a job where I'm the first ops person, “Great everything's on fire.”—I'm an adrenaline junkie in that sense—“Cool. Oh wow, all these problems that I know how to fix.” And then it gets to a reasonable level of working and now it's just care and feeding of same. Okay, now I'm getting slightly bored, so let me look for other problems in other parts of the org.And that doesn't go super well when you're not welcome in those parts of the org which leads to a whole bunch of challenges I've had in my career. This is incidentally why being a consultant aligns so well with me and the way I approach things. It's cool. I'm going to come in; I'm going to fix things, and then I get to leave. On day one, we know this is a time-bound engagement and that's okay.Instead of going down the path of the lies everyone tells themselves where average tenure in this space is 18 to 24 months, but magically we're all going to lie and pretend in the interview that this is their forever job and suddenly you're going to stay here for 25 years and get a pension and a gold watch when you retire. And it's, oh wow that's amazing it sounds like everybody having these conversations wearing old-timey stovepipe hats. There's just so much that isn't realistic in those conversations. So, I talk to people who've been down those paths who've been at the same company for a decade or two, and the common failure mode there is that they have a year or so of experience that they repeat 10 or 20 times. And that's sad; people get stuck. What you say absolutely resonates with me in that every year is a different thing that you're working on. You're not doing the same thing twice. I get antsy when too many days look the same, one to the next.Sean: I definitely hear that. If my every day was, come in join a stand-up, talk about the problem that I had last week and still have today, it wouldn't work for me. I feel lucky that I work for an organization where outside input is actually requested and honored, so if I go to a team and just happened to have noticed something and say, “Hey, this right here you might want to take a look at. And I have some opinions here if you'd like to hear them.” I normally get asked for that opinion, and it normally turns out pretty good.There's definitely times where it's been, like, “No, Sean. You don't know what you're talking about.” And normally they're right. It's definitely not the same. People say you should be at a company for 18 to 24 months. And that's true if your company is totally shortchanging you. When I ask my peers at other companies about, have I gotten stiffed by staying at the same place for this long? It's definitely not. And if that wasn't true, if Twilio was holding back my compensation, maybe this would have gone a different way, but it's not what's happened.Corey: Oh, true and to be clear that is very often the biggest criticism I have of people who stay at one company for a long time. They don't realize what market rate is anymore and they find themselves in a scenario where, “Wow, I could go somewhere else and triple my salary,” which is not an exaggeration and an unpleasant discovery when people realize that they've been taken advantage of. And credit where due, I have had conversations with people at Twilio who have been there a long time. And I have never gotten the impression that that is what's going on there. Your compensation is fair. I want to be very clear here. This is not one of those, “Oh, yeah, I'm just trying to be polite because someone's being taken advantage of and doesn't even know it.” No. They're doing right by their employees. The fact that I have to call them out explicitly as an example of a rarity of a company doing right by its employees, is monstrous.Sean: It is. We've hired a few people recently where I found out that I think their pay was close to doubled just by coming here, and I just wish that it was more okay to call out their prior employees publicly and be like, “Cool. If you work for this company will probably pay you a ton more.”Corey: And that's the other side of it, too, which I did early on in my career. It's, “Oh, I'm leaving this company and screw you all.” “Well, why are you leaving?” “Oh, because I'm getting a 5% raise to change jobs.” I'm not saying that money should not factor into it, but at some point, when all is said and done at that scale, it works out to be 100 bucks a paycheck, or so, is it really worth changing for that? Maybe.If there are things you don't like about the environment, please don't let me dissuade people from interviewing for jobs. You always should be doing that, on some level, just so what the market looks like. But I'm also a big believer in, you don't need to be as mercenary as I was early on in my career. A lot of it was shaped by environments—not Chapman, I want to be clear—that were not particularly kind to staff. And that I felt taken advantage of because I was. And as a result, “Oh, screw me? Screw you.” And it became a very mercenary approach that didn't serve me well. That is now a baked-in aspect of how I view careers in some respects, and that is something of a problem that I wrestle with.Sean: The mercenary thing?Corey: Yeah. I wrestle with the mercenary thing just because when I talk to someone who's having a challenge at work, or something, my default instinctive gut reaction that I've learned to suppress is, “Oh, screw ‘em. Quit and find another job.”Sean: Ah, gotcha.Corey: That's not the most constructive way to work in the context of a company where you're building a career trajectory, and a reputation, and you've been there for five years, and maybe rage-quitting because you didn't wind up getting to pick the title of that presentation isn't the best answer. I can be remarkably petty, for the reasons I'll leave a company. But that's not constructive, and I try very hard to avoid giving that advice unnecessarily to people.Sean: It's definitely just, like, incidents, right? It's never a root cause; it's a contributing factor, and pay is just one contributing factor. I find that a lot of people, even if they're being taken advantage of compensation-wise, they won't leave unless there's something else wrong.Corey: Yeah, compensation is absolutely a symptom, and in most cases, that's not the real reason people are going to leave. I assure you, people who work at The Duckbill Group could make more money, objectively, somewhere else. But there's a question of what people value. We pay people well, but we don't offer FAANG money, with the equity upside and the rest. We're not trying to pull the Netflix and pay absolute top-of-market in all cases to all people.I would love to be able to do that; our margins don't yet support that. Thanks to our sponsors, we're going to continue to ratchet those prices way up. I kid. I kid. But there are business reasons why things are the way they are. What we do offer instead is things that contribute to a workplace we want to work at. More of us are parents than aren't.We don't expect people to work outside of business hours in almost any scenario, short of, you know, re:Invent or something. There's a very human approach to it. We're not VC-backed at all, so we don't ever have to worry about trying to sprint to hit milestones over debt as a company. We have this insane secret approach called ‘revenue' and ‘profitability' that means we can continue to iterate month to month, and as long as the trend line continues, we're happy.Sean: That kind of sustainability is awesome, and is a really good indicator that a company is going to be successful, to me. Especially smaller companies; the decision to not take VC money. And to chase sustainable revenue growth, I know everyone wants to chase the hockey sticks, but at what cost?Corey: Yeah. And I think that people put this on job-seekers way too much. I have been confronted, at one point—I will not name the company—when I was interviewing years ago, and I was asked by the hiring manager, “Well, it seems like you've done a fair bit of job-hopping in the course of your career. What's up with that?” And they pulled up my LinkedIn profile and went through it, and I said, “You realize most of those were contract gigs?” “Well, I don't kno—oh, yeah. I guess it was. Oh, that was—huh. I guess so.”So, it was a failed attempt to call bullshit on my job history. And because I don't take things like that well, I turned it around right back on him, and I said, “No, I appreciate that. Thank you for clarifying.” That's a warning sign is when I thank you for insulting me because what's coming next is always going to sting. But while we're on the topic of turnover, “Your team has lost 80% of its members in the last six months. What's going on with that? Is there a problem here that I should be aware of?” And suddenly, the back-peddling was phenomenal. I did get an offer from that company; I did not accept it.Sean: Good.Corey: You can tell a lot about a company by how they buy their people. And if you're actively being insulted or hazed in the job interview process, no. I want people who I choose not to hire, to come away from the experience feeling respected and that they enjoyed the experience to the point where they would say nice things about us if asked, or even evangelize us without ever even having to be asked. And so far, we've done that because we're very intentional on how we approach things. And man, am I tired of people doing this badly.Sean: When I interview someone, I want them to leave, and then if they don't take the job, I want it to be because it wasn't the right job for them. Or, like, the team wasn't the right fit, not because anything happened in the interview process that was a red flag. That's the worst. I want Twilio to be a spot where there are no red flags. That would be ideal.Corey: Absolutely. I think that so many folks get it wrong, where there's this idea of, “Oh, I'm going to interview you. And oh, you're an ops person. Great. I want you to implement Quicksort on the whiteboard.” And it's, “Question one: do you do that a lot here? And two: no, of course you don't because I've seen your services list. There's no rhyme or reason to the order it appears in. Maybe someone should implement Quicksort in production.”And then there's the other side, too, of, “Oh, great. There's this broad skill set across the entire space. I'm going to figure out where you're weak and then needle you on those.” I don't like hiring for absence of weakness; I like hiring for, you're really good at things we need here and you're acceptable at the things that are non-negotiable, and able to improve in areas where it becomes helpful.Sean: Yeah. The best interview process I ever had, they flew me up to San Francisco, I worked with them for a day on a real problem that they had, like, pair programming. They offered me the job—it didn't work out because I didn't want to move to San Francisco, it turned out—but that interview process was super valuable to me as the candidate because I found out exactly how a day at that job would work; what it would look like.Corey: I had a very similar experience once and the cherry on the top was they paid me a nominal contracting rate for the day—Sean: Same.Corey: —because it was touching things that they were doing. And I think that that's another anti-pattern of, that was a thing that also just happens to be a thing we're going to use in production, but we're not going to tell you that we're not going to compensate you for it. I'll work on toy problems; not production in an interview context.Sean: I wanted to circle back to one thing about leaving a company, like, rage-quitting. It's essentially—if you rage-quit because of a problem, like, a small thing, you're missing an opportunity to grow. And especially if I had one superpower, I would say that it's probably managing up. Part of this is just, I have a lot of privilege that lets me do that, but it is definitely a skill that I wish more people had for their own careers.Corey: I really do, too. We spent all this time practicing how to be a candidate in a job interview, and almost no time training people how to be a good interviewer, and what you're looking for. And you wind up with terrible things like, “I had this problem once in production that I thought was super clever, so I'm going to set it up for you and see how you would solve that problem. And if you don't follow the exact same path that I did, then we're going to go ahead and just keep shooting down anything else you suggest.” No, stop it.Sean: I do a lot of interviewing, and so I love when I learn something from a candidate because I can ask them questions that are like, “How did you figure this out? How did you even notice that this was a problem?”s and you get to go really deep in something they know, the way they know it. We used to do the, like, “Build us an LRU cache in the best big-O notation time.” And if you didn't get it, you didn't get the job, if you did it in slower than optimum time.And I remember leaving one of the interviews and doing the recap, and it was like, if anyone came to work and did this, I would be upset at them for wasting time. This is part of the standard library of all the things that we do. Why are we asking this question? I know for a while we stopped asking the question, which is great. I don't do a lot of code interviews at Twilio, so I don't know if we do something similar there, yet. I should go find out.Corey: I do not know either way, to be clear. None of the stories I'm talking about involved Twilio. Though I will say, I went on an interview years ago at SendGrid in Anaheim, and I don't know if I ever got a formal rejection or not afterwards, but regardless, they did not opt to hire me. In hindsight, good decision.Sean: I wonder if we were in the office at the same time.Corey: It would have been 2006, so I think it might have been a bit before your time.Sean: That was before my time.Corey: And very much, credit where due, I started my career in large-scale email systems, so SendGrid was one of those. Oh, I could probably apply the skill set there. The problem, of course, was that it became pretty apparent, even in those days that eventually there weren't going to be that many companies that needed that skillset. The days of an email admin in every company were drawn to a close, and it was time to evolve or die.Sean: You're welcome.Corey: Of course. And again SendGrid today, under the hood—deep under the hood—does still power Last Week in AWS. You folks send emails and get them where they need to go, for which I thank you, and the rest of the world probably does not most weeks.Sean: [laugh]. Yeah.Corey: Ugh. So, we've covered a lot of wide-ranging topics. If people want to hear more about who you are, and what you have to say, where can they find you?Sean: I'm on Twitter at @log1kal with a one and a K because I hate people who want to find me, apparently. But that's @log1kal. Twitter's probably the only thing.Corey: Excellent. We'll, of course, put a link to that in the [show notes 00:30:39]. Sean, thank you so much for speaking with me today. I really appreciate it.Sean: Thank you so much, Corey. I love having these kinds of conversations. I love that there is no plan; we're just going to have a conversation and record it. I love listening to these kinds of podcasts.Corey: Well, I like creating these kinds of podcasts because the other ones take way too much work.Sean: [laugh].Corey: Sean Kilgore, architect at Twilio. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment explaining how it is almost certainly the fault of Silvia Botros's aura.Sean: [laugh].Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.This has been a HumblePod production. Stay humble.

amazon netflix california google man state san francisco quit daddy cloud orange wifi architects bay vc lighthouses chapman puppets screw last week anaheim accelerate screaming aws devops invent emp sop kubernetes chapman university twilio faang kilgore job titles ec2 sendgrid veem emps corey quinn duckbill group orca security in sean sean it sean you chief cloud economist last week in aws humblepod

Innovations and the Changing DevOps Tides of Tech with Nigel Kersten

Play Episode Listen Later Jul 22, 2021 41:12

About NigelNigel Kersten's day job is Field CTO at Puppet where he leads a group of engineers who work with Puppet's largest customers on cultural and organizational changes necessary for large-scale DevOps implementations - among other things. He's a co-author of the industry-leading State Of DevOps Report and likes to evenly talk about what went right with DevOps and what went wrong based on this research and his experience in the field. He's held multiple positions at Puppet across product and engineering and came to Puppet from the Google SRE organization, where he was responsible for one of the largest Puppet deployments in the world. Nigel is passionate about behavioral economics, electronic music, synthesizers, and Test cricket. Ask him about late-stage capitalism, and shoes.Links: Puppet: https://puppet.com 2020 State of DevOps Report: https://puppet.com/resources/report/2020-state-of-devops-report/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: Your company might be stuck in the middle of a DevOps revolution without even realizing it. Lucky you! Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy in on DevOps practices? Well, download the 2021 State of DevOps report brought to you annually by Puppet since 2011 to explore the trends and blockers keeping evolution firms stuck in the middle of their DevOps evolution. Because they fail to evolve or die like dinosaurs. The significance of organizational buy in, and oh it is significant indeed, and why team identities and interaction models matter. Not to mention weither the use of automation and the cloud translate to DevOps success. All that and more awaits you. Visit: www.puppet.com to download your copy of the report now!Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted episode is sponsored by a long time… I wouldn't even say friends so much as antagonist slash protagonist slash symbiotic company with things I have done as I have staggered through the ecosystem. There's a lot of fingers of blame that I can point throughout the course of my career at different instances, different companies, different clients, et cetera, et cetera, that have shaped me into the monstrosity than I am today. But far and away, the company that has the most impact on the way that I speak publicly, is Puppet.Here to accept the recrimination for what I become and how it's played out is Nigel Kersten, a field CTO at Puppet—or the field CTO; I don't know how many of them they have. Nigel, welcome to the show, and how unique are you?Nigel: Thank you, Corey. Well, I—you know, reasonably unique. I think that you get used to being one of the few Australians living in Portland who's decided to move away from the sunny beaches and live in the gray wilderness of the Pacific Northwest.Corey: So, to give a little context into that ridiculous intro, I was a traveling contract trainer for the Puppet fundamentals course for an entire summer back in I want to say 2014, but don't hold me to it. And it turns out that when you're teaching a whole bunch of students who have paid in many cases, a couple thousand dollars out of pocket to learn a new software where, in some cases, they feel like it's taking their job away because they view their job, rightly or wrongly, is writing the same script again and again. And then the demo breaks and people are angry, and if you don't get a good enough rating, you're not invited to continue, and then the company you're contracting through hits you with a stick, it teaches you to improvise super quickly. So, I wasn't kidding when I said that Puppet was in many ways responsible for the way that I give talks now. So, what do you have to say for yourself?Nigel: Well, I have to say, congratulations for surviving, opinionated defensive nerds who think not only you but your entire product you're demoing could be replaced by a shell script. It's a tough crowd.Corey: It was an experience. And some of these were community-based, and some of them were internal to a specific company. And if people have heard more than one episode of this show, I'm sure they can imagine how that went. I gave a training at Comcast once and set a personal challenge for myself of how many times could I use the word ‘comcastic' in a three-day training. And I would work it in and talk about things like the schedule parameter in Puppet where it doesn't guarantee something's going to execute in a time window; it's the only time it may happen.If it doesn't fire off, and then it isn't going to happen. It's like a Comcast service appointment. And then they just all kind of stared at me for a while and, credit where due, that was the best user rating I ever got from people sitting through one of my training. So, thanks for teaching me how to improve at, basically, could have been a very expensive mistake on Puppet's part. It accidentally worked out for everyone.Nigel: Brilliant, brilliant. Yes, you would have survived teaching the spaceship operator to that sort of a crowd.Corey: Oh, I mostly avoided that thing. That was an advanced Puppet-ism, and this was Puppet fundamentals because I just need to be topically good at things, not deep-dive good at things. But let's dig into that a little bit. For those who have not had the pleasure of working with Puppet, what is it?Nigel: Sure? So, Puppet is a pretty simple DSL. You know, DSLs aren't necessarily in favor these days.Corey: Domain-specific language, for those who have not—Nigel: Yep.Corey: —caught up on that acronym. Yes.Nigel: So, a programming language designed for a specific task. And, you know, instead, we've decided that the world will rest on YAML. And we've absorbed a fair bit of YAML into our ecosystem, but there are things that I will still stand by are just better to do in a programming language. ‘if x then y,' for example, it's just easier to express when you have actual syntax around you and you're not, sort of, forcing everything to be in a data specification language. So, Puppet's pretty simple in that it's a language that lets you describe the state that infrastructure should be.And you can do this in a modular and composable way. So, I can build a little chunk of automation code; hand it to Corey; Corey can build something slightly bigger with it; hand it to someone else. And really, this sort of collaboration is one of the reasons why Puppet's, sort of, being at the center of the DevOps movement, which at its core is not really about tools. It's about reducing friction between different groups.Corey: Back when I was doing my traveling training shtick, I found that I had to figure out a way to describe what Puppet did to folks who were not deep in the space, and the analogy that I came up with that I was particularly partial to was, imagine you get a brand new laptop. Well, what do you do with it? You install your user account and go through the setup; you install the programs that you use, some which have licenses on it; you copy your data onto it; you make sure that certain programs always run on startup because that's the way that you work with these things; you install Firefox because that's the browser of choice that you go with, et cetera, et cetera. Now, imagine having to do that for, instead of one computer, a thousand of them, and instead of a laptop, they're servers. And that is directionally what Puppet does.Nigel: Absolutely. This is the one I use for my mother as well. Like, I was working around Puppet for years before—and the way I explained it was, “You know when you get a new iPad, you've got to set up your Facebook account and your email. Imagine you had ten thousand of these.” And she was like—I was like, “You know, companies like Google, company like big banks, they all have lots and lots and lots of computers.” And she was like, “They run all those things on iPads.” And I was like, “This is not really where my analogy was going.” But.Corey: Right. And increasingly, though, it seems like the world has shifted in some direction where, when you explain that to your mother and she comes back with, “Well, wouldn't they just put the application into Docker and be done with it?” Oh, dear. But that seems to be in many ways that the direction that the zeitgeist has moved in, whether or not that is the reality in many environments, where when you're just deploying containers everywhere—through the miracle of Kubernetes—if you'll pardon the dismissive scorn there, that you just package up your application, shove into a container, and then hurl it from the application team over the operations team, like a dead dog cast into your neighbor's yard for him to worry about. And then it sort of takes up the space of you don't have to manage state anymore because everything is mostly stateless in theory. How have you seen it play out in practice in the last five years?Nigel: I mean, that's a real trend. And, you know, the size of a container should be [laugh] smaller than an operating system. And the reality is, I'm a sysadmin; I love operating systems, I nerded out on operating systems. They're a necessary evil, they're terrible, terrible things: registry keys, config files, they're a pain in the neck to deal with. And if you look at, I think what a lot of operations folks missed about Docker when it started was that it didn't make their life better. It was worse.It was, like, this actual, sort of, terrible toolchain where you sort of tied together all these different things. But really importantly, what it did is it put control into the hands of the developers, and it was the developers who were trying to do stuff who were trying to shift into applications. And I think Docker was a really great technology, in the sense of, you know, developers could ship value on their own. And that was the huge, huge leveling up. It wasn't the interface, it wasn't the user experience, it wasn't all these things, it was just that the control got taken away from the IT trolls in their basement going, “No, don't touch my servers,” and instead given straight to the developers. And that's huge because it let us ship things faster. And that's ultimately the whole goal of things.Corey: The thing that really struck me the most from conducting the trainings that I did was meeting a whole bunch of people across the country, in different technological areas of specialty, in different states of their evolution as technologists, and something that struck me was just how much people wound up identifying with the technology that they worked on. When someone is the AIX admin, and the AIX machines are getting replaced with Linux boxes, there's this tendency to fight against that and rebel, rather than learning Linux. And I get it; I'm as subject to this as anyone is. And in many cases, that was the actual pushback that I saw against adopting something like Puppet. If I identify my job as being the person that runs all these carefully curated scripts that I've spent five years building, and now that all gets replaced with something that is more of a global solution to my local problem, then it feels like a thing that made me special is eroding.And we see that with the migration to cloud as well. When you're the storage admin, and it just becomes an API call to S3, that's kind of a scary thing. And when you're one of the server hugger types—and again, as guilty as anyone of this—and you start to see cloud coming in as, like, a rising tide that eats up what it was that you became known for, it's scary and it becomes a foundational shift in how you view yourself. What I really had a lot of sympathy for was the folks who've been doing this for 20 years. They were, in some cases, a few years away from retirement, and they've been doing basically the same set of tasks every year for 25 years.It's one year of experience repeated 25 times. And they don't have that much time left in their career, intentionally, so they want to retire, but they also don't really want to learn a whole bunch of new technologies just to get through those last few years. I feel for them. But at the same time—Nigel: No, me too, totally. But what are you going to do? But without sounding too dismissive there, I think it's a natural tendency for us to identify with the technology if that's what you're around all the time. You know, mechanics do this, truck drivers with brands of trucks, people, like, to build attachments to the technology they work with because we fit them into this bigger techno-social system. But I have a lot of empathy for the people in enterprise jobs who are being asked to change radically because the cycle of progress is speeding up faster and faster.And as you say, they might be a few years away from retirement. I think I used to feel more differently about this when I was really hot-headed and much more of a tech enthusiast, and that's what I identified with. In terms of, it's okay for a job to just be a job for people. It's okay for someone to be doing a job because they get good health care and good benefits and it's feeding their family. That's an important thing. You can't expect everyone to always be incredibly passionate about technology choices in the same way that I think many of us who live on Twitter and hanging out in this space are.Corey: Oh, I have no problem whatsoever with people who want to show up for 40 hours a week-ish, work on their job, and then go home and have lives and not think about computers at all. There's this dark mass of developers out there that basically never show up on Twitter, they aren't on IRC, they don't go to conferences, and that's fine. I have no problem with that, and I hope I don't come across as being overly dismissive of those folks. I honestly wish I could be content like that. I just don't hold still very well.Nigel: [laugh]. Yeah, so I think you touched on a few interesting things there. And some of those we sort of cover in the State of DevOps Report, which is coming out in the next few weeks.Corey: Indeed, and the State of DevOps Report started off at Puppet, and they've now done it for, what, 10 years?Nigel: This is the 10th year, which is completely crazy. So, I was looking at the stats as I was writing it, and it's 10 years of State of DevOps Reports; I think it's 11 years of DevOps Weekly, Gareth Rushgrove's newsletter; it's 12 or 13 years of DevOpsDays that have been going on. This is longer than I spent in primary and high school put together. It's kind of crazy that the DevOps movement is still, kind of, chugging along, even if it's not necessarily the coolest kid on the block, now that GitOps, SRE flavor of the month, various kinds of permutations of how we work with technology, have perhaps got a little bit cooler. But it's still very, very relevant to a lot of enterprises out there.Corey: Yeah. As I frequently say, legacy is a condescending engineering term for ‘it makes money,' and there's an awful lot of that out there. Forget cloud, there are still companies wrestling with do we explore this virtualization thing? And that was something I was very against back in 2006, let's be very honest. I am very bad at predicting the future of technology.And, “I can see this for small niche edge workload cases, where you have a bunch of idle servers, but for the most part, who's really going to use this in production?” Well, basically everyone because that, in turn, is what the cloud runs on. Yeah, I think we can safely say I got that one hilariously wrong. But hey, if you're aren't going to make predictions, then what's it matter?Nigel: But the industry pushes you in these directions. So, there was this massive bank in Asia who I've been working with for a long time and they were always resistant to adopting virtualization. And then it was only four or five years ago that I visited them; they're like, “Right. Okay. It's time. We're rolling out VMware.” And I was like, “So, I'm really curious. What exactly changed in the last year or two in, like, 2014, 2015 that you decided virtualization was the key?” And I'm like—Corey: Oh, there was this jackwagon who conducted this training? Yeah, no, no, sorry. I can't take credit for that one.Nigel: They couldn't order one rack unit servers with CD drives anymore because their whole process was actually provisioning with CDs before that point.Corey: Welcome to the brave new world of PXE booting, which is kind of hard, so yeah, virtualization is easier. You know, sometimes people have to be dragged into various ways of technological advancement. Which gets to the real thing I want to cover, since this is a promoted episode, where you're talking about the State of DevOps Report, I'm almost less interested in what this year's has to say specifically, than what you've seen over the last decade. What's changed? What was true 10 years ago that is very much not true now? Bonus points if you can answer that without using the word Kubernetes more than twice.Nigel: So, I think one of the big things was the—we've definitely passed peak DevOps team, if you may remember, there was a lot of arguments and there's still regular, is DevOps a job title? Is it a team title? Is it a [crosstalk 00:14:33]—Corey: Oh, I was much on the no side until I saw how much more I would get paid as a DevOps engineer instead of a systems administrator for the exact same job. So, you know, I shut up and I took the money. I figured that the semantic arguments are great, but yeah.Nigel: And that's exactly what we've written in the report. And I think it's great. The sysadmins, we were unloved. You know, we were in the basement, we weren't paid as much as programmers. The running joke used to be for developers, DevOps meant, “I don't need ops anymore.” But for ops people, it was, “I can get paid like a developer.”Corey: In many cases, “Oh, well, systems administrators don't want to learn how to code.” It's, yeah, you're remembering a relatively narrow slice of time between the modern era, where systems administrator types need to be able to write in the lingua franca of everything—which is, of course, YAML, as far as programming languages go—and before that, to be a competent systems administrator, you needed to have a functional grasp of C. And—Nigel: Yeah.Corey: —there is only a limited window in which a bunch of bash scripts and maybe a smidgen of Perl would have carried you through. But the deeper understanding is absolutely necessary, and I would argue, always has been.Nigel: And this is great because you've just linked up with one of the things we found really interesting about the report is that you know when we talk about legacy we don't actually mean the oldest shit. Because the oldest shit is the mainframes; it's a lot of bare metal applications. A lot of that in big enterprises—Corey: We're still waiting for an AWS/400 to replace some of that.Nigel: Well, it's administered by real systems engineers, you know, like, the people who wrote C, who wrote kernel extensions, who could debug things. What we actually mean by legacy is we mean late '90s to late 2000s, early 2010s. Stuff that was put together by kids who, like me, happened to get a job because you grew up with a computer, and then the dotcom explosion happened. You weren't necessarily particularly skilled, and a lot of people, they didn't go through the apprenticeships that mainframe folks and systems engineers actually went through. And everyone just held this stuff together with, you know, duct tape and dental floss. And then now we're paying the price of it all, like, way back down the track. So, the legacy is really just a certain slice of rapid growth in applications and infrastructure, that's sort of an unmanageable mess now.Corey: Oh, here in San Francisco, legacy is anything prior to last night's nightly build. It's turned into something a little ridiculous. I feel like the real power move as a developer now is to get a job, go in on day one, rebase everything in the Git repository to a single commit with a message, ‘legacy code' and then force push it to the main branch. And that's the power move, and that's how it works, and that's also the attitude we wind up encountering in a lot of places. And I don't think it serves anyone particularly well to tie themselves so tightly to that particular vision.Nigel: Yep, absolutely. This is a real problem in this space. And one of the things we found in the State of DevOps Report is that—let me back up a little and give a little bit of methodology of what we actually do. We survey people about their performance metrics, you know, like how quickly can you do deploys? What's your mean time to recovery? Those sorts of things, and what practices do you actually employ?And we essentially go through and do statistical analysis on this, and everyone tends to end up in three cohorts, they separate pretty easily, of low, medium, and high evolution. And so one of the things we found is that everyone at the low level has all sorts of problems. They have issues with what does my team do? What does the team next to me do? How do I talk to the team next to me?How do I actually share anything? How do I even know what my goals are? Like, fundamental company problems. But everyone at all levels of evolution is stuck on two big things: not being able to find enough people with the right skills for what they need, and their legacy infrastructure holding them back.Corey: The thing that I find the most compelling is the idea of not being able to find enough people with the skills that they need. And I'm going to break my own rule and mentioned Kubernetes as a prime example of this. If you are effective at managing Kubernetes in production, you will make a very comfortable living in any geographical location on the planet because it is incredibly complex. And every time we've seen this in previous trends, where you need to get more and more complexity, and more and more expertise just to run something, it looks like a sawtooth curve, where at some point that complexity, it gets abstracted away and compressed down into something that is basically a single line somewhere, or it happens below the surface level of awareness. My argument has been that Kubernetes is something no one's going to care about in roughly three years from now, not because we're not using it anymore, but because it's below the level of awareness that we have to think about, in the same way that there aren't a whole lot of people on the planet these days who have to think about the Linux virtual memory management subsystem. It's there and a few people really care about it, but for the rest of us, we don't have to think about that. That is the infrastructure underneath our infrastructure.Nigel: Absolutely. I used to make a living—and it's ridiculous looking back at this—for a year or two, doing high-performance custom compiled Apaches for people. Like, I was really really good at this.Corey: Well yeah, Apache is a great example of this, where back in the '90s, to get a web server up and running you needed to have three days to spare, an in-depth knowledge of GCC compiler flags, and hope for the best. And then RPM came out and then, okay, then YUM or other things like that—Nigel: Exactly.Corey: —on top of it. And then things like Puppet started showing up, and we saw, all right now, [unintelligible 00:20:01] installed. Great. And then we had—it took a step beyond that, and it was, “Oh, now it's just a Docker-run whatever it is,” and these days, yeah, it's a checkbox in S3.Nigel: So, let me get your Kubernetes prediction down, right. So, you're predicting Kubernetes is going to go away like Apache and highly successful things. It's not an OpenStack failure state; it's Apache invisibility state?Corey: Absolutely. My timeline is a bit questionable, let's be fair, but—it's a little on the aggressive side, but yeah, I think that Kubernetes is inherently too complex for most people to have to wind up thinking about it in that way. And we're not talking small companies; we're talking big ones where you're not in a position, if you're a giant blue-chip Fortune 50, to hire 2000 people who all know Kubernetes super well, and you shouldn't have to. There needs to be some flattening of all of that high level of complexity. Without the management tools, though, with things like Puppet and the things that came before and a bunch of different ways, we would all not be able to get anything done because we'd be too busy writing in assembly. There's always going to be those abstractions on top abstractions on top abstractions, and very few people understand how it works all the way down. But that's, in many cases, okay.Nigel: That's civilization, you know? Do you understand what happens when you plug in something to your electricity socket? I don't want to know; I just want light.Corey: And more to the point, whenever you flip the switch, you don't have that doubt in your mind that the light is going to come on. So, if it doesn't, that's notable, and your first thought is, “Oh, the light bulb is out,” not, “The utility company is down.” And we talk about the cloud being utility computing.Nigel: Has someone put a Kubernetes operator in this light switch that may break this process?Corey: Well, okay, IoT does throw a little bit of a crimp into those works. But yeah. So, let's talk more about the State of DevOps Report. What notable findings were there this year?Nigel: So, one of the big things that we've seen for the last couple of years has been that most companies are stuck in the middle of the evolutionary progress. And anyone who deals with large enterprises knows this is true. Whatever they've adopted in terms of technology, in terms of working methods, you know, agile, various different things, most companies don't tend to advance to the high levels; most places stay mired in mediocrity. So, we wanted to dive into that and try and work out why most companies actually stuck like this when they hit a certain size. And it turns out, the problems aren't technology or DevOps, they really fundamental problems like, “We don't have clear goals. I don't understand what the teams next to me do.”We did a bunch of qualitative interviews as well as the quantitative work in the survey with this report, and we talked to one group of folks at a pretty large financial services company who are like, “Our teams have all been renamed so many times, if I need to go and ask someone for something, I literally page up and down through ServiceNow, trying to find out where to put the change request.” And they're like, “How do I know where to put a network port opening request for this particular service when there are 20 different teams that might be named the right thing, and some are obsolete, and I get no feedback whether I've sent it off to the right thing or to a black hole of enterprise despair?”Corey: I really love installing, upgrading, and fixing security agents in my cloud estate! Why do I say that? Because I sell things, because I sell things for a company that deploys an agent, there's no other reason. Because let's face it. Agents can be a real headache. Well, now Orca Security gives you a single tool that detects basically every risk in your cloud environment -- and that's as easy to install and maintain as a smartphone app. It is agentless, or my intro would've gotten me into trouble here, but it can still see deep into your AWS workloads, while guaranteeing 100% coverage. With Orca Security, there are no overlooked assets, no DevOps headaches, and believe me you will hear from those people if you cause them headaches. and no performance hits on live environments. Connect your first cloud account in minutes and see for yourself at orca.security. Thats “Orca” as in whale, “dot” security as in that things you company claims to care about but doesn't until right after it really should have.Corey: That doesn't get better with a lot of modernization. I mean, I feel like half of my job—and I'm not exaggerating—is introducing Amazonians to one another. Corporate communication between departments and different groups is very far from a solved problem. I think the tooling can help but I've never been a big believer in solving political problems with technology. It doesn't work. People don't work that way.Nigel: Absolutely. One of my earliest times working at Puppet doing, sort of, higher-level sales and services and support, huge national telco walk in there; we've got the development team, the QA team, the infrastructure team. In the course of this conversation, one of them makes a comment about using apt-get, and the others were like, “What do you mean? We're on RHEL.” And it turned out, production was running on RHEL, the QA team running on CentOS and the developers were all building everything on Ubuntu. And because it was Java wraps, they almost didn't have to care. But write once, debug everywhere.Corey: History doesn't repeat, but it rhymes; before Docker, so much of development in startup-land was how do I make my MacBook Pro look a lot more like an EC2 Linux instance? And it turns out that there's an awful lot of work that goes into that maybe isn't the best use of people's time. And we start to see these breakthroughs and these revelations in a bunch of different ways. I have to ask. This is the tenth year that you've done the State of DevOps Report. At this point, why keep doing it? Is it inertia? Are you still discovering new insights every year on top of it? Or is it one of those things where well someone in marketing says we have to do it, so here we are?Nigel: No, actually, it's not that at all. So definitely, we're going to take stock after this year because ten years feels like a really good point to, sort of—it's a nice round number in certain kind of number system. Mainly the reason is, a lot of my job is going and helping big enterprises just get better at using technology. And it's funny how often I just get folks going, “Oh, I read this thing,” like people who aren't on the bleeding edge, constantly discussing these things on Twitter or whatever, but the State of DevOps Report makes its way to them, and they're like, “Oh, I read a thing there about how much better it is if we standardized on one operating system. And that made a really huge difference to what we were actually doing because you had all this data in there showing that that is better.”And honestly, that's the biggest reason why I ended up doing it. It's the fact that it seems to be a tool that has made its way through to very hard to penetrate enterprise folks. And they'll read it and managers will read things that are like, “If you set clear goals for your team and get them to focus on optimizing the legacy environment, you will see returns on it.” And I'm being a little bit facetious in the tone that I'm saying because a lot of this stuff does feel obvious if you're constantly swimming in this stuff day-to-day, but it's not just the practitioners who it's just a job for in a lot of big companies. It's true, a lot of the management chain as well. They're not necessarily going out and reading up on modern agile IT management practices day-to-day, for fun; they go home and do something else.Corey: One of my favorite conferences is Gene Kim's DevOps Enterprise Summit, and the specific reason behind that is, these are very large companies that go beyond companies, in some cases, to institutions, where you have the US Air Force as a presenter one year and very large banks that are 200 years old. And every other conference, it seems, more or less involves people getting on stage, deliver conference-ware and tell stories that make people at those companies feel bad about themselves. Where it's, “We're Twitter for Pets, and this is how we deploy software,” or the ever-popular, “This is how Netflix does stuff.” Yeah, Netflix has basically no budget constraints as far as hiring engineering folks go, and lest we forget, their failure mode is someone can't watch a movie right now. It's not exactly the same thing as the ATM starts spitting out the wrong balance in the streets.And I think that there's an awful lot of discussion where people look at the stories people tell on conference stages and come away feeling bad from it. Very often, I'll see someone from a notable tech companies talk about how they do things. And, “Wow, I wish my group did things like that.” And the person next to me says, “Yeah, me too.” And I check and they work at the same company.And the stories we tell are not necessarily the stories that we live. And it's very easy to come away discouraged from these things. And that goes triply so for large enterprises that are regulated, that have significant downside risk if the technology fails them. And I love watching people getting a chance to tell those stories.Nigel: Let me jump in on that really quickly because—Corey: Please, by all means.Nigel: —one is, you know, having done four years at Google, things are a shitshow internally there, too—Corey: You're talking about it like it's prison. I like it.Nigel: —you know. [laugh]. People get horrified when they turn up and they're like, “Oh, what it's not all gleaming, perfect software artifacts, delivered from the hand of Urs.” But I think what Gene has done with DevOps Enterprise Summit is fantastic in how people share more openly their failure states, but even there—and this is an interesting result we found from a few years ago, State of DevOps Report—even those executives are being more optimistic because it's so beaten into you as the senior executive; you're putting on a public face, and even when they're trying to share the warts-and-all story, they can't help but put a little bit of a positive spin on it. Because I've had exactly the same experience there where someone's up there telling a war story, and then I look, turn to the person next to me, and they work at that same 300-year-old bank, and they're like, “Actually, it's much, much worse than this, and we didn't fix it quite as well as that.” So, I think the big tech companies have terrible inside unless they're Netflix, and the big enterprises are also terrible. But they're also—Corey: No, no, I've talked to Netflix people, too. They do terrible things internally there, too. No one talks about the fact that their internal environments are always tire fires, and there are two stories: the stories we tell publicly, and the reality. And if you don't believe me on that, look at any company in the world's billing system. As much as we all talk about agile and various implementations thereof when it comes to things that charge customers money, we're all doing waterfall.Nigel: Absolutely. [laugh].Corey: Because mistakes show when you triple-charge someone's credit card for the cost of a small country's GDP. It's a problem. I want to normalize those sorts of things more. I'm looking forward to reading this year's report, just because it's interesting to see how folks who are in environments that differ from the ones that I get to see experience in this stuff and how they talk about it.Nigel: Yeah. And so one of the big results I think there for big companies that's really interesting is that one of the, sort of, anti-patterns is having lots of different types of teams. And I kind of touched on this before about having confusing team titles being a real problem. And not being able to cross organizational boundaries quickly is really, really—you know, it's a huge inhibitor and cause, source of friction. But turns out the pattern that is actually really great is one that the Team Topologies guys have discovered.If you've been following what Matthew Skelton and Manuel Pais have been doing for a while, they've basically been documenting a pattern in software organizations of a small number of team types, of a platform team, value stream teams, complicated subtest system teams, and enabling teams. And so we worked with Manuel and Matt on this year's report and asked a whole bunch of questions to try and validate the Team Topologies model, and the results came back and they were just incredibly strong. Because I think this speaks to some of the stuff you mentioned before that no one can afford to hire an army of Kubernetes developers, and whatever the hottest technology is in five years, most big companies can't hire an army of those people either. And so the way you get scale internally before those things become commoditized is you build a small team and create the situation where they can have outsized leverage inside their organization, like get rid of all the blockers to fast flow and make their focus self-service to other people. Because if you're making all of your developers learn distributed systems operations arcane knowledge, that's not a good use of their time, either.Corey: It's really not. And I think that's something that gets lost a lot is, I've never yet seen a company beyond the very early startup stage, where the AWS bill exceeded the cost of the people working on the AWS bill. Payroll is always a larger expense than infrastructure unless you're doing something incredibly strange. And, oh, I want to save some money on the cloud bill is very often offset by the sheer amount of time that you're going to have to pay people to work on that because, contrary to what we believe as engineering hobbyists, people's time is very far from free. And it's also the opportunity cost of if you're going to work on this thing instead of something else, well, is that really the best choice? It comes down to contextualizing what technology is doing as well as with what's happening over in the world of business strategy. And without having a bridge between those, it doesn't seem to work very well.Nigel: Absolutely. It's insane. It's literally insane that, as an industry, we will optimize 5%, 3% of our infrastructure bill or application workload and yet not actually reexamine business processes that are causing your people to spend 10% of their time in synchronous meetings. You can save so much more money and achieve so much more by actually optimizing for fast flow, and getting out of the way of the people who cost lots of money.Corey: So, one last topic that I want to cover before we call it an episode. You talk to an awful lot of folks, and it's easy to point at the aspirational stories of folks doing things the right way. But let's dish for a minute. What are you seeing in terms of people not using the cloud properly? I feel like you might have a story or two on that one.Nigel: I do have a few stories. So, in this year's report, one of the things we wanted to find out of, like, are people using the cloud in the way we think of cloud; you know, elastic, consumption-based, all of these sorts of things. We use the NIST metrics, which I recognize can be a little controversial, but I think you've got to start somewhere as a certain foundation. It turns out just about everyone is using the public cloud. And when I say cloud, I'm not really talking about people's internal VMware that they rebadged as cloud; I'm talking about the public cloud providers.Everyone's using it, but almost no one is taking advantage of the functionality of the cloud. They're instead treating it like an on-premise VMware installation from the mid-2000s, they're taking six weeks to provision instances, they're importing all of their existing processes, they keep these things running for a long time if they fall over, one person is tasked with, “Hey, do you know how pet number 45 is actually doing here?” They're not really treating any of these things in the way that they're actually meant to. And I think we forget about this a lot of the time when we talk about cloud because we jump straight to cloud-native, you know, the sort of bleeding edge of folks in serverless, highly orchestrated containers. I think if you look at the actual numbers, the vast majority of cloud usage, it's still things like EC2 instances on AWS. And there's a reason: because it's a familiar paradigm for people. We're definitely going to progress past there, but I think it's easy to leave the people in the middle behind when we're talking about cloud and how to improve the ecosystem that they all operate in.Corey: Part of the problem, too, is that whenever we look at how folks are misusing cloud, it's easy to lose sight of context. People don't generally wake up and decide I'm going to do a terrible job today unless they work in, you know, Facebook's ethics department or something. Instead, it's very much a people are shaped by the constraints they're laboring under from a bunch of different angles, and they're trying to do the best with what they have. Very often, the reason that a practice or a policy exists is because, once upon a time, there was a constraint that may or may not still be there, and going forward the way that they have seemed like the best option at the time. I found that the default assumption that people are generally smart and doing the right thing with the information they have carries you a lot further, in many respects than what I did is a terrible junior consultant, which is, “Oh, what moron built this?” Invariably to said moron, and then the rest of the engagement rapidly goes downhill from there. Try and assume good faith, and if you see something that makes no sense, ask, “Why is it like this?” Rather than, “Why is it like this?” Tone counts for a lot.Nigel: It's the fundamental attribution bias. It's why we think all other drivers on the road are terrible, but we actually had a good reason for swerving into that lane.Corey: “This isn't how I would have built it. So, it's awful.”Nigel: Yeah, exactly.Corey: Yeah. And in some cases, though, there are choices that are objectively bad, but I tried to understand where they came from there. Company policy, historically, around things like data centers, trying to map one-to-one to cloud often miss some nuances. But hey, there's a reason it's called the digital transformation, not a project that we did.Nigel: [laugh]. And I think you've got to always have empathy for the people on the ground. I quite often have talked to folks who've got, like, a terrible cloud architecture with the deployment and I'm like, “Well, what happened here?” And they went, “Well, we were prepared to deploy this whole thing on AWS, but then Microsoft's salespeople got to the CTO and we got told at the last minute we're redeploying everything on Azure.” And so these people were often—you know, you're given a week or two to pivot around the decision that doesn't necessarily make any sense to them.And there may have been a perfectly good reason for the CTO to do this: they got given really good kickbacks in terms of bonuses for, like, how much they were spending on the infrastructure—I mean, discounts—but people on the ground are generally doing the best with what they can do. If they end up building crap, it's because our system, society, capitalism, everything else is at fault.Corey: [laugh]. I have to say, I'm really looking forward to seeing the observations that you wound up putting into this report as soon as it drops. I'm hoping that I get a chance to speak with you again about the findings, and then I can belligerently tell you to justify yourself. Those are my favorite follow-ups.Nigel: [unintelligible 00:37:05].Corey: If people want to get a copy of the report for themselves or learn more about you, where can they find you?Nigel: Just head straight to puppet.com, and it will be on the banner on the front of the site.Corey: Excellent. And will, of course, put a link to that in the show notes, if people can't remember puppet.com. Thank you so much for taking the time to speak with me. I really appreciate it.Nigel: Awesome. No worries. It was good to catch up.Corey: Nigel Kersten, field CTO at Puppet. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice as well as an insulting comment telling me that ‘comcastic' isn't a funny word, and tell me where you work, though we already know.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

amazon netflix google state san francisco tech innovation australian microsoft fortune portland corporate cloud cd ipads pets cto pacific northwest tone cds gdp iot api puppets screaming atm aws linux java comcast us air force qa devops azure tides apache payroll macbook pro ubuntu yum firefox vmware s3 docker git kubernetes rpm aix nist irc servicenow sre gcc dsl urs kersten invariably apaches openstack gene kim ec2 yaml centos gitops launchdarkly corey quinn amazonians rhel team topologies devopsdays dsls devops report manuel pais duckbill group orca security pxe chief cloud economist last week in aws humblepod

All Along the Shoreline.io of Automation with Anurag Gupta