Podcasts about oltp

Share on
Share on Facebook
Share on Twitter
Share on Reddit
Share on LinkedIn
Copy link to clipboard
  • 35PODCASTS
  • 78EPISODES
  • 38mAVG DURATION
  • 5WEEKLY NEW EPISODES
  • Apr 27, 2022LATEST

POPULARITY

20122013201420152016201720182019202020212022


Best podcasts about oltp

Latest podcast episodes about oltp

Screaming in the Cloud
Data Analytics in Real Time with Venkat Venkataramani

Screaming in the Cloud

Play Episode Listen Later Apr 27, 2022 38:41


About VenkatVenkat Venkataramani is CEO and co-founder of Rockset. In his role, Venkat helps organizations build, grow and compete with data by making real-time analytics accessible to developers and data teams everywhere. Prior to founding Rockset in 2016, he was an Engineering Director for the Facebook infrastructure team that managed online data services for 1.5 billion users. These systems scaled 1000x during Venkat's eight years at Facebook, serving five billion queries per second at single-digit millisecond latency and five 9's of reliability. Venkat and his team also created and contributed to many noted data technologies and open-source projects, including Facebook's TAO distributed data store, RocksDB, Memcached, MySQL, MongoRocks, and others. Prior to Facebook, Venkat worked on tools to make the Oracle database easier to manage. He has a master's in computer science from the University of Wisconsin-Madison, and bachelor's in computer science from the National Institute of Technology, Tiruchirappalli.Links Referenced: Company website: https://rockset.com Company blog: https://rockset.com/blog TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by our friends at Revelo. Revelo is the Spanish word of the day, and its spelled R-E-V-E-L-O. It means “I reveal.” Now, have you tried to hire an engineer lately? I assure you it is significantly harder than it sounds. One of the things that Revelo has recognized is something I've been talking about for a while, specifically that while talent is evenly distributed, opportunity is absolutely not. They're exposing a new talent pool to, basically, those of us without a presence in Latin America via their platform. It's the largest tech talent marketplace in Latin America with over a million engineers in their network, which includes—but isn't limited to—talent in Mexico, Costa Rica, Brazil, and Argentina. Now, not only do they wind up spreading all of their talent on English ability, as well as you know, their engineering skills, but they go significantly beyond that. Some of the folks on their platform are hands down the most talented engineers that I've ever spoken to. Let's also not forget that Latin America has high time zone overlap with what we have here in the United States, so you can hire full-time remote engineers who share most of the workday as your team. It's an end-to-end talent service, so you can find and hire engineers in Central and South America without having to worry about, frankly, the colossal pain of cross-border payroll and benefits and compliance because Revelo handles all of it. If you're hiring engineers, check out revelo.io/screaming to get 20% off your first three months. That's R-E-V-E-L-O dot I-O slash screaming.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today's promoted guest episode is one of those questions I really like to ask because it can often come across as incredibly, well, direct, which is one of the things I love doing. In this case, the question that I am asking is, when you look around at the list of colossal blunders that people make in the course of careers in technology and the rest, it's one of the most common is, “Oh, yeah. I don't like the way that this thing works, so I'm going to build my own database.” That is the siren call to engineers, and it is often the prelude to horrifying disasters. Today, my guest is Venkat Venkataramani, co-founder and CEO at Rockset. Venkat, thank you for joining me.Venkat: Thanks for having me, Corey. It's a pleasure to be here.Corey: So, it is easy for me to sit here in my beautiful ivory tower that is crumbling down around me and use my favorite slash the best database imaginable, which is TXT records shoved into Route 53. Now, there are certainly better databases than that for most use cases. Almost anything really, to be honest with you, because that is a terrifying pattern; good joke, terrible practice. What is Rockset as we look at the broad landscape of things that store data?Venkat: Rockset is a real-time analytics platform built for the cloud. Let me break that down a little bit, right? I think it's a very good question when you say does the world really need another database? Don't we have enough already? SQL databases, NoSQL databases, warehouses, and lake houses now.So, if you really break it down, the first digital transformation that happened in the '80s was when people actually retired pen and paper records and started using a relational database to actually manage their business records and what have you instead of ledgers and books and what have you. And that was the first digital transformation. That was—and Oracle called the rows in a table ‘records' for a reason. They're called records to this date. And then, you know, 20 years later, when all businesses were doing system of record and transactions and transactional databases, then analytics was born, right?This was, like, the whole reason why I wanted to make better data-driven business decisions, and BI was born, warehouses and data lakes started becoming more and more mainstream. And there was really a second category of database management systems because the first category it was very good at to be a system of record, but not really good at complex analytics that businesses are asking to be able to guide their decisions. Fast-forward 20 years from then, the nature of applications are changing. The world is going from batch to real-time, your data never stops coming, advent of Apache Kafka and technologies like that, 5G, IoTs, data is coming from all sorts of nooks and corners within an enterprise, and now customers in enterprises are acquiring the data in real-time at a scale that the world has never seen before.Now, how do you get analytics out of that? And then if you look at the database market—entire market—there are still only two large categories of databases: OLTP databases for transaction processing, and warehouses and data lakes for batch analytics. Now suddenly, you need the speed of OLTP at the scale of batch, right, in terms of, like, complexity of compute, complexity of storage. So, that is really why we thought the data management space needs that third leg, and we call it real-time analytics platform or real-time analytics processing. And this is where the data never stops coming; the queries never stopped coming.You need the speed and the scale, and it's about time we innovate and solve the problem well because in 2015, 2016, when I was researching for this, every company that was looking to solve build applications that were real-time applications was building a custom Rube Goldberg machine of sorts. And it was insanely complex, it was insanely expensive. Fast-forward now, you can build a real-time application in a matter of hours with the simplicity of the cloud using Rockset.Corey: There's a lot to be said that the way we used to do things after the first transformation and we got into the world of batch processing, where—in the days of punch cards, which was a bit before my time and I believe yours as well—where they would drop them off and then the next day, or two days, they would come back later after the run, they would get the results only to figure out syntax error because you put the wrong card first or something like that. And it was maddening. In time, that got better, but still, nightly runs have become a thing to the point where even now, by default, if you wind up looking at the typical timing of a default Linux install, for example, you see that in the middle of the night is when a bunch of things will rotate when various cleanup jobs get done, et cetera, et cetera. And that seemed like a weird direction to go in. One of the most famous Google April Fools Day jokes was when they put out their white paper on MapReduce.And then Yahoo fell for it hook, line, and sinker, built out Hadoop, and we've been stuck with this idea of performing these big query jobs on top of existing giant piles of data, where ideally, you can measure it with a wall clock; in practice, you often measure the calendar in some cases. And as the world continues to evolve, being able to do streaming processing and understand in real-time what is going on, is unlocking different approaches, at least by all accounts. Do you have an example you can give me of a problem that real-time analytics solves for a customer? Because I can sit here and talk all day about how things might theoretically work, but I have to get out of my Route 53-based ivory tower over here, what are customers seeing?Venkat: That's a great question. And I want one hundred percent agree. I think Google did build MapReduce, and I think it's a very nice continuation of what happened there and what is happening in the world now. And built MapReduce and they quickly realized re-indexing the whole world [laugh] every night, as the size of the internet is exploding is a bad idea. And you know how Google index is now? They do real-time indexing.That is how they index the wor—you know, web. And they look for the changes that are happening in the internet, and they only index the changes. And that is exactly the same principle behind—one of the core principles behind Rockset's real-time analytics platform. So, what is the customer story? So, let me give you one of my favorite ones.So, the world's number one or number two buy now, pay later company, they have hundreds of millions of users, they have 300,000-plus merchants, they operate in, like, maybe 100-plus countries, so many different payment methods, you can imagine the complexity. At any given point in time, some part of the product is broken, well, Apple Pay stopped working in Switzerland for this e-commerce merchant. Oh God, like, we got to first detect that. Forget even debugging and figuring out what happened and having an incident response team. So, what did they do as they scale the number of payments processed in the system across the world—it's, like, in millions; first, it was millions in the day, and there was millions in an hour—so like everybody else, they built a batch-based system.So, they would accumulate all these payment records, and every six hours—so initially, it was a day, and then afterwards, you know, you try to see how far I can push it, and they couldn't push it beyond every six hours. Every six hours, some batch job would come and process through all the payments that happened, have some statistical models to detect, hey, here are some of the things that you might want to double-click and follow up on. And as they were scaling, the batch job that they will kick off every six hours was starting to take more than six hours. So, you can see how the story goes. Now, fast-forward, they came to us and say—it's almost like Rockset has, like, a big red button that says, “Real-time this.”And then they kind of like, “Can you make this real-time? Because not only that we are losing millions of potential revenue dollars in a year because something stops working and we're not processing payments, and we don't find out about that up to, like, three hours later, five hours later, six hours later, but our merchants are also very unhappy. We are also not able to protect our customers' business because that is all we are about.” And so fast-forward, they use Rockset, and simply using SQL now they have all the metrics and statistical computation that they want to do, happens in real-time, that are accurate up to the second. All of their anomaly detectors run every minute and the anomaly detectors take, like, hundreds of milliseconds to run.And so, now they've cut down the business observability, I would say. It's not metrics and machine observability is actually the—you know, they have now business observability in real-time. And that not only actually saves them a lot of potential revenue loss from downtimes, that's also allowing them to build a better product and give their customers a better experience because they are now telling their merchants and their customers that something is not working in some part of your e-commerce footprint before even the customers notice that something is wrong. And that allows them to build a better product and a better customer experience than their competitors. So, this is a very real-world example of why companies and enterprises are moving from batch to real-time.Corey: With the stories that you, and frankly, a lot of other data analytics companies tend to fall back on all the time has been stories of the ones you're telling, where you're talking about the largest buy now, pay later lender, for example. These are companies operating at massive scale who have tremendous existing transaction volume, and they're built out already. That's great, but then I wanted to try to cut to the truth of some of these things. And when I visit your pricing page at Rockset, it doesn't have what I would expect if that were the only use case. And what that would be is, “Great. Call here to conta—open up a sales quote, and we'll talk to you et cetera, et cetera, et cetera.”And the answer then is, “Okay, I know it's going to have at least two commas in it, ideally, not three, but okay, great.” Instead, you have a free tier where it's, “Hey, we'll give you a pile of credits, here's some limits on our free account, et cetera, et cetera.” Great. That is awesome. So, it tells me that there is a use case here for folks who have not already, on some level, made a good show of starting the process of conquering the world.Rather, someone with an idea some evening at two in the morning can wind up diving in and getting started. What is the Twitter for Pets, in my garage, spare-time side project story for using something like Rockset? What problem will I have as I wind up building those things out, when I don't have any user traffic or data yet, but I want to, you know for once in my life, do the smart thing in advance rather than building an impressive tower of technical debt?Venkat: That is the first thing we built, by the way. When we finish our product, the first thing we built was self-service. The first thing we built was a free forever tier, which has certain limits because somebody has to pay the bill, right? And then we also have compute instances that are very, very affordable that cost you, like, approximately $1 a day. And so, we built all of that because real-time analytics is not a need that only, like, the large-scale companies have. And I'll give you a very, very simple example.Let's say you're building a game, it's a mobile game. You can use Amazon DynamoDB and use AWS Lambdas and have a serverless stack and, like, you're really only paying… you're kind of keeping your footprint very, very small, and you're able to build a very lively game and see if it gets [wider 00:12:16], and it's growing. And once it grows, you can have all the big company scaling problems. But in the early days, you're just getting started. Now, if you think about DynamoDB and Lambdas and whatnot, you can build almost every part of the game except probably the leaderboard.So, how do I build a leaderboard when thousands of people are playing and all of their individual gameplays and scores and everything is just another simple record in DynamoDB. It's all serverless. But DynamoDB doesn't give me a SQL SELECT *, order by score, limit 100, distinct by the same player. No, this is a analytical question, and it has to be updated in real-time, otherwise, you really don't have this thing where I just finished playing. I go to the leaderboard, and within a second or two, if it doesn't update, you kind of lose people along the way. So, this is one of actually a very popular use case, when the scale is much smaller, which is, like, Rockset augments NoSQL database like a Dynamo or a Mongo where you can continue to use that for—or even a Postgres or MySQL for that case where you can use that as your system of record and keep it small, but cover all of your compute-heavy and analytical parts of your application with Rockset.So, it's almost like kind of a CQRS pattern where you use your OLTP database as your system of record, you connect Rockset to it, and so—Rockset comes in with built-in connectors, by the way, so you don't have to write a single line of code for your inserts and updates and deletes in your transactional database to get reflected in Rockset within one to two seconds. And so now, all of a sudden you have a fully indexed, fast SQL replica of your transactional database that on which you can do all sorts of analytical queries and that's fully isolated with your transactional database. So, this is the pattern that I'm talking about. The mobile leaderboard is an example of that pattern where it comes in very handy. But you can imagine almost everybody building some kind of an application has certain parts of it that is very analytical in nature. And by augmenting your transactional database with Rockset, you can have your cake and eat it too.Corey: One of the challenges I think that at least I've run into when it comes to working with data—and let's be clear, I tend to deal with data in relatively small volumes, mostly. The stuff that's significantly large, like, oh, I don't know, AWS bills from large organizations, the format of those is mostly predefined. When I'm building something out, we're using, I don't know, DynamoDB or being dangerous with SQLite or whatnot, invariably I find that even at small-scale, I paint myself into a corner by data model design or how I wind up structuring access or the rest, and the thing that I'm doing that makes perfect sense today winds up being incredibly challenging to change later. And I still, in production and have a DynamoDB table that has the word ‘test' in its name because of course I do.It's not a great place to find yourself in some cases. And I'm curious as to what you've seen, as you've been building this out and watching customers, especially ones who already had significant datasets as they move to you. Do you have any guidance around how to avoid falling down that particular well?Venkat: I will say a lot of the complexity in this world is by solving the right problems using the wrong tool, or by solving the right problem on the wrong part of the stack. I'll unpack this a little bit, right? So, when your patterns change, your application is getting more complex, it is demanding more things, that doesn't necessarily mean the first part of the application you build—and let's say DynamoDB was your solution for that—was the wrong choice. That is the right choice, but now you're expanded the scope of your application and the demand that you have on your backend transactional database. And now you have to ask the question, now in the expanded scope, which ones are still more of the same category of things on why I chose Dynamo and which ones are actually not at all?And so, instead of going and abusing the GSIs and other really complex and expensive indexing options and whatnot, that Dynamo, you know, has built, and has all sorts of limitations, instead of that, what do I really need and what is the best tool for the job, right? What is the best system for that? And how do I augment? And how do I manage these things? And this goes to the first thing I said, which is, like, this tremendous complexity when you start to build a Rube Goldberg machine of sorts.Okay, now, I'm going to start making changes to Dynamo. Oh, God, like, how do I pick up all of those things and not miss a single record? Now, replicate that to another second system that is going to be search-centric or reporting-centric, and do I have to rethink this once in a while? Do I have to build and manage these pipelines? And suddenly, instead of going from one system to two system, you actually end up going from one system to, like, four different things that with all the pipes and tubes going into the middle.And so, this is what we really observed. And so, when you come in to Rockset and you point us at your DynamoDB table, you don't write a single line of code, and Rockset will automatically scan your Dynamo tables, move that into Rockset, and in real-time, your changes, insert, updates, deletes to Dynamo will be reflected in Rockset. And this is all using Dynamo Streams API, Dynamo Scan API, and whatnot, behind the scenes. And this just gives you an example of if you use the right tool for the job here, when suddenly your application is demanding analytical queries on Dynamo, and you do the right research and find the right tool, your complexity doesn't explode at all, and you can still, again, continue to use Dynamo for what it is very, very good at while augmenting that with a system built for analytics with full-featured SQL and other capabilities that I can talk about, for the parts of your application for which Dynamo is not a good fit. And so, if you use the right tool for the job, you should be in very good place.The other thing is part about this wrong part of the stack. I'll give a very kind of naive example, and then maybe you can extrapolate that to, like, other patterns on how people could—you know, accidental complexities the worst. So, let's just say you need to implement access control on your data. Let's say the best place to implement access control is at the database level, just happens to be that is the right thing. But this database that I picked, doesn't really have role-based access control or what have you, it doesn't really give me all the security features to be able to protect the data the way I want it.So, then what I'm going to do is, I'm going to go look at all the places that is actually having business logic and querying the database and I'm going to put a whole bunch of permission management and roles and privileges, and you can just see how that will be so error-prone, so hard to maintain, and it will be impossible to scale. And this is what is the worst form of accidental complexity because if you had just looked at it that one week or two weeks, how do I get something out, or the database I picked doesn't have it, and then the two weeks, you feel like you made some progress by, kind of like, putting some duct tape if conditions on all the access paths. But now, [laugh] you've just painted yourself into a really, really bad corner.And so, this is another variation of the same problem where you end up solving the right problems in the wrong part of the stack, and that just introduces tremendous amount of accidental complexity. And so, I think yeah, both of these are the common pitfalls that I think people make. I think it's easy to avoid them. I would say there's so much research, there's so much content, and if you know how to search for these things, they're available in the internet. It's a beautiful place. [laugh]. But I guess you have to know how to search for these things. But in my experience, these are the two common pitfalls a lot of people fall into and paint themselves in a corner.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: A question I have, though, that is an extension is this—and I want to give some flavor to it—but why is there a market for real-time analytics? And what I mean by that is, early on in my tenure of fixing horrifying AWS bills, I saw a giant pile of money being hurled over at effectively a MapReduce cluster for Elastic MapReduce. Great. Okay, well, stream-processing is kind of a thing; what about migrating to that? Well, that was a complete non-starter because it wasn't just the job running on those things; there were downstream jobs, and with their own downstream jobs. There were thousands of business processes tied to that thing.And similarly, the idea of real-time analytics, we don't have any use for that because of, oh I don't know, I only wind up pulling these reports on a once-a-week basis, and that's fine, so what do I need that updated for in real-time if I'm looking at them once a week? In practice, the answer is often something aligned with the, “Well, yeah, but you had a real-time updating dashboard, you would find that more useful than those reports.” But people's expectations and business processes have shaped themselves around constraints that now can be removed, but how do you get them to see that? How do you get them to buy in on that? And then how do you untangle that enormous pile of previous constraint into something that leverages the technology that's now available for a brighter future?Venkat: I think [unintelligible 00:21:40] a really good question, who are the people moving to real-time analytics? What do they see? And why can they do it with other tech? Like, you know, as you say… EMR, you know, it's just MapReduce; can't I just run it in sort of every twenty-four hours, every six hours, every hour? How about every five minutes? It doesn't work that way.Corey: How about I spin up a whole bunch of parallel clusters on different timescales so I constantly—Venkat: [laugh].Corey: Have a new report coming in. It's real-time, except—Venkat: Exactly.Corey: You're constantly putting out new ones, but they're just six hours delayed every time.Venkat: Exactly. So, you don't really want to do this. And so, let me unpack it one at a time, right? I mean, we talked about a very good example of a business team which is building business observability at the buy now, pay later company. That's a very clear value-prop on why they want to go from batch to real-time because it saves their company tremendous losses—potential losses—and also allows them to build a better product.So, it could be a marketing operations team looking to get more real-time observability to see what campaigns are working well today and how do I double down and make sure my ad budget for the day is put to good use? I don't have to mention security operations, you know, needing real-time. Don't tell me I got owned three days ago. Tell me—[laugh] somebody is, you know, breaking glass and might be, you know, entering into your house right now. And tell me then and not three days later, you know—Corey: “Yeah, what alert system do you have for security intrusion?” “So, I read the front page of_The New York Times_ every morning and waiting to see my company's name.” Yeah, there probably are better ways to reduce that cycle time.Venkat: Exactly, right. And so, that is really the need, right? Like, I think more and more business teams are saying, “I need operational intelligence and not business intelligence.” Don't make me play Monday morning quarterback.My favorite analogy is it's the middle of the third quarter. I'm six points down. A couple of people, star players in my team and my opponent's team are injured, but there's some in offense, some in defense. What plays do I do and how do I play the game slightly differently to change the outcome of the game and win this game as opposed to losing by six points. So, that I think is kind of really what is driving businesses.You know, I want to be more agile, I want to be more nimble, and take, kind of, being data-driven decision-making to another level. So that, I think, is the real force in play. So, now the real question is, why can they do it already? Because if you go ask a hundred people, “Do you want fast analytics on real-time data or slow analytics on stale data?” How many people are going to say give me slow and stale? Zero, right? Exactly zero people.So, but then why hasn't it happened yet? I think it goes back to the world only has seen two kinds of databases: Transaction processing systems, built for system of record, don't lose my data kind of systems; and then batch analytics, you know, all these warehouses and data lakes. And so, in real-time analytics use cases, the data never stops coming, so you have to actually need a system that is running 24/7. And then what happens is, as soon as you build a real-time dashboard, like this example that you gave, which is, like, I just want all of these dashboards to automatically update all the time, immediately people respond, says, “But I'm not going to be like Clockwork Orange, you know, toothpicks in my eyelids and be staring at this 24/7. Can you do something to alert or detect some anomalies and tap on my shoulder when something off is going on?”And so, now what happens is somebody's actually—a program more than a person—is actually actively monitoring all of these metrics and graphs and doing some analysis, and only bringing this to your attention when you really need to because something is off, right? So, then suddenly what happens is you went from, accumulate all the data and run a batch report to [unintelligible 00:25:16], like, the data never stops coming, the queries never stopped coming, I never stop asking questions; it's just a programmatic way of asking those things. And at that point, you have a data app. This is not a analytics dashboard report anymore. You have a full-fledged application.In fact, that application is harder to build and scale than any application you've ever built before [laugh] because in those situations, again, you don't have this torrent of data coming in all the time and complex analytical questions you're asking on the data 24/7, you know? And so, that I think is really why real-time analytics platform has to be built as almost a third leg. So, this is what we call data apps, which is when your data never stops coming and your queries never stop coming. So, this is really, I think, what is pushing all the expensive EMR clusters or misusing your warehouse, misusing your data lakes. At the end of the day, is what is I think blowing up your Snowflake bills, is what blowing up your warehouse builds because you somehow accidentally use the wrong tool for the job [laugh] going back to the one that we just talked about.You accidentally say, “Oh, God, like, I just need some real-time.” With enough thrust, pigs can fly. Is that a good idea? Probably not, right? And so, I don't want to be building a data app on my warehouse just because I can. You should probably use the best tool for the job, and really use something that was built ground up for it.And I'll give you one technical insight about how real-time analytics platforms are different than warehouses.Corey: Please. I'm here for this.Venkat: Yes. So really, if you think about warehouses and data lakes, I call them storage-optimized systems. I've been building databases all my life, so if I have to really build a database that is for batch analytics, you just break down all of your expenses in terms of let's say, compute and storage. What I'm burning 24/7 is storage. Compute comes and goes when I'm doing a batch data load, or I'm running—an analyst who logs in and tries to run some queries.But what I'm actually burning 24/7 is storage, so I want to compress the heck out of the data, and I want to store it in very cheap media. I want to store it—and I want to make the storage as cheap as possible, so I want to optimize the heck out of the storage use. And I want to make computation on that possible but not efficient. I can shuffle things around and make the analysis possible, but I'm not trying to be compute-efficient. And we just talked about how, as soon as you get into real-time analytics, you very quickly get into the data app business. You're not building a real-time dashboard anymore, you're actually building your application.So, as soon as you get into that, what happens is you start burning both storage and compute 24/7. And we all know, relatively, [laugh] compute and RAM is about a hundred to a thousand times more expensive than storage in the grand scheme of things. And so, if you actually go and look at your Snowflake bill, if you go look at your warehouse bill—BigQuery, no matter what—I bet the computational part of it is about 90 to 95% of the bill and not the storage. And then, if you again, break down, okay, who's spending all the compute, and you'll very quickly narrow down all these real-time-y and data app-y use cases where you can never turn off the compute on your warehouse or your BigQuery, and those are the ones that are blowing up your costs and complexity. And on the Rockset side, we are actually not storage-optimized; we're compute-optimized.So, we index all the data as it comes in. And so, the storage actually goes slightly higher because the, you know, we stored the data and also the indexes of those data automatically, but we usually fold the computational cost to a quarter of what a typical warehouse needs. So, the TCO for our customers goes down by two to four folds, you know? It goes down by half or even to a quarter of what they used to spend. Even though their storage cost goes up in net, that is a very, very small fraction of their spend.And so really, I think, good real-time analytics platforms are all compute-optimized and not storage-optimized, and that is what allows them to be a lot more efficient at being the backend for these data applications.Corey: As someone who spends a lot of time staring into the depths of AWS bills, I think that people also lose sight of the reality that it doesn't matter what you're spending on AWS; it invariably pales in comparison to what you're spending on people to work with these things. The reason to go to cloud is not because it is the cheapest possible way to get computers to do things; it's because it's a capability story. It's about unlocking capacity and capabilities you do not have otherwise. And that dramatically increases your feature velocity and it lets you to achieve things faster, sooner, with better results. And unlocking a capability is always going to be more interesting to a company than saving money on it. When a company cares first, last, and always about just save money, make the bill lower, the end, it's usually a company in decline. Or alternately, something very strange is going on over there.Venkat: I agree with that. One of our favorite customers told us that Rockset took their six-month roadmap and shrunk it to a single afternoon. And their supply chain SaaS backend for heavy construction, 80% of concrete that are being delivered and tracked in North America follows through their platform, and Rockset powers all of their real-time analytics and reporting. And before Rockset, what did they have? They had built a beautiful serverless stack using DynamoDB, even have AWS Lambdas and what-have-you.And why did they have to do all serverless? Because the entire team was two people. [laugh]. And maybe a third person once in a while, they'll get, so 2.5. Brilliant people, like, you know, really pioneers of building an entire data stack on AWS in a serverless fashion; no pipes, no ETL.And then they were like, oh God, finally, I have to do something because my business demands and my customers are demanding real-time reporting on all of these concrete trucks and aggregate trucks delivering stuff. And real-time reporting is the name of the game for them, and so how do I power this? So, I have to build a whole bunch of pipes, deliver it to, like, some Elasticsearch or some kind of like a cluster that I had to keep up in real-time. And this will take me a couple of months, that will take me a couple of months. They came into Rockset on a Thursday, built their MVP over the weekend, and they had the first working version of their product the following Tuesday.So—and then, you know, there was no turning back at that point, not a single line of code was written. You know, you just go and create an account with Rockset, point us at your Dynamo, and then off you go. You know, you can use start using SQL and go start building your real-time application. So again, I think the tremendous value, I think a lot of customers like us, and a lot of customers love us. And if you really ask them what is one thing about Rockset that you really like, I think it'll come back to the same thing, which is, you gave me a lot of time back.What I thought would take six months is now a week. What I thought would be three weeks, we got that in a day. And that allows me to focus on my business. I want to spend more time with my stakeholders, you know, my CPO, my sales teams, and see what they need to grow our business and succeed, and not build yet another data pipeline and have data pipelines and other things coming out of my nose, you know? So, at the end of the day, the simplicity aspects of it is very, very important for real-time analytics because, you know, we can't really realize our vision for real-time being the new default in every enterprise for whenever analytics concern without making it very, very simple and accessible to everybody.And so, that continues to be one of our core thing. And I think you're absolutely right when you say the biggest expense is actually the people and the time and the energy they have to spend. And not having to stand up a huge data ops team that is building and managing all of these things, is probably the number one reason why customers really, really like working with our product.Corey: I want to thank you for taking so much time to talk me through what you're working on these days. If people want to learn more, where's the best place to find you?Venkat: We are Rockset, I'll spell it out for your listeners ROCKSET—rock set—rockset.com. You can go there, you can start a free trial. There is a blog, rockset.com/blog has a prolific blog that is very active. We have all sorts of stories there, and you know engineers talking about how they implemented certain things, to customer case studies.So, if you're really interested in this space, that's one on space to follow and watch. If you're interested in giving this a spin, you know, you can go to rockset.com and start a free trial. If you want to talk to someone, there is, like, a ‘Request Demo' button there; you click it and one of our solutions people or somebody that is more familiar with Rockset would get in touch with you and you can have a conversation with them.Corey: Excellent. And links to that will of course go in the [show notes 00:34:20]. Thank you so much for your time today. I appreciate it.Venkat: Thanks, Corey. It was great.Corey: Venkat Venkataramani, co-founder and CEO at Rockset. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an insulting crappy comment that I will immediately see show up on my real-time dashboard.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Google Cloud Platform Podcast
Spanner Myths Busted with Pritam Shah and Vaibhav Govil

Google Cloud Platform Podcast

Play Episode Listen Later Apr 20, 2022 35:47


This week, we're busting myths around Cloud Spanner with our guests Pritam Shah and Vaibhav Govil. Mark Mirchandani and Max Saltonstall host this episode and learn about the fantastic capabilities of Cloud Spanner. Our guests give us a quick run-down of Spanner database software and its fully-managed offerings. Spanner's unique take on the relational database has sparked some myths. We start by addressing cost and the idea that Spanner is expensive. With its high availability achieved through synchronously replicating data, failures are virtually a non-issue, making the cost well worth it. Our guests describe other features that add to the value of Spanner as well. Workloads of any size are a good fit for Spanner because of its scalability and pricing based on use. Despite rumors, Spanner is now very easy to start using. New additions like the PostgreSQL interface and ORM support have made the usability of Spanner much more familiar. Regional and multi-regional instances are supported, busting the myth that Spanner is only good for global workloads. Our guests offer examples of projects using local and global configurations with Spanner. In the database world, Vaibhav sees trends like the convergence of non-relational and relational databases as well as convergence in the OLTP and OLAP database semantics, and he tells us how Spanner is adapting and growing with these trends. Pritam points out that customers are paying more attention to total cost of ownership, the importance of scalable and reliable database solutions, and the peace of mind that comes with a managed database system. Spanner helps customers with these, freeing up business resources for other things. This year, Spanner has made many announcements about new capabilities coming soon, like PostgreSQL interface on spanner GA, Query Insights visualization tools, cross-regional backups GA, and more. We hear all about these awesome updates. Pritam Shah Pritam is the Director of Engineering for Cloud Spanner. He has been with Google for about four and a half years. Before Spanner, he was the Engineering Lead for observability libraries at Google. That included Distributed Tracing and Metrics at Google scale. His mission was to democratize the instrumentation libraries. That is when he launched Open Census and then took on Cloud Spanner. Vaibhav Govil Vaibhav is the Product lead for Spanner. He has been in this role for the past three years, and before this he was a Product Manager in Google Cloud Storage in Google. Overall, he has spent close to four years at Google, and it has been a great experience. Cool things of the week Our plans to invest $9.5 billion in the U.S. in 2022 blog A policy roadmap for 24⁄7 carbon-free energy blog SRE Prodcast site Meet the people of Google Cloud: Grace Mollison, solutions architect and professional problem solver blog GCP Podcast Episode 224: Solutions Engineering with Grace Mollison and Ann Wallace podcast Interview Spanner site Cloud Spanner myths busted blog PostgreSQL interface docs Cloud Spanner Ecosystem site Spanner: Google's Globally-Distributed Database white paper Spanner Docs docs Spanner Qwiklabs site Using the Cloud Spanner Emulator docs GCP Podcast Episode 62: Cloud Spanner with Deepti Srivastava podcast GCP Podcast Episode 248: Cloud Spanner Revisited with Dilraj Kaur and Christoph Bussler podcast Cloud Spanner federated queries docs What's something cool you're working on? Max is working on a new podcast platform and some spring break projects. Hosts Mark Mirchandani and Max Saltonstall

Screaming in the Cloud
Throwing Houlihans at MongoDB with Rick Houlihan

Screaming in the Cloud

Play Episode Listen Later Mar 24, 2022 40:44


About RickI lead the developer relations team for strategic accounts at MongoDB. My responsibilities include defining technical standards for the global strategic accounts team and consulting with the largest customers and opportunities for the business. My role spans technology sectors and as part of my engagements I routinely provide guidance on industry best practices, technology transformation, distributed systems implementation, cloud migration, and more. I led the architecture and design effort at Amazon for migrating thousands of relational workloads from RDBMS to NoSQL and built the center of excellence team responsible for defining the best practices and design patterns used today by thousands of Amazon internal service teams and AWS customers. I currently operate as the technical leader for our global strategic account teams to build the market for MongoDB technology by facilitating center of excellence capabilities within our customer organizations through training, evangelism, and direct design consultation activities.30+ years of software and IT expertise.9 patents in Cloud Virtualization, Complex Event Processing, Root Cause Analysis, Microprocessor Architecture, and NoSQL Database technology.Links: MongoDB: https://www.mongodb.com/ Twitter: https://twitter.com/houlihan_rick TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance. To learn more visit: snark.cloud/deployandgoCorey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. A year or two before the pandemic hit, I went on a magical journey to a mythical place called Australia. I know, I was shocked as anyone to figure out that this was in fact real. And while I was there, I gave the opening keynote at a conference that was called Latency Conf, which is great because there's a heck of a timezone shift, and I imagine that's what it's talking about.The closing keynote was delivered by someone I hadn't really heard of before, and he started talking about single table design with respect to DynamoDB, which, okay, great; let's see what he's got to say. And the talk started off engaging and entertaining and a high-level overview and then got deeper and deeper and deeper and I felt, “Can I please be excused? My brain is full.” That talk was delivered by Rick Houlihan, who now is the Director of Developer Relations for Strategic Accounts over at MongoDB, and I'm fortunate enough to be able to get him here to more or less break down some of what he was saying back then, catch up with what he's been up to, and more or less suffer my slings and arrows. Rick, thank you for joining me.Rick: Great. Thanks, Corey. I really appreciate—you brought back some memories, you know, trip down memory lane there. And actually, interestingly enough, that was the world's introduction to single table design was that. That was my dry-run rehearsal for re:Invent 2018 is where I delivered that talk, and it has become since the most positive—Corey: This was two weeks before re:Invent, which was just a great thing. I'd been invited to go; why not? I figured I'd see a couple of clients I had out in that direction. And I learned things like Australia is a big place. So, doing a one-week trip, including Sydney, Melbourne, and Perth. Don't do that.Rick: I had no idea that it took so long to fly from one side to the other, right? I mean, that's a long plane [laugh] [crosstalk 00:02:15]—Corey: Oh, yeah. And you were working at AWS at the time—Rick: Absolutely.Corey: —so I can only assume that they basically stuffed you into a dog kennel and threw you underneath the seating area, given their travel policy?Rick: Well, you know, I have the—[clear throat] actually at the time, they just upgraded the policy to allow the intermediate seating, right? So, if you wanted to get the—Corey: Ohhh—Rick: I know—Corey: Big spender. Big spender.Rick: Yes, yes. I can get a little bit extra legroom, so I didn't have my knees shoved into some of these back. But it was good.Corey: So, let's talk about, I guess… we'll call it the elephant in the room. You were at MongoDB, where you were a big proponent of the whole no-SQL side of the world. Then you went to go work at AWS and you carried the good word of DynamoDB far and wide. It made an impression; I built my entire newsletter pipeline production system on top of DynamoDB. It has the same data in three different tables because I'm not good at listening or at computers.But now you're back at Mongo. And it's easy to jump to the conclusion of, “Oh, you're just shilling for whoever it is that happens to sign your paycheck.” And at this point, are you—what's the authenticity story? But I've been paying attention to what you've been saying, and I think that's a bad take because you have been saying the same things all along since before you were on the Dynamo side of it. I do some research for this show, and you've been advocating for outcomes and the right ways to do things. How do you view it?Rick: That's basically the story here, right? I've always been a proponent of NoSQL. You know, what I took—the knowledge—it was interesting, the knowledge I took from MongoDB evolved as I went to AWS and I delivered, you know, thousands of applications and deployed workloads that I'd never even imagined I would have my hands on before I went there. I mean, honestly, what a great place it was to cut your teeth on data modeling at scale, right? I mean, that's the—there is no greater scale.That's when you learn where things break. And honestly, a lot of the lessons I took from MongoDB, well, when I applied them at scale at AWS, they worked with varying levels of success, and we had to evolve those into the sets of design patterns, which I started to propose for DynamoDB customers, which had been highly effective. I still believe in all those patterns. I would never tell somebody that they need to drop everything and run to MongoDB, but, you know, again, all those patterns apply to MongoDB, too, right? A very—a lot—I wouldn't say all of them, but many of them, right?So, I'm a proponent of NoSQL. And I think we talked before the call a little bit about, you know, if I was out there hocking relational technology right now and saying RDBMS is the future, then everybody who criticizes anything I say, I would absolutely have to, you know, say that there's some validity there. But I'm not saying anything different I've ever said. MongoDB announced Serverless, if you remember, in July, and that was a big turning point for me because the API that we offer, the developer experience for MongoDB is unmatched, and this is what I talk to people now. And it's the patterns that I've always proposed, I still model data the same way, I don't do it any different, and I've always said, if you go back to my earlier sessions on NoSQL, it's all the same.It doesn't matter if it's MongoDB, DynamoDB, or any other technology. I've always shown people how to model their data and NoSQL and I don't care what database you're using, I've actually helped MongoDB customers do their job better over the years as well. So.Corey: Oh, yeah. And looking back at some of your early talks as well, you passed my test for, “Is this person a shill?” Because you wound up in those talks, addressing head-on when is a relational model the right thing to do? And then you put the answers up on a slide, and this—and what—it didn't distill down to, “If you're a fool.”Rick: [laugh].Corey: Because there are use cases where if you don't [unintelligible 00:05:48] your access patterns, if you have certain constraints and requirements, then yeah. That you have always been an advocate for doing the right thing for the workload. And in my experience, for my use cases, when I looked at MongoDB previously, it was not a fit for me. It was very much a you run this on an instance basis, you have to handle all this stuff. Like three—you kno, keeping it in triplicate in three different DynamoDB tables, my newsletter production pipeline now, including backups and the rest, of DynamoDB portion has climbed to the princely sum of $1.30 a month, give or take.Rick: A month. Yes, exactly.Corey: So, there's no answer for that there. Now that Mongo Serverless is coming out into the world, oh, okay, this starts to be a lot more compelling. It starts to be a lot more flexible.Rick: I was just going to say, for your use case there, Corey, you're probably looking at the very similar pricing experience now, with MongoDB Serverless. Especially when you look at the pricing model, it's very close to the on-demand table model. It actually has discounted tiering above it, which I haven't really broken it down yet against a provision capacity model, but you know, there's a lot of complexity in DynamoDB pricing. And they're working on this, they'll get better at it as well, but right now you have on-demand, you have provisioned throughput, you have [clear throat] reserved capacity allocations. And, you know, there's a time and place for all of those, but it puts the—again, it's just complexity, right?This is the problem that I've always had with DynamoDB. I just wish that we'd spent more time on improving the developer experience, right, enhancing the API, implementing some of these features that, you know, help. Let's make single table design a first-class citizen of the DynamoDB API. Right now it's a red—it's a—I don't want to say redheaded stepchild, I have two [laugh] I have two redhead children and my wife is redhead, but yeah. [laugh].Corey: [laugh]. That's—it's—Rick: That's the way it's treated, right? It's treated like a stepchild. You know, it's like, come on, we're fully funding the solutions within our own umbrella that are competing with ourselves, and at the same time, we're letting the DynamoDB API languish while our competitors are moving ahead. And eventually, it just becomes, you know, okay, guys, I want to work with the best tooling on the market, and that's really what it came down to. As long as DynamoDB was the king of serverless, yes, absolutely; best tooling on the market.And they still are [clear throat] the leader, right? There's no doubt that DynamoDB is ahead in the serverless landscape, that the MongoDB solution is in its nascency. It's going to be here, it's going to be great, that's part of what I'm here for. And that's again, getting back to why did you make the move, I want to be part of this, right? That's really what it comes down to.Corey: One of the things that I know that was my own bias has always been that if I'm looking at something like—that I'm looking at my customer environments to see what's there, I can see DynamoDB because it has its own line item in the bill. MongoDB is generally either buried in marketplace charges, or it's running on a bunch of EC2 instances, or it just shows up as data transfer. So, it's not as top-of-mind for the way that I view things in… through the lens of you know, billing. So, that does inform my perception, but I also know that when I'm talking to large-scale companies about what they're doing, when they're going all-in on AWS, a large number of them still choose things like Mongo. When I've asked them why that is, sometimes you get the answer of, “Oh, legacy. It's what we built on before.” Cool—Rick: Sure.Corey: —great. Other times, it's a, “We're not planning to leave, but if we ever wanted to go somewhere else, it's nice to not have to reimagine the entire data architecture and change the integration points start to finish because migrations are hard enough without that.” And there is validity to the idea of a strategic exodus being possible, even if it's not something you're actively building for all the time, which I generally advise people not to do.Rick: Yeah. There's a couple things that have occurred over the last, you know, couple of years that have changed the enterprise CIO and CTO's assessment of risk, right? Risk is the number one decision factor in a CTOs portfolio and a CIO's, you know, decision-making process, right? What is the risk? What is the impact of that risk? Do I need to mitigate that risk, or do I accept that risk? Okay?So, right now, what you've seen is with Covid, people have realized that you know, on-prem infrastructure is a risk, right? It used to be an asset; now it's a risk. Those personnel that have to run that on-prem infrastructure, hey, what happens when they're not available? The infrastructure is at risk. Okay.So, offloading that to cloud providers is the natural solution. Great. So, what happens when you offload to a cloud provider and IAD goes down, or you know, us-east-1 goes down—we call it IAD or we used to call it IAD internally at AWS when I was there because, you know, the regions were named by airport codes, but it's us-east-1—how many times has us-east-1 had problems? Do you want to really be the guy that every time us-east-1 goes down, you're in trouble? What happens when people in us-east-1 have trouble? Where do they go?Corey: Down generally speaking.Rick: [crosstalk 00:10:37]—well, if they're well-architected, right, if they're well-architected, what do they do? They go to us-west-2. How much infrastructure is us-west-2 have? So, if everybody in us-east-1 is well-architected, then they all go to us-west-2. What happens in us-west-2? And I guarantee you—and I've been warning about this at AWS for years, there's a cascade failure coming, and it's going to be coming because we're well-architecting everybody to failover from our largest region to our smaller regions.And those smaller regions, they cannot take the load and nobody's doing any of that planning, so, you know, sooner or later, what you're going to see is dominoes fall, okay? [clear throat]. And it's not just going to be us-east-1, it's going to be us-east-1 failed, and the rollover caused a cascade failure in us-west-2, which caused a cascade—Corey: Because everyone's failing over during—Rick: That's right. That's right.Corey: —this event the same way. And also—again, not to dunk on them unnecessarily, but when—Rick: No, I'm not dunking.Corey: —us-east-1 goes, down a lot of the control plane services freeze up—Rick: Oh, of course they do.Corey: —like [unintelligible 00:11:25].Rick: Exactly. Oh, we not single point of failure, right? Uh-huh, exactly. There you go, Route 53, now—and that actually surprised me is DynamoDB instead of Route 53 is your primary database. So, I'm actually must have had some impact on you—Corey: To move one workload off of Dynamo to Route 53 [crosstalk 00:11:39] issue number because I have to practice what I preach.Rick: That's right. Exactly.Corey: It was weird; they the thing slower and little bit less, uh—Rick: [laugh]. I love it when [crosstalk 00:11:45]—yeah, yeah—Corey: —and a little bit [crosstalk 00:11:45] cache-y. But yeah.Rick: —sure. Okay, I can understand that. [laugh].Corey: But it made the architecture diagram a little bit more head-scratching, and really, that's what it's all about. Getting a high score.Rick: Right. So, if you think about your data, right, I mean, would you rather be running on an infrastructure that's tied to a cloud provider that could experience these kinds of regional failures and cascade failures, or would you rather have your data infrastructure go across cloud providers so that when provider has problems, you can just go ahead and switch the light bulb over on the other one and ramp right back up, right? You know? And honestly, you're running active, active configurations and that kind of, [clear throat] you know, deployment, you know, design, and you're never going to go down. You're always going—Corey: The challenge I've had—Rick: —to be the one that stays up.Corey: The theory is sound, but the challenge I've had in production with trying these things is that one, the thing that winds up handling the failover piece is often causes more outage than the underlying stuff itself.Rick: Well, sure. Yeah.Corey: Two, when you're building something to run a workload to run in multiple cloud providers, you're forced to use a lot of—Rick: Lowest common denominator?Corey: Lowest common denominator stuff. Yeah.Rick: Yeah, yeah totally. I hear that all the time.Corey: Unless you're actively running it in both places, it looks like a DR Plan, which doesn't survive the next commit to the codebase. It's the—Rick: I totally buy that. You're talking about the stack, stack duplication, all that kind of—that's an overhead and complexity, I don't worry about at the data layer, right?Corey: Oh, yeah.Rick: The data layer—Corey: If you're talking about—Rick: —[crosstalk 00:12:58]Corey: —[crosstalk 00:12:58] data layer, oh, everything you're saying makes perfect sense.Rick: Makes perfect sense, right? And honestly, you know, let's put it this way: If this is what you want to do—Corey: What do you mean identity management and security handover working differently? Oh, that's a different team's problem. Oh, I miss those days.Rick: Yeah, you know, totally right. It's not ideal. But you know, I mean, honestly, it's not a deal that somebody wants to manage themselves, is moving that data around. The data is the lock-in. The data is the thing that ties you to—Corey: And the cost of moving it around in some cases, too.Rick: That's exactly right. You know, so you know, having infrastructure that spans providers and spans both on-prem and cloud, potentially, you know, that can span multiple on-prem locations, man, I mean, that's just that's power. And MongoDB provides that; I mean, DynamoDB can't. And that's really one of the biggest limitations that it will always have, right? And we talked about, and I still believe in the power of global tables, and multi-region deployments, and everything, it's all real.But these types of scenarios, I think this is the next generation of failure that the cloud providers are not really prepared for, they haven't experienced it, they don't know what it's even going to look like, and I don't think you want to be tied to a single provider when these things start happening, right, if you have a large amount of infrastructure deployed someplace. It just seems like [clear throat] that's a risk that you're running at these days, and you can mitigate that risk somewhat by going with a MongoDB Atlas. I agree, all those other considerations. But you know, I also heard—it's a lot of fun, too, right? There's a lot of fun in that, right?Because if you think about it, I can deploy technologies in ways on any cloud provider, they're going to be cloud provider agnostic, right? I can use, you know, containerized technologies, Kubernetes, I can use—hell, I'm not even afraid to use Lambda functions, and just, you know, put a wrapper around that code and deploy it both as a Lambda or a Cloud Function in GCP. The code's almost the same in many cases, right? What it's doing with the data, you can code this stuff in a way—I used to do it all the time—you abstract the data layer, right? Create a DAL. How about a CAL? A cloud [laugh] cloud access layer, right, you know? [laugh].Corey: I wish, on some level, we could go down some of these paths. And someone asked me once a while back of, “Well, you seem to have a lot of opinions on this. Do you think you could build a better cloud than AWS?” And my answer—Rick: Hell yes.Corey: —look them a bit by surprise of, “Absolutely. Step one, I want similar resources, so give me $20 billion to spend”—Rick: I was going to say, right?Corey: —”then I'm going to hire the smart people.” Not that we're somehow smarter or better or anything else than the people who built AWS originally, but now—Rick: We have all those lessons learned.Corey: —we have fifteen years of experience to fall back on.Rick: Exactly.Corey: “Oh. I wouldn't make that mistake again.”Rick: Exactly. Don't need to worry about that. Yeah exactly.Corey: You can't just turn off a cloud service and relaunch it with a completely different interface and API and the rest.Rick: People who criticize, you know, services like DynamoDB, like—and other AWS services—look, these things are like any kind of retooling of the services, it's like rebuilding the engine on the airplane while it's flying.Corey: Oh, yeah.Rick: And you have to do it with a level of service assurance that—I mean, come on. DynamoDB provides four nines out of the box, right? Five nines if you turn on global tables. And they're doing this at the same time as they have pipeline releases dropping regularly, right? So, you can imagine what kind of, you know, unit testing goes on there, what kind of Canary deployments are happening.It's just, it's an amazing infrastructure that they maintain, incredibly complex, you know? In some ways, these are lessons that we need to learn in MongoDB if we're going to be successful operating a shared backplane serverless, you know, processing fabric. We have to look at what DynamoDB does right. And we need to build our own infrastructure that mirrors those things, right? And in some ways, these things are there, in some ways, they're working on, in some ways, we got a long ways to go.But you know, I mean, it's this is the exciting part of that journey for me. Now, in my case, I focus on strategic accounts, right? Strategic accounts are big, you know, they're the potential to be our whale customers, right? These are probably not customers who would be all that interested in serverless, right? They're customers that would be more interested in provisioned infrastructure because they're the people that I talked to when I was at DynamoDB; I would be talking to customers who are interested in like, reserved capacity allocations, right? If you're talking about—Corey: Yeah, I wanted to ask you about that. You're developer advocacy—which I get—for strategic accounts.Rick: Right.Corey: And I'm trying to wrap my head around—Rick: Why [crosstalk 00:17:19]—Corey: [crosstalk 00:17:19] strategic accounts are the big ones, potential spend lots of stuff. Why do they need special developer advocacy?Rick: [laugh]. Well, yeah, it's funny because, you know, one of the reasons why it started talking to Mark Porter about this, you know, was the fact that, you know, the overlap is really around [clear throat] the engagements that I ran when I was doing the Amazon retail migration, right? When Amazon retail started to move to NoSQL, we deprecated 3000 Oracle server instances, we moved a large percentage of those workloads to NoSQL. The vast majority probably just were lift-and-shift into RDS and whatnot because they were too small, too old, not worth upgrading whatnot, but every single tier, what we call tier-one service, right, every money-making service was redesigned and redeployed on DynamoDB, right? So, we're talking about 25,000 developers that we had to ramp. This is back four years ago; now we have, like, 75,000.But back then we had 25,000 global developers, we had [clear throat] a technology shift, a fundamental paradigm shift between relational modeling and NoSQL modeling, and the whole entire organization needed to get up to speed, right? So, it was about creating a center of excellence, it was about operating as an office of the CTO within the organization to drive this technology into the DNA of our company. And so that exercise was actually incredibly informative, educational, in that process of executing a technology transformation in a major enterprise. And this is something that we want to reproduce. And it's actually what I did for Dynamo as well, really more than anything.Yes, I was on Twitter, I was on Twitch, I did a lot of these things that were kind of developer advocate, you know, activities, but my primary job at AWS was working with large strategic customers, enabling their teams, you know, teaching them how to model their data in NoSQL, and helping them cross the chasm, right, from relational. And that is advocacy, right? The way I do it is I use their workloads. [clear throat]. I use their—the customers, you know, project teams themselves, I break down their models, I break down their access patterns when I leave, essentially—with the whole day of design reviews, we'll walk through 12 or 15 workloads, and when I leave these guys have an idea: How would I do it if I wanted to use NoSQL, right?Give them enough breadcrumbs so that they can actually say, “Okay, if I want to take it to the next step, I can do it without calling up and say, ‘Hey, can we get a professional services team in here?'” right? So, it's kind of developer advocacy and it's kind of not, right? We're kind of recognizing that these are whales, these are customers with internal resources that are so huge, they could suck our Developer's Advocacy Team in and chew it up, right? So, what we're trying to do is form a focus team that can hit hard and move the needle inside the accounts. That's what I'm doing. Essentially, it's the same work I did for [clear throat] AWS for DynamoDB. I'm just doing it for, you know—they traded for a new quarterback. Let's put it that way. [laugh].Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: So, one thing that I find appealing about the approach maps to what I do in the world of cloud economics, where I—like, in my own environment, our AWS bill is creeping up again—we have 14 AWS accounts—and that's a little over $900 a month now. Which, yeah, big money, big money.Rick: [laugh].Corey: In the context of running a company, that no one notices or cares. And our customers spend hundreds of millions a year, pretty commonly. So, I see the stuff in the big accounts and I see the stuff in the tiny account here. Honestly, the more interesting stuff is generally in on the smaller side of the scale, just because you're not going to have a misconfiguration costing a third of your bill when a third of your bill is $80 million a year. So—Rick: That's correct. If you do then that's a real problem, right?Corey: Oh yeah.Rick: [laugh].Corey: It's very much a two opposite ends of a very broad spectrum. And advice for folks in one of those situations is often disastrous to folks on the other side of that.Rick: That's right. That's right. I mean, at some scale, managing granularity hurts you, right? The overhead of trying to keep your costs, you know, it—but at the same time, it's just different, a different measure of cost. There's a different granularity that you're looking at, right? I mean, things below a certain, you know, level stop becoming important when, you know, the budget start to get a certain scale or a certain size, right? Theoretically—Corey: Yeah, for there's certain workloads, things that I care about with my dollar-a-month Dynamo spend, if I were to move that to Mongo Serverless, great, but my considerations are radically different than a company that is spending millions a month on their database structure.Rick: That's right. Really, that's what it comes down to.Corey: Yeah, we don't care about the pennies. We care about is it going to work? How do we back it up? What's the replication factor?Rick: And that—but also, it's more than that. It's, you know, for me, from my perspective, it really comes down to that, you know, companies are spending millions of dollars a year in database services. These are companies that are spending ten times that, five times that, in you know, in developers, you know, expense, right? Building services, maintaining the code that runs—that the services run.You know, the biggest problem I had with MongoDB is the level of code complexity. It's a cut after cut after cut, right? And the way I kind of describe the experience—and other people have described it to me; I didn't come up with this analogy. I had a customer tell me this as they were leaving DynamoDB—“DynamoDB is death by a thousand cuts. You love it, you start using it, you find a little problem, you start fixing it. You start fixing it. You start fixing—you come up with a pattern. Talk to Rick, he'll come up with something. He'll tell you how to do that.” Okay?And you know, how many customers did I would do this with? You know, and it's honestly, they're 15-minute phone calls for me, but every single one of those 15-minute phone calls turns into eight hours of developer time writing the code, debugging it, deploying it over and over again, it's making sure it's going the way it's [crosstalk 00:23:02]—Corey: Have another 15-minute call with Rick, et cetera, et cetera. Yeah.Rick: Another 15—exactly. And it's like okay, that's you know—eventually, they just get tired of it, right? And I actually had a customer that tell me—a big customer—tell me flat out, “Yeah, you proved that the DynamoDB can support our workload and it'll probably do it cheaper, but I don't have a half-a-dozen Ricks on my team, right? I don't have any Ricks on my team. I can't be getting you in here every single time we have to do a complex data model overhaul, right?”And this was—granted, it was one of the more complex implementations that I've ever done. In order to make it work. I had to overload the fricking table with multiple access patterns on the partition key, something I never done in my life. I made it work, but it was just—honestly, that was an exercise to me that taught me something. If I have to do this, it's unnatural, okay?And that's—[laugh] you know what I mean? And honestly, there's API improvements that we could have done to make that less of a problem. It's not like we haven't known since the last, I don't know, I joined the company that a thousand WCUs per storage partition was pretty small. Okay? We've kind of known that for I don't know, since DynamoDB, was invented. As matter of fact is, from what I know, talking to people who were around back then, that was a huge bone of contention back in the day, right? A thousand WCUs, ten gigabytes, there were a lot of the PEs on the team that were going, “No way. No way. That's way too small.” And then there were other people that were like, “Nah, nobody's ever going to need more than that.” And you know, a lot of this was based on the analysis of [crosstalk 00:24:28]—Corey: Oh, nothing ever survives first contact from—Rick: Of course.Corey: —customer, particularly a customer who is not themselves deeply familiar with what's happening under the hood. Like, I had this problem back when I was traveling trainer for Puppet for a while. It was, “Great. Well, Puppet is obviously a piece of crap because everyone I talked to has problems with it.” So, I was one of the early developers behind SaltStack—Rick: Oh nice.Corey: —and, “Ah, this is going to be a thing of beauty and it'll be awesome.” And that lasted until the first time I saw what somebody's done with it in the wild. It was, “Oh, okay, that's an [unintelligible 00:25:00] choice.”Rick: Okay, that's how—“Yeah, I never thought about that,” right? Happy path. We all love the happy path, right? As we're working with technologies, we figure out how we like to use it, we all use it that way. Of course, you can solve any problem you want the way that you'd like to solve it. But as soon as someone else takes that clay, they mold a different statue and you go, “Oh, I didn't realize it could look like that.” Right, exactly.Corey: So, here's one for you that I've been—I still struggle with this from time to time, but why would I, if I'm building something out—well, first off, why on earth would I do that? I have people for that who are good at things—but if I'm building something out and it has a database layer, why would someone choose NoSQL over—Rick: Oh, sure.Corey: —over SQL?Rick: [crosstalk 00:25:38] question.Corey: —and let me be clear here—and I'm coming at this from the perspective of someone who, basically me a few years ago, who has no real understanding of what databases are. So, my mental model of a database is Microsoft Excel, where I can fire up a [unintelligible 00:25:51] table of these things—Rick: Sure. [laugh]. Hey, well then, you know what? Then you should love NoSQL because that's kind of the best analogy of what is NoSQL. It's like a spreadsheet, right? Whereas a relational database is like a bunch of spreadsheets, each with their own types of rows, right? So—[laugh].Corey: Oh, my mind was blown with relational stuff [unintelligible 00:26:07] wait, you could have multiple tables? It's, “What do you think relational meant there, buddy?” My map of NoSQL was always key and value, and that was it. And that's all it can be. And sure, for some things, that's what I use, but not everything.Rick: That's right. So, you know, the bottom line is, when you think about the relational database, it all goes back to, you know, the first paper ever written on the relational model, Edgar Codd—and I can't remember the exact title, but he wrote the distributed model, the data model for distributed systems, something like that. He discussed, you know, the concept of normalization, the power of normalization, why you would want this. And the reason why we wanted this, why he thought this was important, this actually kind of demonstrates how—boy, they used to write killer abstracts to papers, right? It's like the very first sentence, this is why I'm write in this paper. You read the first sentence, you know: “Future users of modern computer systems must have a way to be able to ask questions of the data without knowing how to write code.”I mean, I don't know if those were the words, but that was basically what he said, that was why he invented the normalized data model. Because, you know, with the hierarchical management systems at the time, everyone had to know everything about the data in order to be able to get any answers, right? And he was like, “No, I want to be able to just write a question and have the system answer that.” Now, at the time, a lot of people felt like that's great, and they agreed with his normalized model—it was elegant—but they all believe that the CPU overhead at the time was way too high, right? To generate these views of data on the fly, no freaking way. Storage is expensive. But it ain't that expensive, right?Well, this little thing called Moore's Law, right? Moore's Law balanced his checkbook for, like, 40 years, 50 years, it balanced the relational database checkbook, okay? So, as the CPUs got faster and faster, crunching, the data became less and less of a problem, okay? And so we crunched bigger and bigger data sets, we got very, very happy with this. Up until about 2014.At 2014, a really interesting thing happened. If you look at the top 500, which is the supercomputers, the top 500 supercomputing clusters around the world, and you look at their performance increases year-to-year after 2014, it went off a cliff. No longer beating Moore's Law. Ever since, they've been—and per-core performance, you know, CPU, you know, instructions executed per second, everything. It's just flattening. Those curves are flattening. Moore's Law is broken.Now, you'll get people argue about it, but the reality is, if it wasn't broken, the top 500 would still be cruising away. They're not. Okay? So, what this is telling us is that the relational database is losing its horsepower. Okay?Why is it happening? Because, you know, gate length has an absolute minimum, it's called zero, right? We can't have a logic gate that's the—with negative distance, right? [laugh]. So, you know, these things—but storage, storage, hey, it just keeps on getting cheaper and cheaper, right?We're going the other way with storage, right? It's gigabytes, it's terabytes, it's petabytes, you know, with CPU, we're going smaller and smaller and smaller, and the fab cost is increasing. There's just—it's going to take a next-generation CPU technology to get back on track with Moore's Law.Corey: Well, here's the challenge. Everything you're saying makes perfect sense from where your perspective is. I reiterate, you are working with strategic accounts, which means ‘big.' When I'm building something out in the evenings because I want to see if something is possible, performance considerations and that sort of characteristic does not factor into it. When I'm a very small-scale, I care about cost to some extent—sure, whatever—but the far more expensive aspect of it, in the ways that matter have been what is the expensive—what—the big expensive piece is—Rick: We've talked about it.Corey: —engineering time—Rick: That's what we just talked about, right?Corey: —where it's, “What I'm I familiar with?”Rick: As a developer, right, why would I use MongoDB over DynamoDB? Because the developer experience [crosstalk 00:29:33]—Corey: Exactly. Sure, down the road there are performance characteristics and yeah, at the time I have this super-large, scaled-out, complex workload, yeah, but most workloads will not get to that.Rick: Will not ever get there. Ever get there. [crosstalk 00:29:45]—Corey: Yeah, so optimizing for [crosstalk 00:29:45], how's it going to work when I'm Facebook-scale? It's—Rick: So, first of—no, exactly, Facebook scale is irrelevant here. What I'm talking about is actually a cost ratchet that's going to lever on midsize workloads soon, right? Within the next four to five years, you're going to see mid-level workloads start to suffer from significant performance cost deficiencies compared to NoSQL workloads running on the same. Now you—hell, you see it right now, but you don't really experience it, like you said, until you get to scale, right? But in midsize workloads, [clear throat] that's going to start showing up, right? This cost overhead cannot go away.Now, the other thing here that you got to understand is, just because it's new technology doesn't make it harder to use. Just because you don't know how to use something, right, doesn't mean that it's more difficult. And NoSQL databases are not more difficult than the relational database. I can express every single relationship in a NoSQL database that I express in a relational database. If you think about the modern OLTP applications, we've done the analysis, ad nauseum: 70% of access patterns are for a single object, a single row of data from a single table; another 20% are for a row of datas—a range of rows from a single table. Okay, that leaves only 10% of your access patterns involve any kind of complex table traversal or entity traversals. Okay?And most of those are simple one-to-many hierarchies. So, let's put those into perspective here: 99% of the access patterns in an OLTP application can be modeled without denormalization in a single table. Because single table doesn't require—just because I put all the objects in one place doesn't mean that it's denormalized. Denormalized requires strong redundancies in the stored set. Duplication of data. Okay?Edgar Codd himself said that the normalized data model does not depend on storage, that they are irrelevant. I could put all the objects in the same document. As long as there's no duplication of data, there's no denormalization. I know, I can see your head going, “Wow,” but it's true, right? Because as long as I can clearly express the relationships of the data without strong redundancies, it is a normalized data model.That's what most people don't understand. NoSQL does not require denormalization. That's a decision you make, and it usually happens when you have many-to-many relationships; then we need to start duplicating the data.Corey: In many cases, at least my own experience—because again, I am bad at computers—I find that the data model is not something that is sat out—that you sit down and consciously plan very often. It's rather something—Rick: Oh yeah.Corey: —happens to you instead. I mean—Rick: That's right. [laugh].Corey: —realistically, like, using DynamoDB for this is aspirational. I just checked, and if I look at—so I started this newsletter back in March of 2017. I spun up this DynamoDB table that backs it, and I know it's the one that's in production because it has the word ‘test' in its name, because of course it does. And I'm looking into it, and it has 8700 items in it now and it's 3.7 megabytes. It's—Rick: Sure, oh boy. Nothing, right?Corey: —not for nothing, this could have been just as easily and probably less complex for my level of understanding at the time, a CSV file that I—Rick: Right. Exactly, right.Corey: —grabbed from a Lambda out of S3, do the thing to it, and then put it back.Rick: [unintelligible 00:32:45]. Right.Corey: And then from a performance and perspective side on my side, it would make no discernible difference.Rick: That's right because you're not making high-velocity requests against the small object. It's just a single request every now and then. S3 performance would probably—you might even be less. It might even cost you less to use S3.Corey: Right. And 30 to 100 of the latest ones are the only things that are ever looked at in any given week, the rest of it is mostly deadstock that could be transitioned out elsewhere.Rick: Exactly.Corey: But again, like, now that they have their lower cost infrequent access storage, then great. It's not item level; it's table levels, so what's the point? I can knock that $1.30 a month down to, what, $1.10?Rick: Oh well, yeah, no, I mean, again, Corey for those small workloads, you know what? It's like, go with what you know. But the reality is, look, as a developer, we should always want to know more, and we should always want to know new things, and we should always be aware of where the industry is headed. And honestly, I've heard through—I'm an old, old school, relational guy, okay, I cut my teeth on—oh, God, I don't even know what version of MS SQL Server it was, but when I was, you know, interviewing at MongoDB. I was talking to Dan Pasette, about the old Enterprise Manager, where we did the schema designer and all this, and we were reminiscing about, you know, back in the day, right?Yeah, you know, reality of things are is that if you don't get tuned into the new tooling, then you're going to get left behind sooner or later. And I know a lot of people who that has happened to over the years. There's a reason why I'm 56 years old and still relevant in tech, okay? [laugh].Corey: Mainframes, right? I kid.Rick: Yes, mainframes.Corey: I kid. You're not that much older than I am, let's be clear here.Rick: You know what? I worked on them, okay? And some of my peers, they never stopped, right? They just kind of stayed there.Corey: I'm still waiting for AWS/400. We don't see them yet, but hope springs eternal.Rick: I love it. I love that. But no, one of the things that you just said that I think it hit me really, it's like the data model isn't something you think about. The data model is something that just happens, right? And you know what, that is a problem because this is exactly what developers today think. They think know the relational database, but they don't.You talk to any DBA out there who's coming in after the fact and cleaned up all the crappy SQL that people like me wrote, okay? I mean, honestly, I wrote some stuff in the day that I thought, “This is perfect. There's no way that could be anything better than this,” right? Nice derived table joins insi—and you know what? Then here comes the DBA when the server is running at 90% CPU and 100% percent memory utilization and page swapping like crazy, and you're saying we got to start sharding the dataset.And you know, my director of engineering at the time said, “No, no, no. What we need is somebody to come in and clean up our SQL.” I said, “What do you mean? I wrote that SQL.” He's like, “Like I said, we need someone to come and clean up our SQL.”I said, “Okay, fine.” We brought the guy in. 1500 bucks an hour, we paid this guy, I was like, “There's no way that this guy is going to be worth that.” A day and a half later, our servers are running at 50% CPU and 20% memory utilization. And we're thinking about, you know, canceling orders for additional hardware. And this was back in the day before cloud.So, you know, developers think they know what they're doing. [clear throat]. They don't know what they're doing when it comes to the database. And don't think just because it's a relational database and they can hack it easier that it's better, right? Yeah, it's, there's no substitute for knowing what you're doing; that's what it comes down to.So, you know, if you're going to use a relational database, then learn it. And honestly, it's a hell of a lot more complicated to learn a relational database and do it well than it is to learn how to model your data in NoSQL. So, if you sit two developers down, and you say, “You learn NoSQL, you learn relational,” two months later, this guy is still going to be studying. This guy's going to be writing code for seven weeks. Okay? [laugh]. So, you know, that's what it comes down to. You want to go fast, use NoSQL and you won't have any problems.Corey: I think that's a good place to leave it. If people want to learn more about how you view these things, where's the best place to find you?Rick: You know, always hit me up on Twitter, right? I mean, @houlihan_rick, that's my—underbar rick, that's my Twitter handle. And you know, I apologize to folks who have hit me up on Twitter and gotten no response. My Twitter as you probably have as well, my message request box is about 3000 deep.So, you know, every now and then I'll start going in there and I'll dig through, and I'll reply to somebody who actually hit me up three months ago if I get that far down the queue. It is a Last In, First Out, right? I try to keep things as current as possible. [laugh].Corey: [crosstalk 00:36:51]. My DMs are a trash fire. Apologies as well. And we will, of course, put links to it in the [show notes 00:36:55].Rick: Absolutely.Corey: Thank you so much for your time. I really do appreciate it. It's always an education talking to you about this stuff.Rick: I really appreciate being on the show. Thanks a lot. Look forward to seeing where things go.Corey: Likewise.Rick: All right.Corey: Rick Houlihan Director of Developer Relations, Strategic Accounts at MongoDB. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an upset comment talking about how we didn't go into the proper and purest expression of NoSQL non-relational data, DNS TXT records.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Software Engineering Radio - The Podcast for Professional Software Developers

Frank McSherry, Chief Scientist at Materialize talks to Host Akshay Manchale about Materialize which is a SQL database that maintains incremental views over streaming data. Frank talks about how Materialize can complement analytical systems...

Outrageous Love the Podcast: Our Journeys to Responsiveness
Ben Kingsbury's Journey to Responsiveness, the Educrats Series, 3 of 3

Outrageous Love the Podcast: Our Journeys to Responsiveness

Play Episode Listen Later Mar 16, 2022 58:16


OLTP ends the "educrats series" in California with a look inside the belly of the beast of the California Department of Education (CDE). This time, we hear the perspective of educator Ben Kingsbury. Ben serves as a program consultant in the CDE. He offers deep insights, not only on the bureaucracy but also on his journey, which takes us to Saudi Arabia and we land with a dope reference to the GAP Band. Ben's journey reminds us of the old adage you cannot judge a book by its cover...ever. Fascinating and fun! Dr. Hollie's two cents raises the question, why do we to continue to believe in  an educational system that has perpetually not equitably served all students? How many chances does the system get? Listen in for the answer.

Screaming in the Cloud
Becoming a Pathfinder in Tech with Emily Kager

Screaming in the Cloud

Play Episode Listen Later Mar 3, 2022 36:20


About EmilyEmily is an Android engineer by day, but makes tech jokes and satires videos by night. She lives in San Francisco with two ridiculously fluffy dogs.Links: Uber: https://eng.uber.com/ Blog: https://www.emilykager.com/ Twitter: https://twitter.com/EmilyKager TikTok: https://www.tiktok.com/@shmemmmy TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance query accelerator for the Oracle MySQL Database Service, although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLAP and OLTP—don't ask me to pronounce those acronyms again—workloads directly from your MySQL database and eliminate the time-consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today's episode is a little bit off of the beaten path because, you know, normally we talk to folks doing things in the world of cloud. What is cloud, you ask? Great question. Whatever someone's trying to sell you that day happens to be cloud.But it usually looks like SaaS products, Platform as a Service products, Infrastructure as a Service products, with ridiculous names because no one ever really thought what that might look like to pronounce out loud. But today, we're going in a completely different direction. My guest is Emily Kager, a senior Android engineer at a small scrappy startup called Uber. Emily, thank you for joining me.Emily: Thanks for having me.Corey: So, I'm going to outright come out and say it I know remarkably little about, I don't even want to say the mobile ecosystem in general, but even Android specifically because I fell down the iPhone hole a long time ago, and platform lock-in is a very real thing. Whenever you start talking about technical things, that generally tends to sail completely past me. You're talking about things like Promises and whatnot. And it's like, oh, that sounds suspiciously close to JavaScript, a language that I cannot make sense of to save my life. And it's clear you know an awful lot about what you're doing. It's also clear, I don't know, a whole heck of a lot about that side of the universe.Emily: Well, that's good because I don't know much about the cloud.Corey: Exactly. Which sounds like well, we don't have a whole lot of points of commonality to have a show on, except for this small little thing, where recently, I decided in an attempt to recapture my lost youth and instead wound up feeling older than I ever have before, I joined the TikToks and started making small videos that I would consider humorous, but almost no one else will. And okay, great. I give it a hearty, sensible chuckle and move on, and then I start scrolling to see what else is out there. And I started encountering you, kind of a lot.And oh, my God, this is content that it's relatable, it is educational, dare I say, and most of all, it's engaging without being overbearing. And this is a new type of content creation that I hadn't really spent a lot of time with before. So, I want to talk to you about that.Emily: Awesome. I want to apologize for having to see my face as you're just scrolling throughout your day, but happy to chat about it. [laugh].Corey: No, no, it's—compared to some of the things I wind up on the TikTok algorithm, it is ridiculous. I think it's about 80% confident that I'm a lesbian for some Godforsaken reason. Which hey, power to the people. I don't think I qualify, but you know, that's just how it works. And what I found really interesting about it, what does tie it back to the world of cloud, is that a recurring theme of this show has been, since the beginning, where does the next generation of cloud-engineering-type come from?Because I've been in this space, almost 20 years, and it turns out that my path of working to help desk until you realize that you like the computers, but not so much being screamed at by the general public, then go find a unicorn job somewhere you can bluff your way into because the technical interviewer is out sick that day, and so on and so forth, isn't really a path that is A) repeatable by a whole lot of people, and B) something that exists anymore. So, how do people who are just entering the workforce now or transitioning into tech from other fields learn about this stuff? And we've had a bunch of people talking about approaches to educating people on these sorts of things, but I don't think I've ever spoken to someone who's been as effective at it in minute or less long videos as you are.Emily: That's super kind. Yeah, I think there's actually a whole discussion and joke set on TikTok of people's parents suggesting why don't you just go slide your resume under the CEOs door? Like, why don't you just go get a job [laugh] that way? I think the realities of—what year are we in? 2022? [laugh]—Corey: All year long, I'm told.Emily: Yeah, [laugh] yeah. Yeah. I think that's not going to be the reality anymore, right? You can't just go shake hands with the CEO and work your way up from the mailroom and yeah, that's not the way anymore. So yeah, I think I, you know, started just putting some feelers out, making educational content mostly about my own experiences as a change career person in the tech world.I have some, I would say interesting perspectives on how to enter the industry, you know, either through undergrad or after undergrad, so. And it's done really well. I think people are really interested in tech is a career at this point. Like, it's kind of well known that they're good jobs, well paid, and, you know, pretty, like, good work-life balance, most of the time. So yeah, the youth are interested.Corey: It's something that offers a path forward that lends itself to folks with less traditional backgrounds. For example, you have a master's degree; I have an eighth-grade education on paper. And, yes, I'm proof-positive that it is possible to get into this space and, by some definitions, excel in it without having a degree, but let's also be clear, here, I have the winds of privilege at my back, and I was stupendously lucky. It is harder to do without the credential than it is with the credential.Emily: Yep.Corey: But the credential is not required in the same way that it is if I want to be a surgeon. Yeah, you're going to spend a lot of time in either school or prison with that approach. So, you have really two paths there; one is preferable over the other. Tech, it feels like there's always more than one way to get in. And there's always, it seems, as many stories as there are people out there about how they wound up approaching their own path to it. What was yours?Emily: Yeah. First of all, it's funny, you mentioned surgeons because I actually just today saw on my ‘For You' page some surgeons sharing, you know, their own suturing techniques. And I think it's a really interesting platform even, you know, within different fields and different subsets to kind of share information and keep up to date and connect with people in your own industry. So, beyond learning how to get into [laugh] an industry, it can also be helpful for other things. But sorry, I completely forgot the original question. How—what was my path? Is that what the question was?Corey: Yeah. How did you get here is always a good question. It's the origin stories that we sometimes tell, sometimes we wind up occluding aspects of it. But I find it's helpful to tell these stories just because, if nothing else, it reaffirms to folks who are watching or listening or reading depending on how they want to consume this, that when they feel like well, I tried to get a credential and didn't succeed, or I applied for a job and didn't get it, there are other paths. There is not only one way to get there.Emily: Yeah. And I think it's also super important to talk about failures that we've had, right? So, when I was in undergrad, I was studying neuroscience and I was pre-med. And I thought I wanted to go to med school, kind of decided halfway through, I was only lukewarm about it, and I don't think med school is the type of thing that you want to feel lukewarm about as you're [laugh] approaching, you know, hundreds of thousands of dollars of debt and a ten-plus year commitment to schooling and whatever else, right? So yeah, I felt very lukewarm about the whole thing.Both my parents were doctors, so I just didn't really have exposure to many other careers or job options. I'm from a pretty, like, rural area, so tech had never really [laugh] occurred to me either. So yeah, then I decided to just take a year off after undergrad, felt super lost. I think when you're 22, everything feels so important, [laugh] and you look at everyone else who already has their first job at 22, and I was like, “Wow, I'm a huge failure. I'm never going to have a job.” Which is, you know, hilarious looking back because 22-year-olds are so young. And yeah, just decided to take a year off. I worked at a nonprofit. I hated it, hated the work. Decided, like I, you know, can never do this forever.Corey: I can't do nonprofit stuff. I'm going to do for-profit stuff. And it turns out that most—when you say nonprofit, it doesn't mean what I thought. It ap—usually means, you know, something that's dedicated to a charitable cause, not, you know, a VC-backed company that doesn't know how to make any money.Emily: Yeah. I mean, it could still be very corporate at nonprofit. After that, actually—Corey: Oh, yes. Money is the root of all good as well as evil.Emily: Yeah. And I actually had a task at the nonprofit where I was sorting a ton of things in spreadsheets. And I was like, wow, it'd be easy if there was just, like, some program I could write to, like, do this. So, I actually reached out to my brother, who was a computer science nerd—affectionately—and he helped me write some, like, Excel macros, and I was like, “This is so cool.” And I ended up taking a free course, CS50, which is great, by the way, great course, super high quality from Harvard, totally free to take online.And really liked it, so I did something a little crazy and decided to just dive right in. [laugh]. And I applied to a post-bacc program to kind of take all the courses that a CS undergrad would have taken just after. And that post-bacc turned into a master's program.Corey: And here you are now on the other side of having done it. If—sort of the dangerous questions: If you had known then what you know now, would you have gone down the same path, or would you have done something different to get into the space?Emily: Yeah, I mean, I think it's hard once you've kind of made it, to be like, “I would change all this.” I think I would probably try more things in undergrad. That would be the real answer to that. It obviously would have been a lot easier and more time-efficient if I didn't have to go back to school and do something. But that being said, I don't think that getting a post-bacc or a Master's is the only way into tech; it was just my path.And I try not to… I try not to promote other paths that I don't really know much about independently, right? So—on me. So—but plenty of people are successful going through boot camps or self-teaching, even, I think they're just much more difficult paths because the reality is, like, having a degree is still definitely an easier path when you show up to an interview and you can just kind of show your piece of paper, which, for better or worse, that's the reality sometimes.Corey: My wife's a corporate attorney, so I've been law adjacent for over a decade now, and one of the things that always struck me about that field is the big law approach is you go to a top-tier law school, you wind up putting your nose to the grindstone for all three years, and you hope to get an offer at one of the big law firms. And they all keep their salaries in lockstep. I think right now they're all—they just upgraded again to $235,000 a year starting. And if you don't get one of those rare, prestigious jobs at a number of select firms, it's almost a bimodal distribution where you're making somewhere between 60 and $80,000 a year to start somewhere else. It is the one path to make big money in law as you're fresh out of school, and there are no real do-overs in most cases.So, it's easy to apply that type of thinking to tech, and it's just not true. Talking to folks who have this dream of working at Google and they finally go through the interview process. And it turns out that oh no, they froze when asked to solve Fizz Buzz, or invert a binary tree on a whiteboard, or whatever ridiculous brainteaser question they're being asked, and, “Oh, no, my life is over.” And it's, you know, you can go to, I don't know, Stripe, two blocks down the street and try again. And if that doesn't work, Microsoft, or Amazon, or go down the entire list of tech companies you've heard of and haven't heard of, and they all compensate directionally the same way. It's not a one-shot, ‘this is it' moment in the same way. And I—Emily: Yeah.Corey: —I think that's a unique thing to tech right now.Emily: Yeah, definitely. And I think a lot of kids—I say kids, but really, like, you know, 18 to 20-year-olds—Corey: Oh, believe me, after being on TikTok for a couple of weeks, let me say that every one of you are children, to my perspective. I am now Grandpa Quinn over here.Emily: [laugh]. I'll take it. Yeah, but a lot of them have reached out like, “I didn't get hired at FAANG right out of school. Is my life over? Is my career over?” And I've never worked at a FAANG. [laugh]. I'm pretty happy. I definitely think I have a successful career, and I almost think I'm better for not having gone right into it, you know?I think it can be great for some people. There's great, you know… definitely great salaries, great mentorship options, but it's not the only option. And I think maybe tech is unique in that way, but there's just so many good companies to work at, and so many great opportunities, you really don't need to go to the name brand in the same way that maybe you would have to in law. It's funny you say that because my partner is also a lawyer [laugh] and [crosstalk 00:13:00]—Corey: Oh, dear. We should start a support group of our own, on some level.Emily: I know, yeah. He just went through the whole big law recruiting thing. So, I know much about that. [laugh].Corey: It's always an experience. The way that I have found across the board as well is there's also a shared, I guess, esprit de corps almost across the industry. I mean, you are on the Android side of the world, and I historically was on the DevOps side of the universe, although now mocking cloud services—but not the way test engineers say when they use the term ‘mocking'—is what I do. But there are shared experiences that tie us together, and that's part of what I found so interesting about a lot of your content.Because yes, there is some of the deep dive stuff into Android and, cool, sails right over my head—I hear the whistling sound vaguely as it goes over—but then there's other stories about things that are unique—that are, I guess, a shared experience. For me, one of the things that tied all of tech together, regardless of where in the ecosystem you fit in, is a shared sense of being utterly intimidated to hell by the miracle of Git, where it's like, Git's entire superpower is making you feel dumb. Doesn't matter who you are, from someone who doesn't know what Git is all the way to Linus himself. Someone is go—at some point, you're going to look at it and wonder, “What the hell is going on?” It's just a question of how far you get along the path before it changes your understanding of the universe.And I wound up starting to give talks, in the before times, at front-end conferences about this, which you want to talk about dispiriting things. I would build slides like, you know, a DevOps person would: Black Helvetica text on a white slide. Everyone else has these beautifully pristine, great slides. I have 20 minutes to go.How can I fix it? Change the font to Comic Sans because if you're going to have something that looks crappy, make it look like it was intentionally so.Emily: And did it work?Corey: Oh, it worked swimmingly. It was fantastic. I like the idea of being able to reach people in different areas, no matter where they are in their journey, and one of the things that appeals to me about TikTok in general in your content in particular, is it seems like we have something of a shared perspective on, getting people's attention is required in order to teach them something, and I think we both use the same vehicle for that, which is humor.Emily: Yeah, I would agree. I think the other interesting thing I just wanted to touch on; you were talking about is, we don't really know too much about each other's fields in tech. And I think when you're talking to a younger audience, maybe who you want to get interested in tech, it's really hard to communicate all the different avenues into tech that they can take. And this is something that I'm still struggling with because I know my experience as an Android developer, a mobile developer, I probably medium I understand, you know, back end development, but I don't think I could explain to a college student why or what even is, [laugh] you know, cloud development and how they could get involved in that, or all these other fields that I just really don't know much about. And I think that's kind of what ties a lot of people in tech together as well, right? Because we know our little corners of the world, and you have to start to get comfortable with the things that you don't know. And I think that's really hard to explain to [laugh] the younger generation as you're trying to get them excited about things.Corey: Oh, yeah. And the reality, too, of what we tell people and how the world works is radically different. Like, I want to learn a technology that will absolutely last for an entire career and then some, and I want to be able to be employed anytime, anywhere, at any company. The easy slam dunk answer that I think will not change in either of our lifetimes is Microsoft Excel. It powers the world.People think I'm kidding, but it is the IDE of back-office processes and communications. If Excel were to go away or even worse, Microsoft were to change Excel's interface, people would be storming Redmond by noon.Emily: Yeah, I believe it. Yeah, you know, it's interesting, right? Like, it's hard to tell people—because people will tell to me, “Well, do you have to keep learning things?” And I'm like, “Yeah. You got to keep learning things, like, all the time.”But I don't think that should be, you know, a deterrent from the career; it's just a reality. But to try to manage, like, the fears a lot of people have coming into tech and also encouraging them to still, you know, try it, go after it, I think that's something I struggle with when I'm creating my content for—towards, like, younger people. [laugh].Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: Something I found on Twitter is that among other things that Twitter has going on for it, it doesn't do nuance, it does, effectively, things that are black and white, yes or no, it's always a binary in many respects. And one of those is that, like, should—like, is passion or requirement for working in tech. And there's the, “Yes, you absolutely have to be passionate for this and power through it.” And the answer, “No, you don't need to be passionate about it's okay to do it for the money and not kill yourself working 20 hours a day.” And from my perspective, I take a more moderate stance, which is how you get both sides of that argument to hate you, but it's, I don't think you need to have this all-consuming drive for tech, but I do think you need to like it.Emily: Oh yeah.Corey: I think you need to enjoy what you're doing or it's going to feel like unmitigated toil and misery, and you will not be happy in the space. And if you're not happy, really is the rest of it all worth it?Emily: I think that applies to most careers, though, right? Like that—definitely, when I was looking to switch careers, that was the main thing I was looking for. Number one was like, you know, pretty solid salary. And number two was, do I just not hate it? [laugh]. And I think if you're doing anything and you hate it, you're going to be miserable, right?Like, even if you're doing it to make a paycheck if you actually hate every single day when you wake up in the morning and you dread, you know, going to bed because the next morning, you have to wake up and do it again, like, you're going to be miserable. But I do think, yeah, like, to your point, there's a middle ground in all this, right? You don't have to dream about tech, but I think you do have to realize that, yeah, if you're going to be in this industry for decades, you're going to have to be able to learn and be interested enough in things that, you know, learning isn't a huge slog either. So.Corey: I've never understood the folks who don't want to learn as they go through their career because it just seems like a recipe to do the same thing every year for 40 years, and then you retire with what 40 years of experience—one year experience repeated 40 times. It's a… any technology or any disruption change happens, and suddenly you're in a very uncomfortable situation when we're talking about knowledge workers.Emily: Yeah, I think people—you know, I think we talk a lot about, like, imposter syndrome in our industry right? So, I think people already feel like maybe, “I don't know anything so why would I put myself out there and learn new things?” I mean, I definitely sometimes struggle with this where I'm like, “I'm very comfortable [laugh] in, like, what I do day-to-day. I know what I'm doing.” So yeah, when you have to learn, like, a totally new language or new architecture, whatever, it can feel very overwhelming to be like, wow, I actually am, you know, super stupid. [laugh]. But it's just new things, right? You're learning new things, and—Corey: Like, “Find the imposter. Oh, no, it's me.” Yes, it's a consistent problem.Emily: But it's a really powerful thing to acknowledge that you can feel stupid and you can ask questions and you can be new to something, and that's, like, totally valid. And I started taking a new language course a year or two ago, and showing up every day and speaking a new language and feeling like an idiot, it was actually super empowering because everyone in the class is doing it, you know? We didn't know the language and we were just, you know, talking gibberish to each other, and that's fine. We were learning.Corey: The emotional highs and lows are also—they hit quickly. I have never felt smarter or dumber in a two-minute span of each other than when working on technology. It's one of those, “I will never understand how this works—oh my God, it works. I'm a genius. Just kidding. It doesn't work. Nevermind. Forget everything I just said.” It's a real emotional roller coaster.Emily: [laugh]. There's only two ends of the spectrum, right? Like, there's no middle ground in this situation. It's, “I'm a genius,” or, “I should quit and never work on technology ever again.”Corey: So, I've been experimenting on TikTok a bit and you've been on it significantly longer. You have, as of this recording, something in the direction of 65,000 followers on the TikToks. I have a bit more than that on the Twitters, which only took me a brief 14 years to do. So, great. I've noticed that as I wind up—as you hit certain inflection points on Twitter, your experience definitely changes, when—as far as just, like, the unfortunate comments coming out of the woodwork.Like, I was making fun of LinkedIn at some point, and then there was some troll comment in the comments, and I looked at who the commenter was and it was the official LinkedIn brand account. And okay, well, that's novel, but all right. I'd like to add them to my professional network on TikTok. So, there we go. But have you noticed inflection points as well, in your—experience changes on the platform as you continue to grow?Emily: Yeah. I think—I saw something once that Twitter is only fun if you have less than, like, [laugh] 5000 followers or something. So, I think we both surpassed that a while ago. And yeah, I think it can be a very interesting experience as you start to gain followers. And to be honest, like, I'm on both platforms, just to kind of make content.It's a very, like, creative outlet for me. I don't necessarily care that much about how many followers I have. But it is an interesting progression to see, like, you know, you get a little bit of engagement, and it's usually, like, a back and forth; you're kind of like actually connecting to people, and then as you kind of surpass maybe five or ten-thousand followers, there's all these people who come in who you don't know who they are, they don't know who you are, they make assumptions about you, they are saying really mean things that I think just because you have, like, a high follower account that they're like, “I can say whatever I want to this person.” And it's definitely an interesting change. I think over the years—because I've been fairly public for a number of years now—you kind of get more immune to it. I'm sure you feel the same way, but you're like, whatever, just kind of brush off a lot of these things. But—Corey: Oh, yeah. You become more of a persona to people than an actual person.Emily: Yeah.Corey: And that is—Emily: Yeah.Corey: —people forget that—you know, everyone yells at you about, “That was an unkind thing, express more empathy all the ti”—I mean, you get that all the time when you get—when set a slight foot wrong. And they're right—don't think I'm saying otherwise—but they're not expressing a lot of empathy for you at the same time, either. So, it's one of those you have to disengage and disconnect on certain levels and just start to ignore it. But it's been a wild ride.Emily: I used to wonder, I used to see, like, accounts that have you know, 50, 60,000 followers on Twitter back when I was a smaller account, and they didn't—they never tweeted, and I was like, “How'd they get so many followers? They never tweet.” And now I understand. It's that they gained that many followers and then they left. [laugh]. They're done.Corey: [unintelligible 00:23:18] like, “This platform sucks now.” And it's—a lot of folks, like, “Oh, Twitter's not as good as it used to be.” It's like, well hang on. Has the platform itself changed or has your exposure to it changed? And it's a question that doesn't really have a great answer or way to find out, but it's… it's been a—it's an ongoing struggle for folks. And I do have empathy for that. I try to avoid getting involved in pile-ons wherever possible.Emily: Yeah. That's been a new change for me, too. I think a lot of my early brand on Twitter—as dumb as that word is—was, you know, kind of finding, like, misogynists in tech and really, like, creating a pile-on on them. And, you know, I think there is a space for calling out bad behavior in the industry, but you want to be careful because really, there are other people on the other side of the screen. And unless someone's really implying—like, unless they're really intending ill intent, you know, I think I've kind of now moved less towards that type of [laugh] pile-on. It is fun though. That's the thing. It's fun.Corey: Plus the algorithm rewards engagement. Say horrifying things and get a bunch of attention and more followers. But you don't necessarily want to participate in that.Emily: Yeah, exactly. And that's the other thing I realized that if someone is really saying something stupid, me bringing attention to it is only going to amplify it more. So. Especially as you gain followers and you have more of an audience to whatever you quote, tweet, or retweet, or comment on, right? So.Corey: As I look at, like, the sheer amount of content that you've put out—it's weird because if someone asked me this question, I don't know that I would have a good answer, but I am curious. You are consistently exploring new boundaries in terms of the humor, the content, the topics, the rest. How do you come up with it?Emily: This is going to be a really unsatisfying answer. [laugh]. I don't know. [laugh]. I'm a runner, and a lot of times when I'm running I don't use headphones. A lot of people say I'm sociopathic because I just am by myself in the world, and—this is such, like, a weird answer—but yeah, I just kind of—I'm thinking about things, usually I'm like digesting my day, things that happened, things that were annoying.And to be honest, I think it's pretty easy to identify things that are relatable, right? So, a lot of the gripes that all engineers have, right? So, you're like, “Wow, it was really annoying that I had to make a ticket in Jira today.” And you can kind of think about how is it annoying, and how can I make this funny and relatable to someone else? So—and to be hon—like, when I had, you know, a group of coworkers that I worked really closely in my last job, I would just send them the jokes, and then if they thought it was funny, I would just, like, post it on Twitter.And that's kind of… you know, it's just, like, the basic chit-chat that you do. But now we're all remote, so I found an outlet through Twitter and TikTok, where I would just express all my, you know, stupid engineering jokes to the world. [laugh]. Whether they want it or not.Corey: Something I found is that—and it always has frustrated me, and I figured, one day, I too, would figure out how to solve for this. And no. There are things I will tweet out that I think are screamingly funny and hilarious, and no one cares. Conversely, I'll jot off something right before I dive into a meeting, and I'll come back and find out it's gone around the internet three times. And there seems to be no rhyme or reason to it, other than that my sense of humor is not quite dialed into exactly where most folks in this industries are. It's close enough that could be overlooked, but I still feel like the best jokes go unappreciated.Emily: Oh, I agree. I mean, I send jokes by friends all the time that I'm like, “I'm posting this,” and it gets, like, you know, 20 likes. And I don't even care. I think, you know—I think that's the—you know can—you start to learn as a content creator that you're like, “I'm going to put out the content that I want to put out and hope other people find it funny, but at the end of the day, I don't really care.” So, I'm laughing at my own jokes. I'll admit that. So. I think they're funny. My—Corey: [crosstalk 00:26:58]—Emily: —[crosstalk 00:26:58] funny, too.Corey: —for me because if—I'm keeping myself engaged, otherwise it gets boring, and I lose interest in the sound of my own voice, which is just a terrible sin for me. So, it's—I have to keep it engaging or I'll lose interest.Emily: Yeah, exactly.Corey: Do you find when you're trying to put together content, that—for TikTok, for example—that you've come up with something that, “Huh, this doesn't really fit the video format. Maybe it's more of a blog post or something else.” Do you find that one content venue feeds another? Do you reuse content across multiple platforms? And if so—Emily: Yeah.Corey: —what have you learned from all that?Emily: That's an interesting question. I think—I do maintain a blog, but I don't post so often on it, and I find that the—for the more serious content I'm making that's not jokes, right? I think TikTok just really hits a different audience. Like, people don't find my blog, it's not discoverable, maybe they're not checking it, and I think definitely the younger audience prefers to consume things in video content. And a lot of my content is also aimed towards people who maybe are exploring tech who don't work in tech yet, and so to really hit them, they probably aren't following me and they probably don't know who I am, they probably don't even know what to look for in my blog.So, for example, I have a blog post all about how I transitioned into tech, blah, blah, blah, and people still ask me all the time on TikTok, “How did you transition into tech? How did you”—I'm like, “It's in my blog.” On my—like, you know, linked my bio. But you still have to just kind of—I think, like, I tend to just recreate the content into the different platforms. And it can be a bit tedious, but I try to keep my blog up to date with, like, different stories of things that have happened to me. But these days, I mostly just post on TikTok, to be honest. [laugh].Corey: I had the same problem, but content reuse saved me. I started writing a long-form blog post of roughly 1000 to 1500 words every week, then reading it into a microphone. It became the AWS Morning Brief podcast and emailing out to the newsletter as well. So, it's one piece of content used three different times, which was awesome, but then there's the other side of it, which is, I need to come up with an interesting idea or concept or something to talk about for 1000 words every week, like clockwork. And one of the things that made this way easier is a tip I got from Scott Hanselman that I have been passing on whenever it seems appropriate—like in this conversation—which is if you find yourself explaining something a third time, turn it into a blog post because then you'll just be able to link people to the thing that you wrote where you go into significantly more depth around what you're talking about than you can in a two-tweet exchange, and that in turn, gives you a place to dump that stuff out.And I found that has worked super well for me because once I've written it and gotten it out, I also often find I stopped making the same reference all the time because now I've said it, I've said my piece. Now, I can move on and come up with a second analogy, or a new joke or something.Emily: Yeah. I've also found that um—that's a great idea from Scott; he's also great on the TikToks [laugh]—Corey: Oh, yes he is.Emily: —[crosstalk 00:29:45] [laugh]. Building his account. Yeah, I think another interesting thing is, specifically on TikTok and Twitter because it's more of a conversation between you and your community, I tend to get a lot of ideas just from people asking me questions, right? So, in the comments of something, it could be related to the video I just made and it really helps me expand upon, you know, what I was just saying and maybe answer a follow-up question in a different video. Or maybe it's just a totally unrelated question.So, someone finds, you know, one of my comedy videos and is like, “Hey, you work in tech. Like, what is that like in San Francisco?” Right? So, I think I've found a ton of inspiration just from community people and really what they're asking for, right? Because at the end of the day, you want to make content that people actually care about and want to know the answers to.Corey: Yeah, seems like that does help. If it's, “How do I wind up building a following or getting a lot of traffic or the rest?” And it's Lord knows, once you have a website that has a certain amount of Google juice, you just get besieged by random requests from basically every channel. “Hey, I saw this great article linked to a back issue of the newsletter talking about this thing. Would you mind including my link to it, this would help your readers.” And it's just it's a pure SEO scam.And it's yeah, I don't—my approach to SEO has been this, again, ancient, old-timey idea of I'm going to write compelling original content that ideally other people find valuable and then assume that the rest is going to take care of itself. Because, on some level, that is what all these algorithms are trying to do is surface the useful stuff. I feel like as long as you hold to that, you're not going to go too far wrong.Emily: No, that's true. Also, something funny about reusing content is sometimes I'll post a joke on Twitter, and if it does well, I'll make it into a video format. And you know, sometimes I change the format of the joke around, whatever. But I—a couple times this happened—I'll post something on Twitter, and then, like, a day or two later, I'll make a TikTok about it, and a lot of people will come in and be like, “I already saw this joke on Twitter.” And they won't know it's from me, so they're basically accusing me of joke stealing when really I'm just content-raising is what I should tell them. But it is funny. [laugh].Corey: That's happened me a couple times on Twitter. People are like, “Hey, that's a stolen joke.” And then they'll google it and they'll dig it out. Like, “Here's the original—oh, wait, you said it two years ago.” “Yeah. No one liked it then, so here we are.” “If you liked it then, why didn't you blow it up like you did now?” So.Emily: They remembered it from two years ago, but they didn't remember it was yours. [laugh].Corey: At some level, I feel like I could almost loop my Twitter account and just let it continue to play out again for the next seven years, and other than the live-streaming stuff and the live-tweeting various events, I feel like it would do fairly well, but who knows.Emily: Yeah. Yeah. But at the end of the day, I think there's also a finite amount of funny tech jokes, and we're all just kind of recycling each other's jokes at some point. So, I don't get too offended by that. I'm like, “Sure. We all made the same joke about NFTs. Great.” Like, I don't care. [laugh].Corey: I really want to thank you for taking the time to speak with me today.Emily: [crosstalk 00:32:36] been fun.Corey: If people want to learn more and appreciate some of that awesome content, where's the best place to find you?Emily: Yeah, I'm on the Twitters and the TikToks, just like you.Corey: Excellent. And we will, of course, put links to that in the [show notes 00:32:45].Emily: Had a great time. Thank you so much for having me again.Corey: No, thank you for coming. Emily Kager, senior Android engineer at Uber. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that links to a TikTok video of you ranting for a solid minute, but because computers and phones alike are very hard, you're using the wrong camera, and we just get that video of your floor.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Into the Year of Documentation with Dr. KellyAnn Fitzpatrick

Screaming in the Cloud

Play Episode Listen Later Mar 2, 2022 37:52


About KellyKellyAnn Fitzpatrick is a Senior Industry Analyst at RedMonk, the developer-focused industry analyst firm. Having previously worked as a QA analyst, test & release manager, and tech writer, she has experience with containers, CI/CD, testing frameworks, documentation, and training. She has also taught technical communication to computer science majors at the Georgia Institute of Technology as a Brittain Postdoctoral Fellow.Holding a Ph.D. in English from the University at Albany and a B.A. in English and Medieval Studies from the University of Notre Dame, KellyAnn's side projects include teaching, speaking, and writing about medievalism (the ways that post-medieval societies reimagine or appropriate the Middle Ages), and running to/from donut shops.Links: RedMonk: https://redmonk.com/ Twitter: https://twitter.com/drkellyannfitz TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance query accelerator for the Oracle MySQL Database Service, although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLAP and OLTP—don't ask me to pronounce those acronyms again—workloads directly from your MySQL database and eliminate the time-consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. It's always a good day when I get to sit down and have a chat with someone who works over at our friends at RedMonk. Today is no exception because after trying for, well, an embarrassingly long time, my whining and pleading has finally borne fruit, and I'm joined by Kelly Fitzpatrick, who's a senior industry analyst at RedMonk. Kelly, thank you for, I guess, finally giving in to my always polite, but remarkably persistent requests to show up on the show.Kelly: Great, thanks for having me. It's great to finally be on the show.Corey: So, let's start at the very beginning because I am always shockingly offended whenever it happens, but some people don't actually know what RedMonk is. What is it you'd say it is that you folks do?Kelly: Oh, I love this question. Because it's like, “What do you do,” versus, “What are you?” And that's a very big difference. And I'm going to start with maybe what we are. So, we are a developer-focused industry analyst firm. You put all those things, kind of, together.And in terms of what we do, it means that we follow tech trends. And that's something that many industry analysts do, but our perspective is really interested in developers specifically and then practitioners more broadly. So, it's not just, “Okay, these are things that are happening in tech that you care about if you're a CIO,” but what tech things affect developers in terms of how they're building software and why they want to build software and where they're building software?Corey: So, backing it up slightly because it turns out that I don't know the answer to this either. What exactly is an industry analyst firm? And the reason I bring this up is I've been invited to industry analyst events, and that is entirely your colleague, James Governor's, fault because he took me out for lunch at I think it was Google Next a few years ago and said, “Oh, you're definitely an analyst.” “Okay, cool. Well, I don't think I am. Why should I be an analyst?”“Oh, because companies have analyst budgets.” “Oh, you said, analyst”—protip: Never get in the way of people trying to pay you to do things. But I still feel like I don't know what an analyst is, in this sense. Which means I'm about to get a whole bunch of refund requests when this thing airs.Kelly: I should hope not. But industry analysts, one of the jokes that we have around RedMonk is how do we explain to our families what an industry analyst is? And I think even Steve and James, who are RedMonk's founders, they've been doing this for quite a long time, like, much longer than they ever want to admit that they do, and they still are like, “Okay, how do I explain this to my parents?” Or you know, anyone else who's asking, and partly, it's almost like a very—a term that you'll see in the tech industry, but outside of it doesn't really have that much, kind of, currency in the same way that you can tell someone that you're like, maybe a business analyst or something like that, or any of those, almost like spy-like versions of analyst. I think was it The Hunt for Red October, the actual hero of that is an analyst, but not the type of analyst that I am in any way, shape or form.But you know, industry analyst firms, specifically, it's like we keep up on what tech is out there. People engage with us because they want to know what to buy for the things that they're doing and the things that they're building, or how to better create and sell the stuff that they are building to people who build software. So, in our case, it's like, all right, what type of tools are developers using? And where does this particular tool that our company is building fit into that? And how do you talk about that with developers in a way that makes sense to them?Corey: On some level, what I imagine your approach to this stuff is aligns somewhat with my own. Before you became an industry analyst, which I'm still not entirely sure I know what that is—I'm sorry, not your fault; just so many expressions of it out there—before you wound up down that path, you were a QA manager; you wound up effectively finding interesting bugs in software, documentation, et cetera. And, on some level, that's, I think, what has made me even somewhat useful in the space is I'll go ahead and try and build something out of something that a vendor has released, and huh, the documentation says it should work this way, but I try it and it breaks and it fails. And the response is always invariably the same, which is, “That's interesting,” which is engineering-speak for, “What the hell is that?” I have this knack for stumbling over weird issues, and I feel like that aligns with what makes for a successful QA person. Is that directionally correct, or am I dramatically misunderstanding things and I'm just accident-prone?Kelly: [laugh]. No, I think that makes a lot of sense. And especially coming from QA where it's like, not just making sure that something works, but making sure that something doesn't break if you try to break it in different ways, the things that are not necessarily the expected, you know, behaviors, that type of mindset, I think, for me translated very easily to, kind of, being an analyst. Because it's about asking questions; it's about not just taking the word of your developers that this software works, but going and seeing if it actually does and kind of getting your hands dirty, and in some cases, trying to figure out where certain problems or who broke the build, or why did the build break is always kind of super fun mystery that I love doing—not really, but, like, everyone kind of has to do it—and I think that translates to the analyst world where it's like, what pieces of these systems, or tech stacks, or just the way information is being conveyed about them is working or is not, and in what ways can people kind of maybe see things a different way that the people who are building or writing about these things did not anticipate?Corey: From my position, and this is one of the reasons I sort of started down this whole path is if I'm trying to build something with a product or a platform—or basically anything, it doesn't really matter what—and the user experience is bad, or there are bugs that get in my way, my default response—even now—is not, “Oh, this thing's a piece of crap that's nowhere near ready for primetime use,” but instead, it's, “Oh, I'm not smart enough to figure out how to use it.” It becomes a reflection on the user, and they feel bad as a result. And I don't like that for anyone, for any product because it doesn't serve the product well, it certainly doesn't serve the human being trying to use it and failing well, and from a pure business perspective, it certainly doesn't serve the ability to solve a business problem in any meaningful respect. So, that has been one of the reasons that I've been tilting at that particular windmill for as long as I have.Kelly: I think that makes sense because you can have the theoretically best, most innovative, going to change everyone's lives for the better, product in the world, but if nobody can use it, it's not going to change the world.Corey: As you take a look at your time at RedMonk, which has been, I believe, four years, give or take?Kelly: We're going to say three to four.Corey: Three to four? Because you've been promoted twice in your time there, let's be very clear, and this is clearly a—Kelly: That's a very, very astute observation on your part.Corey: It is a meteoric rise. And what makes that also fascinating from my perspective, is that despite being a company that is, I believe, 19 years old, you aren't exactly a giant company that throws bodies at problems. I believe you have seven full-time employees, two of whom have been hired in the last quarter.Kelly: That's true. So, seven full-time employees and five analysts. So, we have—of that it's five analysts, and we only added a fifth analyst the beginning of this year, with Dr. Kate Holterhoff. [unintelligible 00:08:09], kind of, bring her on the team.So, we had been operating with, like, kind of, six full-time employees. We were like, “We need some more resources in this area.” And we heard another analyst, which if you talk about, okay, we hired one more, but when you're talking about hiring one more and adding that to a team of, like, four analysts, it's such a big difference, just in terms of, kind of, resources. And I think your observation about you ca—we don't just throw bodies at problems is kind of correct. That is absolutely not the way we go about things at all.Corey: At a company that is taking the same model that The Duckbill Group does—by which I mean not raising a bunch of outside money is, as best I can tell—that means that you have to fall back on this ancient business model known as making more money than it costs to run the place every month, you don't get to do this massive scaled out hiring thing. So, bringing on multiple employees at a relatively low turnover company means that suddenly you're onboarding not just one new person, but two. What has that been like? Because to be very clear, if you're hiring 20 engineers or whatnot, okay, great, and you're having significant turnover, yeah, onboarding two folks is not that big of a deal, but this is a significant percentage of your team.Kelly: It is. And so for us—and Kate started at the beginning of this year, so she's only been here for a bit—but in terms of onboarding another analyst, this is something where I haven't done before, but, like, my colleagues have, whereas the other new member of our team, Morgan Harris, who is our Account Engagement Manager, and she is amazing, and has also, like, very interesting background and client success in, like, fashion, which is, you know, awesome when I'm trying to figure out what [unintelligible 00:09:48] fit I need to do, we have someone in-house who can actually give me advice on that. But that's not something that we have onboarded for that role very much in the past, so bringing on someone where they're the only person in their role and, like, having to begin to learn the role. And then also to bring in another analyst where we have a little bit more experience onboarding analysts, it takes a lot of patience for everybody involved. And the thing I love about RedMonk and the people that I get to work with is that they actually have that patience and we function very well as, like, a team.And because of that, I think things that could really have thrown us off course, like losing an account engagement or onboarding one and then onboarding a new analyst, like, over the holidays, during a pandemic, and everything else that is happening, it's going much more smoothly than it could have otherwise.Corey: These are abnormal times, to be sure. It's one of those things where it's, we're a couple years into a pandemic now, and I still feel like we haven't really solved most of the problems that this has laid bare, which kind of makes me despair of ever really figuring out what that's going to look like down the road.Kelly: Yeah, absolutely. And there is very much the sense that, “Okay, we should be kind of back to normal, going to in-person conferences.” And then you get to an in-person conference, and then they all move back to virtual or, as in your case, you go to an in-person conference and then you have to sequester yourself away from your family for a couple of weeks to make sure that you're not bringing something home.Corey: So, I have to ask. You have been quoted as saying that 2022—for those listening, that is this year—is the year of documentation. You're onboarding two new people into a company that does not see significant turnover, which means that invariably, “Oh, it's been a while since we've updated the documentation. Whoops-a-doozy,” is a pretty common experience there. How much of your assertion that this is the year of documentation comes down to the, “Huh. Our onboarding stuff is really out of date,” versus a larger thing that you're seeing in the industry?Kelly: That is a great question because you never know what your documentation is like until you have someone new, kind of, come in with fresh eyes, has a perspective not only on, “Okay, I have no idea what this means,” or, “This is not where I thought it would be,” or, “This, you know, system is not working in any… in any way similar to anything I have ever seen in any other part of my, like, kind of, working career.” So, that's where you really see what kind of gaps you have, but then you also kind of get to see which parts are working out really well. And not to spend, kind of, too much on that, but one of the best things that my coworkers did for me when I started was, Rachel Stephens had kept a log of, like, all the questions that she had as a new analyst. And she just, like, gave that to me with some advice on different things, like, in a spreadsheet, which I think is—I love spreadsheets so much and so does Rachel. And I think I might love spreadsheets more than Rachel at this point, even though she actually has a hat that says, “Spreadsheets.”But when Kate started, it was fascinating to go through that and see what parts of that were either no longer relevant because the entire world had changed, or because the industry had advanced, or because there's all these new things you need to know now that we're not on the list of things that you needed to know three years ago. And then what other, even, topics belong down on that kind of list of things to know. So, I think documentation is always a good, like, check-in for things like that.But going back to, like, your larger question. So, documentation is important, not just because we happened to be onboarding, but a lot of people, I think once they no longer could be in the office with people and rely on that kind of face-to-face conversations to smooth over things began, I think, to realize how essential documentation was to just their everyday to day, kind of, working lives. So, I think that's something that we've definitely seen from the pandemic. But then there are certainly other signals in the software industry-specific, which we can go into or not depending on your level of interest.Corey: Well, something that I see that I have never been a huge fan of in corporate life—and it feels like it is very much a broad spectrum—has been that on one side of the coin, you have this idea that everything we do is bespoke and we just hire smart people and get out of their way. Yeah, that's more uncontrolled anarchy than it is a repeatable company process around anything. And the other extreme is this tendency that companies have, particularly the large, somewhat slow-moving companies, to attempt to codify absolutely everything. It almost feels like it derives from the what I believe to be mistaken belief that with enough process, eventually you can arrive at the promised land where you don't have to have intelligent, dynamic people working behind things, you can basically distill it down to follow the script and push the buttons in the proper order, and any conceivable outcome is going to be achieved. I don't know if that's accurate, but that's always how it felt when you start getting too deeply mired in documentation-slash-process as almost religion.Kelly: And I think—you know, I agree. There has to be something between, “All right, we don't document anything and it's not necessary and we don't need it.” And then—Corey: “We might get raided by the FBI. We want nothing written down.” At which point it's like, what do you do here? Yeah.Kelly: Yeah. Leave no evidence, leave no paper trail of anything like that. And going too far into thinking that processes is absolutely everything, and that absolutely anyone can be plugged into any given role and things will be equally successful, or that we'll just be automated away or become just these, kind of, automatons. And I think that balance, it's important to think about that because while documentation is important, and you know, I will say 2022, I think we're going to hear more and more about it, we see it more as an increasingly valuable thing in tech, you can't solve everything with documentation. You can use it as the, kind of, duct tape and baling wire for some of the things that your company is doing, but throwing documentation at it is not going to fix things in the same way that throwing engineers at a problem is not going to fix it either. Or most problems. I mean, there are some that you can just throw engineers at.Corey: Well, there's a company wiki, also known as where documentation goes to die.Kelly: It is. And those, like, internal wikis, as horrible as they can be in terms of that's where knowledge goes to die as well, places that have nothing like that, it can be even more chaotic than places that are relying on the, kind of, company internal wiki.Corey: So, delving into a bit of a different topic here, before you were in the QA universe, you were what distills down to an academic. And I know that sometimes that can be interpreted as a personal attack in some quarters; I assure you, despite my own eighth grade level of education, that is not how this is intended at all. Your undergraduate degree was in medieval history—or medieval studies and your PhD was in English. So, a couple of questions around that. One, when we talk about medieval studies, are we talking about writing analyst reports about Netscape Navigator, or are we talking things a bit later in the sweep of history than that?Kelly: I appreciate the Netscape Navigator reference. I get that reference.Corey: Well, yeah. Medieval studies; you have to.Kelly: Medieval studies, when you—where we study the internet in the 1990s, basically. I completely lost the line of questioning that you're asking because I was just so taken by the Netscape Navigator reference.Corey: Well, thank you. Started off with the medieval studies history. So, medieval studies of things dating back to, I guess, before we had reasonably recorded records in a consistent way. And also Twitter. But I'm wondering how much of that lends itself to what you do as an analyst.Kelly: Quite a bit. And as much as I want to say, it's all Monty Python references all the time, it isn't. But the disciplinary rigor that you have to pick up as a medievalist or as anyone who's getting any kind of PhD ever, you know, for the most part, that very much easily translated to being an analyst. And even more so tech culture is, in so many ways, like, enamored—there's these pop culture medieval-isms that a lot of people who move in technical circles appreciate. And that kind of overlap for me was kind of fascinating.So, when I started, like, working in tech, the fact that I was like writing a dissertation on Lord of the Rings was this little interesting thing that my coworkers could, like, kind of latch on to and talk about with me, that had nothing to do with tech and that had nothing to do with the seemingly scary parts of being an academic.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats V-U-L-T-R.com slash screaming.Corey: I want to talk a little bit about the idea of academic rigor because to my understanding, in the academic world, the publication process is… I don't want to say it's arduous. But if people subjected my blog post anything approaching this, I would never write another one as long as I lived. How does that differ? Because a lot of what I write is off-the-cuff stuff—and I'm not just including tweets, but also tweets—whereas academic literature winds up in peer-reviewed journals and effectively expands the boundaries of our collective societal knowledge as we know it. And it does deserve a different level of scrutiny, let's be clear. But how do you find that shifts given that you are writing full-on industry analyst reports, which is something that we almost never do on our side, just honestly, due to my own peccadilloes?Kelly: You should write some industry reports. They're so fun. They're very fun.Corey: I am so bad at writing the long-form stuff. And we've done one or two previously, and each time my business partner had to basically hold my nose to the grindstone by force to get me to ship, on some level.Kelly: And also, I feel like you might be underselling the amount of writing talent it takes to tweet.Corey: It depends. You can get a lot more trouble tweeting than you can in academia most of the time. Every Twitter person is Reviewer 2. It becomes this whole great thing of, “Well, did you consider this edge corner case nuance?” It's, “I've got to say, in 208 any characters, not really. Kind of ran out of space.”Kelly: Yeah, there's no space at all. And it's not what that was intended. But going back to your original question about, like, you know, academic publishing and that type of process, I don't miss it. And I have actually published some academic pieces since I became an analyst. So, my book finally came out after I had started as—it came out the end of 2019 and I had already been at RedMonk for a year.It's an academic book; it has nothing to do with being an industry analyst. And I had an essay come out in another collection around the same time. So, I've had that come out, but the thing is, the cycle for that started about a year earlier. So, the timeframe for getting things out in, especially the humanities, can be very arduous and frustrating because you're kind of like, “I wrote this thing. I want it to actually appear somewhere that people can read it or use it or rip it apart if that's what they're going to do.”And then the jokes that you hear on Twitter about Reviewer 2 are often real. A lot of academic publishing is done in, like, usually, like, a double-blind process where you don't know who's reviewing you and the reviewers don't know who you are. I've been a reviewer, too, so I've been on that side of it. And—Corey: Which why you run into the common trope of people—Kelly: Yes.Corey: —suggesting, “Oh, you don't know what you're talking about. You should read this work by someone else,” who is in fact, the author they are reviewing.Kelly: Absolutely. That I think happens even when people do know who [laugh] who's stuff they're reviewing. Because it happens on Twitter all the time.Corey: Like, “Well, have you gotten to the next step beyond where you have a reviewer saying you should wind up looking at the work cited by”—and then they name-check themselves? Have we reached that level of petty yet, or has that still yet to be explored?Kelly: That is definitely something that happens in academic publishing. In academic circles, there can be these, like, frenemy relations among people that you know, especially if you are in a subfield that is very tiny. You tend to know everybody who is in that subfield, and there's, like, a lot of infighting. And it does not feel that far from tech, sometimes. [unintelligible 00:21:52] you could look at the whole tech industry, and you look at the little areas that people specialize in, and there are these communities around these specializations that—you can see some of them on Twitter.Clearly, not all of them exist in the Twitterverse, but in some ways, I think that translated over nicely of, like, the year-long publication and, like, double peer-review process is not something that I have to deal with as much now, and it's certainly something that I don't miss.Corey: You spent extensive amounts of time studying the past, and presumably dragons as well because, you know, it's impossible to separate medieval studies from dragons in my mind because basically, I am a giant child who lives through fantasy novels when it comes to exploring that kind of past. And do you wind up seeing any lessons we can take from the things you have studied to our current industry? That is sort of a strange question, but they say that history doesn't repeat, but it rhymes, and I'm curious to how far back that goes. Because most people are citing, you know, 1980s business studies. This goes centuries before that.Kelly: I think the thing that maybe stands out for me the most the way that you framed that is, when we look at the past and we think of something like the Middle Ages, we will often use that term and be like, “Okay, here's this thing that actually existed, right?” Here's, like, this 500 years of history, and this is where the Middle Ages began, and here's where it ended, and this is what it was like, and this is what the people were like. And we look at that as the some type of self-evident thing that exists when in reality, it's a concept that we created, that people who lived in later ages created this concept, but then it becomes something that has real currency and, really, weight in terms of, like, how we talk about the world.So, someone will say, you know, I like that film. It was very medieval. And it'll be a complete fantasy that has nothing to do with Middle Ages but has a whole bunch of these tropes and signals that we translate as the Middle Ages. I feel like the tech industry has a great capacity to do that as well, to kind of fold in along with things that we tend to think of as being very scientific and very logical but to take a concept and then just kind of begin to act as if it is an actual thing when it's something that people are trying to make a thing.Corey: Tech has a lot of challenges around the refusing to learn from history aspect in some areas, too. One of the most common examples I've heard of—or at least one that resonated the most with me—is hiring, where tech loves to say, “No one really knows how to hire effectively and well.” And that is provably not true. Ford and GM and Coca-Cola have run multi-decade studies on how to do this. They've gotten it down to a science.But very often, we look at that in tech and we're trying to invent everything from first principles. And I think, on some level, part of that comes out as, “Well, I wouldn't do so well in that type of interview scenario, therefore, it sucks.” And I feel like we're too willing in some cases to fail to heed the lessons that others have painstakingly learned, so we go ahead and experiment on our own and try and reinvent things that maybe we should not be innovating around if we're small, scrappy, and trying to one area of the industry. Maybe going back to how we hire human beings should not be one of those areas of innovation that you spend all your time on as a company.Kelly: I think for some companies, I think it depends on how you're hiring now. It's like, if your hiring practices are horrible, like, you probably do need to change them. But to your point, like, spending all of your energy on how are we hiring, can be counterproductive. Am I allowed to ask you a question?Corey: Oh, by all means. Mostly, the questions people ask me is, “What the hell is wrong with you?” But that's fine, I'm used to that one, too. Bonus points if you have a different one.Kelly: Like, your hiring processes at Duckbill Group. Because you've hired, you know, folks recently. How do you describe that? Like, what points of that you think… are working really well?Corey: The things that have worked out well for us have been being very transparent at the beginning around things like comp, what the job looks like, where it starts, where it stops, what we expect from people, what we do not expect from people, so there are no surprises down that path. We explain how many rounds of interviews there are, who they'll be meeting with at each stage. If we wind up declining to continue with a candidate in a particular cycle, anything past the initial blind resume submission, we will tell them; we don't ghost people. Full stop. Originally, we wanted to wind up responding to every applicant with a, “Sorry, we're not going to proceed,” if the resume was a colossal mismatch. For example, we're hiring for a cloud economist, and we have people with PhDs in economics, and… that's it. They have not read the job description.And then when you started doing that people would argue with us on a constant basis, and it just became a soul-sucking time sink. So, it's unfortunate, but that's the reality of it. But once we've had a conversation with you, doing that is the right answer. We try and move relatively quickly. We're honest with folks because we believe that an interview is very much a two-way street.And even if we declined to proceed—or you declined to proceed with us; either way—that you should still think well enough of us that you would recommend us to people for whom it might be a fit. And if we treat you like crap, you're never going to do that. Not to mention, I just don't like making people feel like crap as a general rule. So, that stuff that has all come out of hiring studies.So, has the idea of a standardized interview. We don't have an arbitrary question list that we wind up smacking people with from a variety of different angles. And if you drew the lucky questions, you'll do fine. We also don't set this up as pass-fail, we tend to presume that by the time you've been around the industry for as long as generally is expected for years of experience for the role, we're not going to suddenly unmask you as not knowing how computers work through our ridiculous series of trivia questions. We don't ask those.We also make the interview look a lot like what the job is, which is apparently a weird thing. It's in a lot of tech companies it's, “Go and solve whiteboard algorithms for us.” And then, “Great. Now, what's the job?” “It's going to be moving around some CSS nonsense.”It's like, first that is very different, and secondly, it's way harder to move CSS than to implement quicksort, for most folks. At least for me. So, it's… yeah, it just doesn't measure the right things. That's our approach. I'm not saying we cracked it by any means to be very clear here. This is just what we have found that sucks the least.Kelly: Yeah, I think the, ‘we're not going to do obscure whiteboarding exercises' is probably one of the key things. I think some people are still very attached those personal reasons. And I think the other thing I liked about what you said, is to make the interview as similar to the job as you can, which based on my own getting hired process at RedMonk and then to some levels of being involved in hiring our, kind of, new hires, I really like that. And I think that for me, the process will like, okay, you submit your application. There'd be—I think I'd to do a writing sample.But then it was like, you get on a call and you talk to Steve. And then you get on a call and you talk to James. And talking to people is my job. Like for the most part. I write things, but it's mostly talking to people, which you may not believe by the level of articulate, articulate-ness, I am stumbling my way through in this sentence.And then the transparency angle, I think it's something that most companies are not—may not be able to approach hiring in such a transparent way for whatever reason, but at least the motion towards being transparent about things like salaries, as opposed to that horrible salary negotiation part where that can be a nightmare for people, especially if there's this code of silence around what your coworkers or potential coworkers are making.Corey: We learned we were underpaying our clouds economists, so we wound up adjusting the rate advertised; at the same time we wound up improving the comp for existing team because, “Yeah, we're just going to make you apply again to be paid a fair wage for what you do,” no. Not how we play these games.Kelly: Yeah, which is, you know, one of the things that we're seeing in the industry now. Of course, the term ‘The Great Resignation' is out there. But with that comes, you know, people going to new places partly because that's how they can get, like, the salary increase or whatever it is they want for among other reasons.Corey: Some of the employees who have left have been our staunchest advocates, both for new applicants as well as new clients. There's something to be said for treating people as you mean to go on. My business partner, I've been clear that we aspire for this to be a 20, 25-year company, and you don't do that by burning bridges.Kelly: Yeah. Or just assuming that your folks are going to stay for three years and move on, which tends to be the kind of the lifespan of where people stay.Corey: Well, if they do, that's fine because it is expected. I don't want people to wind up feeling that they owe us anything. If it no longer makes sense for them to be here because they're not fulfilled or whatnot—this has happened to us before we've tried to change their mind, talked to them about what they wanted, and okay, we can't offer what you're after. How can we help you move on? That's the way it works.And like, the one thing we don't do in interviews—and this is something I very much picked up from the RedMonk culture as well—is we do a lot of writing here, so there's a writing sample of here's a list of theoretical findings for an AWS bill—if we're talking about a cloud economist role—great. Now, the next round is people are going to talk to you about that, and we're going to roleplay as if we were a client. But let's be clear, I won't tolerate abusive behavior from clients to our team, I will fire a client if it happens. So, we're not going to wind up bullying the applicant and smacking ‘em around on stuff—or smacking them around to be clear. That was an ‘em not a him, let's be clear.It's a problem of not wanting to even set the baseline expectation that you just have to sit there and take it when clients decide to go down unfortunate paths. And I believe it's happened all of maybe once in our five-and-a-half-year history. So, why would you ever sit around and basically have a bunch of people chip away at an applicant's self-confidence? By virtue of being in the room and having the conversation, they are clearly baseline competent at a number of things. Now, it's just a question of fit and whether their expression of skills is what we're doing right now as a company.At least that's how I see it. And I think that there is a lot of alignment here, not just between our two companies, but between the kinds of companies I look at and can actively recommend that people go and talk to.Kelly: Yeah. I think that emphasis on, it's not just about what a company is doing—like, what is their business, you know, how they're making money—but how they're treating people, like, on their way in and on the way out. I don't think you can oversell how important that is.Corey: Culture is what you wind up with instead of what you intend. And I think that's something that winds up getting lost a fair bit.Kelly: Yeah, culture is definitely not something you can just go buy, right? [laugh], where you can, like—this is what our culture will be.Corey: No, no. But if there is, “Culture-in-a-box. Like, you may not be able to buy it, but I would love to sell it to you,” seems to be the watchwords of a number of different companies out there. Kelly, I really want to thank you for taking the time to speak with me today. If people want to learn more, where can they find you?Kelly: They can find me on Twitter at @drkellyannfitz, that's D-R-K-E-L-L-Y-A-N-N-F-I-T-Z—I apologize for having such a long Twitter handle—or my RedMonk work and of my colleagues, you can find that at redmonk.com.Corey: And we will, of course, include links to that in the [show notes 00:33:14]. Thank you so much for your time. I appreciate it.Kelly: Thanks for having me.Corey: Kelly Fitzpatrick, senior industry analyst at RedMonk. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment telling me how terrible this was and that we should go listen to Reviewer 2's podcast instead.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Commanding the Council of the Lords of Thought with Anna Belak

Screaming in the Cloud

Play Episode Listen Later Mar 1, 2022 33:29


About AnnaAnna has nearly ten years of experience researching and advising organizations on cloud adoption with a focus on security best practices. As a Gartner Analyst, Anna spent six years helping more than 500 enterprises with vulnerability management, security monitoring, and DevSecOps initiatives. Anna's research and talks have been used to transform organizations' IT strategies and her research agenda helped to shape markets. Anna is the Director of Thought Leadership at Sysdig, using her deep understanding of the security industry to help IT professionals succeed in their cloud-native journey.Anna holds a PhD in Materials Engineering from the University of Michigan, where she developed computational methods to study solar cells and rechargeable batteries.How do I adapt my security practices for the cloud-native world?How do I select and deploy appropriate tools and processes to address business needs?How do I make sense of new technology trends like threat deception, machine learning, and containers?Links: Sysdig: https://sysdig.com/ “2022 Cloud-Native Security and Usage Report”: https://sysdig.com/2022-cloud-native-security-and-usage-report/ Twitter: https://twitter.com/aabelak LinkedIn: https://www.linkedin.com/in/aabelak/ Email: anna.belak@sysdig.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance query accelerator for the Oracle MySQL Database Service, although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLAP and OLTP—don't ask me to pronounce those acronyms again—workloads directly from your MySQL database and eliminate the time-consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Once upon a time, I went to a conference talk at, basically, a user meetup. This was in the before times, when that wasn't quite as much of a deadly risk because of a pandemic, and mostly a deadly risk due to me shooting my mouth off when it wasn't particularly appreciated.At that talk, I wound up seeing a new open-source project that was presented to me, and it was called Sysdig. I wasn't quite sure on what it did at the time and I didn't know what it would be turning into, but here we are now, what is it, five years later. Well, it's turned into something rather interesting. This is a promoted episode brought to us by our friends at Sysdig and my guest today is their Director of Thought Leadership, Anna Belak. Anna, thank you for joining me.Anna: Hi, Corey. I'm very happy to be here. I'm a big fan.Corey: Oh, dear. So, let's start at the beginning. Well, we'll start with the title: Director of Thought Leadership. That is a lofty title, it sounds like you sit on the council of the Lords of Thought somewhere. Where does your job start and stop?Anna: I command the Council of the Lords of thought, actually. [laugh].Corey: Supply chain issues mean the robe wasn't available. I get it, I get it.Anna: There is a robe. I'm just not wearing it right now. So, the shortest way to describe the role is probably something that reports into engineering, interestingly, and it deals with product and marketing in a way that is half evangelism and half product strategy. I just didn't feel like being called any of those other things, so they were like, “Director of Thought Leadership you are.” And I was like, “That sounds awesome.”Corey: You know, it's one of those titles that people generally don't see a whole lot of, so if nothing else, I always liked those job titles that cause people to sit up and take notice as opposed to something that just people fall asleep by the time you get halfway through it because, in lieu of a promotion, people give you additional adjectives in your title. And we're going to go with it. So, before you wound up at Sysdig, you were at Gartner for a number of years.Anna: That's right, I spent about six years at Gartner, and there half the time I covered containers, Kubernetes, and DevOps from an infrastructure perspective, and half the time I spent covering security operations, actually, not specifically with respect to containers, or cloud, but broadly. And so my favorite thing is security operations, as it relates to containers and cloud-native workloads, which is kind of how I ended up here.Corey: I wouldn't call that my favorite thing. It's certainly something that is near and dear to the top of mind, but that's not because I like it, let's put it [laugh] that way. It's one of those areas where getting it wrong is catastrophic. Back in 2017, when I went to that meetup in San Francisco, Sysdig seemed really interesting to me because it looked like it tied together a whole bunch of different diagnostic tools, LSOF, strace, and the rest. Honestly—and I mean no slight to the folks who built out this particular tool—it felt like DTrace, only it understood the value of being accessible to its users without basically getting a doctorate in something.I like the idea, and it felt like it was very much aimed at an in-depth performance analysis story or an observability play. But today, it seems that you folks have instead gone in much more of a direction of DevSecOps, if the people listening to this, and you, will pardon the term. How did that happen? What was that product evolution like?Anna: Yeah, I think that's a fair assessment, actually. And again, no disrespect to DTrace of which I'm also a fan. So, we certainly started out in the container observability space, essentially because this whole Docker Kubernetes thing was exploding in popularity—I mean, before it was exploding, it was just kind of like, peaking out—and very quickly, our founder Loris, who is the co-founder of Wireshark, was like, “Hey, there's a visibility issue here. We can't see inside these things with the tools that we have that are built for host instrumentation, so I'm going to make a thing.” And he made a thing, and it was an awesome thing that was open-sourced.And then ultimately, what happened is, the ecosystem of containers and communities evolved, and more and more people started to adopt it. And so more people needed kind of a more, let's say, hefty, serious tool for observability, and then what followed was another tool for security because what we actually discovered was the data that we're able to collect from the system with Sysdig is incredibly useful for noticing security problems. So, that caused us to kind of expand into that space. And today we are very much a tool that still has an observability component that is quite popular, has a security component which is it's fairly broad: We cover CSPM use cases, we cover [CIEM 00:05:04] use cases, and we are very, kind of let's say, very strong and very serious about our detection response and runtime security use cases, which come from that pedigree of the original Sysdig as well.Corey: You can get a fairly accurate picture of what the future of technology looks like by taking a look at what my opinion of something is, and then doing the exact opposite of that. I was a big believer that virtualization, “Complete flash in the pan; who's going to use that?” Public cloud, “Are you out of your tree? No one's going to trust other companies with their holy of holies.” And I also spent a lot of time crapping on containers and not actually getting into them.Instead, I leapfrogged over into the serverless land, which I was a big fan of, which of course means that it's going to be doomed sooner or later. My security position has also somewhat followed similar tracks where, back when you're running virtual machines that tend to be persistent, you really have to care about security because you are running full-on systems that are persistent, and they run all kinds of different services simultaneously. Looking at Lambda Functions, for example, in the modern serverless world, I always find a lot of the tooling and services and offerings around security for that are a little overblown. They have a defined narrow input, they have a defined output, there usually aren't omnibus functions shoved in here where they have all kinds of different code paths. And it just doesn't have the same attack surface, so it often feels like it's trying to sell me something I don't need. Security in the container world is one of those areas I never had to deal with in anger, as a direct result. So, I have to ask, how bad is it?Anna: Well, I have some data to share with you, but I'll start by saying that I maybe was the opposite of you, so we'll see which one of us wins this one. I was an instant container fangirl from the minute I discovered them. But I crapped out—Corey: The industry shows you were right on that one. I think the jury [laugh] is pretty much in on this one.Anna: Oh, I will take it. But I did crap on Lambda Functions pretty hard. I was like, “Serverless? This is dumb. Like, how are we ever going to make that work?” So, it seems to be catching on a little bit, at least it. It does seem like serverless is playing the function of, like, the glue between bits, so that does actually make a lot of sense. In retrospect, I don't know that we're going to have—Corey: Well, it feels like it started off with a whole bunch of constraints around it, and over time, they've continued to relax those constraints. It used to be, “How do I package this?” It's, “Oh, simple. You just spent four days learning about all the ins and outs of this,” and now it's, “Oh, yeah. You just give it a Docker file?” “Oh. Well, that seems easier. I could have just been stubborn and waited.” Hindsight.Anna: Yeah, exactly. So, containers as they are today, I think are definitely much more usable than they were five-plus years ago. There are—again there's a lot of commercial support around these things, right? So, if you're, you know, like, a big enterprise client, then you don't really have time to fool around in open-source, you can go in, buy yourself a thing, and they'll come with support, and somebody will hold your hand as you figure it out, and it's actually quite, quite pleasant. Whether or not that has really gone mainstream or whether or not we've built out the entire operational ecosystem around it in a, let's say, safe and functional way remains to be seen. So, I'll share some data from our report, which is actually kind of the key thing I want to talk about.Corey: Yeah, I wanted to get into that. You wound up publishing this somewhat recently, and I regret that as of the time of this recording, I have not yet had time to go into it in-depth, and of course eviscerate it in my typical style on Twitter—although that may have been rectified by the time that this show airs, to be very clear—but it's the Sysdig “2022 Cloud-Native Security and Usage Report”.Anna: Please at me when you Twitter-shred it. [laugh].Corey: Oh, when I read through and screenshot it, and I'd make what observations that I imagine are witty. But I'm looking forward to it; I've done that periodically with the Flexera, “State of the Cloud” report for last few years, and every once in a while, whatever there's a, “We've done a piece of thought leadership, and written a report,” it's, “Oh, great. Let's make fun of it.” That's basically my default position on things. I am not a popular man, as you might imagine. But not having had the chance to go through it in-depth, what did this attempt to figure out when the study was built, and what did you learn that you found surprising?Anna: Yeah, so the first thing I want to point out because it's actually quite important is that this report is not a survey. This is actual data from our actual back end. So, we're a SaaS provider, we collect data for our customers, we completely anonymize it, and then we show in aggregate what in fact we see them doing or not doing. Because we think this is a pretty good indicator of what's actually happening versus asking people for their opinion, which is, you know, their opinion.Corey: Oh, I love that. My favorite lies that people tell are the lies they don't realize that they're telling. It's, I'll do an AWS bill analysis and, “Great. So, tell me about all these instances you have running over in Frankfurt.” “Oh, we don't have anything there.”I believe you're being sincere when you say this, however, the data does show otherwise, and yay, now we're in a security incident.Anna: Exactly.Corey: I'm a big believer of going to the actual source for things like this where it's possible.Anna: Exactly. So, I'll tell you my biggest takeaway from the whole thing probably was that I was surprised by the lack of… surprise. And I work in cloud-native security, so I'm kind of hoping every single day that people will start adopting these modern patterns of, like, discarding images, and deploying new ones when they found a vulnerability, and making ephemeral systems that don't run for a long time like a virtual machine in disguise, and so on. And it appears that that's just not really happening.Corey: Yeah, it's always been fun, more than a little entertaining, when I wind up taking a look at the aspirational plans that companies have. “Great, so when are you going to do”—“Oh, we're going to get to that after the next sprint.” “Cool.” And then I just set a reminder and I go back a year later, and, “How's that coming?” “Oh, yeah. We're going to get to that next sprint.”It's the big lie that we always tell ourselves that right after we finished this current project, then we're going to suddenly start doing smart things, making the right decisions, and the rest. Security, cost, and a few other things all tend to fall on the side of, you can spend infinite money and infinite time on these things, but it doesn't advance what your business is doing, but if you do none of those things, you don't really have a business anymore. So, it's always a challenge to get it prioritized by the strategic folks.Anna: Exactly. You're exactly right because what people ultimately do is they prioritize business needs, right? They are prioritizing whatever makes them money or creates the trinkets their selling faster or whatever it is, right? The interesting thing, though, is if you think about who our customers would be, like, who the people in this dataset are, they are all companies who are probably more or less born in the cloud or at least have some arm that is born in the cloud, and they are building software, right? So, they're not really just your average enterprises you might see in a Gartner client base which is more broad; they are software companies.And for software companies, delivering software faster is the most important thing, right, and then delivering secure software faster, should be the most important thing, but it's kind of like the other thing that we talk about and don't do. And that's actually what we found. We found that people do deliver software faster because of containers and cloud, but they don't necessarily deliver secure software faster because as is one of our data points, 75% of containers that run in production have critical or high vulnerabilities that have a patch available. So, they could have been fixed but they weren't fixed. And people ask why, right? And why, well because it's hard; because it takes time; because something else took priority; because I've accepted the risk. You know, lots of reasons why.Corey: One of the big challenges, I think, is that I can walk up and down the expo hall at the RSA Conference, which until somewhat recently, you were not allowed to present that or exhibit at unless you had the word ‘firewall' in your talk title, or wound up having certain amounts of FUD splattered across your banners at the show floor. It feels like there are 12 products—give or take—for sale there, but there are hundreds of booths because those products have different names, different messaging, and the rest, but it all feels like it distills down to basically the same general categories. And I can buy all of those things. And it costs an enormous pile of money, and at the end of it, it doesn't actually move the needle on what my business is doing. At least not in a positive direction, you know? We just set a giant pile of money on fire to make sure that we're secure.Well, great. Security is never an absolute, and on top of that, there's always the question of what are we trying to achieve as a business. As a goal—from a strategic perspective—security often looks a lot like, “Please let's not have a data breach that we have to report to people.” And ideally, if we have a lapse, we find out about it through a vector that is other than the front page of The New York Times. That feels like it's a challenging thing to get prioritized in a lot of these companies. And you have found in your report that there are significant challenges, of course, but also that some companies in some workloads are in fact getting it right.Anna: Right, exactly. So, I'm very much in line with your thinking about this RSA shopping spree, and the reality of that situation is that even if we were to assume that all of the products you bought at the RSA shopping center were the best of breed, the most amazing, fantastic, perfect in every way, you would still have to somehow build a program on top of them. You have to have a process, you have to have people who are bought into that process, who are skilled enough to execute on that process, and who are more or less in agreement with the people next door to them who are stuck using one of the 12 trinkets you bought, but not the one that you're using. So, I think that struggle persists into the cloud and may actually be worse in the cloud because now, not only are we having to create a processor on all these tools so that we can actually do something useful with them, but the platform in which we're operating is fundamentally different than what a lot of us learned on, right?So, the priorities in cloud are different; the way that infrastructure is built is a little different, like, you have to program a YAML file to make yourself an instance, and that's kind of not how we are used to doing it necessarily, right? So, there are lots of challenges in terms of skills gap, and then there's just this eternal challenge of, like, how do we put the right steps into place so that everybody who's involved doesn't have to suffer, right, and that the thing that comes out at the end is not garbage. So, our approach to it is to try to give people all the pieces they need within a certain scope, so again, we're talking about people developing software in a cloud-native world, we're focused kind of on containers and cloud workloads even though it's not necessarily containers. So that's, like, our sandbox, right? But whoever you are, right, the idea is that you need to look to the left—because we say ‘shift left'—but then you kind of have to follow that thread all the way to the right.And I actually think that the thing that people most often neglect is the thing on the right, right? They maybe check for compliance, you know, they check configurations, they check for vulnerabilities, they check, blah, blah, blah, all this checking and testing. They release their beautiful baby into the world, and they're like, okay, I wash my hands of it. It's fine. [laugh]. Right but—Corey: It has successfully been hurled over the fence. It is the best kind of problem, now: Someone else's.Anna: It's gone. Yeah. But it's someone else's—the attacker community, right, who are now, like, “Oh, delicious. A new target.” And like, that's the point at which the fun starts for a lot of those folks who are on the offensive side. So, if you don't have any way to manage that thing's security as it's running, you're kind of like missing the most important piece, right? [laugh].Corey: One of the challenges that I tend to see with a lot of programmatic analysis of this is that it doesn't necessarily take into account any of the context because it can't. If I have, for example, a containerized workload that's entire job is to take an image from S3, run some analysis or transformation on it then output the results of that to some data store, and that's all it's allowed to talk to you, it can't ever talk to the internet, having a system that starts shrieking about, “Ah, there's a vulnerability in one of the libraries that was used to build that container; fix it, fix it, fix it,” doesn't feel like it's necessarily something that adds significant value to what I do. I mean, I see this all the time with very purpose-built Lambda Functions that I have doing one thing and one thing only. “Ah, but one of the dependencies in the JSON processing library could turn into something horrifying.” “Yeah, except the only JSON it's dealing with is what DynamoDB returns. The only thing in there is what I've put in there.”That is not a realistic vector of things for me to defend against. The challenge then becomes when everything is screaming that it's an emergency when you know, due to context, that it's not, people just start ignoring everything, including the, “Oh, and by the way, the building is on fire,” as one of—like, on page five, that's just a small addendum there. How do you view that?Anna: The noise insecurity problem, I think, is ancient and forever. So, it was always bad, right, but in cloud—at least some containers—you would think it should be less bad, right, because if we actually followed these sort of cloud-native philosophy, of creating very purp—actually it's called the Unix philosophy from, like, I don't know, before I was born—creating things that are fairly purposeful, like, they do one thing—like you're saying—and then they disappear, then it's much easier to know what they're able to do, right, because they're only able to do what we've told them, they're able to do. So, if this thing is enabled to make one kind of network connection, like, I'm not really concerned about all the other network connections it could be making because it can't, right? So, that should make it easier for us to understand what the attack surface actually is. Unfortunately, it's fairly difficult to codify and productize the discovery of that, and the enrichment of the vulnerability information or the configuration information with that.That is something we are definitely focusing on as a vendor. There are other folks in the industry that are also working on this kind of thing. But you're exactly right, the prioritization of not just a vulnerability, but a vulnerability is a good example. Like, it's a vulnerability, right? Maybe it's a critical or maybe it's not.First of all, is it exposed to the outside world somehow? Like, can we actually talk to this system? Is it mitigated, right? Maybe there's some other controls in place that is mitigating that vulnerability. So, if you look at all this context, at the end of the day, the question isn't really, like, how many of these things can I ignore? The question is at the very least, which are the most important things that I actually can't ignore? So, like you're saying, like, the buildings on fire, I need to know, and if it's just, like, a smoldering situation, maybe that's not so bad. But I really need to know about the fire.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: It always becomes a challenge of prioritization, and that has been one of those things that I think, on some level, might almost cut against a tool that works at the level that Sysdig does. I mean, something that you found in your report, but I feel like, on some level, is one of those broadly known, or at least unconsciously understood things is, you can look into a lot of these tools that give incredibly insightful depth and explore all kinds of neat, far-future, bleeding edge, absolute front of the world, deep-dive security posture defenses, but then you have a bunch of open S3 buckets that have all of your company's database backups living in them. It feels like there's a lot of walk before you can run. And then that, on some level, leads to the wow, we can't even secure our S3 buckets; what's the point of doing anything beyond that? It's easy to, on some level, almost despair, want to give up, for some folks that I've spoken to. Do you find that is a common thing or am I just talking to people who are just sad all the time?Anna: I think a lot of security people are sad all the time. So, the despair is real, but I do think that we all end up in the same solution, right? The solution is defense in depth, the solution is layer control, so the reality is if you don't bother with the basic security hygiene of keeping your buckets closed, and like not giving admin access to every random person and thing, right? If you don't bother with those things, then, like, you're right, you could have all the tools in the world and you could have the most advanced tools in the world, and you're just kind of wasting your time and money.But the flipside of that is, people will always make mistakes, right? So, even if you are, quote-unquote, “Doing everything right,” we're all human, and things happen, and somebody will leave a bucket open on accident, or somebody will misconfigure some server somewhere, allowing it to make a connection it shouldn't, right? And so if you actually have built out a full pipeline that covers you from end-to-end, both pre-deployment, and at runtime, and for vulnerabilities, and misconfigurations, and for all of these things, then you kind of have checks along the way so that this problem doesn't make it too far. And if it does make it too far and somebody actually does try to exploit you, you will at least see that attack before they've ruined everything completely.Corey: One thing I think Sysdig gets very right that I wish this was not worthy of commenting on, but of course, we live in the worst timeline, so of course it is, is that when I pull up the website, it does not market itself through the whole fear, uncertainty, and doubt nonsense. It doesn't have the scary pictures of, “Do you know what's happening in your environment right now?” Or the terrifying statistics that show that we're all about to die and whatnot. Instead, it talks about the value that it offers its customers. For example, I believe its opening story is, “Run with confidence.” Like, great, you actually have some reassurance that it is not as bad as it could be. That is, on the one hand, a very uplifting message and two, super rare. Why is it that so much of the security industry resorts to just some of the absolute worst storytelling tactics in order to drive sales?Anna: That is a huge compliment, Corey, and thank you. We try very hard to be kind of cool in our marketing.Corey: It shows. I'm tired of the 1990s era story of, “Do you know where the hackers are?” And of course, someone's wearing, like, a ski mask and typing with gloves on—which is always how I break into things; I don't know about you—but all right, we have the scary clip art of the hacker person, and it just doesn't go anywhere positive.Anna: Yeah. I mean, I think there certainly was a trend for a while have this FUD approach. And it's still prevalent in the industry, in some circles more than others. But at the end of the day, Cloud is hard and security is hard, and we don't really want to add to the suffering; we would like to add to the solution, right? So, I don't think people don't know that security is hard and that hackers are out there.And you know, there's, like, ransomware on the news every single day. It's not exactly difficult to tell that there's a challenge there, so for us to have to go and, like, exacerbate this fear is almost condescending, I feel, which is kind of why we don't. Like, we know people have problems, and they know that they need to solve them. I think the challenge really is just making sure that A) can folks know where to start and how to build a sane roadmap for themselves? Because there are many, many, many things to work on, right?We were talking about context before, right? Like, so we actually try to gather this context and help people. You made a comment about how having a lot of telemetry might actually be a little bit counterproductive because, like, there's too much data, what do I do well—Corey: Here's the 8000 findings we found that you fail—great. Yeah. Congratulations, you're effectively the Nessus report as a company. Great. Here you go.Anna: Everything is over.Corey: Yeah.Anna: Well, no shit, Nessus, you know. Nessus did its thing. All right. [laugh].Corey: Oh, Nessus was fantastic. Nessus was—for those who are unaware, Nessus was an open-source scanner made by the folks at Tenable, and what was great about it was that you could run it against an environment, it would spit out all the things that it found. Now, one of the challenges, of course, is that you could white-label this and slap whatever logo you wanted on the top, and there were a lot of ‘security consultancies' that use the term incredibly… lightly, that would just run a Nessus report, drop off the thick print out. “Here's the 800 things you need to fix. Pay me.” And wander on off into the sunset.And when you have 800 things you need to fix, you fix none of them. And they would just sit there and atrophy on the shelf. Not to say that all those things weren't valid findings, but you know, the whole, you're using an esoteric, slightly deprecated TLS algorithm on one of your back-end services, versus your Elasticsearch database does not have a password set. Like, there are different levels of concern here. And that is the problem.Anna: Yeah. That is in fact one of the problems we're aggressively trying to solve, right? So, because we see so much of the data, we're actually able to piece together a lot of context to gives you a sense of risk, right? So, instead of showing all the data to the customer—the customer can see it if they want; like, it's all in there, you can look at it—one of the things we're really trying to do is collect enough information about the finding or the event or the vulnerability or whatever, so we can kind of tell you what to do.For example, one you can do this is super basic, but if you're looking at a specific vulnerability, like, let's say it's like Log4j or whatever, you type it in, and you can see all your systems affected by this thing, right? Then you can, in the same tool, like, click to the other tab, and you can see events associated with this vulnerability. So, if you can see the systems that the vulnerability is on and you can see there's weird activity on those systems, right? So, if you're trying to triage some weird thing in your environment, during the Log4j disaster, it's very easy for you to be like, “Huh. Okay, these are the relevant systems. This is the vulnerability. Like, here's all that I know about this stuff.”So, we kind of try to simplify as much as possible—my design team uses the word ‘easify,' which I love; it's a great word—to easify, the experience of the end-user so that they can get to whatever it is they're trying to do today. Like, what can I do today to make my company more secure as quickly as possible? So, that is sort of our goal. And all this huge wealth of information we gather, we try to package for the users in a way that is, in fact, digestible. And not just like, “Here's a deluge of suffering,” like, “Look.” [laugh]. You know?Corey: This is definitely complicated in the environment I tend to operate in which is almost purely AWS. How much more complex is get when people start looking into the multi-cloud story, or hybrid environments where they have data center is talking to things within AWS? Because then it's not just the expanded footprint, but the entire security model works slightly differently in all of those different environments as well, and it feels like that is not a terrific strategy.Anna: Yeah, this is tough. My feelings on multi-cloud are mostly negative, actually.Corey: Oh, thank goodness. It's not just me.Anna: I was going to say that, like, multi-cloud is not a strategy; it's just something that happens to you.Corey: Same with hybrid. No one plans to do hybrid. They start doing a cloud migration, realize halfway through some things are really hard to move, give up, plant the flag, declare victory, and now it's called hybrid.Anna: Basically. But my position—and again, as an analyst, you kind of, I think, end up in this position, you just have a lot of sympathy for the poor people who are just trying to get these stupid systems to run. And so I kind of understand that, like, nothing's ideal, and we're just going to have to work with it. So multi-cloud, I think is one of those things where it's not really ideal, we just have to work with it. There's certainly advantages to it, like, there's presumably some level of mythical redundancy or whatever. I don't know.But the reality is that if you're trying to secure a pile of junk in Azure and a pile of junk in AWS, like, it'd be nice if you had, like, one tool that told you what to do with both piles of junk, and sometimes we do do that. And in fact, it's very difficult to do that if you're not a third-party tool because if you're AWS, you don't have much incentive to, like, tell people how to secure Azure, right? So, any tool in the category of, like, third-party CSPM—Gartner calls them CWPP—kind of, cloud security is attempting to span those clouds because they always have to be relevant, otherwise, like, what's the point, right?Corey: Well, I would argue cynically there's also the VC model, where, “Oh, great. If we cover multiple cloud providers, that doubles or triples our potential addressable market.” And, okay, great, I don't have those constraints, which is why I tend to focus on one cloud provider where I tend to see the problems I know how to solve as opposed to trying to conquer the world. I guess I have my bias on that one.Anna: Fair. But there's—I think the barrier to entry is lower as a security vendor, right? Especially if you're doing things like CSPMs. Take an example. So, if you're looking at compliance requirements, right, if your team understands, like, what it means to be compliant with PCI, you know, like, [line three 00:28:14] or whatever, you can apply that to Azure and Amazon fairly trivially, and be like, “Okay, well, here's how I check in Azure, and here's how I check in Amazon,” right?So, it's not very difficult to, I think, engineer that once you understand the basic premise of what you're trying to accomplish. It does become complicated as you're trying to deal with more and more different cloud services. Again, if you're kind of trying to be a cloud security company, you almost have no choice. Like, you have to either say, “I'm only doing this for AWS,” which is kind of a weird thing to do because they're kind of doing their own half-baked thing already, or I have to do this for everybody. And so most default to doing it for everybody.Whether they do it equally well, for everybody, I don't know. From our perspective, like, there's clearly a roadmap, so we have done one of them first and then one of them second and one of the third, and so I guarantee you that we're better in some than others. So, I think you're going to have pluses and minuses no matter what you do, but ultimately what you're looking for is coverage of the tool's capabilities, and whether or not you have a program that is going to leverage that tool, right? And then you can check the boxes of like, “Okay. Does it do the AWS thing? Does it do this other AWS thing? Does it do this Azure thing?”Corey: I really appreciate your taking the time out of your day to speak with me. We're going to throw a link to the report itself in the [show notes 00:29:23], but other than that, if people want to learn more about how you view these things, where's the best place to find you?Anna: I am—rarely—but on Twitter at @aabelak. I am also on LinkedIn like everybody else, and in the worst case, you could find me by email, at anna.belak@sysdig.com.Corey: And we will of course put links to that in the [show notes 00:29:44]. Thank you so much for taking the time to speak with me today. I appreciate it.Anna: Thanks for having me, Corey. It's been fun.Corey: Anna Belak, Director of Thought Leadership at Sysdig. I'm Cloud Economist Corey Quinn and this is streaming on the cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment telling me not only why this entire approach to security is awful and doomed to fail, but also what booth number I can find you at this year's RSA Conference.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Quantum Leaps in Bioinformatics with Lynn Langit

Screaming in the Cloud

Play Episode Listen Later Feb 24, 2022 36:22


About LynnCloud Architect who codes, Angel InvestorLinks: Lynn Langit Consulting: https://lynnlangit.com/ Groove Capital: https://www.groovecap.com/groove-capital-minnesotas-first-check-fund Twitter: https://twitter.com/lynnlangit GitHub: https://github.com/lynnlangit TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance query accelerator for the Oracle MySQL Database Service, although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLAP and OLTP—don't ask me to pronounce those acronyms again—workloads directly from your MySQL database and eliminate the time-consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. So, I've been doing this podcast for a little while now—by my understanding, this is episode 300 and something—but back when the very first episode aired, I had pre-recorded the first twelve episodes. Episode number ten was with Lynn Langit who is, among many other things, the CEO of Lynn Langit Consulting, she is also the first person to achieve the AWS Community Hero and equivalent designations at all three of the primary tier-one hyperscale cloud providers, which I can't even wrap my head around what it takes to get that at one of those companies. Lynn, thank you so much for agreeing to come back now that I'm no longer scared of the microphone.Lynn: Well, thank you for having me. It's great to be back, Corey.Corey: So, it's been a few years now since we really sat down and caught up. And what an interesting few years it's been. There's been a whole minor global pandemic thing that wound up hitting us from unexpected and unpleasant places. There's been a significant, I would say, not revolution but evolution in how adoption of cloud services has been proceeding. The types of problems that customers are encountering, the conversational discourse has moved significantly away from, “Should we be using cloud?” Into, “Okay, we obviously should be using Cloud. How should we be using it?” And the industry keeps on churning. Sure there's still rough parts, there are still ridiculous aspects of it, but what have you been up to?Lynn: Well, as you might remember, I have an independent consultancy where I do really what my customers need. I work across different clouds, which keeps it interesting and fun, but I've had a focus over the past few years in supporting bioinformatics research. Before the pandemic, it was mostly cancer research. Since the pandemic, it's been all Covid, all the time.Corey: All Covid, all the time sort of has been the unofficial theme of this. And it's weird. I know, we're in 2022, now, but it still feels like on some level, it's like, “Man, this is March 2020; it's still dragging on, on some level.” There have been a number of stories in the world that is, let's say medicine-adjacent, more so than—we're all sort of medicine adjacent these days, but there's been a lot of refocusing away from things like cancer research into Covid and similar pandemic respiratory diseases. Do you think that there's a longer-term story where we're going to start seeing progress stall on things that were previously areas of focus—in your case cancer—in favor of reducing infectious disease, or is it really one of those ‘rising tide lifts all boats' type of scenarios?Lynn: Yeah, it's the latter. It's been really interesting. Without getting too much into the details, you know, you think of genomic research for drug discovery, you know, we started with this idea of different DNA sequencing cohorts. So, like people from the—you know, that started from the United States, people that started from Africa, you know, different cohort as a normative to evaluate the effectiveness of diseases, what was an area of research already was to go down to the level of what's called single-cell RNA. So, look at the expression of the genomics by cell area, so by the different parts of your body.Well, this is similar to what has been done to understand the impact and the efficacy of potential Covid drugs. So, this whole single-cell RNA mapping cohorts of what is normal for different types of populations has resulted in this data explosion that I've never seen before. And I see it as positive for the impact of human health. However, it really drives the need for adoption to the cloud. These research facilities are running out of space if they're still working on-prem.Corey: I spend an awful lot of time thinking about data and its storage from a primarily cost-focused perspective, for obvious reasons, and that is nuanced and intricate and requires, sort of, an end-to-end lifecycle policy. There's this idea of, ideally, you would delete old data you don't need anymore, but failing that you, maybe aspirationally, don't need 500 copies of the same thing lying around. Maybe there are ways to fix that. And that's all within one cloud ecosystem. You work across all of the clouds. How do you keep it all straight in your head trying to figure out things around lifecycles, things around just understanding the capabilities of the various platforms? Because I got to say, from my perspective, it's challenging enough only bounding it to one.Lynn: Yeah, it's the constant problem. The big clients I had over this past year were not on Amazon, they were on other platforms. So, it seems like it sort of goes in cycles. And what I'll sometimes need to do is hire subcontractors that have been working on those platforms because you can't, I mean, you can't even know one platform, much less all of them to the level of complexity in order to implement. One thing that is kind of interesting though, in bioinformatics is—and different than the other domains—is when you talk about data, it's a function of time first and cost second.So, they will run on less computational resources, so that they can, for example, not overspend their research grant, and wait longer for the results. And this has been really an interesting shift in my work because I used to work with FinTech and ad tech, where it's all about, get it out there fast. And we don't really care how much it costs, we just want it super fast. So, this continuum of time or money shifts by vertical. And that's been something that—I don't know, it's kind of obvious, in hindsight, but I didn't really expect until I got into the different domains.Corey: It's always been fascinating to me watching how different organizations and different organization types wind up have interacting with cost. I mean, I've been saying for a while now that cost and architecture are the same thing when it comes to cloud. What are your trade-offs? What are your constraints? In many venture-backed companies, it's when you have a giant pile of other people's money raring to go, and it's a spend it and hit your milestone if you want to get another round of funding, or this has been an incredible journey Medium post in the making, then, yeah, okay, go ahead and make the result happen faster. Save money is not the first, second or third order of business as far as what you're trying to achieve.In academia, where everything's grant powered. And it's a question of, we need to be able to deliver, and we need to be able to show results and be able to go and play the game and understand the cultural context we're operating in, and ideally get another grant next year, it completely shifts the balance of what needs to be prioritized and when. And I don't think there's been a lot of discussion around that because most cloud cost discussions inherently center around industry.Lynn: They do and they focus on the industries where they're willing to spend most. So, most of the reference examples are, they always prioritize for time and money is sort of unlimited. I'll give you an example—this was from a few years back—some work I did with a research group in Australia, and again, it was a genomics example. They were running on-prem, and to do a single query, it took them 500 hours. And I was just like, “Are you kidding me?”And they're like, “Hey, cloud lady, what can you do?” Right? So, we gave two solutions, and the first solution was kind of a more of a lift-and-shift kind of a solution because they didn't know anything about cloud. And it took a few hours. The second solution was what was in our opinion, super elegant, it was one of the earliest data lakes, it took minutes.Well, it was a big hit to the ego that they adopted… the easier solution. But again, it's a learning because another dimension about cloud architecture is usability. The FinTechs are like, “We're going to get it really done fast; we'll hire who we need to hire.” The biotechs, they can't afford to hire who they need to hire because there all being hired by the FinTechs. So, you have these different dimensions you need to optimize for that aren't really obvious if you just work in the industries that optimize for time.Corey: And the thing that always gets overlooked is that in most environments, the people working on things are more expensive than the infrastructure themselves. And back when Lambda and all the serverless joy came out, my first iteration of lastweekinaws.com website was powered entirely by Lambda functions, S3, and other assorted bits of nonsense. Today, it's on WordPress.And it's not because I think that is somehow the superior architecture from a purely technologist point of view, but because I have to find other people who aren't me or one of the other six people in the world at the time who could stuff all that into their head and work on it effectively, should be able to make changes to the website. That is not something I need to be focusing on. There's something to be said for going to where there's a significant talent pool, rather than pushing the frontiers of innovation in areas that don't directly benefit whatever it is your organization is targeting.Lynn: Yeah, it's really interesting, when Covid hit back in 2020—kind of an interesting little story here—one of my clients is the Broad Institute at MIT and Harvard—they're a well-known research organization for, you know, cancer genomic datasets—they were tasked with pivoting their labs so that they could provide Covid testing capability. And I was a long-term contractor with them, so they brought me in for an architectural cloud consultant. I said, “This clearly is a serverless. I know you guys haven't done this before, but this is going to be burstable, you don't know how big this is going to need to go.” And then just to make life interesting, in the middle of the build of that, I was one of the first people in Minnesota to get Covid, so I actually wasn't able to go and complete it, nor was I able to get a test because there weren't tests.I mean, you know, I can't make this stuff up. I was in the ER saying, “Okay, is this the end of me, or can I go back and get you some tests?” [laugh]. So, it's really kind of two things—kind of a weird story. And also, life situations will cause change, and so the Broad did launch that pipeline, and it was serving up to 10% of the Covid tests in the United States.But they had never done anything serverlessly or had considered it before because they didn't need to have that amount of change. It was really, again, a big thing when I came into human health. Prior to that, I was doing all serverless all the time. You know, I came into human health, and they were saying, “Okay, we're going to have massive VMs.” And I was like, “No…” but you know, you have to meet the client where they are.Corey: I think it's the easiest thing in the world, particularly as a junior consultant—because you do not see senior consultants doing this ever, you know, after the first time—to walk into an environment, look around and have zero context into what's going on—because you're a consultant; you haven't been there and say, “This is ridiculous. What fool built this?” Invariably, to said fool. Now, most people don't show up in the morning hoping to do a terrible job at work today, so there are constraints that you are certainly not seeing. And maybe it was an offering wasn't available that maybe they weren't aware of it. Maybe there was a constraint that you're not seeing.But the best case is you're right and you just made them feel terrible, which is not generally a great way to land more consulting projects. It's always frustrating to me because even looking at a bill and having a pretty good idea of what's going on, I always frame it as, “Can you help me understand why this is the case? Had you considered this, or is that not an option?” As opposed to categorically saying, well, this is not the way to do it. Because once you're wrong when you're delivering expertise, it takes a lot to build that back, if it's even possible.Lynn: Well, again, from human health because, you know, they were consuming the vendor information, they thought they wanted to learn how to use Kubernetes, but what they really needed to learn was how to do archiving to reduce their storage costs.Corey: Yes. Kubernetes is a terrific solution for a bunch of problems and create several orders of magnitude more somewhere along the way. My somewhat accurate, somewhat snarky observation is that Kubernetes is great if your primary problem is you want to pretend you work at Google but didn't pass their technical screen. I don't really want to cosplay as a cloud provider myself, most days. That said, there are use cases for which it makes sense, but context is everything, and generally speaking, I don't tend to follow a hype trend to figure out whether or not it's going to solve my particular problem.Lynn: Well, here's the soundbite: “Kubernetes is today's Hadoop.”Corey: Oh, there are people who are not going to like that. I made a tweet, I think—Lynn: Tough.Corey: —three years ago now—Lynn: It's true. [laugh].Corey: Oh, yeah. Tweet three years ago or so that said, “Hot take: In five years, nobody's going to care about Kubernetes.” And I think I have a year or two left on that prediction. And what I said at the time was that not that it's going to go away and not be anywhere—because enterprises do not move that quickly—but it's no longer going to be the sort of thing that everyone is concerned about at a very high level. The Linux kernel has a bunch of aspects to it that we used to have to care about a fair bit. Now, a few people really, really need to care about those things; because of those folks' hard work, the rest of us don't have to think about it at all. And that is the nature of technology, in the fullness of time.Lynn: Well, another way to think about it is Kubernetes is a C++. Certain people are going to be experts in it and need to, and that's valid, right, but what percentage of developers code in C++. Like, ten? Five? You know, it's kind of analogous, right?So, it's one of the signatures of my consultancy. You know, I'm this pragmatic midwesterner, and I love to say, “Look,”—like you said—“If you think you need this, you really need to understand the actual cost of it because it's non-trivial on all clouds.” And I get to say that because I'm independent. You know, they're doing solid work to abstract it into a higher-level implementation, but when I hear a customer say, “I need Kubernetes,” the burden of proof is on them [laugh] before I'm going to build that.Corey: Speaking of hype-driven emerging technologies, you are arguably one of the few people on the planet I can have this conversation with, and I do not mean that as an insult other people operating in this space. For context, a couple of years ago, AWS launched Brakets—which they spelled Braket without a C because it's Amazon and spelling is hard, presumably; I know, I know, there's a reason behind it—and it is their service that enables you to get access to quantum computers the same way we get access to any other AWS service: Through a somewhat janky console and some APIs. And, okay, quantum computing. We've heard a lot about it forever; it always seemed a bit like science fiction and it was never really clearly articulated what kind of value it can solve for us.So, “Aha, now it's here. I don't need to go and build or buy a quantum computer somewhere else.” And I tried using the Quickstart, and it turns out that the Hello World tutorial for quantum computing—at least to my mind—is basically an application for a PhD program at Berkeley. And I am not that type of academic for better or worse, so I kept smacking my head off of that and realizing, okay, whatever this is, is clearly not for me. You have been doing some deep dives in the quantum computing space, but as we've just mentioned, your day job is not, to my understanding, a college professor. You are a consultant, you run your own consultancy, solving data problems, particularly towards bioinformatics. What is the deal—to the layperson—of quantum computing these days?Lynn: Well, yeah, like you, I was introduced years ago and tried to read the books, and I didn't have the math and just, you know, saw it as a curiosity. Last year, I picked up a book from O'Reilly called Practical Quantum Computing, which of course, because the name was attractive to me. I read it, felt like I was getting a little bit more knowledge, implemented a learning JavaScript library with a browser-based editor—so zero-install—and it was a simulator, you couldn't run it on actual QPUs. So, I decided to see if there's any other interest in my tech community, and I got about five other developers and we ran a 15-week long book club because we all just wanted to move forward with our knowledge. Because there is this fundamental difference in the information you can get from a qubit versus a bit because a qubit can basically be, like, a globe, and so it has a superposition, and so you can have all the different mathematical points on the globe, versus a bit is on or off.I mean, that's intuitive, like, “Hey, I could get more information out of that.” So, the potential usages—it's always been tech that leads the way—is on figuring out of what are called NP-hard or computationally complex problems, and, again, this is at the edge of my knowledge, but this is where bioinformatics is. I think of it in an oversimplified way, as [N by N by N by N, all by all by all 00:16:49]. We want to see all possible combinations of all possible inputs. So, for example, we can figure out which Covid drug we should try—which set of drugs we should try—and we want that as fast as possible.So, I wanted to see, okay, you know, where's this at? Plus, like you said, Amazon introduced Braket; when Amazon introduces something, then there's some customers somewhere that are using it. I mean, that's—you know, kind of pay attention to it now. So, as I was doing this book club, I investigated all the different cloud vendors and captured all that learning in a GitHub, and just recently recorded a LinkedIn Learning course. Which again, in the learning ladder is, if this is, you know, Hello World and this is actual implementation, it's like right here.But right here doesn't exist. Like, there's nothing there, so I tried to make something to say, okay, the Amazon Braket example, how does that actually work? What is a Hadamard Gate? Why do you care? What is amplification? How do you measure it? Like, what would you do with that? And so, you know, I tried to interpret some academic papers and do that learning layer in the middle to help move people towards productivity. Am I fully there? No. Did I move further? I hope so. Do you want to come along with me? Great.Corey: You've done something, though, that I don't think anyone else yet has when I had conversations with them about quantum computing, which is we all are shaped by our own needs and our own experiences when we interact with a cloud provider. To me, I, perhaps foolishly, took Amazon seriously when they called it Amazon Web Services. “Oh, okay. Clearly, this is going to be things to help me build websites and website accessories, more or less.” So, it's always odd to me when I'll see something like oh, and here's our IoT solution that winds up powering a fleet of 10,000 robots, and I'm looking around my website going, “I don't really have a problem that could be solved by the 10,000 robots. I have a bunch that could be made a lot worse.”But it feels like it's this orthogonal thing that is removed. But some areas, it's okay. I can see the points of commonality and how you get there from here, and if I think really hard, I can do that with IoT stuff. For example, iRobot is a cloud-connected robot that talks to something that looks like a website and vacuums my house. Whereas with quantum computing, it always felt very isolated, very much an island as far as being connected to anything else that I can recognize. Bioinformatics research, as you describe it, well, yeah, I can see you get the bioinformatics research from web services. And now I can see how you can get to quantum computing through the bioinformatics side of things.Lynn: Well, the other thing that really was useful for me, I am doing TensorFlow, finally. Took me a few years, but for neural networks. And so I am using, with some of my bioinformatics clients, acceleration with GPUs and TPUs, if I happen to be on Google because it's a known thing that when you're training a neural network, again, similar you have complexity, so you have a specialized chip, where you can offload some of the linear algebra onto that chip. So, you split the classic and the tensor portion, if you will, and you do computation on both sides. And so it's not a huge leap to say, “Well, I'm not going to use a GPU, I'm going to use a QPU,” because you split. And that's the way it actually works.There's actually a really interesting paper I put in my GitHub. It is a QCNN, and it is—that's a Quantum Convolutional Neural Network that is used to analyze images of breast cancer. Because again, on the image, you can think of the pixels as what's called a tensor, which is just vectors in multiple dimensions, you need the [all by all by all 00:20:17] again; that's really how it goes in my head. You know, you have the globe of the qubit and you want to get the all possible combinations faster, so that you can analyze all combinations in the, in this case, the image. And they found, not only was it faster, it was more accurate. And that's why I am interested in this.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: The neat part is that this might be one of the first clear-cut stories where, “What could I use a quantum computer for?” And the answer isn't something that's forward-looking or theoretical. I mean, the obvious gag when you said reading about Practical Quantum Computing is that book is probably in pre-release, I would assume.Lynn: [laugh].Corey: But it's a hard thing to solve for, and I do have the awareness that I am not an academic, academia has never been my friend, so I bias heavily for, “Well, can we use this to solve real-world problems slash make money?”—because industry—and academia focuses, ideally and aspirationally on the expansion of the limits of human knowledge. And sometimes it's okay to do those things without an immediate, “Well, how can I turn a profit on it next quarter?” What a dismal, bleak society we have if that's all that we wind up focusing on any given point in time.Lynn: Yeah, that's for sure.Corey: Which, of course, sets us up for one other thing that's a relatively recent change for you. You now have mentioned in your bio, which I believe is new since the last time we spoke, that you are an angel investor. And that is something that I recently found being applied to me as well after I made an investment in a startup that I was very excited about. I talked about in the show previously; it's called Byte Check. But honestly, I didn't realize that what I was doing was called angel investing until I read the press release because ‘strategic angel' are two words that no one ever applies to me, particularly in that order. What happened? What are you doing these days?Lynn: Well, I live in Minneapolis. So—and I moved there in 2019, so you know, my 2020 story is first I had Covid, got over that, and then I was there during the tragedy of George Floyd. So, I wanted to understand more about what were the root causes, and what I could do to make an impact in the recovery of my city. And I was really surprised to find that Minnesota is one of the most charitable states in the United States, it ranks one or two, but yet we have in the Twin Cities of Minneapolis and St. Paul, we have really unacceptable income inequality and poverty. So, something's not working.I'm a pretty charitable person; I always allocate a certain percentage of my money to charity, but I said, “I want to accelerate this.” So, at the same time, there was a new angel investment fund launched, it's called Groove Capital, that was going to focus on women-owned and BIPOC businesses. And I thought, “Hmm, this seems good.”Now, I was super intimidated because I lived in California for so many years, and check sizes in California, you just add a zero. And I thought, you know, “I don't have generational wealth. This is my own money.” You know, I'm well-compensated, but I'm not loaded.Corey: Yeah there's a common trope right now that oh, angel investor is a polite way of saying I am rich—Lynn: Right.Corey: —but I rent my home at this point, living in San Francisco. It is, I am not exactly sitting here diving into a money bin out back, Scrooge McDuck-style either.Lynn: Right. Well, I mean, you know, I'll just be transparent about it. Like everybody else, or many people, I moved out of California because of the cost of doing business there and reduced my cost of living by 40% move into the Midwest, which is awesome. So anyway, I joined this fund, and it's been just fantastic because I've listened to deals on my own and felt just like a complete, like, I don't know what I'm doing. But I'm taking advantage—Corey: How do you evaluate an idea that someone has that's early-stage, barely better in some cases than back-of-an-envelope scrawlings?Lynn: For sure, right. But what I found through the fund is I can contribute both money and time because, you know, I did this cloud expertise, and in addition to writing checks for a couple companies that I really believe in, for example, I got all these companies on the X cloud company for startups program. Because that wasn't just a known thing in my ecosystem. I was like, “Why are you paying a cloud bill? You could be on the startup program for the first year.”So, I'm impacting these new businesses with both my experience and my dollars, and I just really love it. I just really, really love it. And you know, the reasons I want to talk about it is because more people who have expertise in tech should do this because you can really, really be impactful. One of the companies that I invested in is called TurnSignl. They are coming to Los Angeles.It was three attorneys and one of their brothers is a police officer. They wanted to de-escalate situations that happen with traffic stops. So, it's a mobile app, where you push a button and you're connected to an attorney. And they do training for the community and police officers, and the idea to record the conversation and to get an attorney involved to de-escalate and get everybody home safely. And that was my first investment and I'm—it's going national, and I'm like, really, really—the kind of things I want to do you know.Corey: It is simultaneously such a terrific idea and such a stunning indictment of the society that makes something like that necessary.Lynn: Well, you know, we have to find practical solutions. We have to find ways forward.Corey: Oh, please. Don't interpret anything I'm saying a shade on that. It's like, “Well, I wish the world were differently.” Yeah, I think most people do. But you have to deal for better or worse with the hand that you're dealt, and this is, for better or worse, at the time of recording this, the society that we have, and finding the best path forward is often not easy.But it beats just sitting here complaining about everything every day, and not doing anything to be part of that change. The surprising thing I learned as I went through it was that in many cases, the value of individual angel investors is not the check that they're writing, that's basically just almost a formality, on some level. It is the expertise, it is the insight into particular markets, and the rest. The part of what you're saying that surprises me that I hadn't really considered, but of course, it must exist, is the idea of angel funds. Is this generally run by an existing VC firm? Is it a group of like-minded friends who decide, ah, we're going to just basically do the investing equivalent of a giving circle where everyone puts some money in the pot and then that decides where to go? How is it structured?Lynn: Yeah, the way ours worked is you do pay a fee—it's a small fee—to be part of it, and then they have people who vet deals for you. And then what I really like about it is the community aspect because just like in tech, when you're learning something new in tech, you have community, same thing here. We have a Slack, we have a website for each deal, we have in-person meetups when Covid situation allows, and we have chosen to start by investing in Minnesota, although we're going to, in fund two we're going to invest in Upper Midwest. And for example, here's something I would have never known. There's an angel tax credit Minnesota, that for certain businesses, you can get a 25% tax credit. Which hey, do good, be good, get good. I would have never known about that, I would have never known how to do it. All my investments so far have qualified. Fantastic. My money goes further.Corey: Yeah, it's about well, what are you talking about worrying about taxes? That there's about to be doing something good? Yeah, great. If you believe in a cause, take advantage of the tax code as written—I am not advocating tax fraud; pay every cent that you owe, let's be serious here. They have no sense of humor about that—Lynn: [laugh].Corey: —and take advantage of that. That means you have additional money to do good with. I wish that more people had an awareness around that particular school of thought.Lynn: Well, make your money go further, make your money effective.Corey: Oh yes.Lynn: Because like it or not, we run on money. We run on money. And so be smart, from everything where you shop to how you spend. That's how we're going to make change.Corey: One last area I want to explore with you is that for a long time you've been working on, effectively, data pipelines and similar things in that space, tied to your consulting work. You are clearly skilled across all of the various cloud providers and even tieing into the expertise side of what you're doing as an angel investor, you've always been a staunch advocate for, I guess we'll call it doing security the right way. And I've always been tangentially related to security throughout the course of my career. And somewhat recently, I launched another day of my newsletter focused on security within AWS, for folks who are not themselves in the security space of what do you need to know. But so much of it comes down to the do the easy thing now, the right way to do it before you wind up having to do a whole bunch of damage control. And you've been advocating for that since before it was trendy to do so. I imagine you're still somewhat passionate about that perspective.Lynn: Well, I always like to say, you know, Werner Vogels doesn't talk anything about tech; he just talks about, “Please use our security.” And I don't blame him. I mean, you know, I joke that I am an AWS Community Hero because I made a bunch of YouTube videos about securing buckets. And that was, like, seven years ago and I just had a financial client, literally in November, and their buckets, you know, was made public because it was easy for the developer. I'm like, “Ugh, can we just do our foundations?”I don't know why it is not seen as a valuable skill. I mean, I've made craploads of money because people come after they have an incident, but you know, I wish we would be better. And I'm worried because as we start to get more and more of our health information in these big repositories—granted, we have some laws; yay, good—but it's just not valued like coding up a new feature with node or something. And why not? I don't understand.So, I make all these educational resources: I make courses, I have GitHub repos, I have videos. You know, just do it. Plus the people who learned security. I mean, we are always in demand. I'm not a security professional, but I always do security kind of like as a courtesy. And people are like, “Oh, you know, you're great. Oh, my friend needs you.” Dah-dah-dah… I mean, you'll be working forever.Corey: It feels like it's aligned with cost in that it is almost a reactive function. You can spend all your time on it, but it's not going to advance the state of your org further toward its stated goals. You've got to do it, but there's also never really any ‘done' there. It's just easier for me on the cost side because I can very easily quantify the return on investment, whereas with security, it's much more nebulous. And, of course, you wind up with the vendor—I'm going to call it what it is, in some cases—nonsense that is in this space, where, “Oh, you're completely doomed, unless you buy their particular product.” You know, walk up or down the aisle at RSA a few times and your shopping cart is full. And great, are you more secure? You're a lot more complex, but does this get you to a better outcome?And it's, I am so continually frustrated by all of these fancy whiz-bang solutions that are sort of going around the easy stuff—not easy, but it's the baseline level of things: Secure your S3 buckets, or—for users themselves—it's use a password manager that has a strong password on it, use it for everything, use MFA for the important things that you need to use, make sure your email is secure, don't click random nonsense. There's a whole separate pile of things. If I can click the wrong link in an email and it destroys my company, maybe it's not me clicking that link in the email that's the root problem here. Maybe there's an entire security model revisitation that's due. But I'm sorry, I will rant like a loon about the dismal state of security these days, if you let me, and you absolutely should not.Lynn: Well, I would just entreat the audience, basic threat modeling is not complicated. It's like cost modeling. It's just a basic of having successful business on the cloud.Corey: [sigh]. I wish the world work differently than it does, and yet here we are. Lynne, I really want to thank you for taking the time to come on the show a second time. If people want to learn more about what you're up to and talk to you about anything we've discussed, what's the best way to find you?Lynn: So, if you can't find me, you're not looking. I have an internet-easy name. But two places that I'm pretty active: Twitter—just my name, @lynnlangit—and go to my GitHub. In particular, I have a learning cloud kind of meta-repository that has over 100 links to mostly free things on every cloud and just use them. Have at it, learn, be a practitioner, use the cloud more effectively.Corey: And we will, of course, put links to that in the [show notes 00:32:25]. Thanks so much for coming back on. I really appreciate it.Lynn: Thanks for having me. It's been fun.Corey: Lynn Langit, CEO of Lynn Langit Consulting, and oh so much more. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment talking about how security really isn't that important, and right before you submit that comment accidentally type your banking password into the form, too.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Literally Working in the Cloud(s) with Tyler Slove

Screaming in the Cloud

Play Episode Listen Later Feb 22, 2022 34:02


About TylerLifelong learner, passionate coach, obsessed with continuous improvement, avid solver of people puzzles.Links: United Airlines: https://www.united.com/ LinkedIn: https://www.linkedin.com/in/tylerslove/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Couchbase Capella database as a service is flexible, full-featured, and fully managed with built-in access via Key-Value SQL, and full-text search. Flexible JSON documents align to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling, while reducing costs. Capella has the best price-performance of any fully managed document database. Visit couchbase.com/ScreamingintheCloud to try Capella today for free, and be up and running in 3 minutes. No credit card required. Couchbase Capella make your data sing.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance query accelerator for the Oracle MySQL Database Service, although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLAP and OLTP—don't ask me to pronounce those acronyms again—workloads directly from your MySQL database and eliminate the time-consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Calling this show Screaming in the Cloud has been pretty… easy most of the time because that's mostly what I do: I shake my fist and I yell at clouds. And most companies are okay with that. Today's guest is likely a little bit on the other side of that because when I'm screaming at clouds, it's often out the window, when I'm in a plane.Today, I'm joined by Tyler Slove, who's a Senior Manager in the Enterprise Cloud and DevOps Group at United Airlines, a company I spend way too much time dealing with when we're not in the midst of a global pandemic. Tyler, thank you for joining me.Tyler: Yeah. Thanks for the invite, Corey. Really excited to be here.Corey: So, I want to talk a little bit about, first, how glad I am to finally talk to you because airlines are kind of like computers—and particularly cloud—where when you first see it, it is magic; it is transformative, it's endless possibilities, the power of flight slash instant provisioning of computer resources. Okay, so not everyone is going to find those quite the same way. What's novel today is commonplace tomorrow, and then you get annoyed because your plane is 20 minutes late as it hurls you through the sky to the other side of the planet with the miracle of flight while you're on the internet the whole way. And it's one of those problems where it is sort of definitionally, a thankless job. It is either in the background that just empowers things, or everyone's yelling at you on Twitter. So, given that you work with both sides of that, how do you find that commonality to play out in your world?Tyler: Yeah, it's an interesting thought, and I hadn't necessarily connected the dots before. Because I, like you, are just as frustrated when that flight is, like, 20 minutes delayed. It's like, “Oh, I wanted to be—[laugh]—where I wanted to be at that time.” And, you know, when you think about it, it's actually an ongoing joke I have with one of my mentors. Like, airlines should not work; when you think about the maintenance, the aircraft, the crews, the weather, legal stuff, like, it's amazing how complex they are, and it's something that's kept me interested for, you know, the first three years that I've been here.But it is similar, actually, to being in an operational role, right? You do everything right, everything's resilient, you roll through an Amazon, like, region-specific issue without any blips, and no one reaches out to you. But you know, you have one issue, and then it's you're getting out of bed at three in the morning, and everyone's got a big retrospective about why you didn't do something that could have resulted in that not happening. And I can see the parallel.Corey: We all tend to have blind spots, and I more or less had my idea of big enterprise technology fixed a while back. And it occurred to me a few years ago that this is probably no longer accurate because I'm sitting here thinking of, well, United Airlines—with whom I do extortionately large amount of travel, let's be very clear here; we're talking I think I did 140,000 miles domestically flown in 2019, the last year that was even close to normal. Protip: Don't fly that much. It really winds up doing a number on your internal clock and having any semblance of life. But I'm sitting there thinking that it's old-school technology; there's a mainframe that powers all of this, and all of the staff checking me in are using these ancient Unix green screens has always been my assumption.And that thought occurred to me as I'm staring at my iPhone, checking in automatically in the mobile app—that was very modern and working at the same time—and the penny finally dropped for me of this is probably not accurate, how I'm envisioning the technology on the back end working. And there have been announcements that United is moving an awful lot of its systems to AWS specifically. What is that—I don't want to call it modernization because that sends the wrong undertone or subtext to it, but what has that cloud transformation been like?Tyler: So, it's the marrying together of those two things without the time that you would potentially want to just rewrite the functionality that the mainframes that have gotten us to do the amount of you know flights and revenue that we do, and that are rock solid, like, we don't get the chance to shut that thing down for three months and rebuild it—or what would be, realistically, more like three years. So, it's how do we build a—Corey: Yeah, it's a heck of a delay notice to put on the airport flight thing: “Flight delayed?” “Oh, when is it rescheduled to?” “2025.” Yeah, turns out that doesn't usually happen.Tyler: Yeah, and so we've got to do it at the same time. And there's, you know, analogies of, like, changing the tire while you're driving or changing the engine on the jet while it's flying. And we've actually—it's felt like that, but it's been in an exciting way. So, we really are able to decouple the front end from the back end or some of the core systems and then, piece-by-piece, modernize them, and do them in a way that is safe and responsible, given you know, the amount of folks that are relying on us to get to where they want to go every day.So yeah, it's been challenging for sure, but it's also the right thing to do. It's the direction we need to go where we can focus more of our engineering talent, which is scarce or limited, you know, we would rather have folks invested in improving the user experience instead of—what we have is a world-class data center, but you know, the number of people that are focused on making that what it is, I would much rather see that happen—or that investment be put into a higher up the value chain.Corey: It's also, on some level, on a baseline trying to understand how it all fits together. You look at the challenges that an airline has, you have challenges with labor, with press, with you know, the big problem of the logistics of not just the scheduling and the rest of making sure that everything flows throughout an enormous what is effectively logistics network, but also the, you know, the minor detail of keeping the planes in the sky when they're supposed to be in the sky. And it feels like on some other you flip through the list of concerns a company has, and technology in the computer sense feels like it's going to be, like, chapter 47 of that giant book. Obviously, that's not true because technology is an empowering story. It is not just the booking system; it controls, more or less, everything.At some level, I'd like to make fun of big companies saying, “Oh, we're not a”—insert whatever the company really does here—“We're a tech company.” But without technology, I don't think you, at this point, have much of an airline. How do you see yourselves in the broader sense? Are you increasingly a tech company?Tyler: We are increasingly a tech company. I think we're… we're seen as partners with the VPs of the different functional areas, right? It's not a separation of the business and IT the way that maybe we would have thought about it five or ten years ago. It's, both of us can't be successful without each other, and the functions have come to trust that we will spend the time we need to understand the problems that they're solving, and we'll bring different perspectives, we're going to bring technical solutions, but we're also going to bring, you know, potentially system or flow changes and business process improvements. And that takes some getting—that right a few times and building up that trust and spending the time you need to, like, go past, “Oh, here's a set of user stories. Just do them.” Of, like, “What are we trying to solve here? Could we just remove this process? Do we even need to do this thing anymore?” And once you prove yourself, I've never felt like we've been put in a backroom or seen as a lower priority. We're working on the same stuff together, and we win or lose together.Corey: I know a lot about the airline industry because I go to tech conferences, and when I'm at tech conferences, invariably the speaker—who's usually J. Paul Reed, but not always—decides to talk about computers, and incident response, and the rest through the lens of the airline industry, which for some reason has always been one of those neck and neck things that are just completely inseparable for those types of talks. And they talk about airline incidents, and very often it's not even, like, the horrifying headline-making stuff, but things like two aircraft passed closer to one another than they should have, and the NTSB does a full investigation. And they talk about how, “Oh, this is exactly the sort of thing you should do whenever there's a computer-related issue.” And I am curious, given that you do in fact have those investigations with the plane-facing stuff, how much of that culture carries over into the, “Hmm. We took a systems outage on the computer side.” And how much of that is similar versus how much of this is just conference-ware.Tyler: It's actually quite similar; that part of our culture permeates through. And we're actually looking at what's the right level of time to spend to get to the root cause when sometimes it's hard to explain in computers. Or there's so many variables that it's going to take us, you know, weeks or dozens of hours to really get there. But yeah, after any significant incident, we're religious about having a follow-up problem review where we get all the information that we need, and we, kind of, are expected to figure out exactly—like, replay what happened, step-by-step, and what were the controls that were in place to avoid such a thing, and were those complied with or not, et cetera. And earlier at my time in United, definitely was frustrated with how—I'm like, “I just need to get back to delivery. We've got this—this sprint is ending, and I can't spend four hours doing this.”Like, that was a… what was seen as, like, a one-time event. And I don't think that all the things that culminated in that are going to happen again, and I've done a few things that I feel are going to mitigate the risk moving forward, but actually, I've changed my perspective on this now. So, we are forcing—or not even forcing; we're simulating major incidents and then doing that type of a problem review so that we can learn ahead of time and we can make it a heck of a lot more fun [laugh] and open and transparent conversation. So hey, me or someone from my team gets behind the curtain and, like, creates some simulation of a major issue in one of our pre-production environments, and then the team that's responsible for the operations and whatnot of that response.And we look at what alerts went off? What alerts do we expect to go off that didn't? What was maybe a leading indicator that we aren't yet looking at? And kind of so we're calling that a game day, and we took that, you know, from—AWS has influenced our thinking on that, or they contributed to it. And it's a really good way to build those relationships, when there's not a lot on the line, you're not coming around what could be a customer-impacting negative experience, which is, you know, really what drives us to do good work is to make sure that never happens.And it does happen, but you know, we're getting more and more resilient. And this is a way to turn that on its head and be able to take the positive of that, and get the spirit, and get people to collaborate better because they—like, “Hey, I did that fun thing together. Now, when we're in the heat of it, we're going to collaborate better, we're going to be, kind of, more open with the information we're sharing because we understand each other's people and their intentions, and you know, where someone's coming from.” So, yeah, we were pretty excited about that.Corey: I have to admit I'm a little on the envious side about how your timing has worked out. Because back in 2008, when the cloud was still a new thing and some of the early adopters were diving in, the experience really sucked. I mean, this was before CloudFormation and other ways of managing systems. And by migrating over the last few years, so many of those sharp edges have been smoothed, and established patterns and processes, and understanding of how cloud interplays with enterprise IT has evolved dramatically. What has been your experience migrating to AWS? What's worked well and what hasn't?Tyler: Yeah, so the migration itself has been very deliberate. So, we were focused on AWS from the beginning, and it was—we believe that they're a leader, that they're going to give us what we need, but also we didn't want to fragment our engineers across multiple platforms and have them have to pick a team. Like, “Am I going to choose to learn how to build stuff in AWS, or GCP?” So, from just a transformation, and to get everybody on the same page, and upskill the organization, we're focused on AWS. And there's definitely, like, some learning curve, or moving into an environment where there used to be a centralized team that handled a lot of stuff for you and made it magic—like, as an engineer; I just have to make sure that my app builds, and then I can send it to someone, and they're going to deploy it, and it's going to work and then you know, we… shifting the responsibility to, okay, we actually believe that if—we could do that; we could just have the same function that did that in the on-prem world, do that for you in the cloud world, but our belief is that we come up with better software when the engineer understands and can control the entire workload and that it's like, “Hey, I can configure my app to take advantage of this particular portion of the underlying infrastructure.”And that became very clear with, like, Lambda or things like that, where it's… you know, there's only so many configurations, and it doesn't make sense to try to get someone else to do that for you. So, there's mindset changes that had to happen. There's also just, like, proving it out. Like, is this going to be more reliable than our data center, which is extremely reliable? And there have been issues in the cloud, like, where we have something running parallel, and we have a cloud issue and it didn't impact on-prem.So, how do we learn from that? And then how do we kind of continue on and figure out, how do we build resilient workloads in the cloud? How do we make sure that we cover our bases on not just getting it running, but like, getting it running the right way, and then doing the testing that we need to do—like I mentioned earlier on the game days—to really be confident in it so that we can ultimately move away from needing to have any sort of backup in the data center.Corey: I was poking around in an AWS account recently, and it looked like there were seven different ways of managing the systems that have been brought to bear in that account, and different design philosophies, competing approaches. And the sad part is that this was my personal AWS account. No one else has ever built anything in that account except for me. And if I have that problem as one person—admittedly a strange person—I can't imagine what the governance story around something like AWS looks like for an organization that has thousands of people working in your IT org. How do you wind up managing the way to build things appropriately?I can't fathom—even though I am a fan of ClickOps—just letting everyone loose with admin rights in the AWS console. There has to be some form of gating approach. Is that done through patterns? Is that done through some sort of internal platform that abstracts away for folks? How are you managing this?Tyler: Yeah, so this is one of the things that led to a learning curve at the beginning, but I think it's worthwhile. And I can't take credit for this because it was a decision that happened before I came, but we're all-in on infrastructure as code. So, we're not extremely prescriptive about what that means across the entire enterprise, but you cannot deploy anything into an environment, like, higher than a development area without it being defined as CloudFormation and promoted through. And that allows us consistency, auditability, [laugh] and a lot of other things.So, that was kind of phase one, and that's been—I believe—in place since we started in the cloud. Like, maybe there were some pocket accounts and some things that existed before, but once we were all-in, and it was, kind of, official that's been in place. And I'm glad we held to that because there's been a lot of, like, “Oh, just remove that. Let people build stuff through the console because they need to move fast.” And we're like, “Yes, that would move them fast right now, but the level of inconsistency would be extremely risky to be able to handle that, and handle production incidents if you don't have a pre-prod environment to test the patch that you're trying to put in on the fly, that manages hundreds of orders a second.”So, we started with CloudFormation. We were kind of all-in on CloudFormation, and then over the last year or so—maybe a little bit longer—it's become apparent that CloudFormation has some limitations. And it can be also intimidating to have to, in excruciating detail, like, define every single parameter of every resource you're trying to create. And—Corey: It's wordy. It's YAML or JSON, whichever one you hate the most, invariably, is the one you're dealing with today. And yeah, it has its limitations.Tyler: Yeah. And then they're sharing that happens, right? So, it's like, I've got someone that I go to lunch with, that's like, “Oh, I just built this solution. It's all in CloudFormation.” They send it over, and then I'm looking at, it's like, “Can I reuse this? Which parameters here are things that I should change for my app, and which ones are there because security mandated it, or it's part of, like, a corporate compliance thing, or other reasons why?”So, what we are really excited about in the last few months, we've really invested in CDK constructs and being able to define. You know, as my small team, we have visibility and strong, like, partnerships with our cloud engineering group, with our security groups, and whatnot, and we can say, “Hey, if you want to build an ECS cluster, like, this is a good, known way to start.” And you can just provide, like, X number of parameters that are meaningful to you, and you can inherit all the rest. And you're going to get our logging standards, you're going to get our security standards, all that, like, more or less built-in. And we also can version that.So, we can know, hey, this person built off the CDK App 1.1, and then we have some sort of security change, right? So say, now we want to install some other agent on all these things. And it's like, “Okay, all the ones that were deployed on 1.1, we need to move it from 1.1 to 1.2.”And we can test what that upgrade path looks like in a lab environment, and then we can, you know, release it and have, you know, 30 different app teams all consume that update in a relatively self-service manner that means we don't have to do it one by one. And then, yeah, it just gives us the ability to respond to stuff as quickly as we need to in the current environment.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: It's a constant challenge and it's really neat seeing the adoption of things like the CDK, which I've always sort of mentally put on the same stack as, “Oh, yeah, this is something that scrappy tiny startups use.” But you're the exact opposite of that. The fact that you're using it and finding success with it says a lot. I think you're also right there with the most nimble, advanced, tiniest of startups in the world, and you're still trying to figure out how to contextualize this into the broader lifecycle and understand the long-term architectural implications of how this stuff works. If it helps anything, I can assure you, you are very far from alone.If anyone else is feeling that way, exactly the same position. And if you're out there saying, “Oh, yeah. We've solved this. This is how we do it.” Find a second person to agree with you. But then come talk to me. Because everyone solves it locally; no one solves that globally. It's a hard problem.Tyler: Yeah. We've had this vision of, like, a vending machine for stuff. And then we've tried that in different ways and templates, and we think that this is the right pattern.Corey: Yeah, every time AWS builds a vending machine for accounts and whatnot, it's like the worst kind of vending machine; the kind that eats all your money.Tyler: Service catalog. Yeah.Corey: Yeah. It becomes a disaster. So, I want to talk about a couple of other things as well. When we started talking a year or so ago, you were a team lead. Today, you are a senior manager, and it turns out that, unlike when you start your own company and can invent your own made-up title, like, Cloud Economist, those words mean things. So first, congratulations on the promotion, how'd it come about?Tyler: Thank you. Yeah, it came about—I guess, I really have always been passionate about people leadership, but I know that in order to properly lead and, like, have the context, and you need to know what it's like to do these hard things that my team is solving, and be responsible for those, kind of, as an individual. So, you know, I've been spending the last, like, five or so years as an individual contributor, kind of learning how all this stuff works, and then learning from a lot of different managers. You know, I've been really lucky to have some people that, kind of, took me under their wing, coached me, and is just, like, the person that puts the wind in your sails, but like, not in a… not in a fake way, but like actually sees you and puts you into situations that are going to force you to grow and have your back if something goes wrong. And I kind of saw that and I wanted to be that for someone else.So, you know, it's… yeah, it was something that I kind of put my hat in the ring, and a position came and I was tapped to step up and do it. But it was initially for a very small team, right, so a three-person team. But it's since expanded to be six or seven over the next month or so.Corey: One of the things that I found always interesting slash admirable about you is we travel in somewhat similar circles. We both have pitched in from time to time as mentors in Forrest Brazeal's cloud resume challenge, and it's nice to see people who are working at established companies who are very busy with their day jobs, also taking the time out of the day to help, effectively, what is the next generation of cloud engineer find their way within this industry. How did you get onto that track?Tyler: Yeah, so I guess it's, you got to send the elevator back down. I have the experience of, kind of, being on the edge of, like—I was on the waitlist for my university, I had to—also was on the waitlist for my first job as a rotational program, and there was always kind of this, like, I had to claw for it, I had to prove myself, and also had to—I was the first in my family to pursue opportunities like this. And I got the itch for it, then I also see there's so much potential in folks. And like, even looking at my parents as examples, right? My father's an auto mechanic, and he's probably one of the smartest people I know, but didn't really… have the opportunity to get into technology. [unintelligible 00:22:44] kind of in a blue-collar job.But I just feel like there's so much untapped potential, and I am passionate about helping people at least, like, understand what opportunities are available to them. And not just assume that if you don't have an example of someone who's a software engineer in your life, or a sibling, or a parent, like, that's outside of your reach.Corey: I love the phrase, ‘send the elevator back down' because it's true. I feel like the only reason that anyone that you have ever heard of in tech, who you have any modicum of respect for—and I include both of us on that list as well, but basically everyone else in the industry, too—the only reason all of us are here in the roles that we're in is that at some point, someone did a favor for us that they didn't have to, but they did. And it's almost impossible to pay that back, so instead, I've stopped trying. I instead try to do those favors in a forward-looking way for other people whenever I can. And there's a lot to be said for expressing that through a way of helping people find their way and see what happens.Because let's face it, the industry that you and I came up in doesn't really exist in the same way. There is no fleet of help desk positions out there the way there was when I first started getting exposed to technology, that would get me into this direction, so people have to come through alternate paths. And some people try and express that through advice that no longer applies for a world long gone. I try and at least keep up with what's going on in this space.Tyler: Yeah, absolutely. It's a dynamic environment for sure, and when I look at just how challenging it is to try to, like, find a senior cloud engineer, and then looking at, okay, is what we're doing here, like, really rocket science? Does it require ten years of experience? And I think the answer is no, like, we've got a small enough group here, we know what we're doing, and everyone's passionate about bringing other people up and, like, finding their strengths, giving them a problem, not giving them the answer to the problem, and kind of strategically building to bigger, bigger things until the next day, you know—or before you know it, they're able to solve problems that you would have previously thought, like, “Oh, that's something that I have to get my hands on.” And it's just so powerful to see that and to be part of that. So, that's kind of the approach we're taking.Corey: It refreshing to see. So, many companies are requiring that they hire senior talent, and they can't take junior talent because, “Oh, that person would take six months to come up to speed in this environment. We want to hit the ground running.” And the job req has been open for nine months. At some point, building talent becomes the best slash only way forward.I'm still at a scale now where I'm not in a position be able to do that, just because we are dropping principal consultants into dynamic strange situations, and that is a terrible environment for a junior, but as you scale past a certain point—I don't really know what that point is, but yes, United Airlines has scaled past that point—bringing folks up, taking interns, making interns job offers, and continuing to expand what is happening, I think, on some level, one of the big hiring challenges for United and other similarly situated companies has been that, oh, the technology must be ancient caribou-era of trekking across the tundra level of development. But we just talked about using the CDK, and pattern design for things. The public perception and the reality are incredibly divergent.Tyler: Yeah. Maybe I'm strange in this regard. But since college, I've worked only in very, very large organizations. And seeing the satisfaction that you have, or you can get from working with those systems, and being able to churn out a modern customer experience, or modernizing the system for operational efficiency, just it's very satisfying to me to be in that environment. I know that it probably scares other people away.But it's just the scale; it's hard to get that scale somewhere more—I don't know, I guess, like, younger, newer because you don't have years of legacy. But I don't necessarily see that as a bad thing. Like, years of success and technology that's supported that success that you need to figure out how to handle.Corey: One last question that I have for you harkens back to something that I said earlier, where I congratulated you on your promotion to management. It's not really a promotion, at least not the way that I think it should be thought about. Because it's very much an orthogonal skill. You were a great engineer and architect building things yourself. And now you manage a team where if you're diving into fix things by hand, you are misunderstanding the role in many respects, suddenly, your toolkit is no longer doing the thing yourself, but rather delegating the thing to be done and making sure that it gets done and your primary slash only toolkit to do all of that is hiring and developing talent. How have you negotiated that transition? Do you still find yourself itching to dive in and fix the work yourself? Are you better at letting go than I was for a long time? Where do you find yourself on that?Tyler: Yeah, so that the inclination is still there, but I've learned to, like, recognize it and let it go. But I also have told my team members, like, 90% of the time, I'm going to give you all the latitude in the world, and I'm going to spend all my time helping you understand the problem that we're facing as I understand it, and the potential roadblocks, and then there may be some times where I'm going to be like, “I really want it done this way.” And I ask them to give me that… give me that ability. I have yet to really break that one out. But that's the only way that you can scale, and you get so much satisfaction about over… empowering someone to solve a hard challenge, and then seeing that they did it in a way different than you did it, and they did it better. [laugh].And that's a little bit of an ego hit, but you're like, that's what it's about. And then they can build that confidence and then take on larger challenges. And that's what gets me out of bed in the morning; that's what gets me excited is working with people who just really want to do good work. And I can help put the right challenges in front of them, help shield them from stuff that's not adding value, but like, asking for their time, connecting them with others that is going to kind of get that wind in their sails, and just get out of their way.And then once the success is there, do everything I can to get that out and make sure that people know the good work that we're doing. Because as much as you can say your work speaks for itself, in a huge organization, it's not so much the case. Like, good work often goes unacknowledged if there's not someone if you're—like, promoting that. And most individuals aren't comfortable—myself included—promoting my own work. Like, I wouldn't do that, but I'm more than happy to promote the work of someone on my team.Corey: On some level, as managers, you get recognized and evaluated based upon the performance of your team, not the things that you personally achieve. And that has always been a difficult transition. I got to level with you; I never handled it super well. It sounds like you are way better suited for the role than I ever was.Tyler: Well, it's early on, but yeah, I'm very excited.Corey: If I really want to evaluate a manager, all I have to do is really talk to their team, more often than not, and you start to see things when you probe properly. I really want to thank you for taking so much time out of your day to speak with me. If people want to learn more about what you're up to and how you see things, where can they find you?Tyler: I'm probably most active on LinkedIn. So, just tylerslove at LinkedIn.Corey: We'll be sure to add that to both the [show notes 00:29:58], as well as I will add you to my professional network on LinkedIn, which I believe is the catchphrase that they're using. Thanks so much for your time. I appreciate it.Tyler: All right. Thanks, Corey.Corey: Tyler Slove, Senior Manager for Enterprise Cloud and DevOps at United Airlines. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud, of the usual kind. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment disavowing all of this newfangled technology we've been talking about and that's why you only travel via steamship.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
Merewif's Mitigation of Risk with Ana Visneski

Screaming in the Cloud

Play Episode Listen Later Feb 10, 2022 44:02


About AnaAna Visneski is the founder of Merewif, a crisis communications and management consulting firm. She is a veteran of the U.S. Coast Guard where she was a first responder to major disasters from Hurricane Katrina to the BP Oil Spill, and various other incidents. After the USCG, Ana moved on to a whole new disaster that needed an experienced crisis operator - running Launch Operations for AWS. Following that she was the global lead for AWS Disaster Response, overseeing deploying AWS technology response to natural disasters and overseeing the response to COVID. She has a Master of Communication Digital Media and a Master of Communication in Networks from the University of Washington, where she currently teaching Crisis Communications. Links: Mirewif: https://www.themerewif.com/ Oracle HeatWave: https://www.oracle.com/mysql/heatwave/ Twitter: https://twitter.com/acvisneski The—T-H-E—merewif—M-E-R-E-W-I-F dot com: https://www.themerewif.com/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today has been on this show before, generally at a previous point in her career where she was making a transition. That time, she was leaving AWS, as happens to awesome people a fair bit of the time—more than it potentially should—and going to work at H2O.ai, a company that does some sort of machine learning thing that I can't be bothered to remember offhand. I talked to her again, as she has just left that company to start her own thing. Ana Visneski is the Chief Chaos Coordinator at Mirewif. Ana, thank you for joining me yet again.Ana: Oh, I mean, how could I not when you're the one who got me to get off my butt and actually start my own company?Corey: What's fun is that your company is a crisis communications firm, and first that's definitely useful for me because I do put the ‘crisis' in ‘crisis comms,' let's not kid ourselves.Ana: You're not wrong. [laugh].Corey: But I'm also your first customer.Ana: Mm-hm.Corey: And you're in one of the harder niches to get people to stand up and say, “Yeah. Oh, yeah. Can I get a testimonial on this?” “Absolutely not. We hired you because we did something horrible.”And that's not really how I tend to view crisis comms. I mean, it's sort of a similar problem to what I had when I started The Duckbill Group of, “Hey, can I use you as a testimonial about your horrifying AWS bill?” “No.” And I understand how it looks, which is not the reality of it. And in time, I found ways to get people to slap their logo on their website. But I want to be the first logo, the fact that I have a platypus associated is just a nice bonus.Ana: Absolutely. You will be the first logo when I finally get around adding logos. The interesting thing is, that it's not just crisis comms that I'm doing with the company. I also do threat assessment, violence assessment, so risk analysis, basically, on if you have an employee that might be a risk or, for some of my video game or gaming companies, if you have someone in your fan organization that is a potential risk.I also do crisis management planning. So, I will put together an operational plan—similar to what I built when I was at AWS—a top to bottom, this is how you run a crisis to make sure your people don't burn out, make sure your leadership is aware of what's going on and gets the proper daily briefings, that sort of thing. And then lastly, I've actually been doing some consulting with governments on their disaster response technology needs. So, there's a lot of different aspects to it.Corey: Yeah, to be very clear, none of those things are things that I have roped you in for. I don't have employees that I'm looking there with, “Oh, if they blow their stack this is going to be a disaster.” Like that is not the nature of the work we're doing together. What we're doing is more along the lines of, “Okay, great. I have a bad tweet that blows up. How do I handle this without, ‘All right, pass me that shovel. We're digging this puppy deeper. Now, okay. Holes dug nice and deep. Let's work on the edging details a little bit.'”Ana: [laugh]. Yep.Corey: It's the, “How do I avoid making things worse in moments of crisis?” And we're building plans for things that I hope to never need around things like data breaches, like, the stuff that every business should have a plan for. Because when disaster strikes, as it tends to in various ways, I don't want to be sitting here flipping through the Yellow Pages for, “I've messed up.” Like, I don't know what section that would be in. Having a plan ready to go is important.Ana: I would say it's actually critical.Corey: Yeah.Ana: So, that's the thing is, unfortunately—and as Covid taught a lot of people—having that plan in place before things go wrong before the shit hits the fan, is what's going to save you or not. It'll save you millions of dollars, it'll save your employees, and it could potentially save lives. And so what I think a lot of companies have finally figured out is, “Oh, wait. We weren't ready for Covid. We actually need to be ready for the next thing.”But I also teach crisis communications for the communication leadership program for the University of Washington; it's a graduate program. You've been a guest speaker there. You were one of the favorite guest speakers. And there I tell them all the time is that you have to plan. The two critical things before anything even starts is planning and trust.If you don't have plans in place on how you're going to do things, you're going to have people running around like chickens with their heads cut off going, “Oh, what do we do?” And someone's going to do something that makes it worse—inevitably—with the best of intentions. And then the other thing is, if your audience, if your customers don't trust you to be doing the right thing in the first place, then no amount of planning is going to help from that deficit.Corey: It also, in my experience working with you, comes down to avoiding putting your foot in your mouth with the best of intentions.Ana: Yes.Corey: Heaven forbid if you have an employee pass and tweeting out something like, “We are heartbroken to announce the loss of our dear friend and colleague, [Shtephen 00:06:45]. Also, we're hiring.” Like, make sure you don't wind up coming across as the worst example of humanity. It's the basic stuff.Ana: Even more than just that basic of don't put your I'm hiring—because you saw that tweet that was going around with, “So-and-so has passed. Please mourn off the clock.” Whether that was a joke or not—and it's up for debate if it was real or not, like—Corey: We've all known people who would have said such a thing and it would not have been a joke.Ana: Exactly. But the other thing is, it's not even just that. It's knowing the timelines for notifications. So, for example, there should be at least a 24-hour next-of-kin notification window, where if someone is passed, the friends and family grieving can be notified. The last thing you want is a friend of Shtephen to find out that he died because you tweeted about it. That is traumatizing.So, you actually have to have a plan in place of, you've received notification from Shtephen's wife that he has passed. Obviously, you're going to be offering her your support. Say, “Hey, here's the things we can offer you to help.” You have, you know, your package of, like, here's the ways we can help you. But then you also say, “Can you let me know when it's appropriate for me to tell the other employees?”Because the moment you start telling employees—this recently happened; a friend of mine in the Coast Guard passed, and unfortunately, some others found out about his passing because someone posted about it on Facebook. That is not the way you need to find out. So, it's not even the blatantly obvious things, like, “Oh, hey, don't post about hiring,” it's also just the order in which you notify so that things don't leak.Corey: I didn't even know Shtephen was married. I mean, what kind of—Ana: [laugh].Corey: —crappy employer am I here? Yeah, it's the human side of it.Ana: Mm-hm.Corey: And that's one of the things I've always admired about you. It's—and again, when I started doing all these nonsense things, I had a circle of friends that I could run things past of, “Hey, is this tweet a bridge too far?” And in time, I needed to rely on those people a little bit less because it turns out that I have a pretty good eye for what's going to make people feel bad. And that's really the only thing I care about is if it makes someone feel bad, then I'm not thrilled with the tweet most of the time.And I figured out where that line lies. And then I got loud and big enough on Twitter where I started having to think about it again, where, all right, I know it's not mean, but I'm going to hear about it. Is the juice worth the squeeze? And the reason I like working with you on things like that is I've grown well past the point where I'm comfortable asking people to volunteer for basically what amounts to something of my own brand-building exercise. Paying people for advice has always been something that I'm a big fan of, and now I'm able to do that and have a professional way.And I don't think you've ever once been wrong. There are times you've given guidance that I have not followed, but that's what you see anytime you're talking about someone a downside, risk side of the business. That's the entire function of an attorney for a business is to identify risk. If you start letting attorneys, for example, my wife, great attorney, great wife, wound up—Ana: And very tolerant human being. [laugh].Corey: Oh, extraordinarily—living saint. But she wound up editing a proposal that I was going to send out—back when I was independent—once. And I looked at it and she's like, “Oh, well that could go wrong, and that could go wrong and no, we're going to change that and the rest.” It's like, this is—I understand where you're coming from, but this is a sales document. And it was for a proposal, it was something like $7,000 back then.It's like, worst-case scenario, I'm a nice person, I will fall over myself apologizing and give them a full refund. The end. That sort of caps my downside risk here, if they want to be obnoxious and go to court, well, I've been doing this for three months, I guess I'm shutting down the LLC because that's been sued into oblivion. I'm getting a real job. Like that was the risk mitigation there.She's used to doing risk analysis for a company with 250,000 employees, and yeah, they have more to lose than I do in those things, so I get it. But you don't generally have lawyers on your sales team that are proactively over-promising things, for obvious reasons. At least—because there's no way to get a salesperson disbarred. I've checked.Ana: Of course you did. When I'm teaching class, one of the other things I do is I actually have some lawyers come in and talk. And the reason is, I learned this one when I was in the Coast Guard, and I was running District Eight. So, it's basically the entire Gulf Coast and all the way up the Mississippi to the Canadian border. So, all of the units contained in that area, I was in charge of their media relations, their community relations.And this was, like, right after Katrina. I learned pretty quickly that having a very good relationship with my lawyer—so the head of legal—it made us a one-two punch that was unbeatable because I could look at it from the human empathy, communication, subtext aspect, and he'd look at it from the legal aspect, and the two of us would be like, “Okay, you can do this legally, but here's the impact of it if you say it this way, or if you do this.” Or, “Ehh, don't do this one, legally.” Like, it's just a great thing. But risk analysis, from my perspective versus a lawyer's, are slightly different.I do, of course, talk to lawyers, obviously, a lot, and look at the legal side of stuff. But a lot of what I'm looking at is perception, subtext, potential pitfalls. You and I've had many conversations, and you know me well enough to know that most of the time I'm giving you guidance, but if I see one more, I'm like, “Absolutely not. Do not do that.” I will lean into it so heavily, and be like, “Corey, here's the eight ways this is going to go badly for you. You're going to end up in The Times for bad stuff.”Corey: And you say that so infrequently that I definitely pay attention when you do. I don't always listen, I mean, [crosstalk 00:12:14] I wound up posting that Andy Jassy birthday video. But you know—Ana: I helped with that video, though. [laugh].Corey: —you were instrumental behind that video. Thank you for that.Ana: You're welcome. But that's—so what's fun about working with you, and different than my other clients is there are these moments where I get to also express my weird sense of humor, you know, where it's just like, calling Jeff Bezos, a space cowboy. Those moments of getting to find—help you with that line. Because I have that same sense of humor line and I don't get to express it a lot with my other clients because most of them are very, very serious bidness. And not to say your business isn't serious, but you yourself are almost—Corey: But we do have fun with it.Ana: —never serious. Exactly, exactly. And that one, like, I really enjoy that aspect of it. But with a lot of the other stuff, it is incredibly serious. And like the risk analysis that your wife does, versus the risk analysis type I do, I'm actually looking at emotional stuff.So, when we're talking about acts of violence, for example, acts of violence are, almost to a one, about power. So, what I do is I actually sit and look at okay, this person is lashing out. What power dynamic has them wanting to lash out? So like, if you look at a lot of the school shootings, it's about kids who feel bullied, they want to regain power by showing they have power or the guys who write their manifesto about hating women, et cetera, et cetera. So, it's always about a power dynamic.So, it's not about, is it legal to go in and shoot the office? It's clearly not. But has the system taught them that they can push the line far enough that this sort of behavior, they might get famous for it? Or might get away with it? And then how do you mitigate that particular power dynamic? And so that gets real tricky. And luckily, with you, I have not had to deal with that one.Corey: For better or worse, I come out from a good place to place a good intention. I'm trying to imagine if I just said, “To hell with it,” and decided to just take off the gloves and be a complete bully every time I felt like it. I could do some damage at this point. But… no.Ana: You could, but the thing is remember what I said at the very beginning: It's about trust. What has made you so very successful, what has made you so good at what you do is you're very intentional and very careful. Not to say you're not a pain in the ass. I will agree with some—Corey: And I do get wrong. Let's be clear. I'm no saint.Ana: Oh, no, no, no. No. You've gotten stuff wrong, but you immediately apologize for it. So, when I'm talking about this from a space of trust, it's not that you're not obnoxious; you totally can be.Corey: Extraordinarily so.Ana: You can totally be a snarky pain in the ass. Like I said, your wife is a saint. And sometimes—like, we were talking about recently, backing off on mocking people for working for Facebook because you and I both saw what it did to Chloe. And it's just not cool to do that to someone who's making a career choice, whether we agree with it or not. I personally have companies I would never work for. You and I have discussed contracts—not with you, but contracts I wouldn't take. Me personally, it's in my contract, I will not defend someone who is a sexual harasser or sexual assaulter. Like, I won't defend them. If they do #MeToo stuff—Corey: Mm-hm. The way that we've codified that—Ana: —I won't do it.Corey: —here is generally speaking—and this is a truism, I would encourage everyone in business to consider is, if you don't respect a client's business, you probably should not take their money. And—Ana: [laugh].Corey: —that leads to a lot of things.Ana: Yeah. I wish that was more common. [laugh].Corey: Yeah. It's—and again, I've never once shamed a company for this. I have declined to work with a number of companies in different capacities. And I've never been very open about this because I don't want companies to be listening to this and think, “Ohh, we sell ads. He might not want to work with us, so we're not going to reach out.” First, I will never mention, name, or drag anyone publicly.Ana: Oh, yeah. Same.Corey: Secondly, there's no such thing as any saint in these industries.Ana: Oh, no.Corey: I'm not talking about, “Oh, you display ads to people? [tsking noise].” No, I'm talking about, “You make landmines.” Let's be clear here. This is a whole other side of the universe. And I still never drag the companies that I declined to work with, in public, for having the temerity to reach out. Just seems like it's the wrong incentive structure if I start down that path.Ana: I was just talking to a client that I firmly believe we're at a pivot point in the way businesses are run. I was calling 2022 the Year of Transparency. And the reason I'm saying that is because in the last couple years with people working from home, with Covid, with Black Lives Matter, with all the stuff that's been going on in the world, and then, like, Activision Blizzard, and the lawsuits, and pay disparity, and Paizo unionizing—Paizo is a tabletop company that makes Pathfinder RPG—Corey: Mmm.Ana: —you know, all these companies. So, we're starting to see the game industry see unionization, we're seeing Starbucks employees want to unionize. People are not going to accept, “No comment,” anymore. They're not going to accept, “We're just not going to answer this.” And I can already see your brain ticking on who you're about to—I know where you're thinking.But my point is, when I've been talking to some of them, “I'm like, you have to be prepared that the old-school mentality of people not sharing their pay, like, not sharing how much they make compared to the person sitting next to them, that's gone.” People share that information now. There are companies where they are having spreadsheets. Now, one thing I did like about AWS was I always knew, like, my peers and I were encouraged if we want—my manager was awesome—my first manager was like, “If you guys want to talk about what you're making, go ahead.” And I was able to find out that because I had the masters, and more experience, and all this other stuff, I was actually—in my level group—the highest-paid one, even though I was the only woman at first. That's pretty cool to know.Corey: That's the kind of story that never makes the rounds.Ana: Well, and the thing is, we're not going to see people accepting obfuscation anymore. I think that's done. It's too easy to share information now for companies to think that their dirty laundry isn't going to come out, to think that they can lie and get away with stuff. As you know, we've talked about this a bit, I'm actually working on a book with a comic book artist—I didn't get his permission to say his name, so I'm not going to say it yet—and it's literally a picture book on how to not screw things up in today's digital media age when it comes to how you communicate with people. It's called Oh, Noes: A Picture Book for Execs. [laugh]. Um, but you know, you got to focus on the fact that people aren't going to accept obfuscation and lies anymore. They're not going to accept, “Oh, we're the company. We've got your best interests at heart.” It's not how it works anymore.Corey: That's what I see in this entire industry, where there's this idea that we're not going to say anything, we're just going to do our thing and not comment on any of these things. Which, okay, it's a strategy. But customers and the community and loud obnoxious—Ana: They talk to each other.Corey: —people on Twitter are going to comment in your absence. And that becomes a problem.Ana: Have you seen the movie—what is it?—John Tucker Must Die?Corey: I have not.Ana: It's a movie about three girls at the same high school who find out the guy is dating all three of them, and how they plot to destroy him. And every time I see one of these things happen where a big tech company—or any company—doesn't say anything, but then their customers start talking to each other going, “Wait a second,” I always think of that movie. And it's like, you can't think that people aren't going to talk to each other anymore.Especially once you get huge. When you're looking at these big, big companies, people want to take you down. Like, they're over this idea of monopolization and this idea that you can do things and there's no accountability. So yeah, I've been calling this the Year of Transparency because I think we're going to see huge shifts in what is and isn't okay to hide from your customers. Trust is your most valuable asset. And it can be lost in seconds.Corey: It's the easiest thing in the world to get, and it's incredibly easy to lose it, and almost impossible to regain it once you've lost it.Ana: Yes. And I think my students get sick of me saying this because I say it every week: “Trust is easy to get if you do it right, but you got to do it right.” You actually have to be honest, you have to, you know—and I'm not saying share secrets. But you can be—like, a good example with AWS is, they do great COEs after they have a big splat. You know, 2017, when they had a service disruption, and the latest ones, like, they do a good COE. Being able to rely on that sort of thing is critical.Corey: For me, it's one of the things that we do here just because of the sensitive information with which we are entrusted, and the way that we operate in the industry, we hold ourselves to a bar that is pretty similar to what you'll see in regulated industries and the rest. I periodically disclose all of my investments, which is nowhere near as interesting as most people would think.Ana: [laugh].Corey: I make it clear exactly where my interests are. This is the reason we have no partners with any company in this space, just because it is the perception of conflict of interest is huge. I mean, half our consulting business is doing contract negotiation on behalf of customers, with AWS directly. As soon as it comes out that we have a back channel deal with someone, everyone's going to question what's going on. It's easier never to enter into those engagements rather than having to try and back-walk it later. No. Does that leave opportunities on the table? Sometimes. But I think this is the better long-term play if I can think beyond next quarter's numbers.Ana: Yeah, absolutely. And that's, like, similar for me is that I have to be mindful of not taking contracts with companies that are in conflict with each other. And I don't mean conflict like they're at war, but like, where my working with each of them puts me in a position where there could be questions on who my loyalties are to.Corey: On the sponsorship side of our business, we refuse to do anything that even looks like an exclusivity contract, of, “All right. None of our direct competitors will be allowed to sponsor for a fixed period of ti”—sure, if you buy out the ads you don't want them to take, I guess, sure. But you don't get editorial control, either. It's the same approach: You can buy my attention, but never my opinion. Paying me does not make me say nicer things about you, directly.It does force me to look more closely into what your company does, and no one's purely good or purely evil. I will talk more about what I see, good and bad. That is the nature of what you get with me, and that is something that I don't think a number of folks realize, out of that ecosystem.Ana: Well, there's a level of professional maturity that goes with taking criticism. And when you have worked on something for a very, very, very long time, and it is your baby and you're getting criticized, it can be natural to have an emotional response. And that's something that, as a crisis communicator, I look at. Are the attacks coming in—and attacks, or commentary, or negative press—is it coming in, in an emotional way, like, what's happening is there's been a nerve hit because there's an emotional investment in whatever's going on? Or is it an impact of concern over finances, concern over jobs? So, there's different reasons why people will react and things. And that's one of the things I have to always keep in mind when I'm looking at stuff. As you well know. We've had many conversations about this. [laugh].Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance query accelerator for the Oracle MySQL Database Service, although I insist on calling it, “My squirrel.” While MySQL has long been the worlds most popular open-source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLAP and OLTP—don't ask me to pronounce those acronyms again—workloads directly from your MySQL database and eliminate the time-consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: One of the things that I always admired about you—and I have never once incidentally tried to change this in any way—but you have never leaked confidential information to me about anyone or anything. And to be clear, I have never asked. Back when you're running launch operations at AWS, I don't want to know things that are coming out if I can avoid it because then it gets very challenging for me to remember what I can talk about versus what I can't. My insight into AWS product roadmaps is not much better than anyone else in the industry. I just pay attention and I have a knack for being able to see what's coming.But because of the perception that I have the inside track, I don't break the news; I don't create the news; I just talk about what other people have already written about publicly. It's safer that way for me, and I've always appreciated your ability to respect confidentiality because for stuff like this, it matters more than anything else.Ana: Absolutely. My confidentiality is huge thing. I just don't talk about stuff. And in fact, like, my husband doesn't even know who half my clients are. He knows the number of clients I have, but he doesn't know who I'm working with. And that's because, you know, I don't need him to know. And it's a confidentiality thing. And you know, spouse, you're my husband, I have ten clients. And that's what I'll say. You know? He knows about you, obviously.Corey: Well, I should hope so. He's lovely. I was at your wedding, lovely though it was.Ana: That is true.Corey: So, one other thing that you're in the process of launching as we speak is apparently your own podcast. Loathe though I am to drive people to the competition, tell me about it.Ana: [laugh]. It's not actually competition, and we do have to give you credit for the name. One of your superpowers is giving really funny, punny names to just about anything. Next time we get a pet, I'm going to be like, “I want a pun that goes around this. What can I name the dog?”Corey: And how long did it take me to name your podcast?Ana: God, like, two minutes. It's so annoying because I'd been—Corey: It took that long?Ana: You let me finish typing.Corey: Yeah, that was nice of me, I thought.Ana: Yeah, you let me finish typing, and then you're like—okay, so not even two minutes. Like, a minute.Corey: It's not that I'm that good at naming things. It's just that I've never worked at AWS, and people who are so bad at it, that when some—they just encounter someone who's average with these things, we look like wizards from the future.Ana: So yeah, we're launching a podcast called [Disasterpiece Theater 00:26:15]. And it's actually a podcast where we're going to have subject matter experts from NASA, from medical fields, cybersecurity folks, we're going to actually have a shark expert so we can talk about The Meg and how that works. But the whole point of the podcast is taking pop culture movies—so like Jurassic Park, Alien: Covenant, all of these—and talking about how they'd actually work in the real world. How would Alien: Covenant have gone down if these people were trained the way people on ships are trained now? Or The Meg, what would you actually do if you had a shark in that scene where there's hundreds of thousands of people in the water? How would that actually go? I mean, the shark wouldn't be three miles long, but same concept.So, it's going to be a lot of fun, just kind of going through. One of my favorite guests is my dad. My dad's going to be talking to us about the movie 2012. My dad's a naval architect marine engineer. And he and I had the most fascinating conversation after watching that movie on how ships like that would actually be built, and what would happen, and what would have to happen, and the different rules and regulations that would have to change, and how you would actually—like, and his pet peeves with lazy things the writers did. So, it's going to be a lot of fun. We're doing it as a short run to see how it sticks. It'll be eight episodes to start, and then if there's a desire for more, we'll do a second season.Corey: I'm really looking forward to seeing how it comes out. I'd ask, “What's going on? You're starting a company and this side project for funsies? What's the point?” But I started this podcast show not too long after I started what became The Duckbill Group. So yeah.Ana: What's funny is, this has all kind of cascaded in weird ways because next month, the company's—Merewif's been around a year next month.Corey: Wow. Hard to believe.Ana: Which is totally crazy to think about. But I only—I was doing it as a side gig while I was at H2O until October—end of September. So, it's only been full-time since October. The podcast idea—Corey: Why now? Why now, though? What drove you—Ana: [laugh].Corey: —you went from giant company to start-up to launching your own thing. And you're launching your own thing in the same way that I launched my own company, which I'm going to shorthand to ‘the dumb way,' which is right now there is so much constipated capital sloshing around the VC ecosystem, and we both started companies that are absolutely never going to be a VC-scale opportunity because, you know, what can you do with $4 billion in investment? Oh, something monstrous, for damn sure. But there's no—there's no good answer to that. But we're never going to be the VC-scale opportunity.Ana: [laugh]. I dread to think what you would figure out what to do with, like, $400 million. It's terrifying.Corey: Oh, the video would be ridiculous. We're talking, like, Pixar quality…ridiculousness, making fun of various things in this industry, on a lark.Ana: Oh, I can imagine. I can imagine. I can imagine a weekly game show with you, too, where you brought in engineers from the different services and ask them random questions, kind of like Jeopardy, but with, like, the floor dropping out underneath them. Then they just get replaced with the next engineer or whatever. Like, they get an answer wrong; they drop through the floor; the next one slides in.Corey: I like that, yeah. That has legs.Ana: And this is why you and I are not allowed to come up with ideas together.Corey: Yeah this is—Ana: Anyway.Corey: —what we do to break on us from time to time. Yeah.Ana: [laugh]. So, the timing was a couple things. One, I've wanted to do this company since I got out of the Coast Guard. Like, it's something I wanted to do, but I needed to get more private experience. Because up until 2016, all of my experience was public sector. It was military, it was Coast Guard.So, while I'd worked with—in disasters, I had been side-by-side with BP dealing with that disaster and all sorts of stuff, I didn't actually have the experience myself. And I kept going, “Oh, well. I'll do it eventually. I'll do it eventually. I'm not ready yet. I'm not ready yet.”And then literally, you and one other person were like, “No, I literally need you to set this up right now because I need your help with something and I need an official way to pay you.”Corey: It seemed like the right thing to do. Yeah. Yeah.Ana: So, it was, like, “Okay.” And the name Merewif actually means ‘sea witch' or ‘siren' in Old English. Little known fact: I majored in English specializing in medieval and ancient literature when I was in college.Corey: That explains your depth of insight into the AWS documentation.Ana: [laugh]. Yeah. I can read like nobody's business. And so, in traditional stories, a lot of times, the hero will go to a witch or a sea witch for advice, or for knowledge, or for medicines, or whatever. So, it kind of tied together the fact that I was in the Coast Guard—so I've always been around oceans—my Old English, Middle English background.And yeah, it just—the name made sense to me. So, it was like, “Well, I have a name now. Let's just do it.” And so I did it. And then as the year went on, I started getting a lot of interest, different friends in the industry found out what I was doing, or they found out through a friend, an alumni classmate of mine pinged me going, “Hey, this company really needs your help. Can I do an introduction?” I said, “Okay.”And so it started taking off. And so by September, I was like, “Well, if I can get a couple things lined up, I'm going to have too much to do with the job I love, which is Merewif, to stay at a day job that I'm like, ‘Ehh. It's a job.'” And it's been incredible. Like, it's busy. It sometimes means waking up at two in the morning to see what you're up to.Corey: It happens, sometimes. To be clear, that is out of your own choice. The beautiful thing about my business is that it's strictly a business hours problem.Ana: Yes, except I knew that the video was launching today, and I wanted to take one more scrub on it to make sure that [laugh] there wasn't anything over the line.Corey: Yeah. We go right up to it, but try not to cross it.Ana: Yes. And so—and that's the killer thing is, like, I'm loving every day. Like, it's crazy. It's different things. I do hate being my own finance department. But you know.Corey: Fractional CFOs are one of our first strategic hires that we made here, and it was a bit of a stretch, and it's a, “We think we can afford it because, Dan”—who's been a guest on this show—“As our CFO says we can, and that's sort of his job, so all right. Let's see what happens.” And sure, it's great way to fail if he's not good at his job, but he was right. And it has been an absolute Godsend just for the things I don't have to worry about that have been taken off of my head, are—it's like not having to plan a wedding anymore. That level of relief.Ana: [laugh].Oh, yeah. Covid messed up my wedding, too. So, that ended up being in our backyard. But you know, at the end of the day, every day I'm doing work that I've spent my whole career becoming really good at, and becoming an expert at, and being able to talk with [countries 00:32:30] that can't necessarily afford to hire someone like me full time, but to be able to walk them through, “All right, here's the cloud technologies that are available for you, but you're also going to want to have, for example, a snowball edge in your area because you're going to lose connectivity.” And, “Oh, hey, talk to the guys over at Project OWL.”It's a cool one if you haven't looked at it. They're basically these floating little—they look like little ducks; well, the original versions of them did—and they basically allow—they're WiFi repeaters in some ways, where they float. So, if you disperse them in an area where disasters happen, even if it flooded, it's going to keep that wireless network up and available in that entire area, for everyone who's impacted. Which is a huge problem in the last mile. So, getting to do this stuff that I love anyway, it was just time.And I'm loving teaching at the UW. I'm back at the program I actually graduated from. And this will be of no shock to you, at some point in the near future, I'm going to be applying to do my PhD. It's been a goal of mine since I was little to be the first PhD in my family. Were weirdly competitive about very strange things.Corey: I will be extremely disappointed if your dissertation does not feature the word ‘shitposting,' and of course, a link to something that cites my work.Ana: Actually shitposting could end up in there because what I really want to study is the impact of emerging technologies, including social media and things like that, and how they're impacting the ability of responders to have a common operating picture. So, it's clouding the ability. So, a common operating picture is how the Coast Guard and the Fish and Wildlife and the local fire department all know what's going on when a disaster happens, right? That's great, but they now all have separate systems. And if you think the local fire department or the local fisheries guys have the same level of security as, say, the Coast Guard does on their systems, they don't.So, how do you get them into the same common operating picture? And then what happens if it's a hurricane, and you have people tweeting pictures of the hurricane, and they're not even in the area from the hurricane? So, you have all this additional noise, you have all these additional security needs that weren't there, say, during Katrina, when we were doing everything by, like—no joke—a lot of faxing and text messaging and driving things back and forth. How do you deal with that? So yeah, that's actually what I'm looking at doing.So yeah, shitposting might end up in there as a what do you do when you're in a disaster and you have shitposting cluttering up your mess? So yeah, that's what I'm hoping to do at some point. But I've got so much work right now with Merewif that, right now, I don't have time to get the PhD. [laugh]. So.Corey: Industry and academia tend to be a little on the different side. And for what it's worth, like, there are a lot of companies doing PR, crisis comms work, et cetera, et cetera. The reason that there was really—this was one of those no-bid contracts because you understand this industry in a way that few people do. You've worked within it, you understand the dynamics within it, as well as adjacent industries like gaming, for example. Having someone who understands the moving parts of an industry, who the major players are and how that all fits together, it's something that you can't take some random comms firm off the street and expect them to understand it in the evolving way that social media, among others, has really shifted the entire narrative. So, I don't know of anyone else who's doing it the way that you do. They're certainly not talking about it the same.Ana: Way. There are a few firms that do something similar, but they're bigger and they have a lot of people and they're not as specialized as I am. So, they have an idea of it, but they're not necessarily from that industry. Or, you know, I've been playing video games since I was—what—ten. And I've been very involved. I do panels about women in the military, and how we're represented in video games and comic books, I do those quite often.Actually, real quick, that reminds me back to the PhD thing.Corey: Of course.Ana: The other reason I want to get the PhD is because, as a woman, having that extra boost of not only have I been doing this for—oh God, almost 20 years; that makes me feel really old—almost 20 years, but I also have a PhD in this specific technique. In order to get this PhD, I have to convince a university to let me combine an IT PhD, like, either an information technology or an IS tech—like, a science PhD and a communication PhD into one. There is no school that quite offers what I want, so I'm going to actually have to combine them. But I will say that one of the other reasons I really want to do it, other than the fact that I get to look at my little brother—who you know—and go, “Pttht, I got it first,” is because as a woman, it does give me one more way to keep the door open that my male counterparts don't necessarily need. And as you know, in this industry, that's a lot. I mean, it's not easy being a younger-looking blue-haired woman who's like, “Hi, I know my shit.”Corey: Meanwhile, I am presumed competent in a way that people who aren't over-represented are not. And when I say something, it is presumed true, as opposed to being nibbled to death by ducks with, “Well, can you back up that assertion?” Because sometimes, no. I'm speculating, but I am presumed to be right as a default.Ana: Yep.Corey: And people love to say that, “Oh, yeah, privilege isn't really a thing.” Let's be very clear here. I did have to build a lot of the stuff that's here. None of this was handed to me. But I didn't have a headwind at fighting against me every step of the way the I would have if I didn't look like this.Ana: One of the things I've joked about a lot is my being a veteran, has actually helped me with some of those headwinds because there are assumptions made about my personality—[laugh] the fact that I'm blunt, the fact that I—Corey: No.Ana: —tend to be very straightforward. And I believe my very first meeting with Ariel Kelman when he was a VP at Amazon—at AWS—was, in one of the meetings, the very first one was the words, “Are you shitting me?” Came out of my mouth over something. [laugh]. Could help it; just came out of my mouth.I am very good at filtering when I need to, but in that moment, whoof, I couldn't have. So, being a veteran does help a bit because there's some personality assumptions that other women deal with the, “Oh, she's a bitch.” With me. It's, “Oh, she's scary because she was a veteran.” I'm like, “All right. [laugh]. Cool. We'll lean into that. We will tell you this has been my personality since I was five. We'll let you think it was the Coast Guard that made me this way.”Corey: You joined early. Got it.Ana: Oh, totally. Joined at five. Well, my dad was Coast Guard, so let's just count that. I grew up in the Coast Guard.Corey: I just never grew up. It was easier.Ana: You know, when they're going to let you drive a 378-foot ship, you kind of have to grow up a little bit.Corey: One would hope anyway.Ana: [laugh]. Well, and I mean, you know, there's the other factor is that, you know—actually, in my AWS interview, I think I scared my Bar Raiser by telling one of these stories—there were times where when I made a decision, someone could get killed if I was wrong.Corey: So, that does happen at Amazon scale, but less frequently than it does in the armed services.Ana: Well, yeah. I mean, there it's you're literally being dumb and leaving people in place in front of a tornado, which I'm not going to get into. I'm very—Corey: Or a power bus is—a safety isn't put on and someone gets electrocuted. But it's always small-scale stuff, not—it's not as common.Ana: Yeah. And when you're doing—like, I was a search and rescue controller, and I had to know the area I was operating in the winds, the potential risks, what type of vessels were in that area, and then we had a computer software called SAROPS that helped me search. But, like, growing up in an industry where if I screwed up someone could die gives you a completely different perspective on a lot of things.Corey: Compared to that, there is no stress in the computer industry. There really isn't.Ana: I used to joke at launch when people were freaking out—and I told Ariel this once and I thought he was going to snort his coffee—but we were sitting there and people were like, “Oh, my gosh,” for re:Invent I was like, “Is the building flooding?” “No.” “Is it on fire?” “No.” “Is anyone shooting at us?” “No.” “Okay, cool. Chill out. [laugh]. It'll be okay.”Corey: Yeah, “You can weather some mean tweets. I promise. It'll be okay.”Ana: “Deep breaths.” But you know, at the same time, on the empathy scale is understanding that not everyone has that experience. So, that's the other thing that's critical to understand as a crisis communicator or as a leader of any kind, is that the stresses and crazy things I've been through have made me who I am. The stresses and crazy things you've been through have made you who you are, right? Well, what you find—what will trigger your brain to go, “This is fight or flight. Oh, my gosh, this is terrifying. Oh, gosh, I could”—you know, for some of these people at re:Invent, “Oh, my gosh, I could lose my job. If I lose my job, I can't feed my family.”So, even though I don't panic because I'm like, “Meh, no one's shooting at me. Cool.” Understanding that for the person next to them, they could physically be having that response of fight or flight is a critical part of leadership and crisis comms. You know, I think too often people are like, “Oh, my hardship beats your hardship.” Well, yeah.Not everyone has been in 60-foot seas where they literally bounce off bulkheads and pass a mushroom through their nose because, by the way, you can get that seasick. But it's true. And if you look at some of the younger people you're hiring, what they consider as, “Oh, my gosh, this could be a problem.” You're like, “Well, okay. We're going to be okay. Take a breath.”Corey: Perspective is one of those things that comes with experience, for better or worse.Ana: [laugh]. Yeah, right?Corey: So, I want to thank you for taking so much time to speak with me today.Ana: Oh, absolutely.Corey: If people want to learn more, where can they find you?Ana: So, I am @acvisneski, on Twitter. And also, my webpage is the—T-H-E—merewif—M-E-R-E-W-I-F dot com. Those are the two best places.Corey: And we'll put them in the [show notes 00:42:00], of course.Ana: Awesome.Corey: Thank you so much for joining me today. I really appreciate it.Ana: Oh, happy to. It's always fun.Corey: It really is. Ana Visneski, Chief Chaos Coordinator at the Merewif. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment telling me that this was the worst possible way to find out that Shtephen was no longer with us.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Screaming in the Cloud
How to Investigate the Post-Incident Fallout with Laura Maguire, PhD

Screaming in the Cloud

Play Episode Listen Later Feb 8, 2022 30:57


About LauraLaura leads the research program at Jeli.io.  She has a Master's degree in Human Factors & Systems Safety and a PhD in Cognitive Systems Engineering. Her doctoral work focused on distributed incident response practices in DevOps teams responsible for critical digital services. She was a researcher with the SNAFU Catchers Consortium from 2017-2020 and her research interests lie in resilience engineering, coordination design and enabling adaptive capacity across distributed work teams. As a backcountry skier and alpine climber, she also studies cognition & resilient performance in high risk, high consequence mountain environments.  Links: Howie: The Post-Incident Guide: https://www.jeli.io/howie-the-post-incident-guide/ Jeli: https://www.jeli.io Twitter: https://twitter.com/lauramdmaguire TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the things that's always been a treasure and a joy in working in production environments is things breaking. What do you do after the fact? How do you respond to that incident?Now, very often in my experience, you dive directly into the next incident because no one has time to actually fix the problems but just spend their entire careers firefighting. It turns out that there are apparently alternate ways. My guest today is Laura Maguire who leads the research program at Jeli, and her doctoral work focused on distributed incident response in DevOps teams responsible for critical digital services. Laura, thank you for joining me.Laura: Happy to be here, Corey, thanks for having me.Corey: I'm still just trying to wrap my head around the idea of there being a critical digital service, as someone whose primary output is, let's be honest, shitposting. But that's right, people do use the internet for things that are a bit more serious than making jokes that are at least funny only to me. So, what got you down this path? How did you get to be the person that you are in the industry and standing in the position you hold?Laura: Yeah, I have had a long circuitous route to get to where I am today, but one of the common threads is about safety and risk and how do people manage safety and risk? I started off in natural resource industries, in mountain safety, trying to understand how do we stop things from crashing, from breaking, from exploding, from catching fire, and how do we help support the people in those environments? And when I went back to do my PhD, I was tossed into the world of software engineers. And at first I thought, now, what do firefighters, pilots, you know, emergency room physicians have to do with software engineers and risk in software engineering? And it turns out, there's actually a lot, there's a lot in common between the types of people who handle real-time failures that have widespread consequences and the folks who run continuous deployment environments.And so one of the things that the pandemic did for us is it made it immediately apparent that digital service delivery is a critical function in society. Initially, we'd been thinking about these kinds of things as being financial markets, as being availability of electronic health records, communication systems for disaster recovery, and now we're seeing things like communication and collaboration systems for schools, for businesses, this helps keep society functioning.Corey: What makes part of this field so interesting is that the evolution in the space where, back when I first started my career about a decade-and-a-half ago, there was a very real concern in my first Linux admin gig when I accidentally deleted some of the data from the data warehouse that, “Oh, I don't have a job anymore.” And I remember being surprised and grateful that I still did because, “Oh, you just learned something. You going to do it again?” “No. Well, not like that exactly, but probably some other way, yeah.”And we have evolved so far beyond that now, to the point where when that doesn't happen after an incident, it becomes almost noteworthy in its own right and it blows up on social media. So, the Overton window of what is acceptable disaster response and incident management, and how we learn from those things has dramatically shifted even in the relatively brief window of 15 years. And we're starting to see now almost a next-generation approach to this. One thing that you were, I believe the principal author behind is Howie: The Post-Incident Guide, which is a thing that you have up on jeli.io—that's J-E-L-I dot I-O—talking about how to run post-incident investigations. What made you decide to write something like this?Laura: Yeah, so what you described at the beginning there about this kind of shift from blameless—blameful-type approaches to incident response to thinking more broadly about the system of work, thinking about what does it mean to operate in continuous deployment environments is really fundamental. Because working in these kinds of worlds, we don't have an established knowledge base about how these systems work, about how they break because they're continuously changing, the knowledge, the expertise required to manage them is continuously changing. And so that shift towards a blameless or blame-aware post-incident review is really important because it creates this environment where we can actually share knowledge, share expertise, and distribute more of our understandings of how these systems work and how they break. So that, kind of, led us to create the Howie Guide—the how we got here post-incident guide. And it was largely because companies were kind of coming from this position of, we find the person who did the thing that broke the system and then we can all rest easy and move forward. And so it was really a way to provide some foundation, introduce some ideas from the resilience engineering literature, which has been around for, you know, the last 30 or 40 years—Corey: It's kind of amazing, on some level, how tech as an industry has always tried to reinvent things from first principles. I mean, we figured out long before we started caring about computers in the way we do that when there was an incident, the right response to get the learnings from it for things like airline crashes—always a perennial favorite topic in this space for conference talks—is to make sure that everyone can report what happened in a safe way that's non-accusatory, but even in the early-2010s, I was still working in environments where the last person to break production or break the bill had the shame trophy hanging out on their desk, and it would stay there until the next person broke it. And it was just a weird, perverse incentive where it's, “Oh if I broke something, I should hide it.”That is absolutely the most dangerous approach because when things are broken, yes, it's generally a bad thing, so you may as well find the silver lining in it from my point of view and figure out, okay, what have we learned about our systems as a result of the way that these things break? And sometimes the things that we learn are, in fact, not that deep, or there's not a whole lot of learnings about it, such as when the entire county loses power, computers don't work so well. Oh, okay. Great, we have learned that. More often, though, there seem to be deeper learnings.And I guess what I'm trying to understand is, I have a relatively naive approach on what the idea of incident response should look like, but it's basically based on the last time I touched things that were production-looking, which was six or seven years ago. What is the current state of the art that the advanced leaders in the space as they start to really look at how to dive into this? Because I'm reasonably certain it's not still the, “Oh, you know, you can learn things when your computers break.” What is pushing the envelope these days?Laura: Yeah, so it's kind of interesting. You brought up incident response because incident response and incident analysis are the, sort of like, what do we learn from those things are very tightly coupled. What we can see when we look at someone responding in real-time to a failure is, it's difficult to detect all of the signals; they don't pop up and wave a little flag and say, like, “I am what's broken.” There's multiple compounding and interacting factors. So, there's difficulty in the detection phase; diagnosis is always challenging because of how the systems are interrelated, and then the repair is never straightforward.But when we stop and look at these kinds of things after the fact, of really common theme emerges, and that it's not necessarily about a specific technical skill set or understanding about the system, it's about the shared, distributed understanding of that. And so to put that in plain speak, it's what do you know that's important to the problem? What do I know that's important to the problem? And then how do we collectively work together to extract that specific knowledge and expertise, and put that into practice when we're under time pressure, when there's a lot of uncertainty, when we've got the VP DMing us and being like, “When's the system going to be back up?” and Twitter's exploding with unhappy customers?So, when we think about the cutting edge of what's really interesting and relevant, I think organizations are starting to understand that it's how do we coordinate and we collaborate effectively? And so using incident analysis as a way to recognize not only the technical aspects of what went wrong but the social aspects of that as well. And the teamwork aspects of that is really driving some innovation in this space.Corey: It seems to me, on some level, that the increasing sophistication of what environments look like is also potentially driving some of these things. I mean, again, when you have three web servers and one of them's broken, okay, it's a problem; we should definitely jump on that and fix it. But now you have thousands of containers running hundreds of microservices for some Godforsaken reason because what we decided this thing that solves the problem of 500 engineers working on the same repository is a political problem, so now we're going to use microservices for everything because, you know, people. Great. But then it becomes this really difficult to identify problem of what is actually broken?And past a certain point of scale, it's no longer a question of, “Is it broken?” so much as, “How broken is it at any given point in time?” And getting real-time observability into what's going on does pose more than a little bit of a challenge.Laura: Yeah, absolutely. So, the more complexity that you have in the system, the more diversity of knowledge and skill sets that you have. One person is never going to know everything about the system, obviously, and so you need kind of variability in what people know, how current that knowledge is, you need some people who have legacy knowledge, you have some people who have bleeding edge, my fingers were on the keyboard just moments ago, I did the last deploy, that kind of variability in whose knowledge and skill sets you have to be able to bring to bear to the problem in front of you. One of the really interesting aspects, when you step back and you start to look really carefully about how people work in these kinds of incidents, is you have folks that are jumping, get things done, probe a lot of things, they look at a lot of different areas trying to gather information about what's happening, and then you have people who sit back and they kind of take a bit of a broader view, and they're trying to understand where are people trying to find information? Where might our systems not be showing us what's going on?And so it takes this combination of people working in the problem directly and people working on the problem more broadly to be able to get a better sense of how it's broken, how widespread is that problem, what are the implications, what might repair actually look like in this specific context?Corey: Do you suspect that this might be what gives rise, sometimes, to it seems middle management's perennial quest to build the single pane of glass dashboard of, “Wow, it looks like you're poking around through 15 disparate systems trying to figure out what's going on. Why don't we put that all on one page?” It's a, “Great, let's go tilt at that windmill some more.” It feels like it's very aligned with what you're saying. And I just, I don't know where the pattern comes from; I just know I see it all the time, and it drives me up a wall.Laura: Yeah, I would call that pattern pretty common across many different domains that work in very complex, adaptive environments. And that is—like, it's an oversimplification. We want the world to be less messy, less unstructured, less ad hoc than it often is when you're working at the cutting edge of whatever kind of technology or whatever kind of operating environment you're in. There are things that we can know about the problems that we are going to face, and we can defend against those kinds of failure modes effectively, but to your point, these are very largely unstructured problem spaces when you start to have multiple interacting failures happening concurrently. And so Ashby, who back in 1956 started talking about, sort of, control systems really hammered this point home when he was talking about, if you have a world where there's a lot of variability—in this case, how things are going to break—you need a lot of variability in how you're going to cope with those potential types of failures.And so part of it is, yes, trying to find the right dashboard or the right set of metrics that are going to tell us about the system performance, but part of it is also giving the responders the ability to, in real-time, figure out what kinds of things they're going to need to address the problem. So, there's this tension between wanting to structure unstructured problems—put those all in a single pane of glass—and what most folks who work at the frontlines of these kinds of worlds know is, it's actually my ability to be flexible and to be able to adapt and to be able to search very quickly to gather the information and the people that I need, that are what's really going to help me to address those hard problems.Corey: Something I've noticed for my entire career, and I don't know if it's just unfounded arrogance, and I'm very much on the wrong side of the Dunning-Kruger curve here, but it always struck me that the corporate response to any form of outage has is generally trending toward oh, we need a process around this, where it seems like the entire idea is that every time a thing happens, there should be a documented process and a runbook on how to perform every given task, with the ultimate milestone on the hill that everyone's striving for is, ah, with enough process and enough runbooks, we can then eventually get rid of all the people who know all this stuff works, and basically staff at up with people who'd know how to follow a script and run push the button when told to buy the instruction manual. And that's always rankled, as someone who got into this space because I enjoy creative thinking, I enjoy looking at the relationships between things. Cost and architecture are the same thing; that's how I got into this. It's not due to an undying love of spreadsheets on my part. That's my business partner's problem.But it's this idea of being able to play with the puzzle, and the more you document things with process, the more you become reliant on those things. On some level, it feels like it ossifies things to the point where change is no longer easily attainable. Is that actually what happens, or am I just wildly overstating the case? Either as possible. Or a third option, too. You're the expert; I'm just here asking ridiculous questions.Laura: Yeah, well, I think it's a balance between needing some structure, needing some guidelines around expected actions to take place. This is for a number of reasons. One, we talked about earlier about how we need multiple diverse perspectives. So, you're going to have people from different teams, from different roles in the organization, from different levels of knowledge, participating in an incident response. And so because of that, you need some form of script, some kind of process that creates some predictability, creates some common ground around how is this thing going to go, what kinds of tools do we have at our disposal to be able to either find out what's going on, fix what's going on, get the right kinds of authority to be able to take certain kinds of actions.So, you need some degree of process around that, but I agree with you that too much process and the idea that we can actually apply operational procedures to these kinds of environments is completely counterproductive. And what it ends up doing is it ends up, kind of, saying, “Well, you didn't follow those rules and that's why the incident went the way it did,” as opposed to saying, “Oh, these rules actually didn't apply in ways that really matter, given the problem that was faced, and there was no latitude to be able to adapt in real-time or to be able to improvise, to be creative in how you're thinking about the problem.” And so you've really kind of put the responders into a bit of a box, and not given them productive avenues to, kind of, move forward from. So, having worked in a lot of very highly regulated environments, I recognize there's value in having prescription, but it's also about enabling performance and enabling adaptive performance in real-time when you're working at the speeds and the scales that we are in this kind of world.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance query accelerator for the Oracle MySQL Database Service, although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLAP and OLTP—don't ask me to pronounce those acronyms again—workloads directly from your MySQL database and eliminate the time-consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: Yeah, and let's be fair, here; I am setting up something of a false dichotomy. I'm not suggesting that the answer is oh, you either are mired in process, or it is the complete Wild West. If you start a new role and, “Great. How do I get started? What's the onboarding process?” Like, “Step one, write those docs for us.”Or how many times have we seen the pattern where day-one onboarding is, “Well, here's the GitHub repo, and there's some docs there. And update it as you go because this stuff is constantly in motion.” That's a terrible first-time experience for a lot of folks, so there has to be something that starts people off in the right direction, a sort of a quick guide to this is what's going on in the environment, and here are some directions for exploration. But also, you aren't going to be able to get that to a level of granularity where it's going to be anything other than woefully out of date in most environments without resorting to draconian measures. I feel like—Laura: Yeah.Corey: —the answer is somewhere in the middle, and where that lives depends upon whether you're running Twitter for Pets or a nuclear reactor control system.Laura: Yeah. And it brings us to a really important point of organizational life, which is that we are always operating under constraints. We are always managing trade-offs in this space. It's very acute when you're in an incident and you're like, “Do I bring the system back up but I still don't know what's wrong or do I leave it down a little bit longer and I can collect more information about the nature of the problem that I'm facing?”But more chronic is the fact that organizations are always facing this need to build the next thing, not focus on what just happened. You talked about the next incident starting and jumping in before we can actually really digest what just happened with the last incident; these kinds of pressures and constraints are a very normal part of organizational life, and we are balancing those trade-offs between time spent on one thing versus another as being innovating, learning, creating change within our environment. The reason why it's important to surface that is that it helps change the conversation when we're doing any kind of post-incident learning session.It's like, oh, it allows us to surface things that we typically can't say in a meeting. “Well, I wasn't able to do that because I know that team has a code freeze going on right now.” Or, “We don't have the right type of, like, service agreement to get our vendor on the phone, so we had to sit and wait for the ticket to get dealt with.” Those kinds of things are very real limiters to how people can act during incidents, and yet, don't typically get brought up because they're just kind of chronic, everyday things that people deal with.Corey: As you look across the industry, what do you think that organizations are getting, I guess, it's the most wrong when it comes to these things today? Because most people are no longer in the era of, “All right. Who's the last person to touch it? Well, they're fired.” But I also don't think that they're necessarily living the envisioned reality that you described in the Howie Guide, as well as the areas of research you're exploring. What's the most common failure mode?Laura: Hmm. I got to tweak that a little bit to make it less about the failure mode and more about the challenges that I see organizations facing because there are many failure modes, but some common issues that we see companies facing is they're like, “Okay, we buy into this idea that we should start looking at the system, that we should start looking beyond the technical thing that broke and more broadly at how did different aspects of our system interact.” And I mean, both people as a part of the system, I mean processes part of the system, as well as the software itself. And so that's a big part of why we wrote the Howie Guide, is because companies are struggling with that gap between, “Okay, we're not entirely sure what this means to our organization, but we're willing to take steps to get there.” But there's a big gap between recognizing that and jumping into the academic literature that's been around for many, many years from other kinds of high-risk, high-consequence type domains.So, I think some of the challenges they face is actually operationalizing some of these ideas, particularly when they already have processes and practices in place. There's ideas that are very common throughout an organization that take a long time to shift people's thinking around, the implicit biases or orientations towards a problem that we as individuals have, all of those kinds of things take time. You mentioned the Overton window, and that's a great example of it is intolerable in some organizations to have a discussion about what do people know and not know about different aspects of the system because there's an assumption that if you're the engineer responsible for that, you should know everything. So, those challenges, I think, are quite limiting to helping organizations move forward. Unfortunately, we see not a lot of time being put into really understanding how an incident was handled, and so typically, reviews get done on the side of the desk, they get done with a minimal amount of effort, and then the learnings that come out of them are quite shallow.Corey: Is there a maturity model, where it makes sense to begin investing in this, whereas if you've do it too quickly, you're not really going to be able to ship your MVP and see what happens; if you go too late, you have a globe-spanning service that winds up being down all the time so no one trusts it. What is the sweet spot for really started to care about incident response? In other words, how do people know that it's time to start taking this stuff more seriously?Laura: Ah. Well… you have kids?Corey: Oh, yes. One and four. Oh yeah.Laura: Right—Corey: Demons. Little demons whom I love very much.Laura: [laugh]. They look angelic, Corey. I don't know what you're talking about. Would you not teach them how to learn or not teach them about the world until they started school?Corey: No, but it would also be considered child abuse at this age to teach them about the AWS bill. So, there is a spectrum as far as what is appropriate learnings at what stage.Laura: Yeah, absolutely. So, that's a really good point is that depending on where you are at in your operation, you might not have the resources to be able to launch full-scale investigations. You may not have the complexity within your system, within your teams, and you don't have the legacy to, sort of, draw through, to pull through, that requires large-scale investigations with multiple investigators. That's really why we were trying to make the Howie Guide very applicable to a broad range of organizations is, here are the tools, here are the techniques that we know can help you understand more about the environment that you're operating in, the people that you're working with, so that you can level up over time, you can draw more and more techniques and resources to be able to go deeper on those kinds of things over time. It might be appropriate at an early stage to say, hey, let's do these really informally, let's pull the team together, talk about how things got set up, why choices were made to use the kinds of components that we use, and talk a little bit more about why someone made a decision they did.That might be low-risk when you're small because y'all know each other, largely you know the decisions, those conversations can be more frank. As you get larger, as more people you don't know are on those types of calls, you might need to handle them differently so that people have psychological safety, to be able to share what they knew and what they didn't know at the time. It can be a graduated process over time, but we've also seen very small, early-stage companies really treat this seriously right from the get-go. At Jeli, I mean, one of our core fundamentals is learning, right, and so we do, we spend time on sharing with each other, “Oh, my mental model about this was X. Is that the same as what you have?” “No.” And then we can kind of parse what's going on between those kinds of things. So, I think it really is an orientation towards learning that is appropriate any size or scale.Corey: I really want to thank you for taking the time to speak with me today. If people want to learn more about what you're up to, how you view these things and possibly improve their own position on these areas, where can they find you?Laura: So, we have a lot of content on jeli.io. I am also on Twitter at—Corey: Oh, that's always a mistake.Laura: [laugh]. @lauramdmaguire. And I love to talk about this stuff. I love to hear how people are interpreting, kind of, some of the ideas that are in the resilience engineering space. Should I say, “Tweet at me,” or is that dangerous, Corey?Corey: It depends. I find that the listeners to this show are all far more attractive than the average, and good people, through and through. At least that's what I tell the sponsors. So yeah, it should be just fine. And we will of course include links to those in the [show notes 00:27:11].Laura: Sounds good.Corey: Thank you so much for your time. I really appreciate it.Laura: Thank you. It's been a pleasure.Corey: Laura Maguire, researcher at Jeli. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please give a five-star review on your podcast platform of choice along with an angry, insulting comment that I will read just as soon as I get them all to display on my single-pane-of-glass dashboard.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Web Dev 101 - Front End, Back End, Full Stack
#1 – Intro to DataNation, what is OLTP and OLAP

Web Dev 101 - Front End, Back End, Full Stack

Play Episode Listen Later Feb 5, 2022 9:36


Alex Merced introduces the new DataNation podcast that will be available on all podcast providers the discusses what is OLTP and what is OLAP and why they matter. Join the DataNation community at https://www.DataNation.click Register for the Subsurface Conference at - https://www.dremio.com/subsurface/live/winter2022/?utm_medium=social&utm_source=dremio&utm_term=alexmercedsocial&utm_content=na&utm_campaign=event-subsurface-2022 Join the Subsurface Slack Community at - https://join.slack.com/t/subsurfaceworkspace/shared_invite/zt-ghkyk4ox-8FmydCM_6xGdx9Li0qo3Jg

Screaming in the Cloud
The Proliferation of Ways to Learn with Serena (@shenetworks)

Screaming in the Cloud

Play Episode Listen Later Feb 3, 2022 34:31


About Serena Serena is a Network Engineer who specializes in Data Center Compute and Virtualization. She has degrees in Computer Information Systems with a concentration on networking and information security and is currently pursuing a master's in Data Center Systems Engineering. She is most known for her content on TikTok and Twitter as Shenetworks. Serena's content focuses on networking and security for beginners which has included popular videos on bug bounties, switch spoofing, VLAN hoping, and passing the Security+ certification in 24 hours.Links: Cisco cert Discord study group:https://discord.com/invite/uXQ8yWnN8a Beacons:https://beacons.page/shenetworks TikTok:https://www.tiktok.com/@shenetworks sysengineer's TikTok:https://www.tiktok.com/@sysengineer Twitter:https://twitter.com/notshenetworks TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They've also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today's guest was on relatively recently, but it turns out that when I have people on the show to talk about things, invariably I tend to continue talking to them about things and that leads down really interesting rabbit holes. Today is a stranger rabbit hole than most. Joining me once again is @SheNetworks or Serena [DiPenti 00:00:51]. Thanks for coming back and subjecting yourself to, basically, my nonsense all over again in the same month.Serena: Thanks for having me back. Excited.Corey: So, you have a, I think study group is the term that you're using. I don't know how to describe it in a way that doesn't make me sound ridiculous and describing and speaking with my hands and the rest. It's a Discord, as the kids of today tend to use. There are some private channels on an existing Discord group, and we'll get to the mechanics of that in a second. But it's a study group for various Cisco certifications, which it's been a while since I had one; my CCNA is something I took back in 2009. I've checked, it's expired to the point where they can't even look it up anymore to figure out who I might have been, once upon a time. What is this group and where did it come from?Serena: Yeah, so the Discord itself is kind of a collective of a bunch of people that are creators on TikTok. And it's just, like, a cool place to connect, especially people from TikTok join, people from Twitter join, they want to interact, you know, a great place to get resources if you're early in your career. I—you know, new year, new me resolution was [laugh] I wanted to start studying for the CCNP a little bit, and I've been doing it pretty loosely for a while, but I kind of was like, all right, time to actually sit down and dedicate some real time to this. And I put on Twitter, you know, if anybody else was interested—I know there's other various study groups out there and things like that, but I was just like, hey, you know, it was anyone interested and a study group and I got really good response. Of course, a lot of people are at the CCNA level, so I made a channel for CCNA and CCNP, so whatever level you're at, you can come in and ask questions. It's really great.Corey: One thing that irked me when I first joined, as well, there's no CCENT which was sort of the entry-level Cisco cert, the first half of the CCNA, and I did a bit of googling before shooting my mouth off. And it turns out that Cisco sunset that cert a while back, so CCNA is now the entry-level cert, as I understand it.Serena: Yeah. So, when I did my CCNA, I did the C-C-E-N-T—the CCENT, and then the ICND2, and that's how I got my CCNA. And then I went and got the Data Center CCNA, which was two exams… two? Or maybe it was just one. I can't remember fully. But they basically got rid of all of their CCNAs and created one new one that's just the CCNA Ente